A data lifecycle archival decision method incorporating a time value model

Abstract

The application provides a data lifecycle archiving decision method combined with a time value model, comprising: constructing a multi-dimensional blood relationship graph containing data entities, processing processes and business target nodes, calculating a dynamic conversion ambiguity index calibrated by an abnormal frequency, building an entity horizontal correlation with a collaborative verification weight, inputting a business target entropy representing business change uncertainty, obtaining an entity initial time value based on the graph, synthesizing upstream data value, conversion ambiguity and collaborative weight, constructing a time value decay model coupled with business target entropy and a regulation decay rate, setting an archiving threshold changing with storage occupancy and data entity storage cost indicators, and continuously monitoring data time value; when the data value is lower than the dynamic threshold and all associated business target entropies meet the standard, automatic archiving is performed, realizing intelligent archiving decision considering data blood relationship, business fluctuation and storage cost. The application realizes dynamic balance between data availability and storage operation and maintenance cost.

Description

Technical Field

[0001] This invention relates to the field of data archiving technology, and specifically to a data lifecycle archiving decision-making method that incorporates a time value model. Background Technology

[0002] Enterprises continuously accumulate large amounts of multi-source heterogeneous data during their daily operations. This data serves as a crucial core asset, supporting key aspects such as business analysis, intelligent decision-making, and operational management. As the scale of data continues to expand, the pressure on enterprise data storage and computing resources increases significantly, and system maintenance costs rise rapidly. To control storage costs while ensuring the rational use of data, data lifecycle management, especially data archiving decisions, has become an important task in the field of data governance. The intrinsic value of data during its lifecycle typically diminishes over time and with changes in the business environment. Traditional data archiving decisions often rely on manual experience or simple time-cycle rules, such as setting a uniform retention period before archiving. This single, static approach severs the deep connection between data and actual business processes, making it difficult to objectively assess the true application value of data. It is also unable to adapt to complex and ever-changing data application scenarios, easily leading to the premature archiving of high-value data or the long-term occupation of high-cost storage resources by low-value data.

[0003] In modern, complex data flow environments, the generation, processing, and application of data typically involve a series of extraction, transformation, and loading processes, thereby forming a data lineage network. Existing data value assessment and archiving methods mostly still focus on single data entities as the analysis object, making it difficult to fully explore the value transfer relationships between upstream and downstream nodes in the data lineage network, and failing to effectively consider the potential collaborative cross-validation effects between data at the same level. At the same time, the transformation complexity of data processing, the frequency of downstream anomalies, and the uncertainty of changes in the enterprise's macro-business objectives all have a significant impact on the long-term reliability and potential value of the underlying data entities. If only static value decay models and fixed archiving thresholds are used, it is not only difficult to depict the true decay law of data value changes due to business fluctuations, but also impossible to dynamically adjust according to the current storage resource pressure of the system. Summary of the Invention

[0004] To improve the scientific rigor and business relevance of data time value assessment, and to reduce overall storage costs while maximizing data business value, this invention provides a data lifecycle archiving decision-making method that incorporates a time value model. This method includes:

[0005] S1. Acquire multi-source heterogeneous data and construct a multi-dimensional lineage graph containing data entity nodes, data processing process nodes, and business target nodes; calculate a dynamic transformation ambiguity index for each data processing process node; and calibrate the dynamic transformation ambiguity index based on the frequency of anomalies in downstream data entities; establish horizontal associations representing cross-validation relationships among data entity nodes and assign collaborative validation weights; input business target entropy for business target nodes; the business target entropy is used to represent the uncertainty of future changes to business targets; S2. Calculate the initial time value of data entities based on the multi-dimensional lineage graph; the initial time value is composed of upstream data value, transformation reliability coefficient, and collaborative validation... The weights are jointly determined; a time value decay model for data entities is constructed; the decay rate of the time value decay model is controlled by a decay factor coupled with the business target entropy, and the decay factor is inversely proportional to the weighted average of the business target entropy of all associated business target nodes; S3, an archive value threshold positively correlated with the system storage resource utilization rate and the data entity storage cost index is set; the time value of data entities is continuously tracked, and when the time value is lower than the threshold and the business target entropy of all associated business target nodes is lower than the preset entropy threshold, the archive operation of the data entity is triggered; wherein, for data entities without associated business target nodes, the preset default comprehensive business target entropy is used to determine the entropy threshold.

[0006] Furthermore, the construction of the multidimensional lineage graph, which includes data entity nodes, data processing process nodes, and business target nodes, specifically includes: parsing the database metadata and calculation job scripts of the business system, extracting table structure and data flow information, and generating data entity nodes and data processing process nodes; identifying and generating business target nodes by recognizing the front-end routing requests and feedback page parameters of the business application system; establishing directed edges between the data entity nodes and data processing process nodes according to the data flow information, and associating the business target nodes with the end data entity nodes that provide data support for the nodes, thereby generating the multidimensional lineage graph.

[0007] Furthermore, the calculation of the dynamic conversion ambiguity index for each data processing node includes: extracting the control flow graph of the data processing node, counting the number of conditional judgment statements as the number of conditional branches; calculating the vector cosine distance between the input data and the output data in terms of field type and field number as the data pattern difference; obtaining the number of version releases of the software components that the data processing node depends on within a preset time window as the frequency of dependent component changes; normalizing and weighting the number of conditional branches, the data pattern difference, and the frequency of dependent component changes to obtain the initial value of the dynamic conversion ambiguity index; counting the total number of errors triggered by downstream data entities in data quality detection, using the total number of errors to construct a penalty coefficient, and multiplying the initial value by the penalty coefficient to complete the reverse calibration.

[0008] Furthermore, the step of establishing a horizontal association representing a cross-validation relationship between data entity nodes and assigning collaborative validation weights specifically includes: identifying pairs of data entity nodes with the same primary key field but belonging to different data tables in a multidimensional lineage graph; statistically analyzing the frequency of joint occurrence of the data entity node pairs in system call records within a preset historical time period; calculating the Pearson correlation coefficient between the data records of the data entity node pairs; normalizing the joint occurrence frequency; and then weighting and summing the result with the absolute value of the Pearson correlation coefficient to obtain the collaborative validation weights for the horizontal association.

[0009] Further, the step of inputting business target entropy for business target nodes includes: statistically analyzing the frequency of access to each business target node and the number of modifications to associated business configurations within a preset period; fitting the frequency and number of modifications using a time series prediction algorithm to calculate the future access probability distribution and change probability distribution, respectively; calculating the first information entropy of the future access probability distribution and the second information entropy of the change probability distribution based on the Shannon information entropy formula, and normalizing the first and second information entropies according to the maximum entropy value corresponding to the number of prediction time steps to obtain the first normalized information entropy and the second normalized information entropy; and weightedly fusing the first and second normalized information entropies to generate the business target entropy used to represent the uncertainty of changes.

[0010] Furthermore, the calculation of the initial time value of a data entity based on a multidimensional kinship graph includes: obtaining the transfer value of an upstream data entity, wherein the transfer value is positively correlated with the time value of the upstream data entity; calculating a conversion reliability coefficient based on the dynamic conversion ambiguity index of the data processing nodes connecting the upstream data entity and the current data entity, wherein the conversion reliability coefficient is inversely proportional to the dynamic conversion ambiguity index; multiplying the transfer value by the conversion reliability coefficient to obtain the main link value; extracting other data entities that have a horizontal association with the current data entity, multiplying the time value of the entity in the previous calculation cycle by the corresponding collaborative verification weight, and summing them to obtain the collaborative gain value; and adding the main link value and the collaborative gain value to obtain the initial time value of the current data entity.

[0011] Further, the construction of the time value decay model for the data entity includes: acquiring all business target nodes with associated paths to the current data entity; if no business target nodes have associated paths to the current data entity, then using a preset default comprehensive business target entropy as the comprehensive business target entropy of the current data entity; if business target nodes have associated paths to the current data entity, then calculating the path decay weight of each business target node based on the hop count of the associated path, and using the path decay weight to perform a weighted average calculation of the business target entropy of all business target nodes to obtain the comprehensive business target entropy; when the comprehensive business target entropy is lower than a preset minimum entropy value, using the preset minimum entropy value instead of the comprehensive business target entropy as the lower limit of the denominator; dividing a preset benchmark decay rate constant matching the residence time unit by the processed comprehensive business target entropy to obtain the decay factor; wherein, when the residence time is in days, the unit of the preset benchmark decay rate constant is the reciprocal of days; using an exponential decay function, with the initial time value as the base and the decay factor as the exponential decay rate, constructing a time value decay model.

[0012] Furthermore, the setting of the archive value threshold positively correlated with the system storage resource utilization rate and the data entity storage cost index includes: real-time monitoring of the total disk capacity and used capacity of the storage cluster through the system daemon process to calculate the system storage resource utilization rate; obtaining the unit storage cost of the storage medium where the current data entity is located, as well as the occupied capacity of the current data entity, and calculating the data entity storage cost index based on the unit storage cost and occupied capacity; multiplying the system storage resource utilization rate by a preset elasticity coefficient, and adding the product of the normalized value of the data entity storage cost index and a preset weighting coefficient to obtain a resource pressure index; multiplying the preset basic archive threshold by the resource pressure index to generate the archive value threshold.

[0013] Furthermore, the triggering of the archiving operation for the data entity includes: when the triggering condition is met, migrating the data entity from an online high-performance storage node to an offline low-cost storage node; updating the storage location attribute of the data entity node in the multidimensional lineage graph and marking it as an archived state; automatically generating a data routing mapping record containing the original storage path, the new storage path, and the archived timestamp, and updating it in the system's global routing gateway to maintain the topological integrity of the data lineage graph.

[0014] The present invention has the following technical effects: This invention constructs a multi-dimensional lineage graph encompassing data entities, processing procedures, and business objectives. It establishes horizontal relationships between data entity nodes and introduces business objective entropy into business objective nodes, thus characterizing the complexity of data flow and the uncertainty of future changes in business objectives. By combining upstream data value, conversion ambiguity index, and collaborative verification weights to calculate the initial time value of data entities, and dynamically controlling the decay rate of data value using a decay factor coupled with business objective entropy, the data value assessment results more closely align with the actual business flow process. Furthermore, by setting an archiving value threshold positively correlated with system resource consumption and data entity storage cost indicators, and by continuously tracking data time value and triggering archiving after dual judgment based on business objective entropy, the invention improves the scientific rigor and precision of data lifecycle archiving decisions, reduces the risk of mis-archiving high-potential business value data, optimizes system storage resource allocation efficiency, and achieves a dynamic balance between data availability and storage operation and maintenance costs. Attached Figure Description

[0015] Figure 1 A flowchart illustrating a data lifecycle archiving decision-making method incorporating a time value model, provided as an embodiment of the present invention; Figure 2 This is a schematic diagram of the probability distribution of service access in the next seven predicted time steps provided by an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the change of the archive value threshold with storage resource utilization rate under different unit storage costs provided in this embodiment of the invention; Figure 4 This is a schematic diagram comparing the changes in cluster storage resource utilization during the test period under different archiving strategies provided in this embodiment of the invention. Detailed Implementation

[0016] This invention provides a data lifecycle archiving decision-making method that incorporates a time value model, referring to... Figure 1 Including S1-S3: S1: Data Acquisition and Preprocessing.

[0017] Specifically, the process involves acquiring multi-source heterogeneous data and constructing a multi-dimensional lineage graph containing data entity nodes, data processing process nodes, and business target nodes; calculating a dynamic transformation ambiguity index for each data processing process node; calibrating the dynamic transformation ambiguity index based on the frequency of anomalies in downstream data entities; establishing horizontal associations representing cross-validation relationships among data entity nodes and assigning collaborative validation weights; and inputting business target entropy into the business target nodes, which represents the uncertainty of future changes to the business target. First, using the data integration tool Apache NiFi, relational databases and distributed file systems are connected via JDBC drivers to extract structured and unstructured data. Next, NetworkX is used to initialize a graph object supporting multiple edge types, where data flow edges are set as directed edges, and lateral cross-validation relationships are represented by bidirectional equal-weighted edges or undirected association attributes. Data entity nodes, data processing process nodes, and business target nodes are created and added to the graph object to form a multi-dimensional lineage graph. Then, AST is used to parse the data processing script, traversing the abstract syntax tree to count the number of nodes in conditional statements as the number of conditional branches. Finally, metadata feature vectors for input and output data are constructed using field types and field counts. The cosine distance between these metadata feature vectors is calculated as the data pattern difference. By calling Git's log query commands, the number of code commits or version releases of the target component within a set time window is counted as the dependency component change frequency.

[0018] After standardizing the number of conditional branches, data pattern differences, and frequency of dependent component changes using the max-min normalization algorithm, the initial value of the dynamic transformation ambiguity index is obtained by weighted summation. The number of times the primary key corresponding to the downstream data entity appears in the error log is counted using the distributed log search engine Elasticsearch as the anomaly frequency. The anomaly frequency is mapped to an anomaly feedback coefficient between 0 and 1 using the Sigmoid activation function. The initial value of the dynamic transformation ambiguity index is multiplied by the calibration factor obtained by adding 1 to the anomaly feedback coefficient to obtain the reverse-calibrated dynamic transformation ambiguity index. The reverse-calibrated dynamic transformation ambiguity index is used to represent the transformation uncertainty amplified by anomaly feedback. Its value is allowed to be greater than 1 and is used for subsequent transformation reliability coefficient calculation.

[0019] Extract business indicator numerical sequences of different data entities under the same time dimension, and use the Pearson correlation coefficient algorithm to calculate the correlation coefficient between the indicator numerical sequences. When the absolute value of the correlation coefficient is greater than the set correlation threshold, create undirected edges between the corresponding data entity nodes as horizontal associations to represent cross-validation relationships. A positive correlation coefficient indicates a unidirectional cross-validation relationship, and a negative correlation coefficient indicates a reverse cross-validation relationship. Further, combine the normalized value of the joint occurrence frequency and the absolute value of the Pearson correlation coefficient to calculate the collaborative validation weight, keeping the collaborative validation weight within the range of 0 to 1. Obtain time-series data formed by the frequency of access to business targets and the number of times related business configurations have been modified in historical periods, and predict the future access probability distribution and the future change probability distribution respectively. Use the Shannon information entropy algorithm to calculate the access uncertainty and change uncertainty respectively, and use the change uncertainty as the main weight to fuse and obtain the business target entropy.

[0020] In one embodiment, multi-source heterogeneous data is acquired, and a multi-dimensional lineage graph containing data entity nodes, data processing process nodes, and business target nodes is constructed. This includes: parsing the database metadata and calculation job scripts of the business system, extracting table structure and data flow information, and generating data entity nodes and data processing process nodes; identifying and generating business target nodes by recognizing the front-end routing requests and feedback page parameters of the business application system; establishing directed edges between the data entity nodes and data processing process nodes according to the data flow information, and associating the business target nodes with the end data entity nodes that provide data support for the nodes, thereby generating a multi-dimensional lineage graph.

[0021] This approach leverages database connectivity to connect to a relational database. It reads table structures, primary keys, and foreign key constraints from the database metadata dictionary to generate attribute sets for data entity nodes. Simultaneously, it uses an abstract syntax tree parsing tool to parse the query scripts in the extraction, transformation, and loading tasks, extracting keywords such as selection, join, and insertion. It identifies field-level data flow information and encapsulates this information into data processing nodes. For example, for a data processing job containing an order total amount field, the processing job is instantiated as an intermediate transformation node. It intercepts front-end routing requests from the business application system through gateway logs, parses resource locator paths and feedback page structure parameters, and automatically generates business target nodes using regular expression matching algorithms, such as daily active user statistics or user growth dashboards. In the graph database, directed edges are created with data flow as the directional attribute, and terminal data entity nodes, such as application-layer wide tables after aggregation and cleaning, are directly connected to the corresponding business target nodes through supporting edges, constructing a multi-dimensional lineage graph with both technical and business logic attributes.

[0022] In one embodiment, the dynamic transformation ambiguity index is calculated for each data processing node, including: extracting the control flow graph of the data processing node and counting the number of conditional judgment statements as the number of conditional branches; calculating the vector cosine distance between input data and output data in terms of field type and field number as the data pattern difference; obtaining the number of version releases of the software components that the data processing node depends on within a preset time window as the dependent component change frequency; normalizing and weighting the number of conditional branches, data pattern difference, and dependent component change frequency to obtain the initial value of the dynamic transformation ambiguity index; counting the total number of errors triggered by downstream data entities in data quality detection, constructing a penalty coefficient greater than or equal to 1 using the total number of errors, and multiplying the initial value by the penalty coefficient to complete the reverse calibration.

[0023] For complex logic code in data processing nodes, a static code analysis framework is used to extract the control flow graph and count the number of conditional branch statements in the code. The metadata structure of the input and output data is transformed into feature vectors. For example, the input contains 5 integer fields and 2 character fields, and the output contains 3 integer fields. The cosine distance between the two feature vectors is used to represent the data pattern difference, with a value between 0 and 1. Simultaneously, the continuous integration pipeline platform interface is invoked to obtain the number of code commits or version releases of dependent components within a preset time window, such as 30 days. The extreme value normalization method is used to map the number of conditional branches (e.g., 8), the data pattern difference (e.g., 0.45), and the frequency of dependent component changes (e.g., 3) to the same dimension interval, and corresponding weight coefficients are assigned, such as 0.4, 0.4, and 0.2, respectively. A weighted sum is then calculated to obtain the initial ambiguity index, such as 0.68. For reverse calibration, the total number of null value or format out-of-bounds errors triggered by the data quality probes of downstream nodes in the scheduling platform in the past seven days is queried (e.g., 5 errors). The formula is then used to calculate the initial ambiguity index. Calculate the penalty coefficient, where The initial value is multiplied by this coefficient to obtain the dynamic conversion ambiguity index after being corrected for the actual business situation, representing the number of errors reported.

[0024] In one embodiment, a horizontal association representing a cross-validation relationship is established between data entity nodes, and collaborative validation weights are assigned. This includes: identifying pairs of data entity nodes with the same primary key field but belonging to different data tables in a multidimensional lineage graph; counting the frequency of joint occurrence of the data entity node pairs in system call records within a preset historical time period; calculating the Pearson correlation coefficient between the data records of the data entity node pairs; and after normalizing the joint occurrence frequency, weighting and summing it with the absolute value of the Pearson correlation coefficient to obtain the collaborative validation weights of the horizontal association.

[0025] The system executes queries in the graph database, traversing table attributes to find fields with the same name marked as primary keys, or fields that meet a preset high cardinality threshold after uniqueness testing and are marked as equivalent primary keys, such as user identification numbers or device addresses. It matches and identifies data entity node pairs belonging to different database instances or different tables. It then accesses the audit log cluster to statistically analyze the number of times the same transaction or query request simultaneously hits the aforementioned node pairs within a preset historical time period, such as the past 90 days, calculating the frequency of joint occurrences, for example, an average of 500 joint queries per day. To assess the coupling degree of the data's intrinsic value, it extracts core numerical metrics fields from the data entity node pairs within the past week, such as the transaction amount sequence of one table and the active points sequence of another, calculating their correlation coefficient. The Pearson correlation coefficient is used to represent the degree of linear correlation. If the Pearson correlation coefficient is negative, its sign is retained as a reverse correlation attribute. However, in the calculation of the collaborative verification weight, its absolute value is used to represent the stable reverse verification strength. The joint occurrence frequency is mapped to an interval using the max-min normalization method and normalized to between 0 and 1 to avoid negative or unbounded values ​​from the standard score normalization. The normalized joint occurrence frequency and the absolute value of the Pearson correlation coefficient are weighted and summed according to a preset ratio coefficient to obtain the collaborative verification weight. For example, the ratio coefficients are 0.6 and 0.4, respectively, and the absolute value of the Pearson correlation coefficient is 0.75. The example value of this calculation result is 0.82, which is persistently stored as the collaborative verification weight attribute of the horizontal undirected edge between the two data nodes in the graph, thereby representing the importance of cross-verification between the two tables in the actual business ecosystem.

[0026] In one embodiment, inputting business target entropy for a business target node includes: counting the frequency of access to each business target node within a preset period and the number of times associated business configurations are modified; fitting the frequency and number of modifications using a time series prediction algorithm to calculate the future access probability distribution and change probability distribution, respectively; calculating the first information entropy of the future access probability distribution and the second information entropy of the change probability distribution based on the Shannon information entropy formula, and normalizing the first and second information entropies according to the maximum entropy value corresponding to the number of prediction time steps to obtain the first normalized information entropy and the second normalized information entropy; and weighting and fusing the first normalized information entropy and the second normalized information entropy to generate a business target entropy representing the uncertainty of changes.

[0027] The system utilizes monitoring metrics and operation audit logs from the configuration center to statistically analyze pre-defined periodic data, such as the access frequency of each business target node within the last thirty days, and the number of modifications to related business configurations. For example, it calculates the total frequency of calls to the core dashboard page via the front-end application programming interface, and the number of changes to business rule thresholds. The access frequency and modification counts are then arranged chronologically to form a time-series feature vector, which is then input into a pre-defined time-series prediction model for fitting. This model can employ a Long Short-Term Memory (LSTM) network, an exponential smoothing model, a grey prediction model, or an autoregressive moving average model to output the predicted access frequency and modification counts for each time step in the next seven days. The predicted access frequency and modification counts are then non-negatively and normally processed to obtain the future access probability distribution and the future change probability distribution, respectively, for use in subsequent business target entropy calculations. A schematic diagram of the business access probability distribution for the next seven predicted time steps is shown below. Figure 2 As shown.

[0028] After fitting and training, the network outputs a predicted sequence for seven consecutive time steps, i.e., the next week. First, the predicted access frequency and predicted modification frequency are non-negatively processed. When the sum of the seven-day predictions is zero, a preset low-entropy distribution is used as a substitute probability distribution, or the corresponding normalized information entropy is set to 0. Then, the predicted sequence is normalized by dividing the daily predicted value by the total seven-day predicted value, converting them into statistically consistent future access probability distribution arrays and change probability distribution arrays. After obtaining the probability distribution arrays, the Shannon information entropy calculation model is called to process the data, and the calculation results are normalized using the maximum entropy value corresponding to the seven predicted time steps, ensuring that both the first and second normalized information entropies are within the range of 0 to... Within the range of 1; for example, if the predicted access probability distribution within a week is uniform, the calculated first normalized information entropy will be higher, i.e., close to 1; conversely, if the changes are concentrated on a certain day, the second normalized information entropy will be lower, i.e., indicating high certainty; a smoothing adjustment mechanism is adopted, assigning a weight coefficient of 0.3 to the first normalized information entropy and a weight coefficient of 0.7 to the second normalized information entropy, and linearly superimposing and fusing the two to obtain the specific business target entropy value. This value represents the complexity of the business scenario's fluctuations in the short term, where the second normalized information entropy corresponds to the uncertainty of business configuration changes, serving as the main component representing the uncertainty of future changes in the business target; the first normalized information entropy is used to supplement the representation of the uncertainty of the future access time sequence distribution of the business target.

[0029] S2: Determination of initial time value.

[0030] Specifically, the initial time value of data entities is calculated based on a multidimensional lineage graph; the initial time value is jointly determined by the upstream data value, the conversion reliability coefficient, and the collaborative verification weight; a time value decay model for data entities is constructed; the decay rate of the time value decay model is controlled by a decay factor coupled with the business target entropy, and the decay factor is inversely proportional to the weighted average of the business target entropy of all associated business target nodes.

[0031] Before performing the initial time value recursive calculation of data entities, the topology is first sorted according to the directed lineage edges. If a directed cycle is detected, the nodes within the cycle are first shrunk into composite nodes according to the strongly connected components, or split into cross-cycle directed edges according to the data generation timestamp before the topology is sorted. For source table nodes or original file nodes that do not have upstream data entities, the basic time value is obtained by normalizing and weighting according to their data quality score, recent access frequency, business tag level, and compliance retention requirements, and is used as the starting point for recursion. For data entities with horizontal undirected associations, only the time value snapshot formed in the previous calculation cycle is read to participate in the collaborative gain calculation. The newly calculated collaborative gain is not written back to participate in other horizontal edge calculations in the same calculation cycle to avoid the circular accumulation caused by bidirectional horizontal associations.

[0032] A depth-first search algorithm is used to traverse the multidimensional lineage graph to locate all direct upstream data entities of the current data entity and obtain the time value of all upstream data entities. Based on the dynamic conversion ambiguity index of the data processing nodes connecting each upstream data entity to the current data entity, a conversion reliability coefficient is calculated. For example, adding 1 to the dynamic conversion ambiguity index yields the reliability denominator, and dividing 1 by the reliability denominator gives the conversion reliability coefficient. The time value of each upstream data entity is multiplied by its corresponding conversion reliability coefficient and then summed to obtain the vertical transfer value. All horizontal association nodes of the current data entity are traversed, and the time value of each horizontally associated data entity in the previous calculation cycle is multiplied by its corresponding collaborative verification weight and then summed to obtain the horizontal collaborative value. The vertical transfer value and the horizontal collaborative value are then added to obtain the initial time value of the current data entity.

[0033] A time value decay model is constructed based on the exponential decay law. Real-time time value equals the initial time value multiplied by the power of the natural logarithm base, where the exponent is the negative of the product of the decay factor and the time difference since the data entity's creation. The shortest path algorithm is used to calculate the graph theory shortest path distance from the current data entity node to its downstream business target node in the multidimensional lineage graph. If no reachable downstream business target node is found, a preset default comprehensive business target entropy is used as the comprehensive business target entropy of the current data entity; if a reachable downstream business target node is found, the graph theory shortest path algorithm is applied. The path attenuation weight is calculated based on distance and a preset damping coefficient. The weighted average of the business target entropy of all associated business target nodes is then calculated based on the path attenuation weight as the comprehensive business target entropy. When the comprehensive business target entropy is 0 or lower than a preset minimum entropy value, the preset minimum entropy value is used instead of the comprehensive business target entropy as the lower limit of the denominator. The preset basic attenuation rate constant is divided by the processed comprehensive business target entropy to obtain the attenuation factor that regulates the attenuation rate. When the comprehensive business target entropy is 0 or lower than a preset minimum entropy value, the preset minimum entropy value is used as the lower limit of the denominator to avoid division by zero or abnormal amplification of the attenuation factor.

[0034] In one embodiment, calculating the initial time value of a data entity based on a multidimensional kinship graph includes: obtaining the transfer value of an upstream data entity, wherein the transfer value is positively correlated with the time value of the upstream data entity; calculating a conversion reliability coefficient based on the dynamic conversion ambiguity index of the data processing nodes connecting the upstream data entity and the current data entity, wherein the conversion reliability coefficient is inversely proportional to the dynamic conversion ambiguity index; multiplying the transfer value by the conversion reliability coefficient to obtain the main link value; extracting other data entities that have a horizontal association with the current data entity, multiplying the time value of the entity in the previous calculation cycle by the corresponding collaborative verification weight, and summing them to obtain the collaborative gain value; and adding the main link value and the collaborative gain value to obtain the initial time value of the current data entity.

[0035] Based on the lineage graph, a depth-first traversal algorithm is executed to extract the predecessor node of the current data entity. If the current data entity does not have a predecessor node, it is identified as a source table node, and the source table basic time value is generated according to the preset basic value initialization rule. If the current data entity has a predecessor node, the time value of each predecessor node in the previous calculation cycle or the current topology level is read. The time value of the upstream data entity, for example, 100, is multiplied by a preset fixed loss coefficient, for example, 0.9, to determine the transmission value. The dynamic conversion ambiguity index attached to the data processing node (operator) connecting the upstream and downstream relationships is obtained, for example, the calculated value is 0.85. The conversion reliability coefficient is calculated using the dynamic conversion ambiguity index, for example, the conversion reliability coefficient is 1 / (1+0.85), which is about 0.54. Then, the transmission value is multiplied by the conversion reliability coefficient to complete the information attenuation conversion of the data from processing to output, generating the main link value, for example, about 48.6, which represents the vertical value retained after the data is processed by specific business rules.

[0036] In the graph, find all horizontal sibling data entity nodes that are related to the current data entity through foreign keys or business logic. Extract the time value of these horizontal entities in the previous calculation period. For example, the value of node B is 50. Multiply this value with the collaboration verification weight between the two nodes, for example, 0.78. Accumulate the calculation results of all eligible horizontal entities to obtain the collaboration gain value. For example, the gain of a single node is 39. Add the vertical main link value to the horizontal collaboration gain value, for example, 87.6. Calculate the initial time value of the current data entity that integrates upstream and downstream lineage and business relationships. Since the horizontal collaboration gain only references the value snapshot of the previous calculation period, there will be no cyclic calculation of mutual amplification between sibling nodes in the current period.

[0037] In one embodiment, constructing a time value decay model for a data entity includes: acquiring all business target nodes with associated paths to the current data entity; if no business target nodes have associated paths to the current data entity, then using a preset default comprehensive business target entropy as the comprehensive business target entropy of the current data entity; if business target nodes have associated paths to the current data entity, then calculating the path decay weight of each business target node based on the hop count of the associated path, and using the path decay weight to perform a weighted average calculation of the business target entropy of all business target nodes to obtain the comprehensive business target entropy; when the comprehensive business target entropy is lower than a preset minimum entropy value, using the preset minimum entropy value instead of the comprehensive business target entropy as the lower limit of the denominator; dividing a preset benchmark decay rate constant matching the dwell time unit by the processed comprehensive business target entropy to obtain a decay factor; wherein, when the dwell time is in days, the unit of the preset benchmark decay rate constant is the reciprocal of days; and constructing a time value decay model using an exponential decay function, with the initial time value as the base and the decay factor as the exponential decay rate.

[0038] In the multidimensional lineage graph, a breadth-first search algorithm is initiated. Starting from the currently evaluated data entity node, it scans downstream along the directed data flow edges to all reachable end-point business target nodes, recording the shortest path hop count from the starting point to each business node as k. If the scan result is empty, the current data entity is identified as a data entity not yet associated with a business target, and the preset default comprehensive business target entropy is read as the comprehensive business target entropy of this data entity, and the time value decay model continues to be constructed. If the scan result is not empty, the path decay weight of the path's impact on the business is calculated using the exponential decay formula, the formula being: The damping coefficient The value is set to 0.8, which assigns a lower impact weight to edge businesses far from the current entity. The entropy attribute values ​​of the business targets attached to all related business target nodes are extracted, and the smoothed comprehensive business target entropy is obtained through a normalized weighted average algorithm of path decay weights; for example, the comprehensive entropy is 0.773. The model configuration module has a built-in benchmark decay rate constant representing the theoretical extreme value of data depreciation, for example, set in the range of 0.01 to 0.1, taking 0.05. When the residence time t is in days, the unit of the benchmark decay rate constant is the reciprocal of days. By dividing this constant by the previously calculated comprehensive business target entropy, i.e., 0.05 divided by 0.773, a decay factor that can reflect the uncertainty characteristics of the business is generated, calculated to be approximately 0.064. The initial time value is used as the initial value and denoted as... The decay factor is denoted as the negative power decay rate in an exponential function. A continuous-time, continuously differentiable time value decay model function is constructed. , where t represents the number of days the data has resided since its generation, thus depicting the curve trajectory of value declining exponentially over time.

[0039] S3: Entropy threshold determination and archiving.

[0040] Specifically, an archive value threshold is set that is positively correlated with the system storage resource utilization rate and data entity storage cost indicators; the time value of data entities is continuously tracked, and when the time value is lower than the threshold and the business target entropy of all associated business target nodes is lower than the preset entropy threshold, the archive operation of the data entity is triggered; among them, the entropy threshold judgment is made by using the preset default comprehensive business target entropy for data entities without associated business target nodes.

[0041] First, using the application programming interface (API) of the open-source system monitoring and alarm framework Prometheus, the current disk usage and total disk capacity of the distributed file system are collected. The current disk usage is divided by the total disk capacity to obtain the system storage resource utilization rate. Next, the storage unit price per GB / month for cloud storage is obtained through the cloud service provider's billing API, and the current occupied capacity of data entities is read. The data entity storage cost index is calculated based on the storage unit price and occupied capacity. The system storage resource utilization rate is multiplied by a preset elasticity coefficient, and the normalized data entity storage cost index is multiplied by a preset weighting coefficient. The sum of these two values ​​yields the resource pressure index. Finally, a preset basic archiving threshold is multiplied by the resource pressure index to obtain the archiving value threshold. The variation of the archiving value threshold with storage resource utilization rate under different unit storage costs is illustrated below. Figure 3 As shown.

[0042] A time value tracking task is set up daily using the Apache Airflow scheduled task framework. The time value decay model is used to calculate the current real-time time value of the data entity and determine if this real-time time value is less than the current archive value threshold. Simultaneously, the current business target entropy of all business target nodes associated with the data entity is queried in the multi-dimensional lineage graph, and it is determined whether the entropy of all associated business targets is less than a preset entropy threshold. If the data entity is marked as not supporting a business target, its preset default comprehensive business target entropy is read and compared with a preset entropy threshold to determine if the business target entropy judgment condition is met. When both the real-time time value and the business target entropy are below the archive value threshold and the preset entropy threshold are met, the storage policy interface, data replication interface, or migration tool of the distributed file system is invoked to migrate the data file or table corresponding to the data entity from the high-frequency storage node where the solid-state drive resides to the cold storage node where the mechanical hard drive resides, triggering and executing the archive operation for the data entity.

[0043] In one embodiment, an archive value threshold positively correlated with system storage resource utilization and data entity storage cost index is set, including: real-time monitoring of the total disk capacity and used capacity of the storage cluster through a system daemon process to calculate system storage resource utilization; obtaining the unit storage cost of the storage medium where the current data entity is located, and the occupied capacity of the current data entity, and calculating the data entity storage cost index based on the unit storage cost and occupied capacity; multiplying the system storage resource utilization by a preset elasticity coefficient, and adding the product of the normalized value of the data entity storage cost index and a preset weighting coefficient to obtain a resource pressure index; multiplying the preset basic archive threshold by the resource pressure index to generate the archive value threshold.

[0044] By deploying an underlying daemon process, the underlying file system or cloud object storage cluster is polled every 5 minutes via system commands to collect the block usage quota and total available capacity of the storage volume. The system storage resource utilization rate is calculated by dividing the two, for example, reaching 75% in the current high-water level state. The infrastructure resource ledger is queried to obtain the unit storage cost corresponding to the underlying hardware where the current data table resides, and the occupied capacity of the current data entity is read to calculate the data entity storage cost index. For example, if the monthly storage cost of a high-performance solid-state drive is 0.5 yuan / GB·month, and the current data entity occupies 100GB, the corresponding storage cost index is 50 yuan / month. The data entity storage cost index is mapped to the 0 to 1 range using an extreme value mapping method to obtain a normalized cost value. For example, the solid-state drive is normalized to... 1. The internal engine extracts the elasticity coefficient representing the cluster scaling tolerance from the configuration items. For example, the parameter is between 1 and 2, and an example is 1.5. The system storage resource utilization rate is multiplied by the elasticity coefficient to obtain 0.75 × 1.5. The product of the normalized data entity storage cost index and the weight coefficient for controlling financial sensitivity, for example, a preset value of 0.8, is 1 × 0.8. The two are added together to generate the comprehensive resource pressure index of the current storage environment, which is 1.925. The model extracts the built-in basic archiving threshold. For example, if the basic archiving threshold is set to 30, the basic value is multiplied by the dynamically calculated resource pressure index, which is 30 × 1.925, to generate a floating archiving value threshold, which is 57.75. This allows the archiving threshold to be automatically increased when storage pressure is tight, triggering a larger-scale offline archiving.

[0045] In one embodiment, triggering the archiving operation for the data entity includes: when the triggering condition is met, migrating the data entity from an online high-performance storage node to an offline low-cost storage node; updating the storage location attribute of the data entity node in the multidimensional lineage graph and marking it as archived; automatically generating a data routing mapping record containing the original storage path, the new storage path, and the archive timestamp, and updating it in the system's global routing gateway to maintain the topological integrity of the data lineage graph.

[0046] When the scheduling engine detects that the value of a data entity is lower than the archiving threshold and the business entropy condition is met, an asynchronous data migration work order is generated. By calling the batch processing synchronization framework, the heterogeneous data source migration instruction is executed, and the data tables on the online high-performance hot nodes are extracted in full or incrementally, compressed and encoded into columnar storage format, and then dumped into the offline low-cost storage resource pool. After the migration is completed, the graph database is written back through the interface to modify the archiving status flag of the data entity node to true, and the storage address field bound to the node is updated to reflect the true cold storage location. To ensure transparent querying of the upper-layer business intelligence application system during the data media replacement process, formatted routing redirection metadata is generated in real time. This metadata contains information pairs of source data mapped to target addresses and corresponding standard format archiving timestamps. These records are sent to the hot-loaded memory of the global routing gateway component of the system architecture layer in real time. Through this mechanism, not only is it ensured that the underlying migration does not cause the multidimensional lineage graph to break, but also automatic rewriting and intelligent routing distribution are realized in the historical traceability scenario.

[0047] The experiment used a real data warehouse environment of a large e-commerce company for comparative verification. The dataset consisted of 1.5 million data records from three core business domains: historical orders, user behavior trajectories, and product logs. The experiment was divided into a control group and an experimental group. The control group adopted a traditional fixed-time-span archiving strategy based on the most recent access time, while the experimental group adopted the archiving mechanism proposed in this invention based on a multi-dimensional lineage graph and a time value decay model. Both groups were run independently in a distributed storage cluster containing 100 data nodes, with a test period of 90 consecutive days to evaluate and compare the actual impact of different strategies on storage resource utilization and business data access. After a 90-day testing period, system performance metrics for both groups were collected and analyzed. In the control group, storage resource utilization gradually decreased from an initial 75% and remained at 68%. Data write-back requests due to erroneous archiving caused by static rules reached an average of 4200 times per day, and the average query response time for core reports was 1.2 seconds. In the experimental group, storage resource utilization steadily decreased to 42%. Thanks to multidimensional lineage analysis and the time value model, the number of data write-back requests caused by erroneous archiving plummeted to an average of 150 times per day, and the query response time for core reports was shortened to an average of 0.3 seconds. Furthermore, the historical data retrieval rate triggered in the offline low-cost cold storage nodes in the experimental group was 0.5%. The comparison of cluster storage resource utilization changes during the testing period under different archiving strategies is illustrated below. Figure 4 As shown.

[0048] The archiving mechanism has improved both storage cost reduction and critical business performance. The experimental group adjusted the archiving value threshold by monitoring the system storage resource utilization rate, the unit storage cost of the underlying media, and the data entity storage cost index formed by the data entity's occupied capacity in real time. This reduced the system storage waste caused by mechanical time rules and safely released nearly half of the high-performance online storage space. By combining the initial time value assessment of upstream and downstream main links and horizontal collaborative entities with multi-dimensional lineage graphs, and then using business target entropy to deduce the value decay trajectory, it is possible to identify the true value retention of different data tables in the current business ecosystem, ensuring that high-value business flow data resides on high-performance nodes and reducing the computational overhead of repeated data migration.

[0049] The above are preferred embodiments of the present invention. It should be noted that, for those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A data lifecycle archiving decision-making method combining a time value model, characterized in that, include: S1. Acquire multi-source heterogeneous data and construct a multi-dimensional lineage graph containing data entity nodes, data processing process nodes, and business target nodes; calculate the dynamic transformation ambiguity index for each data processing process node; calibrate the dynamic transformation ambiguity index based on the frequency of anomalies in downstream data entities; establish horizontal associations representing cross-validation relationships among data entity nodes and assign collaborative validation weights; input business target entropy for business target nodes; business target entropy is used to represent the uncertainty of future changes to business targets. S2: Obtain the transmission value of the upstream data entity, which is positively correlated with the time value of the upstream data entity; calculate the conversion reliability coefficient based on the dynamic conversion ambiguity index of the data processing nodes connecting the upstream data entity and the current data entity, which is inversely proportional to the dynamic conversion ambiguity index; multiply the transmission value by the conversion reliability coefficient to obtain the main link value; extract other data entities that have horizontal correlation with the current data entity, multiply the time value of the entity in the previous calculation cycle by the corresponding collaborative verification weight, and sum them to obtain the collaborative gain value; add the main link value and the collaborative gain value to obtain the initial time value of the current data entity; the initial time value is composed of the upstream data value and the conversion... The reliability coefficient and collaborative verification weight are jointly determined; all business target nodes with associated paths to the current data entity are obtained. If no business target node with associated paths to the current data entity exists, the preset default comprehensive business target entropy is used as the comprehensive business target entropy of the current data entity; if business target nodes with associated paths to the current data entity exist, the path attenuation weight of each business target node is calculated based on the hop count of the associated path, and the business target entropy of all business target nodes is weighted and averaged using the path attenuation weight to obtain the comprehensive business target entropy; when the comprehensive business target entropy is lower than the preset minimum entropy value, the preset minimum entropy value is used instead of the comprehensive business target entropy as the lower limit of the denominator. The decay factor is obtained by dividing the preset baseline decay rate constant, which matches the residence time unit, by the processed comprehensive business target entropy. When the residence time is in days, the unit of the preset baseline decay rate constant is the reciprocal of days. Using an exponential decay function, a time value decay model is constructed with the initial time value as the base and the decay factor as the exponential decay rate. The decay rate of the time value decay model is controlled by the decay factor coupled with the business target entropy, and the decay factor is inversely proportional to the weighted average of the business target entropies of all associated business target nodes. S3 sets an archiving value threshold that is positively correlated with the system storage resource utilization rate and data entity storage cost indicators. The time value of data entities is continuously tracked; when the time value is lower than the threshold and the business target entropy of all associated business target nodes is lower than the preset entropy threshold, an archiving operation is triggered on the data entity. For data entities without associated business target nodes, the preset default comprehensive business target entropy is used for entropy threshold judgment.

2. The data lifecycle archiving decision-making method combining a time value model according to claim 1, characterized in that, The construction of the multidimensional lineage graph, comprising data entity nodes, data processing process nodes, and business target nodes, specifically includes: parsing the database metadata and computational job scripts of the business system, extracting table structure and data flow information, and generating data entity nodes and data processing process nodes; identifying and generating business target nodes by recognizing the front-end routing requests and feedback page parameters of the business application system; establishing directed edges between the data entity nodes and data processing process nodes according to the data flow information, and associating the business target nodes with the end data entity nodes that provide data support for the nodes, thereby generating the multidimensional lineage graph.

3. The data lifecycle archiving decision-making method combining a time value model according to claim 1, characterized in that, The calculation of the dynamic transformation ambiguity index for each data processing node includes: extracting the control flow graph of the data processing node and counting the number of conditional judgment statements as the number of conditional branches; calculating the vector cosine distance between the input data and the output data in terms of field type and field quantity as the data pattern difference degree; obtaining the number of version releases of the software components that the data processing node depends on within a preset time window as the frequency of dependent component changes; normalizing and weighting the number of conditional branches, the data pattern difference degree, and the frequency of dependent component changes to obtain the initial value of the dynamic transformation ambiguity index; counting the total number of errors triggered by downstream data entities in data quality detection, using the total number of errors to construct a penalty coefficient, and multiplying the initial value by the penalty coefficient to complete the reverse calibration.

4. The data lifecycle archiving decision-making method combining a time value model according to claim 1, characterized in that, The step of establishing a horizontal association representing a cross-validation relationship between data entity nodes and assigning collaborative validation weights specifically includes: identifying pairs of data entity nodes with the same primary key field but belonging to different data tables in a multidimensional lineage graph; statistically analyzing the frequency of joint occurrence of the data entity node pairs in system call records within a preset historical time period; calculating the Pearson correlation coefficient between the data records of the data entity node pairs; normalizing the joint occurrence frequency and then weighting and summing it with the absolute value of the Pearson correlation coefficient to obtain the collaborative validation weights of the horizontal association.

5. The data lifecycle archiving decision-making method combining a time value model according to claim 1, characterized in that, The step of inputting business target entropy for business target nodes includes: statistically analyzing the frequency of access to each business target node and the number of modifications to associated business configurations within a preset period; fitting the frequency and number of modifications using a time series prediction algorithm to calculate the future access probability distribution and change probability distribution, respectively; calculating the first information entropy of the future access probability distribution and the second information entropy of the change probability distribution based on the Shannon information entropy formula; normalizing the first and second information entropies according to the maximum entropy value corresponding to the number of prediction time steps to obtain the first normalized information entropy and the second normalized information entropy; and weightedly fusing the first and second normalized information entropies to generate the business target entropy representing the uncertainty of changes.

6. The data lifecycle archiving decision-making method combining a time value model according to claim 1, characterized in that, The setting of the archive value threshold positively correlated with system storage resource utilization and data entity storage cost index includes: real-time monitoring of the total disk capacity and used capacity of the storage cluster through the system daemon process to calculate the system storage resource utilization; obtaining the unit storage cost of the storage medium where the current data entity is located, as well as the occupied capacity of the current data entity, and calculating the data entity storage cost index based on the unit storage cost and occupied capacity; multiplying the system storage resource utilization by a preset elasticity coefficient, and adding the product of the normalized value of the data entity storage cost index and a preset weighting coefficient to obtain a resource pressure index; multiplying the preset basic archive threshold by the resource pressure index to generate the archive value threshold.

7. The data lifecycle archiving decision-making method combining a time value model according to claim 1, characterized in that, The triggering of the archiving operation for the data entity includes: when the triggering condition is met, migrating the data entity from an online high-performance storage node to an offline low-cost storage node; updating the storage location attribute of the data entity node in the multidimensional lineage graph and marking it as archived; automatically generating a data routing mapping record containing the original storage path, the new storage path, and the archive timestamp, and updating it in the system's global routing gateway to maintain the topological integrity of the data lineage graph.

Images