Data hotness and data bloodline-based storage strategy optimization method and device

Abstract

The embodiment of the application provides a storage strategy optimization method and device based on data heat and data blood relationship. The dynamic optimization of the storage strategy is realized through the innovative design of a multi-dimensional heat calculation model, node attribute analysis and a graph attention network. A directed graph structure is constructed, a reliable dependency relationship management system is established by combining blood relationship analysis, a layered storage strategy is introduced, and the storage efficiency is ensured through heat division and dependency relationship optimization. The method effectively solves the deficiencies of traditional technologies in heat evaluation, blood relationship analysis and storage optimization, and provides technical support for software system storage management.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically to a method and apparatus for optimizing storage strategies based on data popularity and data lineage. Background Technology

[0002] The evolution of traditional data storage strategies has progressed from storing single structured data to supporting diverse data types. Early storage strategies were primarily based on relational database models, with their core logic being a table-based data organization structure and optimizing read / write performance through indexing mechanisms (such as B+ trees). This storage method performed well in transaction processing scenarios, but with the advent of the big data era, the storage of massive amounts of data and the demands of high-concurrency access exposed its inherent limitations, especially the disk I / O bottleneck, which significantly impacted overall system performance. To address this challenge, all-in-memory engines have gradually become an important storage solution. Database systems such as Hyper, Peloton, and SAP HANA effectively reduce update costs and improve query efficiency by storing hot data in small pages in memory and compressing cold data for storage on larger pages.

[0003] Meanwhile, to address the needs of Hybrid Transactional and Analytical Processing (HTAP) scenarios, academia and industry have proposed various innovative storage strategies. These strategies can be divided into two categories: memory column selection technology based on main row storage and load-driven hybrid row-column storage technology. The former dynamically adjusts the column storage data in memory, enabling frequently accessed columns to be retrieved quickly; the latter automatically switches between row and column storage modes based on data frequency to adapt to different query loads. However, existing research still has certain limitations. For example, manually specifying the set of memory columns is not adaptable enough to changes in query load, while automated selection methods may lead to excessive computational overhead due to complex algorithm design. Therefore, how to reduce the complexity of storage strategies while ensuring efficient querying remains an important research topic.

[0004] Furthermore, the emergence of distributed storage systems has further enriched the choices of data storage strategies. For example, HBase-based distributed storage systems achieve efficient storage and load balancing of industrial time-series data through pre-partitioning and hot / cold data classification strategies. Although these strategies alleviate the shortcomings of traditional storage methods to some extent, they often depend on specific application scenarios and are difficult to directly extend to other fields. Therefore, exploring a storage strategy that combines versatility and flexibility has become an important direction for future research. Summary of the Invention

[0005] To address the problems in existing technologies, this application provides a storage strategy optimization method and apparatus based on data popularity and data lineage, which can effectively solve the shortcomings of traditional technologies in popularity assessment, lineage analysis and storage optimization, and provide technical support for software system storage management.

[0006] To solve at least one of the above problems, this application provides the following technical solution:

[0007] Firstly, this application provides a storage strategy optimization method based on data popularity and data lineage, including:

[0008] The system collects data packet size, communication frequency, data type, and timestamp information of nodes in the software system, statistically analyzes the access frequency, access time, and user behavior of the nodes, calculates the access frequency popularity, time decay popularity, and user behavior popularity of the nodes, and combines the access frequency popularity, time decay popularity, and user behavior popularity to generate a node popularity value.

[0009] Construct a directed graph with the nodes as vertices and the dependencies between nodes as edges. Use the node popularity value as a node attribute and the computing resource consumption of the nodes as edge weights. Use a graph attention network to iteratively train the directed graph to generate a dynamic data lineage graph.

[0010] Based on the heat value of nodes in the dynamic data lineage graph, nodes are divided into cold nodes, warm nodes, and hot nodes. A priority caching strategy is adopted for hot nodes, while cold nodes are compressed and stored. The direct dependencies between cold nodes and hot and warm nodes are retained, while the indirect dependencies between cold nodes are deleted. The node representation vector of the graph attention network is dynamically updated based on the heat value of the nodes.

[0011] Furthermore, it also includes: reading the software system operation log, extracting node identifiers, data packet size, communication frequency, data type and timestamp information from the operation log, establishing a node data indicator matrix, dividing the node data indicator matrix according to a preset time window, generating a multidimensional data feature sequence, and normalizing the multidimensional data feature sequence;

[0012] The access frequency, access time, and user dwell time of nodes within the time window are statistically analyzed. The access frequency is divided by the maximum access frequency to obtain the access frequency popularity. The difference between the current time and the most recent access time is calculated and time decay popularity is obtained based on the decay function. The weighted sum of the user dwell time and the number of clicks is divided by the maximum weighted sum to obtain the user behavior popularity. The access frequency popularity, time decay popularity, and user behavior popularity are linearly combined to obtain the comprehensive popularity value.

[0013] Furthermore, it also includes: calculating the historical popularity value distribution of the node within multiple time windows, determining the weight coefficients of access frequency popularity, time decay popularity and user behavior popularity based on the historical popularity value distribution, multiplying the weight coefficients by the corresponding popularity values ​​and summing them to obtain the initial popularity value of the node, and constructing a node popularity evaluation matrix;

[0014] The node popularity evaluation matrix is ​​subjected to eigenvalue decomposition to extract principal component feature vectors. A popularity mapping function is constructed based on the principal component feature vectors. The initial popularity value is input into the popularity mapping function for normalization processing to generate the final popularity value of the node. The final popularity value is used as the popularity attribute of the node.

[0015] Furthermore, it also includes: collecting the input and output data streams, data processing tasks, and inter-node communication relationships of each node in the software system; constructing a node dependency matrix based on the data streams and communication relationships; converting the node dependency matrix into an adjacency matrix; establishing directed connections between nodes; and adding the node heat value as an attribute feature of the node.

[0016] Monitor the CPU usage time, memory usage, and number of I / O operations during data transmission between the nodes. Calculate the resource consumption coefficient between the nodes by weighting the CPU usage time, memory usage, and number of I / O operations. Use the resource consumption coefficient as the weight value of the directed edge to generate a weighted dependency graph.

[0017] Furthermore, it also includes: inputting the node attribute features and edge weight values ​​into the graph attention network, calculating the attention coefficients between nodes, weighting and aggregating the features of adjacent nodes according to the attention coefficients, generating the node representation vector through nonlinear transformation, constructing a multi-head attention layer for the nodes based on the representation vectors, and concatenating and normalizing the output of the multi-head attention layer;

[0018] The normalized node representation is iteratively updated, the loss function value of the node representation is calculated, the parameters of the graph attention network are adjusted based on the loss function value, the parameter adjustment process is repeated until the loss function value converges, the trained graph attention network is applied to the dynamic update of the node representation, and a dynamic data lineage map is output.

[0019] Furthermore, it also includes: extracting the heat value of nodes from the dynamic data pedigree, establishing a probability distribution model of the heat value, calculating the mean and standard deviation of the heat value, setting a heat threshold range based on the mean and standard deviation, marking nodes with heat values ​​higher than the upper threshold as hot nodes, marking nodes with heat values ​​between the upper and lower thresholds as warm nodes, and marking nodes with heat values ​​lower than the lower threshold as cold nodes;

[0020] Subgraph structures are constructed for the hot nodes, warm nodes, and cold nodes respectively. The connectivity index and node importance score of the subgraph are calculated. Node state transition rules are established based on the connectivity index and importance score. The hot and cold attribute labels of the nodes are dynamically adjusted according to the state transition rules, and the hierarchical structure of the data lineage graph is updated.

[0021] Furthermore, it also includes: constructing a multi-level cache structure, storing the hot node data in a high-speed cache, encoding the cold node data using a lossless compression algorithm, migrating the compressed cold node data to a low-speed storage device, scanning the dependencies of the cold nodes, retaining the first-order adjacency relationships with the hot and warm nodes, removing the connection edges between the cold nodes, and updating the adjacency matrix of the dependency graph.

[0022] The real-time heat value of a node is obtained, and the weight coefficient of the node in the graph attention network is adjusted based on the real-time heat value. The attention distribution of the node is recalculated, and the attention distribution is weighted and combined with the historical representation vector of the node to generate the updated representation vector of the node. The updated representation vector is then input into the graph attention network for online learning.

[0023] Secondly, this application provides a storage strategy optimization device based on data popularity and data lineage, comprising:

[0024] The node popularity determination module is used to collect data packet size, communication frequency, data type and timestamp information of nodes in the software system, to statistically analyze the access frequency, access time and user behavior of the nodes, to calculate the access frequency popularity, time decay popularity and user behavior popularity of the nodes, and to combine the access frequency popularity, time decay popularity and user behavior popularity to generate a node popularity value.

[0025] The lineage graph construction module is used to construct a directed graph with the nodes as vertices and the dependencies between nodes as edges. The node popularity value is used as a node attribute, the computational resource consumption of the nodes is used as an edge weight, and the directed graph is iteratively trained using a graph attention network to generate a dynamic data lineage graph.

[0026] The strategy storage module is used to divide nodes into cold nodes, warm nodes, and hot nodes based on the heat value of nodes in the dynamic data lineage graph, adopt a priority caching strategy for hot nodes, compress and store cold nodes, retain the direct dependencies between cold nodes and hot nodes and warm nodes, delete the indirect dependencies of cold nodes, and dynamically update the node representation vector of the graph attention network based on the heat value of the nodes.

[0027] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the storage strategy optimization method based on data popularity and data lineage.

[0028] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the storage strategy optimization method based on data popularity and data lineage.

[0029] Fifthly, this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the storage strategy optimization method based on data popularity and data lineage.

[0030] As described above, this application provides a storage strategy optimization method and apparatus based on data popularity and data lineage. Through an innovative design of a multi-dimensional popularity calculation model, and by using node attribute analysis and graph attention networks, it achieves dynamic optimization of the storage strategy. A directed graph structure is constructed, and combined with lineage analysis, a reliable dependency management system is established. A hierarchical storage strategy is introduced, and storage efficiency is ensured through popularity partitioning and dependency optimization. This method effectively addresses the shortcomings of traditional technologies in popularity assessment, lineage analysis, and storage optimization, providing technical support for software system storage management. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating the storage strategy optimization method based on data popularity and data lineage in an embodiment of this application.

[0033] Figure 2 This is a structural diagram of the storage strategy optimization device based on data popularity and data lineage in an embodiment of this application;

[0034] Figure 3 This is a schematic diagram of the structure of the electronic device in the embodiments of this application.

[0035] Figure label:

[0036] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0039] To address the problems existing in current technologies, this application provides a storage strategy optimization method and apparatus based on data popularity and data lineage. By innovatively designing a multi-dimensional popularity calculation model and utilizing node attribute analysis and graph attention networks, dynamic optimization of the storage strategy is achieved. A directed graph structure is constructed, and combined with lineage analysis, a reliable dependency management system is established. A hierarchical storage strategy is introduced, ensuring storage efficiency through popularity partitioning and dependency optimization. This method effectively solves the shortcomings of traditional technologies in popularity assessment, lineage analysis, and storage optimization, providing technical support for software system storage management.

[0040] To effectively address the shortcomings of traditional technologies in areas such as data popularity assessment, data lineage analysis, and storage optimization, and to provide technical support for software system storage management, this application provides an embodiment of a storage strategy optimization method based on data popularity and data lineage. See [link to embodiment]. Figure 1 The storage strategy optimization method based on data popularity and data lineage specifically includes the following:

[0041] Step S101: Collect data packet size, communication frequency, data type and timestamp information of nodes in the software system, statistically analyze the access frequency, access time and user behavior of the nodes, calculate the access frequency heat, time decay heat and user behavior heat of the nodes, and combine the access frequency heat, time decay heat and user behavior heat to generate a node heat value.

[0042] Optionally, in one embodiment, this application is based on a data platform, focusing on step S101, to organize the "raw traffic and operation logs" into calculable node heat values, which are then used in subsequent lineage mapping and storage layering. First, the data collection criteria are clearly defined. This application synchronizes data from three sources: network probes collect data packet sizes and timestamps between nodes (based on mirror ports or bypass TAPs, targeting nodes such as PLCs, HMIs, historical servers, and report services); system and middleware operation logs provide communication frequency and data types (protocol families, table / file / field-level identifiers, ETL job types); and front-end applications and portals record user behavior (login, query, dwell time, clicks). The collection end adds a node identifier (node_id), peer identifier (peer_id), direction (in / out), protocol type, and data type (such as time-series writing, batch report export, field query) to each record, and performs unified time synchronization on the timestamps to avoid "false peaks" caused by cross-device clock drift. Considering the shift system and process cycle, this application adopts a sliding time window strategy. The window width is preset by industry experience (e.g., 15 minutes) and a step size (e.g., 5 minutes) is reserved to balance sensitivity and robustness.

[0043] This embodiment performs a round of normalization and noise reduction before data is written to disk; otherwise, the popularity index will be skewed by "mechanical noise." Data packet size and communication frequency are aggregated within a window into upstream / downstream byte counts, packet counts, and session counts (number of data interactions). Multiple extreme points are truncated rather than smoothed out by the mean, preserving the shape of sudden events. Data types are fused with protocol decoding and task tags to obtain fine-grained categories (read, write, subscribe, export, rerun, etc.). Timestamps are encoded to time positions, marking day / night, shift boundaries, and maintenance identifiers. User behavior logs from the front-end and BI system are first session-based, merging the dwell time and click count of consecutive operations, filtering out bots or batch script accounts to avoid miscounting non-human behavior in "user behavior popularity." These processes align with the natural meaning of subsequent popularity: nodes that are frequently accessed, recently accessed, and have strong user interaction should be considered "hotter."

[0044] This embodiment assembles the above indicators into a window-level "node data indicator matrix" and then calculates three types of popularity. Access frequency popularity reflects the degree to which a node is reached under the current business load. It is calculated by dividing the number of accesses (or sessions) within the window by the maximum number of accesses for all nodes within the same window and then normalizing to 0-1. Time decay popularity emphasizes "recentness." This application takes the interval between the current time and the most recent access time of the node, substitutes it into an exponential decay function, and the shorter the interval, the higher the popularity, which aligns with people's intuition about "popularity." User behavior popularity combines dwell time and click count, obtained through a weighted sum and normalization. The weights are derived from the correlation analysis of "dwell time changes," "click count changes," and resource usage in historical windows, avoiding the one-sided use of clicks to create "popularity." Here, the three popularity metrics are not simply considered mutually exclusive; they describe different dimensions: frequency reflects quantity, decay reflects time, and behavior reflects "human intention."

[0045] This embodiment does not rush into linear addition, but first reviews the historical data. The time series of heat values ​​for each node across multiple windows is aggregated into a historical distribution, which is used to determine the weight coefficients of the three components. The logic is simple: during nighttime when scheduling tasks are intensive, access frequency is high but user behavior is weak, so frequency and attenuation should account for a larger proportion; during daytime shifts with more interaction, the behavior component should be increased. This application performs robust regression on the correlation between the three components of each node over the past week (or one process cycle) and downstream resource usage (CPU, IO) to obtain a role-oriented weighted reassembly, which is then substituted into the current window to generate the initial heat value. Considering the skewness of the heat value distribution of different nodes, this application further performs eigenvalue decomposition on the evaluation matrix composed of the initial heat values ​​of all nodes, extracts the main direction, constructs a monotonic heat value mapping, and pushes the extreme long tail back to a reasonable range. The resulting heat value is both relatively stable to the node itself and comparable on a global scale.

[0046] This embodiment retains only one easily understandable expression at the formula level: , where H(i) is the overall popularity of node i in the current window; F(i) is the access frequency popularity (the value of the number of visits within the window divided by the maximum number of visits in the window); T(i) is the time decay popularity (obtained by the difference between the current time and the most recent access time through an exponential decay function); B(i) is the user behavior popularity (the value of the weighted normalization of dwell time and number of clicks). , , The weighting coefficients for the three components are determined by robust regression of the node's role and historical window, and all three are non-negative and sum to 1. This combination does not deviate from natural laws: if a node has just been accessed in large numbers (high F, high T), or if users have stayed for a long time (high B), its overall popularity should increase.

[0047] This embodiment also considers edge scenarios. When logs are missing or collection is delayed, this application does not fill in the access frequency out of thin air, but marks the window as "insufficient evidence", temporarily uses the popularity of the previous window for short-term extrapolation, and reduces its sample weight in subsequent GAT training to prevent "pseudo-popularity" from polluting the model. For collection sites with strong periodicity, simple frequency normalization will mix "stable high frequency" with "short-term abnormal peaks". This application introduces a rhythmic baseline: a baseline curve is built for each node by hour and shift, and the frequency popularity is decomposed into "baseline component + deviation component". F(i) in the formula actually takes the deviation part. In this way, short-term anomalies can be seen, and long-term stability will not be overstated.

[0048] In the output phase of this embodiment, the three components and overall popularity of each node in the current window are written into the "popularity metric library," recording the weight source, window timestamp, and original evidence summary. Subsequently, in S201, popularity is injected into the lineage graph as a node attribute. The caching and storage strategy module reads the popularity threshold to classify nodes as cold / warm / hot, and makes hierarchical storage decisions in conjunction with the resource consumption weights of lineage edges. In practice, this application additionally generates a "popularity fluctuation label." When the variance of the overall popularity of multiple consecutive windows exceeds the threshold, it prompts the subsequent graph attention network to reduce the aggregation intensity of the node's neighborhood during dynamic updates, preventing fluctuations from "heating up" the neighbors.

[0049] This embodiment also provides two alternative implementation paths. First, if real user behavior cannot be obtained and calls can only be statistically analyzed based on interface tokens, this application degenerates B(i) into an "interaction intention proxy," replacing dwell time and clicks with API call diversity and parameter change rate; Second, for workshops where privacy is extremely sensitive, user behavior dimensions are disabled, this application will... The upward adjustment emphasizes relevance and relies more on the combination of F and T in caching strategies. Neither path changes the core principle: popularity is a comprehensive characterization of "high usage, current usage, and human operation", rather than a single-point amplification of a certain indicator.

[0050] This embodiment ultimately solves the problem of transforming "chaotic access and log signals" into interpretable, computable, and traceable heat values. It not only statistically aligns with real-world usage habits but also connects to subsequent data lineage and resource consumption models, providing clean and stable drivers for cold compression, hot caching, and temperature-level online queries. When facing policy disputes, the engineering team can refer back to the three components and evidence chain in this window to confirm the source of weights and anomaly labels, thereby avoiding arbitrary threshold allocation.

[0051] Step S102: Construct a directed graph with the nodes as vertices and the dependencies between nodes as edges, use the node heat value as a node attribute, use the computing resource consumption of the nodes as edge weights, and use a graph attention network to iteratively train the directed graph to generate a dynamic data lineage graph.

[0052] Optionally, this embodiment unfolds step S102 in a real-world data platform operation scenario, mapping the node heat values ​​obtained in S101 to the graph structure and solidifying the relationships between nodes ("who depends on whom, and at what cost") into edge weights. Finally, iterative training is performed using a graph attention network to obtain a dynamic data lineage graph that can be updated over time. Let's start with the source of dependencies. This application reads ETL orchestration platforms, data lake metadata services, and SQL audit logs, extracting a three-segment chain: "input table / file → processing operator → output table / file." Field-level lineage is drilled down to column mappings when parsable. For streaming jobs, connections are established using topic / partition and operator DAGs as intermediary nodes. To prevent treating network layer sessions as business dependencies, this application only adopts relationships aligned by task ID, job instance number, or query hash, constructing a node dependency matrix, which is then converted into a directed adjacency matrix A and stored in both the graph database and the tensor backend to ensure consistency between training and visualization.

[0053] In this embodiment, the node heat value H(i) is written into the node attributes. In addition to the original technical metadata and structural features (in / out degree, local clustering coefficients), the timestamp and source window of H(i) are additionally recorded to facilitate downstream training by time slice. The definition of edge weights should be "cost-oriented" rather than a vague count. This application continuously samples CPU usage time, peak memory usage, and I / O operation counts during the data transmission and processing phase between nodes, aggregates these by job instance, and then sums them by weight after normalization to obtain the resource consumption coefficient. Considering that different job types have different sensitivities to resource dimensions (e.g., batch writes are more I / O-dependent, while data cleaning is more CPU-dependent), the weighting coefficients are derived from historical regression or operational experience databases, and are recalibrated during major operational transitions. The intuitive basis for this approach is simple: the more "laborious" the link, the more cautious the subsequent storage strategy should be towards its upstream and downstream components.

[0054] In this embodiment, after preparing the directed graph G=(V,E,W), a Graph Attention Network (GAT) is introduced for iterative training. The purpose is not to show off, but to allow the node representation to truly absorb the "value and cost" information of its neighborhood, so that subsequent hot / cold stratification and link pruning are based on sound reasoning. On the input side, the initial feature vector xi of a node is composed of four parts: static metadata, structural features, embedding of the heat scalar H(i), and temporal location encoding; edge-added features are also included. As a bias term for attention, the calculation of the attention coefficient follows the simple idea of ​​"paying attention to neighbors that are strongly correlated with me and whose costs cannot be ignored." This application introduces two types of modifications to the original attention score: one is a heat coupling term, which uses the mean of H(i) and H(j) as a weighting factor, giving greater attention to hot-hot and hot-warm dependencies; the other is a resource penalty term, when... When the value is significantly high, the model should "see" the existence of this expensive edge so that subsequent strategies can avoid mistakenly storing cold data on expensive paths in high-level storage.

[0055] The training objective of this embodiment is to quantify the loss that "propagates correctly." This application constructs two types of supervision signals: if historical annotations or semi-automatic labels exist (e.g., high-value nodes inferred from query logs, critical paths marked by SLAs), cross-entropy or ranking loss is used to advance the separation of embeddings on these nodes; more commonly, there are no annotations, and this application introduces a joint objective of contrastive learning and edge reconstruction: within adjacency, consistency constraints are applied to the same node in the enhanced view, limiting the variation in the representation of the same node within temporally adjacent windows; for edge reconstruction, the model uses node representations to predict the existence of directed edges and their weight ranges; the more accurate the prediction, the better the representation "explains" lineage. This setting aligns with natural laws: truly dependent and frequently used node pairs should be closer in the representation space, and edges with larger weights are easier to distinguish during decoding.

[0056] This embodiment considers that the data is dynamic and not suitable for large-batch training followed by "sealing the box". This application adopts small-batch incremental iteration: each time slice, the new window's H(i) and newly added / changed edges are patched into the graph. To avoid catastrophic forgetting, a "temperature" parameter is introduced into the attention layer to limit the deviation of a single update from historical representations, and a smaller learning rate is set for nodes with drastic fluctuations. The outputs of multi-head attention are spliced ​​and then subjected to layer normalization and residuals to stabilize the representations within an interpretable range. The training process uses an early stopping strategy to monitor the edge reconstruction AUC and the recurrence rate of the critical path, avoiding overfitting to a local pattern of a certain shift.

[0057] This embodiment retains only one formula to express how attention weights combine popularity and resource cost: .

[0058] In the formula, Let be the attention coefficient from i to its neighbor j; , is the initial feature of the node; W and a are the learnable weight matrix and vector; H(i) and H(j) are the heat values ​​of two nodes; edge weight The normalized value of ; η is the gain coefficient of thermal coupling, ζ is the coefficient of resource penalty; softmax is normalized according to the neighbor set of i. The physical meaning of this formula is: the more "hot" the two ends are, the easier they are to be aggregated, and the more "expensive" the edges should be aggregated with caution to avoid indiscriminately spreading high-cost signals.

[0059] This embodiment incorporates several "firefighting" preparations at the engineering level. First, for edges temporarily lacking proof (newly appearing but with incomplete resource sampling), this application sets ŵ as an interval variable and assigns it a large uncertainty, reducing its weight in the training loss, and increasing its weight again after the next batch of sampling completes the process. Second, for abnormal noise (such as extreme values ​​w caused by short-term batch reruns), this application uses quantile truncation instead of hard pruning on the edge feature side, preserving the order but eliminating extreme values. Third, cross-domain lineage (such as cross-system ETL) can cause misconnections when clocks are not synchronized. This application performs event alignment before graph construction and introduces "virtual edges" for bridging when necessary to prevent the Graph Attention Network (GAT) from sticking two unrelated nodes together.

[0060] This embodiment outputs two types of products after training: first, a node representation vector h(i), which, in S103, uses a threshold to classify nodes into cold / warm / hot categories and provides a sorting basis for the caching strategy; second, the dynamic data lineage graph itself, including the versioned attribute of popularity over time, the time-series curve of edge weights, and a snapshot of attention distribution. These are written into the graph database, along with the evidence chain and parameter version number, for easy backtracking and manual review. This application adds an extra layer of "pruning suggestions" to the graph, based on h(i) and... The combined scores filter out indirect edges with low popularity, low attention, and high cost, and hand them over to the strategy engine to make indirect dependency deletion plans.

[0061] This embodiment also adds two implementation variants. For older systems where column-level lineage is incomplete, this application takes a fallback approach, training the GAT with table-level edges, but adding aggregated statistics of field access popularity to node features as a "proximity for fine-grained missing features"; for pure query-type data marketplaces with incomplete computational resource indicators, ŵ is replaced by a ratio approximating query execution time and cache miss rate, still forming a meaningful "cost" dimension. Neither variant changes the core principle: popularity drives attention to focus on value paths, resource costs constrain the aggregation and diffusion range, and iterative training ensures temporal self-consistency between the representation and the graph structure.

[0062] This embodiment addresses three practical problems: kinship is not merely a matter of "existence or non-existence," but rather a strong or weak relationship coupled with popularity and cost; node importance is not arbitrary, but rather emerges from the surrounding context through attention aggregation; and the graph is not a static drawing, but a "living entity" that steadily updates with shifts and operational rhythms. In scenarios where load fluctuates significantly between day and night, engineers can use this graph to answer two simple questions: which data should currently remain in fast storage, and which links are incurring excessive costs without generating value, and can trace the answers back to the specific values ​​and sources of popularity, cost, and attention.

[0063] Step S103: Based on the heat value of the nodes in the dynamic data lineage graph, the nodes are divided into cold nodes, warm nodes and hot nodes. A priority caching strategy is adopted for the hot nodes, and the cold nodes are compressed and stored. The direct dependencies between the cold nodes and the hot nodes and warm nodes are retained, and the indirect dependencies between the cold nodes are deleted. The node representation vector of the graph attention network is dynamically updated based on the heat value of the nodes.

[0064] Optionally, this embodiment follows the output of S101 and S102, with the input being a dynamic data lineage graph, where each node has a final heat value H(i), static / dynamic attributes, and edge resource consumption weights. This application aims to transform these quantities into an executable hierarchical storage and caching strategy, and to link the node representation of the Graph Attention Network (GAT) with its hot and cold states for updates. First, the partitioning criteria are established: this application does not directly use fixed numerical thresholds, but rather determines the upper and lower thresholds based on the current form of the heat distribution and role priors. In engineering, a robust heat distribution is first fitted (excluding points with insufficient evidence), the mean and scale are calculated, and then role biases are superimposed. For example, HMI and reporting services are sensitive to user response, so the thresholds are wider; historical servers and archives are sensitive to throughput, so the thresholds are slightly tighter. After obtaining the upper threshold θhigh and the lower threshold θlow, H(i) ≥ θhigh is recorded as a hot node, θlow ≤ H(i) < θhigh as a warm node, and H(i) < θlow as a cold node. To avoid temperature fluctuations, this application sets a small hysteresis band, where nodes only change their level when they exceed the threshold for several consecutive windows. This allows the cache layer to stabilize and prevents frequent relocations due to short-term peaks.

[0065] This embodiment implements a priority caching strategy for hot nodes, the key being to clearly define "who gets priority, how much they occupy, and when they are evicted." The input end reads the in-degree, path betweenness, and edge resource consumption of hot nodes in the lineage graph to determine the cache priority. Simply put, nodes that "contribute significantly to subsequent accesses and have high link costs" are prioritized. Capacity allocation uses soft quotas; hot nodes occupy high-speed cache (such as NVMe or memory) based on priority and historical hit contributions. Sub-objects (fields, columns, partitions) are sliced ​​at the access granularity to avoid moving the entire table. The eviction strategy is not a simple LRU, but a combined eviction based on both hotness decay and lineage importance: when H(i) rapidly declines and there are downstream links that can be replaced by warmer nodes, its cache share is released first, making the resources available to newly emerging hot nodes. Cache hits and origin-following costs are written back to the lineage graph for the next round of threshold self-calibration.

[0066] This embodiment employs a compressed storage path for cold nodes, logically divided into two layers: data ontology compression and graph structure pruning. For the data ontology, a lossless compression algorithm (columnar compression, dictionary encoding, or temporal difference) is chosen because industrial data still requires accuracy for post-analysis and traceability. Simultaneously, cold nodes are migrated to low-cost media (SATA disks or object storage), retaining the minimum set of indexes to maintain reactivation capability. In terms of the graph structure, this application performs "cold node pruning": only first-order dependencies between the node and hot / warm nodes are retained, while indirect dependencies between cold nodes are deleted. The rationale for pruning is twofold: firstly, to significantly reduce the computational load of GAT training and graph queries; and secondly, to prevent low-value links from "amplifying noise" during attention aggregation. However, a security check is performed before edge deletion. If a cold-to-cold link carries information necessary for auditing (such as full traceability required by regulations), it is marked as a "retained edge," thus excluding it from the pruning rules. After pruning, the adjacency matrix and edge weights are updated, and version numbers and differences are recorded for later re-uploading if needed.

[0067] The core patent point of this embodiment lies in the closed-loop coupling of "heat - storage - graph learning". The attention distribution learned by GAT in the previous training reflects the strength of dependence at a certain time period; when nodes switch between hot and cold, if the representation is not updated, subsequent link prediction and policy decision will lag. This application directly embeds H(i) into the attention scoring item, and dynamically adjusts the weight and normalization strategy of nodes during aggregation according to the hot and cold level. Specifically, the attention score of each edge i→j is first multiplied by the heat factor related to H(i) and H(j). Hot-to-hot links are moderately amplified, and cold-participation links are compressed, but not forcibly zeroed out; warm nodes act as a "buffer layer", and their factors remain neutral to avoid the network overfitting to extreme hot spots. After aggregation, the node representation vector is smoothly combined with the historical representation. Hot nodes adopt low inertia (quick response to business changes), cold nodes adopt high inertia (avoid meaningless oscillations caused by small fluctuations), and warm nodes are in between. These adjustments are consistent with natural laws: frequently used nodes are more sensitive to neighborhood information, while cold nodes maintain low-frequency stability.

[0068] This embodiment also needs to control the impact of "edge deletion" and "hotness changes" on GAT training within an interpretable range; otherwise, graph learning will diverge due to structural abrupt changes. This application adopts an incremental learning process: each window only retrains the local subgraph where the hierarchy has changed, while other regions retain the old parameters; for the reduction of neighbors caused by pruning, the output is stabilized by a mask in attention normalization to prevent weights from concentrating on a single neighbor; for newly emerging hot nodes, the number of multi-head attention is increased in the short term, allowing the model to quickly establish its representation ability in different semantic neighborhoods, and then converges to a unified number of heads. A structure fidelity term is added to the loss function to encourage new and old representations to remain close in unchanged regions and avoid unnecessary drift.

[0069] This embodiment links three processes together: "priority caching—compression pruning—online GAT update," with traceable evidence for each step. The cache side stores hit rate, origin latency, and object-level occupancy; the compression side records compression ratio and warm-up latency; and the GAT side records the heat sensitivity curve of attention weights. When faced with the question "Why was this table put into the cache today?", operations personnel can see the decision-making logic through H(i), the cost of the critical path of lineage, and the evidence of attention amplification, rather than relying on a black-box judgment.

[0070] This embodiment also considers two scenario variants. One is for time-series wide table scenarios, where popularity is often concentrated in a few time partitions. The caching adopts a "partition popularity" granularity, and the popularity factor of GAT is also refined to the partition node level. Pruning is performed on cold partitions to preserve key lineages across partitions. The other is for financial or energy scenarios with strict regulatory compliance, where cold-cold indirect dependencies cannot be deleted. This application downgrades their weight but retains them, and they participate in aggregation in GAT in a "frozen neighbor" manner (without backpropagation), thereby neither compromising traceability nor controlling training overhead.

[0071] In this embodiment, during the output stage, the hierarchical results of the three types of nodes, caching and storage actions, pruning differences, and updated node representation vectors are written back to the metadata database and graph storage, and batch numbers are assigned. Subsequent resource orchestration and query acceleration in S104 directly consume these products. Overall, this application solves the coupling problem of three things: how to stably divide hot and cold nodes, how to align storage actions with lineage pruning, and how to ensure the graph model keeps up with policy adjustments. The resulting effect is that the caching layer focuses on real hot spots, the cold layer's occupancy converges, the graph representation is more sensitive to business changes, and each step is reviewable and interpretable.

[0072] As described above, the storage strategy optimization method based on data popularity and data lineage provided in this application can achieve dynamic optimization of storage strategies through the innovative design of a multi-dimensional popularity calculation model, node attribute analysis, and graph attention networks. It constructs a directed graph structure and, combined with lineage analysis, establishes a reliable dependency management system. A hierarchical storage strategy is introduced, ensuring storage efficiency through popularity partitioning and dependency optimization. This method effectively addresses the shortcomings of traditional technologies in popularity assessment, lineage analysis, and storage optimization, providing technical support for software system storage management.

[0073] In one embodiment of the storage strategy optimization method based on data popularity and data lineage in this application, it may further include the following:

[0074] Step S201: Read the software system operation log, extract node identifier, data packet size, communication frequency, data type and timestamp information from the operation log, establish a node data index matrix, divide the node data index matrix according to a preset time window, generate a multi-dimensional data feature sequence, and normalize the multi-dimensional data feature sequence.

[0075] Step S202: Statistically count the access frequency, access time and user dwell time of nodes within the time window. Divide the access frequency by the maximum access frequency to obtain the access frequency popularity. Calculate the difference between the current time and the most recent access time and obtain the time decay popularity based on the decay function. Divide the weighted sum of the user dwell time and the number of clicks by the maximum weighted sum to obtain the user behavior popularity. Linearly combine the access frequency popularity, time decay popularity and user behavior popularity to obtain the comprehensive popularity value.

[0076] Optionally, this embodiment elaborates on S201 and S202 for the data platform. The input is a mixture of historical and real-time operation logs, and the output is a multi-dimensional feature sequence and three-component heat and comprehensive heat values ​​that can be consumed by subsequent lineage and storage strategies. This application first clarifies "what can be obtained from the logs and how to unify the interpretation". The platform covers multiple subsystems such as PLC / HMI / historical database / ETL scheduling / BI portal, and the log sources include: network-side session logs (data packet size, direction, protocol), application-side operation logs (data type tags such as query, write, export, etc.), task operation logs (ETL job ID, input / output objects, batch time), and front-end session logs (user ID, dwell time, click time, active window). This embodiment retrieves raw records from the log bus in batches, performing five tasks on each record: unifying the timestamp to the NTP baseline; parsing and extracting the node identifier (node_id, unified namespace at four levels: table / file / field / operator); extracting basic counts of packet size and communication frequency (number of packets, number of bytes, number of sessions); standardizing data type labels (read / write / subscribe / export / rerun / DDL); and preserving the original log fingerprint for backtracking. Considering the clock offsets across different systems, this application performs event alignment after retrieval, pairing multiple logs of the same transaction across systems with adjacent time windows based on task ID to avoid duplicate or missed counts.

[0077] This embodiment constructs a node data indicator matrix X, with the row index being node i and the column index being indicator dimension f. The indicator dimensions are not arbitrarily chosen; this application divides them into three groups: communication strength (uplink / downlink packet count, uplink / downlink byte count, number of sessions, retransmission ratio), behavioral semantics (data type distribution, function code / SQL operation distribution, failure ratio), and session interaction (number of active users, average dwell time, number of clicks, session concurrency). X is divided according to a preset time window. Within each window Wτ, a multi-dimensional data feature sequence O(i,τ)={o1,…,oT} with a step length of T is generated for each node. The step length can be on the minute level, covering day-night cycles and shift rhythms. Missing values ​​are not hard-filled; instead, a mask matrix M is attached to preserve the fact of the "evidence gap." Subsequent models will reduce their weight to avoid being swayed by noise.

[0078] This embodiment focuses on the normalization process; otherwise, indicators at different scales would obscure each other in subsequent heat and graph learning. Intensity indicators within the window are scaled using quantile scaling (stretched from p5 to p95 to [0,1]) to suppress extreme spikes while preserving the ranking. Classification and distribution indicators (data types, function codes) are represented by probability vectors, naturally normalized. User behavior such as dwell time and clicks are first processed using session-level Winsor mapping to remove obvious script behavior before being mapped to the [0,1] interval. Furthermore, this application introduces location encoding (hourly / shift one-hot or sine / cosine) as an additional dimension to indicators with distinct day / night and shift cycles, so that subsequent heat calculation and learning processes can distinguish the natural pattern of "busy during the day, idle at night" and avoid misjudging low-frequency events at night as "cold."

[0079] This embodiment proceeds to S202, the purpose of which is to convert the observable metric into three complementary heat components. The access frequency heat F(i,τ) emphasizes "how many times it has been accessed during this period." This application counts the number of accesses (or sessions) of nodes within the Wτ window, normalizes it using the maximum value of all nodes in the window, and obtains a 0-1 metric to prevent absolute values ​​from becoming incomparable due to system size changes. The time decay heat T(i,τ) emphasizes "whether it has been recently accessed." This application takes the difference Δt between the current time now and the node's most recent valid access time t_last(i), substitutes it into the exponential decay function to obtain T(i,τ)=exp(−β·Δt), where β is the decay coefficient, given based on section cycle time experience or calibrated by historical verification. The smaller Δt is, the closer T is to 1, conforming to the common sense that "more recently used is hotter." User behavior heat index B(i,τ) emphasizes "human willingness to interact with data". It is obtained by taking the weighted sum of the average dwell time si and click ci of the node within the Wτ window and dividing it by the maximum weighted sum of all nodes within the window. The weight μ comes from historical regression and measures the explanatory power of dwell time and clicks on backend resource consumption and query latency. This step has practical significance in production line scenarios: long dwell time often corresponds to exploratory analysis or report drill-down and is more sensitive to caching and hot storage.

[0080] This embodiment does not simply add the three together; instead, it first performs a robustness check. This application uses features such as session continuity, window focus, and operation trajectory to determine the proportion of human sessions. Scaling B(i,τ) to To prevent scripts from artificially inflating the popularity of certain behaviors, the second step is rhythmic baseline decomposition. This involves performing seasonal-trend decomposition on F and B, extracting the deviation components relative to the baseline, and using these deviations in the combination to prevent frequently occurring sites from being permanently classified as "hot" and thus masking short-term abnormal hotspots.

[0081] This embodiment uses a linear model in the combination stage for a simple reason: the three components are physically complementary, and linear weights are easier to interpret and tune. This application will reorganize the weights obtained from S101 or historical statistics { , , Loading is done by node role; for example, HMI-type nodes have slightly higher wb (whole number) scores, while historical database-type nodes have higher wf (whole number) and wt (whole number) scores. Then, the overall popularity is calculated. To avoid scale drift between different windows, this application performs a robust normalization (such as quantile stretching) on ​​all H values ​​of the current window and adds a timestamp. The overall heat and its three components are written back to the "heat measurement library" along with the window ID, weight source, mask, and log evidence summary for easy traceability.

[0082] This embodiment also incorporates anomaly and missing certificate handling into the routine process. If the log collection within the window is incomplete (mask ratio exceeds the threshold), this application does not calculate new F and B, but instead uses the upper window value for short-term extrapolation and adjusts it downwards. To prevent artificially inflated heat due to passively increasing t_last, if a node's F is temporarily boosted by a large number of reruns, but its B' and resource consumption do not increase accordingly, this application adds a "batch peak" label to the output to remind S102's GAT to reduce the propagation intensity of that node during attention aggregation, thus avoiding the spread of "artificial heat".

[0083] This embodiment supplements two scenario-based implementations on the engineering side. First, in a repository with complete column-level lineage, `node_id` can be refined to the field / partition level, with O(i,τ) generated by column or partition dimension. This allows F, T, and B' to distinguish between hot and cold data in different columns / partitions within the same table, ensuring the caching strategy is precise down to the column / partition level. Second, when front-end behavior logs cannot be directly collected due to privacy restrictions, API diversity, parameter change rate, and result set size change rate are used as proxies for B. μ is recalibrated based on offline correlation, maintaining an approximation of the physical meaning of "human intent."

[0084] The ultimate problem solved by this embodiment is to compress heterogeneous, multi-granularity operation logs into a time-ordered, comparable multi-dimensional feature sequence, and to produce a three-component heat value and a comprehensive heat value under the constraints of natural laws. The production team obtains not only a score, but also a clear chain of evidence: which window, which type of behavior, and which sessions "heated up" the node; and the subsequent S102 will bring these heat values ​​into attention aggregation, and S103 will link the heat values ​​with costs to caching and compression actions, forming a closed loop from observation to decision-making.

[0085] In one embodiment of the storage strategy optimization method based on data popularity and data lineage in this application, it may further include the following:

[0086] Step S301: Calculate the historical popularity value distribution of the node within multiple time windows, determine the weight coefficients of access frequency popularity, time decay popularity and user behavior popularity based on the historical popularity value distribution, multiply the weight coefficients by the corresponding popularity values ​​and sum them to obtain the initial popularity value of the node, and construct the node popularity evaluation matrix.

[0087] Step S302: Perform eigenvalue decomposition on the node popularity evaluation matrix, extract principal component feature vectors, construct a popularity mapping function based on the principal component feature vectors, input the initial popularity value into the popularity mapping function for normalization processing, generate the final popularity value of the node, and use the final popularity value as the popularity attribute of the node.

[0088] Optionally, this embodiment builds upon the node three-component heat sequence obtained in S101 and the role context in S102, clarifying the two key points of "how to weight the three components and how to resolve scaling and skewness" in S301 and S302, and outputting the results as heat attributes that can be directly consumed by subsequent graph training and storage strategies. First, organize the input. Each node i has access frequency heat F(i,t), time decay heat T(i,t), and user behavior heat B(i,t) in the most recent M time windows, along with role labels (such as HMI, reports, ETL operators, historical databases) and work status calendars (shifts, maintenance, monthly settlement). This application segments nodes by role and constructs their historical heat value distribution: for each component, it calculates quantiles, coefficient of variation, and Pearson / Spearman correlation with resource consumption (CPU, IO), and removes "insufficient evidence windows." The simple reason for doing this is that component weights should not be arbitrary, but should be linked to their ability to explain costs and values; for example, when batching jobs at night, F and T are better able to explain IO overhead, while during human-computer interaction during the day shift, B is closer to query latency.

[0089] This embodiment determines the component weighting coefficients based on the above statistics. This application sets up a robust regression task, using resource costs and SLA events as dependent variables, and historical sequences and time position codes of F, T, and B as independent variables. The regression coefficients are standardized, non-negative, and normalized. , , One or more weights are generated based on role and operating condition. If a component exhibits low correlation and high volatility for a prolonged period under a given role, this application lowers its upper weight limit to prevent noise amplification. For short-term structural changes (such as maintenance), the weights smoothly transition from the "normal group" to the "maintenance group" to avoid abrupt changes. After determining the weights, the three components are linearly synthesized in each window to obtain the initial heat index. Concatenate the H0 values ​​of all nodes within the given window into a vector, then stack this vector across M windows to form a matrix, thus obtaining the node popularity evaluation matrix. (An N-row, M-column matrix over the real number field R, where N rows represent the number of nodes and M columns represent time.) The matrix entries have a clear source, making them easy to audit.

[0090] This embodiment recognizes that the heat sequences of different nodes exhibit scaling differences, skewness, and rhythmicity. Directly using H0 would subject the threshold division to long-tail effects. In step S302, this application performs eigenvalue decomposition on Y to extract stable principal directions. Specifically, it first removes the mean from the rows (nodes) and performs robust standardization (using the median and median absolute deviation (MAD) instead of the mean and variance). Then, it performs seasonal decomposition along the time direction, separating obvious daily / weekly rhythms from the matrix and retaining only the residual principal components, avoiding mistaking predictable rhythms for "heat". Subsequently, it performs principal component analysis (PCA) or truncated SVD (taking the first K singular values / vectors) on the processed matrix to extract the first K principal component eigenvectors (eigenvectors along the time direction and loading vectors along the node directions), which are used to construct the heat mapping function. Intuitively, the first few principal components characterize patterns such as "simultaneous global heating" and "resonant heating in certain sections," which can suppress local noise. Therefore, robust standardization is first performed using the median plus MAD to reduce the impact of extreme values ​​and long tails; then seasonality stripping is performed to remove daily / weekly rhythms; finally, PCA or truncated SVD is used to extract the top K "common variation patterns" from the residual matrix. These principal directions can describe simultaneous heating at the global or section level and also suppress local noise, providing a stable, concise, and interpretable coordinate system for the subsequent heat mapping function.

[0091] This embodiment does not directly use the principal component scores as heat, but instead uses them to construct a monotonic, interpretable mapping function φ. The goal of this application is clear: to make the initial heat... It achieves a unified scale both globally and locally, while preserving relative order. This application uses nodal directional loads as a basis. Project the data to obtain the contribution of each node to the "main heat axis", then combine this contribution with... The original values ​​are bivariately calibrated, and a piecewise monotonic function is fitted: maintaining approximate linearity in the middle interval, and using mild logarithmic or logistic compression at the tail to prevent extreme values ​​from dominating the threshold decision. For ease of understanding, only a general expression is given: , where H(i) is the final heat of node i; is the initial heat of its current window; u(i) is the load projection of the node on the main heat axis (reflecting its consistency with the global thermal pattern); a, b, and c are calibration coefficients (selected by cross-validation of historical windows); It is a monotonic sigmoid function used for tail compression. The physical meaning of each quantity in the formula is clear: This provides an intuitive view of the window's heat level, and u(i) modifies its relationship with group resonance. Limiting extreme stretching, a, b, and c determine the scale and center.

[0092] This embodiment also addresses cross-role comparability. ETL operators and HMIs... Different numerical distributions, if not aligned, will produce systematic biases under a unified threshold. This application introduces "role-layer bias" in φ, aligning with the principal component centers and scales of each role, and then returning to the global scale to ensure semantic consistency of hot, warm, and cold classifications across roles. For cross-system scenarios (such as data marketplaces and time-series warehouses), PCA can be performed separately for each system before fusion using the alignment layer to reduce cross-domain noise.

[0093] In this embodiment, the final heat H(i) is written back to the node attributes of the graph database, along with the version number, mapping coefficients, and principal component summaries, so that S103 can call the threshold stratification and caching / compression strategy. To verify reliability, this application performs two types of health checks in the background: one is "resource consistency," which checks whether nodes with high H are located on high-resource-cost paths for a long time; the other is "query consistency," which checks whether high H corresponds to higher hit rates and lower origin pull rates. If a systematic deviation occurs, the tail compression region of weight learning and φ is traced back to determine whether it is a mismatch of component weights or over-mapping.

[0094] This embodiment provides two further extension paths. First, if user behavior logs are unavailable, B degenerates into a proxy for API diversity and parameter changes, and weight learning will naturally... Even with a lowered value, φ can still function. Secondly, if a work section is highly seasonal (e.g., end-of-month settlement), this application uses STL decomposition to explicitly remove the monthly rhythm before SVD, preventing H from collectively surging at the end of the month. The two paths adhere to the principle of consistency: using historical distribution and main direction to determine weights and benchmarks, avoiding reliance on a single indicator, and not sacrificing interpretability.

[0095] The ultimate problem solved by this embodiment is to transform the three-component heat distribution from "each singing its own tune" to "using the same standard," and this standard can change with operating conditions and roles. The output H can drive the hot / cold stratification and cache compression of S103, and can also be used by engineers to explain "why this node is hot at this time." The entire process leaves a clear chain of evidence: component history—weight learning—evaluation matrix—eigenvalue decomposition—mapping function—final heat distribution, and subsequent disputes can be verified step by step.

[0096] In one embodiment of the storage strategy optimization method based on data popularity and data lineage in this application, it may further include the following:

[0097] Step S401: Collect the input and output data streams, data processing tasks, and inter-node communication relationships of each node in the software system; construct a node dependency matrix based on the data streams and communication relationships; transform the node dependency matrix into an adjacency matrix; establish directed connections between nodes; and add the node heat value as a node attribute feature.

[0098] Step S402: Monitor the CPU usage time, memory usage, and number of I / O operations during the data transmission process between the nodes, weight the CPU usage time, memory usage, and number of I / O operations to calculate the resource consumption coefficient between the nodes, and use the resource consumption coefficient as the weight value of the directed edge to generate a weighted dependency graph.

[0099] Optionally, this embodiment focuses on S401 and S402 for the industrial control data platform. The inputs are the node heat values ​​H(i) and windowed log data obtained from S201 / S202 and S301 / S302. The goal is to systematically precipitate the "who depends on whom" and "what the cost" between nodes into a directed, weighted, and traceable dependency graph, under the premise of business semantic alignment. First, the three types of evidence to be collected are implemented: input and output data streams come from the ETL orchestration and data integration bus (including job ID, upstream object, downstream object, field mapping summary); data processing tasks come from the scheduling system (DAG, instance number, start and end time, rerun flag, execution node); and inter-node communication relationships come from network and middleware auditing (client / server, protocol, session ID, port, transaction boundary). After data collection, event alignment and deduplication are performed: using the job instance number and transaction hash as the primary key, records of the same business event in the three sources are aligned; when streaming tasks and micro-batch tasks are mixed, the window time and topic / partition are used as the common key to prevent “topics with the same name in adjacent time periods” from being linked together.

[0100] This embodiment adopts a two-step approach to relationship construction: "operator mediation + object merging". The first step uses processing operators (ETL tasks, SQL operators, stream operators) as mediator nodes, incorporating the two-segment chain of "upstream data object → operator → downstream data object". The second step merges objects for operators that only undergo morphological transformations but do not change the data identity (such as column renaming and compression encoding), folding the mediator nodes into direct dependencies of "object A → object B". When field-level lineage is complete, fields are prioritized as nodes; if the old system only has table-level information, field popularity statistics are attached to the node attributes to ensure interpretable granularity. In the resulting node dependency matrix D, Dij=1 indicates the existence of a business dependency event from node i to j within the observation window, and the source evidence includes job instances or query hashes. Handshakes / heartbeats on the network side alone are not accepted, avoiding misjudging "physical connectivity" as "data dependency".

[0101] This embodiment transforms the node dependency matrix into an adjacency matrix A through a simple binarization process. This application first accumulates edge counts based on event frequency, then discounts repeated micro-batches within the same window to prevent streaming high-frequency edges from "filling" the area. For cross-system edges, only those with successfully aligned version stamps at both ends are counted. Subsequently, A is standardized into a directed connection graph and stored in the database along with node attributes. Node attributes include: technical metadata (type, creation / update frequency), structural features (in-degree / out-degree, local clustering coefficients), role labels, and the current window's popularity value H(i) and its source weight. The popularity value in the graph is not merely decorative; it is a core signal for subsequent attention and strategy selection, expressing "the necessity of paying attention to this node at this moment."

[0102] This embodiment recognizes that industrial environments exhibit a mixture of "random connections" and "inherent dependencies," and treating them all the same would lead to an overly dense graph. This application introduces edge stability labeling: it statistically analyzes the occurrence rate and interval of edges over the past K windows. Edges with low stability and low cost are initially labeled as "weak dependencies," participating only in visualization and not in training; edges with high stability or co-occurring with critical tasks are labeled as "strong dependencies" and enter training and the policy engine. This selection aligns with natural law: dependencies that frequently co-occur and are linked to critical operations deserve serious consideration in subsequent resource allocation.

[0103] In step S402 of this embodiment, the "business significance of dependency strength" needs to be quantified into a numerical scale of "resource cost". Three metrics are collected: CPU usage time, memory usage, and I / O operation count. These are aggregated by job instance and edge direction (i→j). For convection scenarios, peak values ​​and quantiles are accumulated by window slices to reduce the disturbance of spikes to single-point estimation. Considering the different sensitivities of different task types to these three metrics, this application first performs a robust regression on each task type in the historical period to obtain the explanatory coefficients from the three-dimensional metrics to execution time / back-to-source overhead, and then normalizes to generate a weight set (αCPU, αMEM, αIO). The resource consumption coefficient is obtained by weighting and combining the three-dimensional observations of each edge. And record the calculation details to ensure traceability.

[0104] This embodiment does not simply treat w as a constant, but rather as a time-varying statistic. For each edge, an EWMA (Exponentially Weighted Moving Average) and fluctuation range are maintained, ensuring that short-term spikes do not immediately alter the perception of long-term costs. For cross-domain links (such as cross-cluster or cross-regional links), an additional "transmission distance / network type" penalty is added, ensuring w reflects the true delivery cost. To avoid incomparability of w across different batches of the same service due to differences in data volume, this application normalizes the original three-dimensional indicators of the edges according to the batch size of this window, ensuring that w reflects the marginal cost per unit of data.

[0105] In this embodiment, resource consumption coefficients are written as weights of directed edges into the dependency graph, forming a weighted graph G=(V,E,W). Two layers of indexes are attached when the graph is stored: one is a range index based on edge weights, facilitating the rapid extraction of "high-cost paths"; the other is a sliding window index based on node popularity, supporting the "upstream / downstream cost profile of current hotspots". These two indexes will be directly accessed in the subsequent graph attention training in S102 and the caching / compression decision in S103, ensuring consistency between the training and policy sides.

[0106] The key technical point of this embodiment lies in simultaneously mapping the "authenticity of business dependencies" and the "comparability of resource costs" onto the graph. Authenticity is ensured through three-source alignment and intermediate node folding, avoiding the inadvertent inclusion of network physical connectivity or temporary tool operations into the graph's lineage. Comparability is ensured through scale normalization and task type sensitivity weighting, preventing certain types of tasks from inherently "consuming more CPU" than other dimensions. By combining these two approaches, attention in subsequent graph learning can linger longer on "valuable and expensive" edges, rather than being drawn away by noise.

[0107] This embodiment also provides two on-site variants. One is a data warehouse with complete column-level lineage, where nodes employ a three-tiered structure of "table / partition / column," and the construction of nodes (D) prioritizes the finest granularity, with edge weights of upper-level nodes aggregated from lower-level nodes; thus... It can directly drive "column-level caching" and "partition-level compression". Secondly, in pure query-type data markets, task logs lack CPU / IO details. w uses query execution time, cache miss rate and scanned data volume as proxies, and the weights are determined by offline correlation analysis, which can still characterize the cost per unit of data.

[0108] This embodiment adds three safeguards to ensure engineering robustness. The first is "insufficient evidence" labeling: if the missing rate of a certain three-dimensional indicator exceeds the threshold, w is marked as an interval estimate and its weight is reduced before participating in downstream training. The second is "abnormal peak mitigation": extreme values ​​of w in a single window are synthesized using quantile truncation to preserve relative ordering but prevent numerical explosion. The third is a "legally mandated edge retention" list: the list is matched before pruning or reducing weight to ensure uninterrupted auditing.

[0109] This embodiment ultimately converges the discrete logs and operational facts into a weighted directed graph that clearly explains cause and effect and costs. For production line engineers, when faced with questions like "Why is a certain report slow?" or "Why is a certain chain expensive?", they can follow the upstream of the graph to see not only the current popularity H(i) of a node but also the cost of each hop. This is supported by the underlying CPU / MEM / IO evidence. For the subsequent GAT, the input is no longer a simple adjacency, but a rich map carrying heat and cost. Its attention aggregation during training will naturally follow the intuition of "focusing on current hot topics and cautiously propagating expensive paths", thus providing a solid and interpretable basis for S103's caching and compression decisions.

[0110] In one embodiment of the storage strategy optimization method based on data popularity and data lineage in this application, it may further include the following:

[0111] Step S501: Input the node attribute features and edge weight values ​​into the graph attention network, calculate the attention coefficients between nodes, perform weighted aggregation on the features of adjacent nodes according to the attention coefficients, generate the node representation vector through nonlinear transformation, construct a multi-head attention layer for the nodes based on the representation vectors, and concatenate and normalize the output of the multi-head attention layer.

[0112] Step S502: Iteratively update the normalized node representation, calculate the loss function value of the node representation, adjust the parameters of the graph attention network based on the loss function value, repeat the parameter adjustment process until the loss function value converges, apply the trained graph attention network to the dynamic update of the node representation, and output the dynamic data lineage map.

[0113] Optionally, in this embodiment, steps S501 and S502 revolve around the data platform. The input is a pre-constructed directed weighted graph: the node side carries the heat value H(i), static metadata, structural features, and time location encoding; the edge side carries the resource consumption weight. This application feeds this information into a graph attention network (GAT), allowing the model to consider both "value" (popularity) and "cost" (resource consumption) when aggregating neighborhood information. Then, through multi-head attention and normalization, it generates stable and interpretable node representations for dynamic updates in subsequent link scheduling and storage decisions.

[0114] This embodiment does not skimp on the construction of the input tensor. The initial node feature xi is composed of four parts: technical metadata (size, type, refresh rate), structural features (in / out degree, local clustering coefficient, path betweenness), heat embedding φH(H(i)) (aligned with a cross-role scale using a monotonic mapping), and time-location encoding (shift / hour sine and cosine). Edge features include normalized resource weights. This application incorporates dependency type embedding (ETL / SQL / streaming). Edge weights are used as negative biases for attention, and dependency types as semantic biases, integrated during the attention scoring stage. The intuitive motivation is that when the model "looks at its neighbors," it should both favor hotter paths and recognize high-cost edges to avoid indiscriminate diffusion.

[0115] This embodiment uses learnable linear projection and Leaky ReLU scoring to calculate the attention coefficient, and introduces heat and cost corrections. For each outgoing edge i→j, the basic score sij=aT[Wxi||Wxj] is first calculated, and then... thermal coupling term, subtract The cost suppression term, obtained by LeakyReLU and then normalized by softmax along the neighbor dimension, is... The physical meaning of this step is quite simple: the hotter the two ends, the more eligible they are to be aggregated; the more expensive the edge, the more cautious the aggregation process. Multi-head attention runs in parallel with K heads, each capturing a different "semantic subspace" using independent (Wk, ak, ηk, ζk). For example, one head might be structure-oriented, another behavior-oriented, and yet another focuses on the differences in expensive links. The aggregation vectors output by each head are... Finally, the nodes are concatenated, and layer normalization and residuals are applied to output a single-layer representation. The nonlinear transformation σ is performed using ELU or GELU, and layer normalization helps mitigate layer explosion and gradient instability.

[0116] This embodiment also needs to address the issue of "different response speeds between hot and cold nodes" after S501. This application introduces an intra-layer adaptive temperature τ(i) after normalization, derived monotonically from H(i). Hot nodes use a lower smoothing coefficient (for faster service response), cold nodes use a higher smoothing coefficient (for less sensitivity to noise), and warm nodes are in the middle. In implementation, this is represented by an exponentially smoothed combination of new and historical representations. Hot nodes tend to trust new neighborhoods more, while cold nodes rely more on historical inertia, which aligns with operational experience.

[0117] This embodiment enters the iterative update stage S502. The loss function does not rely on a single objective but is synthesized from multiple components to ensure interpretability and stability. The first component is the edge reconstruction loss Lrec, which requires using node representations to predict the existence and weight range of directed edges, reinforcing the principle that "a good representation is one that can reconstruct lineage." The second component is the heat consistency loss Lhot, which constrains the limited change in the representation direction of hot nodes within adjacent time windows, preventing hotspot fluctuations from causing policy instability. The third component is the semantic classification or ranking loss Lsem (if some key paths or high-value nodes are labeled), allowing the embedding space to separate key nodes. After loss synthesis, the AdamW optimizer is used for mini-batch training, with the learning rate decaying according to the time window. Early stopping is based on the edge AUC and key path recall on the validation set. To avoid oscillations caused by structural abrupt changes, this application introduces a neighbor mask to protect pruned isolated nodes, freezing their gradients in the short term until the next round of heat or edge evidence is sufficient.

[0118] The training process in this embodiment adopts an incremental strategy: at each new window, only a few steps of fine-tuning are performed on the local subgraph where the heat level changes or new edges appear, while other regions directly reuse the old parameters. To enable the model to "remember the cost," this application weights high-cost edges in Lrec, encouraging the model to accurately characterize expensive links; otherwise, downstream systems may mistakenly treat high-cost cold links as cacheable objects. During the online inference phase, GAT reads the updated xi and edge set, outputs a new h(i), and distributes the attention... The version number is written back to the graph database to form a snapshot of "representation + evidence".

[0119] This embodiment retains only one general formula to reflect the core of attention convergence: , .in The heat coupling term is defined as follows: η is the heat gain coefficient, and ζ is the cost penalty coefficient. Here, W, a, and Wk are the normalized values ​​of the resource weights, and they are learnable parameters. σ represents the attention coefficients from i to j, hk_i is the aggregation vector output by the k-th head, and σ is a nonlinear function. Each symbol in the formula has a clear physical meaning: heat drives aggregation, cost inhibits diffusion, and splicing and multiple heads ensure diversity of expression.

[0120] This embodiment also considers two common variants. First, in a pure query-based data marketplace, some edges do not have CPU / memory sampling; in this application, query time is approximated by cache miss rate. First, η remains unchanged, while ζ is tuned using the validation set. Second, in older systems with incomplete column-level lineages, xi lacks fine-grained attributes. This application adds "field popularity statistics aggregation" as a proxy dimension to prevent the model from only looking at structural metrics. Neither variant changes the core logic: attention is balanced between value and cost tensions.

[0121] This embodiment ultimately delivers a dynamic, "breathing" data lineage graph. After training, the node representation h(i) is periodically updated, the attention distribution reflects the current value path, and the downstream hierarchical storage and caching engine reads h and combines it with a heat threshold to make decisions. Operations personnel can see a clear chain of evidence: why a certain hot node is heavily aggregated (high Ĥij), why a certain edge is suppressed (high ŵ, large ζ), and parameter versions and loss curves are all documented. This binds statistical learning with physical intuition, making policy stability easier to interpret.

[0122] This embodiment observed two types of feedback paths after deployment. One type is policy write-back: cache hits and source latency are sent back to the training phase, the edge weight prediction error of Lrec is corrected, and η and ζ gradually approach the true cost surface; the other type is the impact of the operating calendar: within a large maintenance window, Lhot is relaxed, allowing faster representation migration; it is tightened again after production resumes. Overall, this S501-S502 process integrates node attributes, edge costs, heat, and time context into a single entity. The output graph is not a static data structure, but a real-time reference system supporting storage strategy decisions.

[0123] In one embodiment of the storage strategy optimization method based on data popularity and data lineage in this application, it may further include the following:

[0124] Step S601: Extract the heat value of the nodes from the dynamic data pedigree map, establish a probability distribution model of the heat value, calculate the mean and standard deviation of the heat value, set the heat threshold range based on the mean and standard deviation, mark the nodes with heat values ​​higher than the upper threshold as hot nodes, mark the nodes with heat values ​​between the upper and lower thresholds as warm nodes, and mark the nodes with heat values ​​lower than the lower threshold as cold nodes.

[0125] Step S602: Construct subgraph structures for the hot nodes, warm nodes, and cold nodes respectively, calculate the connectivity index and node importance score of the subgraph, establish node state transition rules based on the connectivity index and importance score, dynamically adjust the hot and cold attribute labels of the nodes according to the state transition rules, and update the hierarchical structure of the data lineage graph.

[0126] Optionally, this embodiment follows the dynamic data lineage map obtained in the previous steps, where nodes already carry the final heat value H(i), role, and structural attributes. Regarding S601 and S602, this application clarifies "how to classify hot and cold nodes" and "how to make the hot and cold status change self-consistently with structural evidence." First, the scope of the heat distribution is defined. This application extracts H(i) of all nodes from the graph of the current time window, removes nodes with insufficient evidence and frozen nodes (such as nodes that are legally required to be retained but have no current observations), and establishes a probability distribution model of the heat value for the remaining set. In engineering, a normal distribution is not directly assumed; robust estimation is used in one step: first, a coarse distribution shape is judged using quantiles (skewed / long-tailed); if skewness is obvious, a Box-Cox or logarithmic mapping is performed on H, and then the median and MAD are used to estimate the center and scale, obtaining robust substitutes for the mean μ and standard deviation σ; if the shape is close to symmetrical, the classical mean and standard deviation are reverted to. This approach avoids a small number of extremely hot nodes "driving up" the threshold, and also prevents the threshold from being too low in a globally cold scene.

[0127] This embodiment sets the heat threshold range using (μ,σ) as the scale. The upper threshold θhigh and lower threshold θlow are not mechanically fixed kσ, but are fine-tuned by combining role priors and load calendar. For HMIs and reports that are sensitive to interaction latency, θlow is slightly lower to prevent false degradation; for historical databases and ETL operators that are sensitive to throughput, θhigh is slightly higher to highlight true hotspots. Considering the differences in business rhythms across different workshops, this application performs local calibration for each subdomain and then maps it back to the global scale to ensure cross-domain comparability. Nodes are marked as hot / warm / cold according to the threshold, and hysteresis bands and minimum dwell times are set to avoid frequent migrations caused by threshold jitter. For nodes that have been consistently hot for a long time but suddenly cool down, if their upstream remains hot and the path cost is high, this application delays the degradation within one window and waits for confirmation in the next window to reduce misjudgments.

[0128] This embodiment proceeds to S602, where the three types of nodes are extracted into subgraphs for structure and importance assessment. Subgraph construction follows the principle of "retaining key cross-layer edges to reflect real upstream and downstream constraints": hot subgraphs retain hot-to-hot connections, along with warm / cold incoming edges pointing to hot nodes as boundary evidence; warm subgraphs retain warm-to-warm connections and strongly coupled hot / cold first-order edges; cold subgraphs only retain first-order edges pointing to hot / warm nodes, with cold-to-cold edges only retained within the compliance list. Connectivity metrics (number of weakly connected components, average path length, edge density) and node importance scores (betweenness, eigenvector centrality, PageRank, or influence score under resource constraints) are calculated for each subgraph. The inherent meaning of these quantities is: if a node connects to a critical path and carries a large amount of throughput in a hot subgraph, it should be prioritized; even if a node has a high degree in a cold subgraph, if it connects to cold nodes and does not connect to hot / warm nodes, its value to current business is limited, and it is suitable for compression.

[0129] This embodiment establishes node state transition rules based on the above indicators, characterizing the "hot ↔ warm ↔ cold" change as an evidence-based finite state machine. The rules in this application consider not only H(i), but also structural evidence and path cost. For example, the condition for warm → hot typically includes H exceeding θhigh and the node's betweenness number in the hot subgraph reaching the quantile threshold, or its shortest path weight (cumulative resource cost) to the hot node being significantly high, indicating it is on an expensive critical chain; the condition for hot → warm is that H falls below the hysteresis upper bound and its α-class importance (e.g., PageRank) in the hot subgraph declines, or it is replaced by a new hot node on the path; warm → cold requires H to remain below θlow and the node not serving as a critical bridge to hot / warm in the cold subgraph. To avoid rigid hard thresholds, this application makes the above conditions a fraction, superimposed with a temporal stability term and compliance constraints, triggering a transition only when the composite threshold is exceeded.

[0130] In this embodiment, to integrate state transition and graph learning, importance scores and connectivity metrics are fed back into the online update of GAT. During the update, this application does not directly modify the model structure, but injects information in two ways: first, it embeds importance scores into the structural channels of node features, allowing the attention layer to perceive which nodes are more critical in the current subgraph; second, it adds a reconstruction weighting of warm-hot edges to the training loss, allowing the model to more accurately distinguish between critical paths and edge paths. This feedback loop conforms to natural laws: when a node structurally acts as a "bridge," the model should pay more attention to its neighborhood information; when a path is costly and runs through hotspots, the policy layer should tend to retain its cache to avoid frequent backtracking.

[0131] This embodiment considers two types of on-site differences during implementation. For time-series warehouses, popularity is often concentrated in the most recent partitions. Subgraph construction is refined to the partition-level nodes, and importance is assessed using "time-sensitive PageRank". Newer partitions are given weights with less decay, and state transitions allow for rapid recovery at the partition level. For cross-system lineage (data marketplace - workshop warehouse), due to different measurement methods, this application aligns the subgraph connectivity indicators: first, they are calculated within the system, and then standardized and merged using bridging edge weights to avoid diluting the critical chains of another system due to the natural "more edges" in one system.

[0132] This embodiment sets up two safeguards for risk control. The first is the "compliance retention" list, which ensures that any cold-to-cold links that require full traceability under regulations, even if they have low connectivity and weak importance, will not trigger pruning or degradation, but will only reduce their weight in aggregation and strategy decisions. The second is "abnormal disturbance" identification. If a major overhaul or batch rerun occurs within a window, the connectivity of the heatmap will be temporarily disrupted. This application uses a working condition calendar to reduce the sensitivity of state transitions, making the cold and hot markings more stable within this range.

[0133] This embodiment writes the results of the above rules back to the graph database: updating the hot / cold attributes, residency count, and most recent state change time of each node; archiving the connectivity summary and importance distribution of the three subgraphs; and recording the evidence information triggering the transition (heat threshold crossing, betweenness change, path cost, etc.). The next round of graph attention training reads these attributes and automatically adjusts the attention distribution; the storage engine stratifies accordingly: hot nodes are cached, warm nodes are kept online, and cold nodes are compressed according to a predetermined strategy while retaining the first-order relationship pointing to hot / warm nodes.

[0134] Finally, this embodiment includes a small formula to illustrate the composition of the synthesis transfer fraction, for easier understanding: Where Hz(i) is the heat score z of node i (normalized to the current (μ,σ)); Cz(i) is its connectivity score z in the corresponding subgraph (normalized value such as betweenness or PageRank); Pz(i) is the inverse score of the shortest path cost to the hot core (the lower the cost, the higher the score); R(i) is the compliance and disturbance penalty term (the penalty increases if it is in the maintenance window or compliance is retained); λc, λp, and λr are composite coefficients, calibrated according to role and operating condition. When the score exceeds the corresponding state threshold, a temperature-to-hot or cold-to-temperature transition is triggered, with reverse transitions using symmetrical thresholds and hysteresis. The physical meanings of each term are straightforward: heat represents current use, structure represents criticality, path represents convenience of approaching the hotspot, and penalty represents external constraints.

[0135] The practical effectiveness of this embodiment is reflected in three aspects: the hot / cold node classification is no longer based solely on scores, but rather on group distribution and role context; state transitions are no longer based solely on single windows, but rather on structural and path evidence; and hierarchical structure updates are not "one-off decisions," but form a closed loop with graph learning and storage strategies, ensuring robust handling of shift changes, maintenance, or sudden business disruptions. If engineers inquire about "why a certain node became hot," they can find concrete evidence in the records, such as H exceeding the threshold, betweenness increase, and a shortened path to the hot core, ensuring informed decision-making.

[0136] In one embodiment of the storage strategy optimization method based on data popularity and data lineage in this application, it may further include the following:

[0137] Step S701: Construct a multi-level cache structure, store the hot node data in a high-speed cache, encode the cold node data using a lossless compression algorithm, migrate the compressed cold node data to a low-speed storage device, scan the dependencies of the cold nodes, retain the first-order adjacency relationships with the hot and warm nodes, remove the connection edges between the cold nodes, and update the adjacency matrix of the dependency graph.

[0138] Step S702: Obtain the real-time heat value of the node, adjust the weight coefficient of the node in the graph attention network based on the real-time heat value, recalculate the attention distribution of the node, weight and combine the attention distribution with the historical representation vector of the node to generate the updated representation vector of the node, and input the updated representation vector into the graph attention network for online learning.

[0139] Optionally, this embodiment follows the hot and cold stratification results and weighted lineage graph after S103, and applies the "strategy execution - graph representation backflow" closed loop to the engineering scenario around S701 and S702. First, the multi-level cache structure is built. This application divides the data into three levels based on media performance: L1 is a memory or NVMe direct-connect cache for millisecond-level response; L2 is a local NVMe / SSD page cache for second-level hot data; L3 is object storage or SATA disk for cold data. Hot nodes are prioritized according to their heat H(i), critical path betweenness, and unit cost (path aggregation of edge weight w). Quota allocation is not a one-size-fits-all approach, but rather allocated at the sub-object granularity (field / column / partition) to avoid relocating the entire table. The expulsion strategy adopts a two-factor approach of "heat decay + path value": when H(i) is downlinked and its expensive upstream dependencies have not been accessed recently, L1→L2 are gradually rolled back; conversely, newly emerging hot nodes are first "upgraded" from L2 to L1, and cached records of hit and back-to-origin delay serve as evidence for the next window weight calibration.

[0140] This embodiment performs lossless compression and migration on cold nodes. Columnar data uses dictionary encoding + RLE or Delta + Zstd, while time-series data uses lossless schemes such as GORILLA / Delta-of-Delta. Compression ratio, encoding time, and warm-up time are recorded together. The migration strategy is performed according to idle windows to avoid contention for I / O with daytime queries. After migration, cold and hot levels are marked, and the object index is updated. The read path of the cold node is pointed to the L3 gateway. If necessary, a "pre-warm-up" threshold is set, and when H(i) continuously rises to the temperature zone boundary, a decompressed copy is prepared in advance in L2.

[0141] This embodiment prunes the graph structure of cold nodes to reduce computational and storage overhead without sacrificing traceability. This application scans the first-order adjacencies of cold nodes, retaining direct edges pointing to hot / warm nodes. Edges between cold nodes are added to a candidate deletion set, and then each edge is checked against the regulatory list and audit strategy. Edges with mandatory traceability are marked as "frozen" and not deleted. The remaining cold-to-cold edges are removed using a differential method and written to the change log, and the adjacency matrix A is updated accordingly. Considering the impact of structural abrupt changes on learning, this application informs the neighborhood mask of the GAT of the impact range of the deleted edges. In the next training cycle, the learning rate is reduced for affected nodes to prevent attention from suddenly focusing on a few remaining neighbors.

[0142] In this embodiment, S702 integrates "real-time heat → attention weight → node representation". Each new window obtains the latest value of H(i), converts it into aggregate gain using a monotonic mapping φH, and then applies the edge weights... Attention scoring is embedded as a cost-suppression term. This application recalculates α for each attention k. Then, compared with the attention distribution α in the previous version... For smoothing, hot nodes use a smaller smoothing coefficient (for fast response), cold nodes use a larger smoothing coefficient (for stability), and warm nodes fall somewhere in between. After obtaining the new aggregated output, the node representation is not directly replaced, but rather combined with the historical representation h_prev(i) using an evidence-weighted approach. ρ(i) is determined by the fluctuations in node popularity and the effects of pruning. Larger fluctuations and more structural changes reduce ρ, allowing new information to account for a larger proportion. The combined h_new enters the online learning pipeline for several incremental updates.

[0143] This embodiment of online learning does not rely on large-scale offline replays; it only requires the subgraph of the current window and a small amount of nearest-neighbor history. On the loss side, this application retains three constraints: edge reconstruction loss allows the representation to recover existing directed edges and their weight ranges; temporal consistency loss limits representation drift under stable conditions and relaxes it on maintenance days; and popularity consistency loss requires the nearest-neighbor composition of hot nodes to remain coherent for a short period, avoiding caching strategy jitter. Parameter updates employ small step sizes, weight decay, and early stopping, specifically using the edge AUC of the validation set within the window and critical path recall as dual indicators for stopping.

[0144] This embodiment also considers two types of field variants. For warehouses with a high proportion of column / partition level caching, H(i) drills down to the partition node, and attention calculation is performed simultaneously at the three levels of "table-partition-column". Pruning only reaches cold partitions and does not touch the global structure of hot tables. For periods with incomplete log collection, this application uses the H of the previous window and the EWMA of the edge weights as a fallback, and adjusts the sample weights of nodes with missing data during learning to avoid "false heat" misleading the model.

[0145] This embodiment establishes a feedback loop between strategy and learning. Cache hit rate, origin retrieval latency, and compression / decompression time are aggregated into the evidence base, and the next window is used to correct the scale of φH and the cost penalty coefficient in attention. Changes in query paths caused by pruning are also fed back in. If the deletion of a cold-cold edge leads to an abnormally high origin retrieval cost for hot nodes, a dual self-healing mechanism of "edge replacement" and "level promotion" is triggered. All actions are written to the audit log, including node ID, previous and subsequent levels, migration batch, list of deleted edges, and GAT version number, facilitating engineer review.

[0146] The tangible results of this implementation are immediately apparent: hot nodes are more likely to appear in L1 / L2, queries bypass high-cost cold links, and graph learning remains sensitive to the latest business rhythm without excessive fluctuation. More importantly, every caching, compression, pruning, and representation update is based on verifiable evidence—hotness trajectories, edge weight evidence, attention distribution, and loss curves are linked together. The strategy is no longer a "black box conclusion," but a verifiable process that explains "why this is done at this moment."

[0147] To effectively address the shortcomings of traditional technologies in areas such as data popularity assessment, lineage analysis, and storage optimization, and to provide technical support for software system storage management, this application provides an embodiment of a storage strategy optimization device based on data popularity and lineage for implementing all or part of the aforementioned storage strategy optimization method. See [link to embodiment]. Figure 2 The storage strategy optimization device based on data popularity and data lineage specifically includes the following components:

[0148] The node popularity determination module 10 is used to collect data packet size, communication frequency, data type and timestamp information of nodes in the software system, to statistically analyze the access frequency, access time and user behavior of the nodes, to calculate the access frequency popularity, time decay popularity and user behavior popularity of the nodes, and to combine the access frequency popularity, time decay popularity and user behavior popularity to generate a node popularity value.

[0149] The lineage graph construction module 20 is used to construct a directed graph with the nodes as vertices and the dependencies between nodes as edges. The node popularity value is used as a node attribute, the computing resource consumption of the node is used as an edge weight, and the directed graph is iteratively trained using a graph attention network to generate a dynamic data lineage graph.

[0150] The strategy storage module 30 is used to divide nodes into cold nodes, warm nodes, and hot nodes based on the heat value of nodes in the dynamic data lineage graph, adopt a priority caching strategy for hot nodes, compress and store cold nodes, retain the direct dependency relationship between cold nodes and hot nodes and warm nodes, delete the indirect dependency relationship of cold nodes, and dynamically update the node representation vector of the graph attention network based on the heat value of the nodes.

[0151] As described above, the storage strategy optimization device based on data popularity and data lineage provided in this application can achieve dynamic optimization of storage strategies through innovative design of a multi-dimensional popularity calculation model, node attribute analysis, and graph attention networks. It constructs a directed graph structure and, combined with lineage analysis, establishes a reliable dependency management system. A hierarchical storage strategy is introduced, ensuring storage efficiency through popularity partitioning and dependency optimization. This method effectively addresses the shortcomings of traditional technologies in popularity assessment, lineage analysis, and storage optimization, providing technical support for software system storage management.

[0152] From a hardware perspective, in order to effectively address the shortcomings of traditional technologies in areas such as heat assessment, lineage analysis, and storage optimization, and to provide technical support for software system storage management, this application provides an embodiment of an electronic device for implementing all or part of the aforementioned storage strategy optimization method based on data heat and data lineage. The electronic device specifically includes the following components:

[0153] The system comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the storage strategy optimization device based on data popularity and data lineage and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the storage strategy optimization method based on data popularity and data lineage, and the embodiments of the storage strategy optimization device based on data popularity and data lineage in the embodiments, the contents of which are incorporated herein, and repeated details will not be described again.

[0154] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0155] In practical applications, the storage strategy optimization method based on data popularity and data lineage can be partially executed on the electronic device side as described above, or all operations can be completed on the client device. The specific choice depends on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed on the client device, the client device may further include a processor.

[0156] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0157] Figure 3 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 3 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 3This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0158] In one embodiment, the storage strategy optimization method based on data popularity and data lineage can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following controls:

[0159] Step S101: Collect data packet size, communication frequency, data type and timestamp information of nodes in the software system, statistically analyze the access frequency, access time and user behavior of the nodes, calculate the access frequency heat, time decay heat and user behavior heat of the nodes, and combine the access frequency heat, time decay heat and user behavior heat to generate a node heat value.

[0160] Step S102: Construct a directed graph with the nodes as vertices and the dependencies between nodes as edges, use the node heat value as a node attribute, use the computing resource consumption of the nodes as edge weights, and use a graph attention network to iteratively train the directed graph to generate a dynamic data lineage graph.

[0161] Step S103: Based on the heat value of the nodes in the dynamic data lineage graph, the nodes are divided into cold nodes, warm nodes and hot nodes. A priority caching strategy is adopted for the hot nodes, and the cold nodes are compressed and stored. The direct dependencies between the cold nodes and the hot nodes and warm nodes are retained, and the indirect dependencies between the cold nodes are deleted. The node representation vector of the graph attention network is dynamically updated based on the heat value of the nodes.

[0162] As described above, the electronic device provided in this application innovatively designs a multi-dimensional heat calculation model and achieves dynamic optimization of storage strategies through node attribute analysis and graph attention networks. It constructs a directed graph structure and, combined with lineage analysis, establishes a reliable dependency management system. A hierarchical storage strategy is introduced, ensuring storage efficiency through heat partitioning and dependency optimization. This method effectively addresses the shortcomings of traditional technologies in heat assessment, lineage analysis, and storage optimization, providing technical support for software system storage management.

[0163] In another embodiment, the storage strategy optimization device based on data popularity and data lineage can be configured separately from the central processing unit 9100. For example, the storage strategy optimization device based on data popularity and data lineage can be configured as a chip connected to the central processing unit 9100, and the storage strategy optimization method function based on data popularity and data lineage can be implemented through the control of the central processing unit.

[0164] like Figure 3As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 3 All components shown; in addition, the electronic device 9600 may also include Figure 3 For components not shown, please refer to existing technologies.

[0165] like Figure 3 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0166] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0167] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0168] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0169] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0170] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0171] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored audio via the speaker 9131.

[0172] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the storage strategy optimization method based on data popularity and data lineage, where the execution subject is a server or client, as described in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the storage strategy optimization method based on data popularity and data lineage, where the execution subject is a server or client, as described in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0173] Step S101: Collect data packet size, communication frequency, data type and timestamp information of nodes in the software system, statistically analyze the access frequency, access time and user behavior of the nodes, calculate the access frequency heat, time decay heat and user behavior heat of the nodes, and combine the access frequency heat, time decay heat and user behavior heat to generate a node heat value.

[0174] Step S102: Construct a directed graph with the nodes as vertices and the dependencies between nodes as edges, use the node heat value as a node attribute, use the computing resource consumption of the nodes as edge weights, and use a graph attention network to iteratively train the directed graph to generate a dynamic data lineage graph.

[0175] Step S103: Based on the heat value of the nodes in the dynamic data lineage graph, the nodes are divided into cold nodes, warm nodes and hot nodes. A priority caching strategy is adopted for the hot nodes, and the cold nodes are compressed and stored. The direct dependencies between the cold nodes and the hot nodes and warm nodes are retained, and the indirect dependencies between the cold nodes are deleted. The node representation vector of the graph attention network is dynamically updated based on the heat value of the nodes.

[0176] As described above, the computer-readable storage medium provided in this application, through an innovative design of a multi-dimensional heat calculation model, achieves dynamic optimization of storage strategies via node attribute analysis and graph attention networks. It constructs a directed graph structure and, combined with lineage analysis, establishes a reliable dependency management system. A hierarchical storage strategy is introduced, ensuring storage efficiency through heat partitioning and dependency optimization. This method effectively addresses the shortcomings of traditional technologies in heat assessment, lineage analysis, and storage optimization, providing technical support for software system storage management.

[0177] Embodiments of this application also provide a computer program product capable of implementing all steps of the storage strategy optimization method based on data popularity and data lineage, where the execution subject is a server or client, as described in the above embodiments. When executed by a processor, this computer program / instruction implements the steps of the storage strategy optimization method based on data popularity and data lineage. For example, the computer program / instruction implements the following steps:

[0178] Step S101: Collect data packet size, communication frequency, data type and timestamp information of nodes in the software system, statistically analyze the access frequency, access time and user behavior of the nodes, calculate the access frequency heat, time decay heat and user behavior heat of the nodes, and combine the access frequency heat, time decay heat and user behavior heat to generate a node heat value.

[0179] Step S102: Construct a directed graph with the nodes as vertices and the dependencies between nodes as edges, use the node heat value as a node attribute, use the computing resource consumption of the nodes as edge weights, and use a graph attention network to iteratively train the directed graph to generate a dynamic data lineage graph.

[0180] Step S103: Based on the heat value of the nodes in the dynamic data lineage graph, the nodes are divided into cold nodes, warm nodes and hot nodes. A priority caching strategy is adopted for the hot nodes, and the cold nodes are compressed and stored. The direct dependencies between the cold nodes and the hot nodes and warm nodes are retained, and the indirect dependencies between the cold nodes are deleted. The node representation vector of the graph attention network is dynamically updated based on the heat value of the nodes.

[0181] As described above, the computer program product provided in this application innovatively designs a multi-dimensional heat calculation model and achieves dynamic optimization of storage strategies through node attribute analysis and graph attention networks. It constructs a directed graph structure and, combined with lineage analysis, establishes a reliable dependency management system. A hierarchical storage strategy is introduced, ensuring storage efficiency through heat partitioning and dependency optimization. This method effectively addresses the shortcomings of traditional technologies in heat assessment, lineage analysis, and storage optimization, providing technical support for software system storage management.

[0182] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0183] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0184] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0185] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0186] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A storage strategy optimization method based on data popularity and data lineage, characterized in that, The method includes: The system collects data packet size, communication frequency, data type, and timestamp information of nodes in the software system, statistically analyzes the access frequency, access time, and user behavior of the nodes, calculates the access frequency popularity, time decay popularity, and user behavior popularity of the nodes, and combines the access frequency popularity, time decay popularity, and user behavior popularity to generate a node popularity value. A directed graph is constructed with the nodes as vertices and the dependencies between nodes as edges. The node popularity value is used as a node attribute, and the computational resource consumption of the nodes is used as edge weights. The directed graph is iteratively trained using a graph attention network to generate a dynamic data lineage graph. This includes: collecting the input and output data streams, data processing tasks, and communication relationships between nodes in the software system; constructing a node dependency matrix based on the data streams and communication relationships; transforming the node dependency matrix into an adjacency matrix; establishing directed connections between nodes; and adding the node popularity value as a node attribute feature. The CPU time, memory usage, and number of I / O operations during data transmission between nodes are monitored, and the CPU time, memory usage, and number of I / O operations are weighted and combined to calculate... The resource consumption coefficient between nodes is used as the weight value of directed edges to generate a weighted dependency graph. The node attribute features and edge weight values ​​are input into the graph attention network to calculate the attention coefficient between nodes. The features of adjacent nodes are weighted and aggregated according to the attention coefficient. A node representation vector is generated through nonlinear transformation. A multi-head attention layer is constructed based on the representation vector. The output of the multi-head attention layer is concatenated and normalized. The normalized node representation is iteratively updated, and the loss function value of the node representation is calculated. The parameters of the graph attention network are adjusted based on the loss function value. The parameter adjustment process is repeated until the loss function value converges. The trained graph attention network is applied to the dynamic update of node representation to output a dynamic data lineage map. Based on the heat value of nodes in the dynamic data lineage graph, nodes are divided into cold nodes, warm nodes, and hot nodes. A priority caching strategy is adopted for hot nodes, while cold nodes are compressed and stored. The direct dependencies between cold nodes and hot and warm nodes are retained, while the indirect dependencies between cold nodes are deleted. The node representation vector of the graph attention network is dynamically updated based on the heat value of the nodes.

2. The storage strategy optimization method based on data popularity and data lineage according to claim 1, characterized in that, The data acquisition software system collects data packet size, communication frequency, data type, and timestamp information of nodes, and statistically analyzes the access frequency, access time, and user behavior of these nodes. It calculates the access frequency popularity, time decay popularity, and user behavior popularity of these nodes, including: Read the software system operation log, extract node identifiers, data packet size, communication frequency, data type and timestamp information from the operation log, establish a node data indicator matrix, divide the node data indicator matrix according to a preset time window, generate a multi-dimensional data feature sequence, and normalize the multi-dimensional data feature sequence. The access frequency, access time, and user dwell time of nodes within the time window are statistically analyzed. The access frequency is divided by the maximum access frequency to obtain the access frequency popularity. The difference between the current time and the most recent access time is calculated and time decay popularity is obtained based on the decay function. The weighted sum of the user dwell time and the number of clicks is divided by the maximum weighted sum to obtain the user behavior popularity. The access frequency popularity, time decay popularity, and user behavior popularity are linearly combined to obtain the comprehensive popularity value.

3. The storage strategy optimization method based on data popularity and data lineage according to claim 1, characterized in that, The step of combining the access frequency popularity, time decay popularity, and user behavior popularity to generate a node popularity value includes: Calculate the historical popularity value distribution of the node within multiple time windows, determine the weight coefficients of access frequency popularity, time decay popularity and user behavior popularity based on the historical popularity value distribution, multiply the weight coefficients by the corresponding popularity values ​​and sum them to obtain the initial popularity value of the node, and construct the node popularity evaluation matrix. The node popularity evaluation matrix is ​​subjected to eigenvalue decomposition to extract principal component feature vectors. A popularity mapping function is constructed based on the principal component feature vectors. The initial popularity value is input into the popularity mapping function for normalization processing to generate the final popularity value of the node. The final popularity value is used as the popularity attribute of the node.

4. The storage strategy optimization method based on data popularity and data lineage according to claim 1, characterized in that, The process of classifying nodes into cold nodes, warm nodes, and hot nodes based on the heat value of nodes in the dynamic data pedigree graph includes: The heat value of the nodes is extracted from the dynamic data pedigree, a probability distribution model of the heat value is established, the mean and standard deviation of the heat value are calculated, and a heat threshold range is set based on the mean and standard deviation. Nodes with heat values ​​higher than the upper threshold are marked as hot nodes, nodes with heat values ​​between the upper and lower thresholds are marked as warm nodes, and nodes with heat values ​​lower than the lower threshold are marked as cold nodes. Subgraph structures are constructed for the hot nodes, warm nodes, and cold nodes respectively. The connectivity index and node importance score of the subgraph are calculated. Node state transition rules are established based on the connectivity index and importance score. The hot and cold attribute labels of the nodes are dynamically adjusted according to the state transition rules, and the hierarchical structure of the data lineage graph is updated.

5. The storage strategy optimization method based on data popularity and data lineage according to claim 1, characterized in that, The process of prioritizing caching of hot nodes, compressing and storing cold nodes, retaining direct dependencies between cold nodes and hot and warm nodes, deleting indirect dependencies of cold nodes, and dynamically updating the node representation vector of the graph attention network based on the node's popularity value includes: A multi-level cache structure is constructed, the hot node data is stored in the high-speed cache, the cold node data is encoded using a lossless compression algorithm, the compressed cold node data is migrated to a low-speed storage device, the dependencies of the cold nodes are scanned, the first-order adjacency relationship with the hot and warm nodes is retained, the connection edges between the cold nodes are removed, and the adjacency matrix of the dependency graph is updated. The real-time heat value of a node is obtained, and the weight coefficient of the node in the graph attention network is adjusted based on the real-time heat value. The attention distribution of the node is recalculated, and the attention distribution is weighted and combined with the historical representation vector of the node to generate the updated representation vector of the node. The updated representation vector is then input into the graph attention network for online learning.

6. A storage strategy optimization device based on data popularity and data lineage, characterized in that, The device includes: The node popularity determination module is used to collect data packet size, communication frequency, data type and timestamp information of nodes in the software system, to statistically analyze the access frequency, access time and user behavior of the nodes, to calculate the access frequency popularity, time decay popularity and user behavior popularity of the nodes, and to combine the access frequency popularity, time decay popularity and user behavior popularity to generate a node popularity value. A lineage graph construction module is used to construct a directed graph with the nodes as vertices and the dependencies between nodes as edges. The module uses the node popularity value as a node attribute and the computational resource consumption of the nodes as edge weights. It iteratively trains the directed graph using a graph attention network to generate a dynamic data lineage graph. This includes: collecting the input and output data streams, data processing tasks, and communication relationships between nodes in the software system; constructing a node dependency matrix based on the data streams and communication relationships; converting the node dependency matrix into an adjacency matrix; establishing directed connections between nodes; adding the node popularity value as a node attribute feature; and monitoring CPU time, memory usage, and I / O operation counts during data transmission between nodes, and weighting the CPU time, memory usage, and I / O operation counts. The process involves: calculating the resource consumption coefficient between nodes; using this coefficient as the weight of directed edges to generate a weighted dependency graph; inputting the node attribute features and edge weights into a graph attention network to calculate the attention coefficient between nodes; weighting and aggregating the features of adjacent nodes based on the attention coefficient; generating node representation vectors through nonlinear transformation; constructing a multi-head attention layer for nodes based on these representation vectors; concatenating and normalizing the outputs of the multi-head attention layer; iteratively updating the normalized node representations; calculating the loss function value of the node representations; adjusting the parameters of the graph attention network based on the loss function value; repeating the parameter adjustment process until the loss function value converges; and applying the trained graph attention network to the dynamic update of node representations to output a dynamic data lineage graph. The strategy storage module is used to divide nodes into cold nodes, warm nodes, and hot nodes based on the heat value of nodes in the dynamic data lineage graph, adopt a priority caching strategy for hot nodes, compress and store cold nodes, retain the direct dependencies between cold nodes and hot nodes and warm nodes, delete the indirect dependencies of cold nodes, and dynamically update the node representation vector of the graph attention network based on the heat value of the nodes.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the storage strategy optimization method based on data popularity and data lineage as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the storage strategy optimization method based on data popularity and data lineage as described in any one of claims 1 to 5.

Images