Data consanguinity and hotspot awareness driven global data channel analysis method and data consanguinity and hotspot awareness driven global data channel analysis system

Abstract

The invention discloses a data consanguinity and hotspot awareness driven global data table analysis method and system, and relates to the technical field of data processing, and the method comprises the steps: reading a data flow log, ETL job metadata and a query dependency relationship of a global data table, and carrying out the analysis to form a multi-level data consanguinity map; log features are extracted to construct a popularity time sequence and a popularity spectrum; fusing the sensitivity spectrum and the popularity spectrum to construct a double-spectrum hotspot sensing model; and inputting real-time data characteristics into the model to obtain a node comprehensive influence score, and performing node strategy correction management. According to the method, the technical problems that the data dependency relationship of a global data platform is difficult to accurately describe, data hotspots cannot be comprehensively sensed, data node influence evaluation is inaccurate and resource allocation and data value are not matched are solved, and the purposes of clear data dependency, comprehensive hotspot sensing, accurate node influence evaluation and reasonable resource allocation are achieved. Therefore, the technical effects of accurate evaluation and efficient management and control of the middle table are achieved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for analyzing a full-domain data platform driven by data lineage and hotspot perception. Background Technology

[0002] The global data platform is the core carrier of data processing. The accuracy of its data dependency traceability and the effectiveness of its control over hot data directly impact data processing efficiency, resource utilization, and the quality of business decisions. Existing data processing technologies often employ table-level data lineage analysis and single-dimensional hotspot judgment. While these methods can be effective in environments with simple data chains and stable scenarios, they become increasingly inadequate as the scale of global data expands and the flow of data becomes more multi-layered. Traditional technologies exhibit significant limitations when applied to global data platforms, failing to meet the demands for precise analysis and efficient management. Summary of the Invention

[0003] This application provides a method and system for analyzing a full-domain data platform driven by data lineage and hotspot perception, which solves the technical problems of full-domain data platforms, such as difficulty in accurately depicting data dependencies, inability to fully perceive data hotspots, inaccurate assessment of the impact of data nodes, and mismatch between resource allocation and data value.

[0004] The first aspect of this application provides a data lineage and hotspot perception-driven method for analyzing a full-domain data platform. The method includes: reading data flow logs, ETL job metadata, and query dependencies from the full-domain data platform; performing field-level, table-level, and model feature-level dependency path parsing based on the reading results to form a multi-level data lineage graph, where each directed edge of the multi-level data lineage graph is labeled with a causal weight parameter reflecting the intensity of data transmission; extracting change features and indicator refresh frequency features from the data flow logs to construct a heat time series; performing joint analysis of the short-term and long-term windows of the heat time series to construct a heat spectrum; calculating the sensitivity spectrum of each data node using the causal weight parameters; fusing the sensitivity spectrum with the heat spectrum to construct a dual-spectrum hotspot perception model; inputting real-time data features as input data into the dual-spectrum hotspot perception model; using the model output as the node weight input of the multi-level data lineage graph; performing attenuation influence propagation calculations to establish a node comprehensive influence score; and performing node strategy correction management based on the node comprehensive influence score.

[0005] The second aspect of this application provides a comprehensive data platform analysis system driven by data lineage and hotspot awareness. The system includes: a multi-level data lineage graph acquisition module, used to read data flow logs, ETL job metadata, and query dependencies from the comprehensive data platform; and to perform field-level, table-level, and model feature-level dependency path parsing based on the read results to form a multi-level data lineage graph, wherein each directed edge of the multi-level data lineage graph is labeled with a causal weight parameter reflecting the intensity of data transmission; a heat time series construction module, used to extract change features and indicator refresh frequency features from the data flow logs to construct a heat time series; and a heat spectrum construction module. The system is used to perform joint analysis of the short-term and long-term windows of the heat time series to construct a heat spectrum; the dual-spectrum hotspot perception model construction module is used to calculate the sensitivity spectrum of each data node using the causal weight parameters, and fuse the sensitivity spectrum with the heat spectrum to construct a dual-spectrum hotspot perception model; the node comprehensive influence score construction module is used to input real-time data features as input data into the dual-spectrum hotspot perception model, use the model output as the node weight input of the multi-level data lineage graph, perform attenuation influence propagation calculation, and establish a node comprehensive influence score; the node strategy correction management execution module is used to perform node strategy correction management based on the node comprehensive influence score.

[0006] One or more technical solutions provided in this application have at least the following technical effects or advantages: This application reads the data flow logs, ETL job metadata, and query dependencies of the full-domain data platform to parse multi-level data dependency paths, constructs a lineage graph with causal weights, extracts data change and indicator refresh features to construct a heat time series and generate a heat spectrum, calculates the data node sensitivity spectrum by combining causal weights and integrates them into a dual-spectrum hotspot perception model, calculates the comprehensive impact score of nodes by inputting real-time data features, and then executes strategy correction. This enables traceable data dependencies, accurate hotspot perception, and quantitative assessment of node impact in the full-domain data platform, making resource allocation more reasonable and operation and management more efficient. It achieves clear data dependencies, comprehensive hotspot perception, accurate node impact assessment, and reasonable resource allocation, thereby supporting the platform's accurate assessment and efficient management. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1This is a flowchart illustrating the data lineage and hotspot perception-driven full-domain data platform analysis method provided in this application embodiment.

[0009] Figure 2 This is a schematic diagram of the structure of the full-domain data platform analysis system driven by data lineage and hotspot perception provided in the embodiments of this application.

[0010] Figure labeling: Module 1 for obtaining multi-level data lineage map, Module 2 for constructing heat time series, Module 3 for constructing heat spectrum, Module 4 for constructing dual-spectrum hotspot perception model, Module 5 for constructing node comprehensive impact score, and Module 6 for node strategy correction management execution. Detailed Implementation

[0011] This application provides a method and system for analyzing a full-domain data platform driven by data lineage and hotspot perception, which solves the technical problems of full-domain data platforms, such as difficulty in accurately depicting data dependencies, inability to fully perceive data hotspots, inaccurate assessment of the impact of data nodes, and mismatch between resource allocation and data value.

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0013] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0014] Example 1, as Figure 1 As shown, a comprehensive data platform analysis method driven by data lineage and hotspot awareness is described, wherein the method includes: The system reads the data flow logs, ETL job metadata, and query dependencies of the full-domain data platform. Based on the reading results, it performs field-level, table-level, and model feature-level dependency path parsing to form a multi-level data lineage graph. Each directed edge of the multi-level data lineage graph is marked with a causal weight parameter that reflects the strength of data transmission.

[0015] In this embodiment, the global data platform is an enterprise-level data management and service platform that integrates data from various business lines, centrally stores, manages, processes, and provides data services to external parties. The data flow log is a detailed record file that documents the time, source, destination, and operational behavior of data at each stage within the global data platform, from generation, transmission, storage to use. ETL job metadata describes the basic information of ETL (Extract, Transform, Load) tasks within the global data platform, including task name, execution cycle, data source and target data location, data transformation rules, and execution status. Query dependencies are the relationships and references between different data tables and fields involved in a query statement when a user or business system performs a data query operation within the global data platform.

[0016] Specifically, in the data acquisition phase, log collection tools such as Flume or Logstash are used to capture data flow logs from the servers and data processing components of the global data platform. These logs cover the timestamps, source, and destination information of each stage of data generation, transmission, and storage. The logs are then stored in the HDFS distributed storage system to ensure traceability. Next, the API interfaces of metadata management platforms such as Atlas and DolphinScheduler are called to read ETL job metadata, including job execution cycle, table names, field mappings, and execution status. Furthermore, SparkSQL's query log parsing tool is used to parse the FROM clauses and JOIN conditions in the SQL statements, extracting the table and field dependencies involved in the query operations, ultimately completing the acquisition of three types of core data.

[0017] Next, granular operations are performed. Table-level dependency resolution uses the Antlr SQL syntax parser to analyze the SQL scripts that query data flow logs and ETL job metadata, identify the source and target data tables, and establish direct dependencies between tables. Field-level dependency resolution uses Fastjson to parse the JSON configuration file of the ETL job metadata, extract the correspondence between source and target fields, and then combine it with the SELECT field association logic of the SQL statement to supplement indirect dependencies, forming a complete field-level path. Model feature-level dependency resolution uses a Python script to read the training configuration file of the machine learning model, extract the data source corresponding to the model features, clarify the dependencies between features and underlying tables and fields, and complete the resolution of the three-layer dependency path.

[0018] Then, Neo4j graph database was selected, with specific fields from the actual business as field-level nodes, such as the "order amount" field in the order table; data tables as table-level nodes, such as the order table; and features required by the model as model feature-level nodes, such as the "average monthly spending" feature in the user consumption model. Next, directed edges were established based on the dependencies of the actual data flow. For example, since the "order amount" field belongs to the order table, a directed edge was established from the order amount to the order table; and since the order table provides data support for the "average monthly spending" feature, a directed edge was established from the order table to the average monthly spending, thus constructing the basic lineage graph.

[0019] Next, a weighted average method is used to calculate the causal weight parameters. First, the frequency of data transmission is statistically analyzed, such as the number of times node A synchronizes data to node B during the monthly ETL process. This number is then divided by the maximum number of synchronizations between all nodes during the same period to obtain the frequency score. The data volume percentage is calculated by dividing the amount of data transmitted from node A to node B by the total amount of data transmitted to node B to obtain the percentage score. The business importance of each node is obtained from a business tagging system. For example, core business-related nodes are scored 1 point, general business nodes 0.5 points, and non-core business nodes 0.3 points. The business tagging system is a tool or system used by enterprises or organizations to label business-related data, nodes, tasks, etc., with business attributes, priorities, and uses, facilitating classification management and rapid identification of their business value. Then, the causal weight parameter for each directed edge is calculated using the formula: Frequency Score × 0.4 + Data Volume Percentage Score × 0.4 + Business Importance Score × 0.2. Finally, this weight is labeled on the corresponding directed edge, thus completing the construction of the multi-level data lineage graph.

[0020] By combining data collection, analysis, graph database construction, and weighted calculation methods, the system step-by-step sorting out of multi-granularity data dependencies and quantification of data transmission intensity in the full-domain data platform was realized, providing a precise dependency relationship foundation for subsequent data heat analysis and hotspot perception.

[0021] Change features and indicator refresh frequency features are extracted from the data flow logs to construct a popularity time series.

[0022] In this embodiment, the change characteristic is the total number of times data in the global data platform is modified, added, or deleted within a fixed time period, which is a quantitative information reflecting the activity level of data changes. The indicator refresh frequency characteristic is the number of times specific business indicators in the global data platform are updated within a unit of time, which is a quantitative representation of the frequency of indicator updates.

[0023] Optionally, preprocess the data flow log first. First, check the storage format of the data flow log. Typically, data flow logs are presented as one record per line, containing key information such as timestamp, operation type, involved data metric name, and operation result. Use a spreadsheet tool like Excel to check the data flow log line by line, directly deleting invalid records with missing timestamps or data metric names. Then, rearrange the remaining valid records according to the chronological order of the timestamps to ensure that the data flow log is presented in an orderly manner along the time dimension, avoiding time-related issues during subsequent feature extraction.

[0024] Next, change characteristics are extracted. Based on the preprocessed, ordered log data, the three types of data change operations—data update, data insertion, and data deletion—are first identified. A fixed statistical time interval of one hour is set, dividing the day into 24 one-hour periods, from 00:00-01:00 to 23:00-24:00. Then, within each period, the total number of records belonging to the above three change operations is counted row by row. The total number of records is the change characteristic result for the corresponding period, used to reflect the data change activity level within that period.

[0025] Next, the refresh frequency characteristics of the metrics are extracted. For each data metric involved in the data flow log, all log records containing the metric name are first collected separately to form a dedicated record set for each metric. Within each metric's record set, the timestamps of adjacent records are examined sequentially, and the time difference between them is calculated. For example, if the previous record is at 10:00 and the next is at 12:00, the time difference is 2 hours. The average of the time differences of all adjacent records for that metric is then calculated to obtain the average refresh interval. The reporter divides 1 by the average refresh interval to obtain the refresh frequency of the metric. For example, if the average refresh interval is 2 hours, the refresh frequency is 0.5 times per hour. This process completes the extraction of the refresh frequency characteristics for a single data metric. The above operation is then repeated for all data metrics.

[0026] Finally, a popularity time series is constructed, maintaining the same 1-hour time granularity as the feature extraction described above. Each 1-hour period is used as a time node, and the change characteristics statistically analyzed within that period—namely, the total number of change operations and the refresh frequency of all data indicators within that period—are integrated. First, the arithmetic mean of the refresh frequencies of all indicators within each time period is calculated. Then, the three pieces of information—time node, total number of change operations, and average refresh frequency—are recorded and arranged in chronological order into a structured table. This table constitutes the popularity time series, clearly displaying the data popularity-related characteristics at different time nodes.

[0027] By employing simple and easy-to-implement methods such as basic log organization, segmented statistics, and manual calculation, the originally scattered data flow logs are transformed into an ordered time series of popularity data, providing clear and usable basic data for further analysis of data popularity changes.

[0028] Perform a joint analysis of the short-term and long-term windows of the heat time series to construct a heat spectrum.

[0029] In one embodiment of this application, the heat time series is first segmented over time to establish a short-term window for capturing sudden changes and a long-term window for extracting stable trends. Within the short-term window, the transient response intensity is calculated by identifying the coupling relationship between indicator refresh frequency, access aggregation, and abnormal fluctuations. Within the long-term window, trend stability and historical inertia factors are extracted by monitoring the periodic patterns of node activity and the baseline drift trend. Subsequently, the transient response intensity, trend stability, and historical inertia factors are mapped across windows to form a two-layer time spectrum structure containing dynamic fluctuation components and equilibrium evolution components. Finally, based on this two-layer time spectrum structure, the energy distribution density and response delay characteristics of each node at different time scales are calculated, a time-frequency domain complex feature set is extracted, and the heat spectrum is ultimately generated.

[0030] The sensitivity spectrum of each data node is calculated using the causal weight parameters, and the sensitivity spectrum is fused with the heat spectrum to construct a dual-spectrum hotspot perception model.

[0031] Specifically, firstly, based on the real-time dependency characteristics (including real-time causal weight parameters) of the model input and a multi-level data lineage graph, a graph traversal method is used. Starting from each data node, the upstream dependent nodes are identified one by one along the directed edges, including the nodes that provide data to that node and the downstream dependent nodes, i.e., the nodes that depend on the data of that node, thus clarifying the dependency scope of each data node. Next, a weighted summation is performed to calculate the real-time sensitivity value of each data node. The real-time causal weight parameters of all upstream dependent nodes pointing to that node are accumulated, and the real-time causal weight parameters of that node pointing to all downstream dependent nodes are also accumulated. The arithmetic mean of the two summations is taken as the real-time sensitivity value of that node, which can intuitively reflect the degree of influence of the node in the real-time data dependency chain. Subsequently, the data nodes are grouped according to the node level at the field level, table level, and model feature level. The unique identifier of each data node is matched with its corresponding real-time sensitivity value and recorded one by one, forming a structured sensitivity spectrum, which provides a clear sensitivity feature basis for subsequent fusion.

[0032] Then, combining the real-time node behavior characteristics and real-time time scale parameters input into the model, according to the set real-time time scales, such as a short-term window of 1 hour and a long-term window of 24 hours, the total real-time behavior of each data node within the corresponding time scale is statistically analyzed. This includes the number of real-time access queries, the number of real-time data updates, and the number of real-time computation calls. Next, by dividing the total real-time behavior of each data node within a certain time scale by the total real-time behavior of all data nodes within the same time scale, the popularity ratio of that node at that time scale is obtained. Finally, the popularity ratios of each data node at different time scales are organized into structured data according to the node identifier, forming a node popularity distribution spectrum corresponding to the sensitivity spectrum level.

[0033] Subsequently, using the unique node identifier as the association key, the real-time sensitivity value of each data node in the sensitivity spectrum is correlated one-to-one with the node's heat distribution spectrum at different time scales. Next, the real-time sensitivity value and the corresponding heat distribution spectrum of each data node are combined into a sub-vector. For example, the sub-vector of node A includes the 1-hour heat distribution spectrum, the 24-hour heat distribution spectrum, and the real-time sensitivity value. The correlation between the real-time sensitivity value and the heat distribution spectrum constitutes the core content of the data node sensitivity resonance spectrum. Finally, the sub-vectors of all data nodes are arranged sequentially according to the data node hierarchy from high to low or the data node identifier from small to large, forming a multi-dimensional weight vector containing the node heat distribution spectrum and the sensitivity resonance spectrum. This multi-dimensional weight vector is the core output structure of the dual-spectrum hotspot perception model, thus completing the construction of the dual-spectrum hotspot perception model.

[0034] By employing graph structure traversal, weighted summation, statistical counting, normalization, feature association matching, and vector concatenation, the real-time data features input to the model are transformed into multi-dimensional weight vectors containing bispectral data, ultimately constructing a bispectral hotspot perception model that can support hotspot identification of data nodes.

[0035] Real-time data features are used as input data and fed into the bispectral hotspot perception model. The model output is used as the node weight input of the multi-level data lineage map to perform attenuation effect propagation calculation and establish a node comprehensive influence score.

[0036] Specifically, firstly, real-time data features include real-time node behavior features, such as the number of real-time accesses, updates, and calls; real-time dependency features, such as real-time upstream and downstream dependency links and causal weights; and real-time time scale parameters, such as 1-hour and 24-hour short and long-term windows. These real-time data features are then input into the bispectral hotspot perception model.

[0037] Next, the real-time data features are remapped in the joint frequency domain space of the heat spectrum and sensitivity spectrum to establish real-time response spectrum components. Then, energy density difference analysis and phase drift matching are performed on the real-time response spectrum components and the reference spectrum components in the model to identify anomalous resonance regions. When the energy surge magnitude in the anomalous resonance region exceeds a preset spectral threshold and the phase drift trend continues to strengthen, an anomalous burst feature weight is generated for the corresponding node. Finally, this anomalous burst feature weight is added as supplementary data to the model output.

[0038] The output of the dual-spectrum hotspot perception model is then defined as a multi-dimensional weight vector containing the node heat distribution spectrum and the sensitivity resonance spectrum. This multi-dimensional weight vector is then spectrally normalized and mapped to the feature dimensions of the multi-level data lineage graph to establish the initial node weight matrix. Subsequently, the initial node weight matrix is ​​injected into the node feature fields of the multi-level data lineage graph according to timestamps, forming a time-stamped set of node weights. Finally, this set of node weights is used as the zeroth round input, and a discrete iterative propagation mechanism is used to perform attenuation influence propagation calculations, thereby establishing a comprehensive node influence score.

[0039] Node strategy correction management is carried out based on the comprehensive impact score of the nodes.

[0040] Specifically, the computational resource consumption parameters for each node are first read. Then, based on the established comprehensive node impact score, a matching analysis is performed on the node's influence and resource input to establish a resource-efficiency ratio. Subsequently, a resource reallocation strategy is executed based on this resource-efficiency ratio. This strategy includes reducing computational quotas, extending task scheduling intervals, and migrating computational tasks to a low-priority computation pool.

[0041] Furthermore, the method provided in this application embodiment includes: The heat index time series is segmented over time to establish a short-term window for capturing sudden changes and a long-term window for extracting stable trends. Within the short-term window, the coupling relationship between indicator refresh frequency, access concentration, and abnormal fluctuations is identified, and transient response intensity is calculated. Within the long-term window, the periodic patterns and baseline drift trends of node activity are monitored, and trend stability and historical inertia factors are extracted. After cross-window correlation mapping of the transient response intensity, trend stability, and historical inertia factors, a two-layer time spectrum structure containing dynamic fluctuation components and equilibrium evolution components is formed. Based on the two-layer time spectrum structure, the energy distribution density and response delay characteristics of each node at different time scales are calculated, and a time-frequency domain verification feature set is extracted to generate a heat index spectrum.

[0042] Specifically, the time series of popularity data is first segmented according to a fixed time granularity. The time ranges of short-term and long-term windows are determined first. A 1-hour window is chosen as the short-term unit, which allows for timely capture of sudden changes in data without causing data fragmentation due to its short duration. A 24-hour window is chosen as the long-term unit, aligning with the daily cycle patterns of data in most business scenarios and facilitating the extraction of stable trends. Then, based on these two time units, the popularity time series is further divided into several consecutive short-term and long-term time periods. For example, a day's popularity time series is divided into 24 1-hour short-term windows and 1 24-hour long-term window, thus establishing the two types of windows.

[0043] Next, the coupling relationships are analyzed and transient response intensity is calculated within a short-term window. For each 1-hour short-term window, the indicator refresh frequency (total number of data indicator updates) and access clustering (total number of data queries and calls) are statistically analyzed within that time period. These two values ​​are then compared with the average indicator refresh frequency and average access clustering of the same short-term window over the past 3 days. If the difference exceeds 50% of the average, it is considered an abnormal fluctuation. The transient response intensity for each short-term window is then calculated using the formula: Transient Response Intensity = (Current Window Indicator Refresh Frequency + Current Window Access Clustering) - (Average Indicator Refresh Frequency + Average Access Clustering). A higher value indicates a more pronounced sudden change.

[0044] Then, within a long-term window, cyclical patterns and baseline drift trends are monitored and features are extracted. For a 24-hour long-term window, the node activity data (i.e., the total number of times a node is operated on) within the same long-term window for seven consecutive days is first compiled into a line chart. Cyclical patterns are identified by observing the fluctuations in the line chart. For example, if the activity level between 9 AM and 12 PM is higher than other time periods each day, then that time period is determined to be an active cycle. Next, the average node activity level for the same long-term window each week is calculated and compared with the average of the two adjacent weeks. If the difference exceeds 20% of the average of the previous week, a baseline drift trend is determined to exist. Then, trend stability is calculated by dividing the maximum fluctuation value of activity in each long-term window within a week by the average activity level of that week. The smaller the value, the more stable the trend. The historical inertia factor is calculated by using the ratio of the average activity level of the following week to the average activity level of the previous week. The closer the ratio is to 1, the stronger the historical inertia. This completes the extraction of two types of features.

[0045] The subsequent cross-window association mapping step involves first establishing a time correspondence table to associate the time points of each short-term window with the time range of its corresponding long-term window. For example, the short-term window of 9:00-10:00 on a certain day is associated with the long-term window of 0:00-24:00 on the same day. Next, the transient response intensity of the short-term window is mapped to the time correspondence table as a dynamic fluctuation component, reflecting short-term sudden changes in the data. The trend stability and historical inertia factor of the long-term window are mapped to the time correspondence table and combined as a balanced evolution component, reflecting the long-term stable trend of the data. This forms a two-layer time spectrum structure containing both dynamic fluctuation and balanced evolution components.

[0046] Based on the aforementioned two-layer time spectrum structure, the energy distribution density is calculated for each data node at three time scales: short-term (1 hour), long-term (24 hours), and ultra-long-term (7 days). At the short-term (1 hour) scale, the energy distribution density is calculated by dividing the sum of the index refresh frequency and access clustering of the data node within one hour by the sum of the index refresh frequency and access clustering of all data nodes within one hour. At the long-term (24 hours) scale, the energy distribution density is calculated by dividing the total number of operations performed on the data node within 24 hours (including data changes, index refreshes, and access queries) by the sum of the total number of operations performed on all data nodes within 24 hours. At the ultra-long-term (7 days) scale, the energy distribution density is calculated by dividing the cumulative total number of operations performed on the data node each day within 7 days by the sum of the cumulative total number of operations performed on all data nodes each day within 7 days.

[0047] Finally, the response delay feature is calculated, specifically by recording the time difference between data generation and its presentation on the global data platform, and taking the average delay value within each time scale. Then, the energy distribution density and response delay features at different time scales are compiled into a set, forming a time-frequency domain verification feature set. This feature set is then visualized using tables or line graphs to generate a heat spectrum.

[0048] By employing methods such as fixed time granularity division, basic data statistics, trend observation, data association and integration, and feature visualization, we completed the dual-window analysis and heat spectrum construction of the heat time series, providing a comprehensive and accurate basis for heat feature perception in subsequent data hotspots.

[0049] Furthermore, the method provided in this application embodiment includes: The model output is a multidimensional weight vector containing a node heat distribution spectrum and a sensitivity resonance spectrum. The multidimensional weight vector is mapped to the feature dimensions of the multi-level data lineage graph through spectral normalization to establish an initial node weight matrix. The initial node weight matrix is ​​injected into the node feature fields of the multi-level data lineage graph according to the timestamp to form a node weight set with time stamp. The node weight set is used as the zeroth round input, and decay propagation calculation is performed through a discrete iterative propagation mechanism to establish a node comprehensive influence score.

[0050] In this embodiment, the node heat distribution spectrum is the distribution of heat indicators such as data access, processing, and flow of each data node in the multi-level data lineage diagram of the global data platform under a specific time or business scenario. The sensitivity resonance spectrum is the sensitivity attribute formed by each data node in the multi-level data lineage diagram of the data platform based on data security level, business importance, etc., and the correlation response status between them during data interaction.

[0051] Specifically, the output of the dual-spectrum hotspot perception model is defined as a multi-dimensional weight vector containing a node heat distribution spectrum and a sensitivity resonance spectrum. Each element in this vector corresponds to the heat proportion and sensitivity resonance value of different data nodes at a specific time scale, forming the basic data for subsequent node weight calculation.

[0052] Subsequently, the Min-Max normalization method is used to process the multidimensional weight vector. First, the values ​​of all dimensions in the multidimensional weight vector are traversed to determine the maximum and minimum values ​​of each dimension. Then, the values ​​of each dimension in the vector are uniformly mapped to the range of 0-1 using the formula: (current value - minimum value of dimension) / (maximum value of dimension - minimum value of dimension). At the same time, the feature dimensions of the multi-level data lineage graph are defined, including the weight storage dimensions corresponding to the field-level, table-level, and model feature-level nodes. The normalized multidimensional weight vector is mapped one-to-one with the feature dimensions of the multi-level data lineage graph according to the node level, constructing an initial node weight matrix where each row corresponds to a data node and each column corresponds to a feature dimension weight.

[0053] Next, the timestamp information of the initial weight matrix of the nodes is obtained, and this timestamp is recorded in the standard format of the current system time. The normalized weight value of each row of data in the initial weight matrix of the nodes, corresponding to a single data node, is matched with the node feature fields of the multi-level data lineage graph. The matching basis is the unique identifier of the node, such as the node ID. The normalized weight value and the corresponding timestamp of each row are written into the node feature fields of the multi-level data lineage graph, such as the node weight field and the time identifier field, so that the feature fields of each data node contain both weight data and time information, and finally form a set of node weights with time identifiers.

[0054] Finally, based on the defined time-topology composite decay kernel function, the time decay and topology decay of node weights are fused to quantify the degree of weight decay during propagation. Simultaneously, a discrete iterative propagation mechanism is employed, using a set of time-labeled node weights as the zeroth-round input, iterating continuously until the change in node weights is less than a preset threshold. The final output is a comprehensive impact score for each data node, thereby improving the score's adaptability to data lineage dependencies and the time dimension.

[0055] Furthermore, the method provided in this application embodiment includes: Based on the directed edge causal weights and inter-node hierarchical distances of the aforementioned multi-level data lineage graph, a time-topological composite decay kernel function is defined as follows: ;in, Characterizing the decay kernel, For time indexing, upstream node The time difference between the feature update time and the current propagation calculation time. Representation Nodes To the node The causal weight of the directed edge. The time decay coefficient, The topological attenuation coefficient is... Representation Nodes To the node The hierarchical distance between nodes; based on the time-topology composite decay kernel function, using the node weight set as the zeroth round input, the following recursive relationship is used to calculate the... Wheel node influence vector: ;in, Characterizing the first Nodes during round iteration The propagation impact value, These are the initial weight retention coefficients. Representation Nodes At any moment The initial weight vector, Representation Nodes The set of upstream neighbors, It is a non-linear activation function. Characterizing the first Nodes during round iteration The propagation impact value; perform attenuation propagation calculation based on the node impact vector.

[0056] Specifically, firstly, based on the causal weights of directed edges and the hierarchical distances between nodes in a multi-level data lineage graph, a time-topology fusion modeling method is used to define a time-topology composite decay kernel function. Couple time decay and topology decay when used for node impact analysis: first extract the nodes. arrive Directed edge causal weight It is labeled through data dependencies. Calculate the upstream node. Time difference between feature update time and the current time Introducing a time decay coefficient And through the exponential function Characterize the time decay effect. Calculate the hierarchical distance between nodes. Introducing topological attenuation coefficient And through the exponential function Characterize the topological decay effect. Finally, multiply the three factors together to obtain the complex decay kernel. This enables the joint quantization of decay using both time and topological dimensions.

[0057] Next, a discrete iterative propagation algorithm is used, with the set of node weights as the input for the zeroth round, following the recursive relationship. Perform decay propagation calculations. Discrete iteration is a classic form of iterative optimization, specifically implemented by first determining the nodes. initial weight vector It comes from the set of node weights. Set the initial weight retention coefficient. Control the initial weight retention ratio. Traverse nodes. upstream neighbor set For each upstream node Call the non-linear activation function Processing its k-th round propagation impact value Then, with time-topological complex decay kernel Multiply and sum; combine the sum with the initial weight retention term to obtain the node of round k+1. propagation impact value The process continues iterating until the node influence values ​​converge, ultimately outputting a comprehensive influence score for each node, thus achieving dynamic transmission and precise accumulation of decaying influence in the data lineage graph.

[0058] Furthermore, the method provided in this application embodiment includes: During the discrete iterative propagation process, local energy normalization is performed on each node after each iteration, as follows: ;in, Characterizing the first Nodes during round iteration The propagation impact value, To prevent division by zero constant, Representation by nodes A local subset graph centered on the central point. Characterizing the first Nodes during round iteration The propagation impact value.

[0059] Specifically, in the discrete iterative propagation process described above, after each iteration, local energy normalization is performed on each node to avoid numerical overflow and distribution imbalance. Specifically, the propagation influence value of node v at the k-th iteration is first obtained. This value is the quantification result of the node influence calculated using the time-topological composite decay kernel function and the recursive relation. Next, a local subset graph centered on node v is defined. Collect the propagation influence values ​​of all nodes u in the subset graph in the kth round. And sum them up, while introducing a zero constant to prevent division. To prevent the denominator from being zero, finally use the formula... Calculate the normalized propagation effect value Normalization is a way to ensure the numerical stability of the iterative algorithm. By adapting it to the discrete iterative propagation scenario of data lineage, the energy normalization within the local subset graph mentioned above preserves the local correlation characteristics between nodes and avoids the unlimited amplification of the influence value of high-heat nodes. This ensures that the distribution of node propagation influence value is balanced and the value is controllable after each iteration, laying a stable data foundation for the accurate calculation of the subsequent node comprehensive influence score.

[0060] Furthermore, the method provided in this application embodiment includes: The real-time data features are remapped in the joint frequency domain of the heat spectrum and sensitivity spectrum to establish a real-time response spectrum component. Energy density difference analysis and phase drift matching are performed on the real-time response spectrum component and the reference spectrum component in the dual-spectrum hotspot perception model to identify abnormal resonance regions. When the energy surge magnitude in the abnormal resonance region exceeds a preset spectral threshold and the phase drift trend continues to strengthen, abnormal burst feature weights for the corresponding nodes are generated. The abnormal burst feature weights are added to the model output as supplementary data.

[0061] Optionally, real-time data features, such as real-time node behavior features, real-time dependency features, and real-time timescale parameters, are first transformed into a joint frequency domain space of heat spectrum and sensitivity spectrum. Specifically, the time-domain features of real-time data are converted into a frequency-domain representation using Fast Fourier Transform (FFT), and then fused with the frequency-domain parameters of heat spectrum and sensitivity spectrum to construct a real-time response spectrum component that contains the frequency-domain features of real-time data. This component can simultaneously reflect the frequency-domain performance of real-time data in the dimensions of heat and sensitivity, where heat spectrum reflects the frequency-domain features of node heat distribution, and sensitivity spectrum reflects the frequency-domain features of node sensitivity resonance.

[0062] Next, the real-time response spectral components and the baseline spectral components in the dual-spectral hotspot perception model are compared. The baseline spectral components are the frequency domain characteristic spectra determined during model training under normal conditions. Energy density difference analysis calculates the difference in energy density between the two spectral components at each frequency point, where energy density is the square of the amplitude at that frequency point. The energy density differences at all frequency points are integrated to obtain the energy density difference distribution. Phase drift matching calculates the phase difference between corresponding frequency points of the two spectral components and tracks the phase change trend. When the energy density difference in a certain region is significant and the phase drift shows a continuous deviation, it can be identified as an abnormal resonance region. This region represents an abnormal frequency domain correlation between the real-time data and the baseline state.

[0063] Subsequently, the energy surge magnitude and phase drift trend in the anomalous resonance region were quantitatively evaluated. A preset spectral threshold was set at 30% of the energy density difference exceeding the normal fluctuation range. When the energy surge magnitude in the anomalous resonance region exceeded this preset spectral threshold, and the phase drift trend continued to intensify, such as a sustained increase in phase difference over multiple consecutive time windows, an anomalous burst feature weight was generated for the corresponding node. This anomalous burst feature weight is a quantitative identifier of the node's anomalous frequency domain performance, including information such as the degree of energy surge and phase drift intensity.

[0064] Finally, the generated abnormal burst feature weights are added to the output of the bispectral hotspot perception model as additional fields, so that the model output not only includes the node's heat distribution spectrum and sensitivity resonance spectrum information, but also reflects the node's real-time abnormal frequency domain characteristics, thus providing a more comprehensive basis for subsequent node strategy correction, abnormal early warning, etc.

[0065] By employing techniques such as frequency domain remapping, energy density analysis, phase matching, threshold judgment, and feature addition, the identification and integration of abnormal frequency domain features in real-time data were achieved, thereby enhancing the bispectral hotspot perception model's ability to perceive abnormal scenes.

[0066] Furthermore, the method provided in this application embodiment includes: Read the computing resource consumption parameters corresponding to the node, perform a matching analysis of node influence and resource input based on the node's comprehensive influence score and the computing resource consumption parameters, and establish a resource benefit ratio; execute a resource reallocation strategy according to the resource benefit ratio, the resource reallocation strategy includes reducing computing quota, extending task scheduling interval, or migrating computing from low-priority computing pools.

[0067] In one embodiment, the built-in system resource monitoring tool of the global data platform is first activated. This tool is a resource data acquisition component that reads the system operation logs and resource allocation records of data nodes in real time to obtain the computing resource consumption parameters corresponding to each data node. Specifically, these parameters include CPU utilization, memory usage, storage read / write speed, and task execution time per unit time. These parameters collectively constitute the quantitative basis for the current resource investment scale of the node. Next, a linear weighted comparison is used to correlate the node's comprehensive impact score with the corresponding computing resource consumption parameters. The node's comprehensive impact score reflects the quantitative result of the node's business value and data influence. First, the arithmetic mean of all computing resource consumption parameters of the data node is calculated. Then, the node's comprehensive impact score is divided by this arithmetic mean, and the resulting value is the resource benefit ratio. The resource benefit ratio directly reflects the magnitude of the influence generated by the node's unit resource investment.

[0068] Subsequently, a resource efficiency ratio threshold is preset in the resource scheduling system of the global data platform. This threshold is determined based on historical resource usage efficiency data and business demand priority, and the resource efficiency ratio of each node is compared with this threshold. When the resource efficiency ratio of a node is lower than the threshold, a resource reallocation strategy is executed according to the node's business attributes. For data nodes with non-core business and low real-time requirements, the computing quota is reduced by adjusting resource configuration parameters. This is done by modifying the CPU and memory allocation limits of the node in the resource scheduling system to reduce the computing resources allocated to it per unit time. For node tasks with high execution frequency but not real-time requirements, the task scheduling interval is extended by modifying scheduling cycle parameters. This is done by adjusting the execution cycle value in the task scheduling configuration file to increase the time interval between two task starts. For data nodes with high resource consumption and consistently low efficiency ratios, the computing tasks are migrated to a low-priority computing pool using a task process migration method. The task process of the data node in the original computing pool is stopped by the resource scheduling tool, and the same task process is initialized and started in the low-priority computing pool to complete the migration and deployment of the computing tasks.

[0069] Furthermore, the method provided in this application embodiment includes: After the calibration management is completed, the calibration evaluation and the corresponding node calibration strategy are read to generate a feedback mapping set; the model parameters of the bispectral hotspot perception model are updated in reverse according to the node state changes and resource allocation results in the feedback mapping set, and adaptive update optimization is performed.

[0070] Optionally, after the calibration management is completed, two core pieces of information are read using the log collection tool of the full-domain data platform. One part is calibration evaluation data, specifically including the changes in the overall impact score of each node after calibration, fluctuations in computing resource consumption, and business response efficiency indicators. This data is used to quantify the actual effect of calibration management. The other part is the corresponding node calibration strategy, i.e., the specific operation records performed on each data node in the aforementioned steps, such as reducing computing quotas, extending task scheduling intervals, or migrating computing. Subsequently, the calibration evaluation data of the same data node and the corresponding node calibration strategy are associated and matched using the node's unique identifier to ensure that each node calibration strategy corresponds to the node state change result it produces. Finally, this is integrated to form a structured feedback mapping set containing node identifiers, node calibration strategies, and post-calibration state data, providing data support for subsequent model updates.

[0071] Next, a gradient descent optimization method is employed to update the model parameters of the bispectral hotspot perception model in reverse based on the data in the feedback map set. First, key supervision signals are extracted from the feedback map set, including the node comprehensive impact score deviation value in the node state change data (i.e., the difference between the corrected actual score and the model's original predicted score) and the deviation between the actual resource efficiency ratio and the model's expected value in the resource allocation results. These deviation values ​​are used as optimization targets for updating the model parameters. Then, based on the iterative calculation logic of the gradient descent algorithm, the gradient direction of the deviation values ​​is calculated for the weight parameters related to heat spectrum generation, the correlation coefficients involved in sensitivity resonance spectrum calculation, and the threshold parameters of the baseline spectral components. The values ​​of these parameters are adjusted according to a preset learning rate to make the model's prediction results closer to the corrected actual node states. After each parameter adjustment, the model's prediction accuracy is verified using partial sample data from the feedback map set. If the accuracy does not meet the preset standard, iteration continues until the model's prediction deviation is less than the set threshold, thus completing the adaptive update and optimization of the bispectral hotspot perception model.

[0072] Furthermore, the method provided in this application embodiment includes: Anomaly scores are accumulated for each node in the global data platform. The accumulated anomaly scores are then used to perform a joint anomaly analysis on the accumulated total value and transient change value, and an anomaly warning is issued.

[0073] In one embodiment, the single anomaly score of each data node in the global data platform is first cumulatively calculated. The single anomaly score comes from the abnormal resonance zone identified by the bispectral hotspot perception model and the generated abnormal burst feature weights, etc. The single anomaly scores of each data node within the preset time window are added one by one according to the preset time window, such as hourly or daily, to obtain the cumulative anomaly score of each data node within the corresponding time window. This result intuitively reflects the degree of anomaly accumulation of the node within a specific time period.

[0074] Next, two types of key indicators are extracted from the cumulative results of anomaly scoring. One type is the cumulative total value, which is the sum of the cumulative results of anomaly scoring for each node within a complete statistical period, such as a business cycle, and is used to measure the long-term anomaly accumulation level of the node. The other type is the transient change value, which is calculated by subtracting the cumulative result of the previous adjacent time window from the cumulative result of the current time window to obtain the transient change value of each node, and is used to capture the short-term sudden change trend of the node's anomaly degree.

[0075] Subsequently, a joint anomaly analysis is performed, with cumulative total thresholds and transient change thresholds pre-set based on historical anomaly data and business risk levels. For each data node, its cumulative total value is compared with the cumulative total threshold, and its transient change value is compared with the transient change threshold. If the cumulative total value exceeds the cumulative total threshold, or the transient change value exceeds the transient change threshold, or both exceed the corresponding thresholds, then the data node is determined to have an anomaly risk.

[0076] Finally, the early warning triggering mechanism of the full-domain data platform is activated. When an abnormal risk is detected in a data node, an early warning message is automatically generated, which includes the node identifier, the type of abnormality (cumulative or transient), and the specific value of the abnormal indicator. The warning is then reported through system message push, email notification, and other means to promptly remind operation and maintenance personnel to investigate and handle the abnormality.

[0077] In summary, the data lineage and hotspot perception-driven full-domain data platform analysis method provided in this application has the following technical effects: This application integrates real-time data features into a multi-level data lineage graph of a global data platform and inputs them into a dual-spectrum hotspot perception model. Through spectral normalization mapping, discrete iterative propagation attenuation calculation, and energy density difference analysis, it obtains the node heat distribution spectrum, sensitivity resonance spectrum, and abnormal burst characteristic weights. It calculates the node's comprehensive impact score and resource benefit ratio, and combines node resource consumption parameters, correction evaluation, and cumulative abnormal score results to perform strategy correction and joint abnormality analysis. This enables node resource reallocation and abnormality early warning, making the resource scheduling accuracy and abnormality perception reliability of the global data platform more precise and reliable. It achieves clear data dependencies, comprehensive hotspot perception, accurate node impact assessment, and reasonable resource allocation, thus supporting the platform's accurate assessment and efficient management.

[0078] Example 2, as Figure 2 As shown, based on the same inventive concept as in Embodiment 1 above, this application provides a full-domain data platform analysis system driven by data lineage and hotspot awareness, the system comprising: The multi-level data lineage graph acquisition module 1 is used to read the data flow logs, ETL job metadata and query dependencies of the full-domain data platform. Based on the reading results, it performs field-level, table-level and model feature-level dependency path parsing to form a multi-level data lineage graph. Each directed edge of the multi-level data lineage graph is marked with a causal weight parameter that reflects the data transmission strength.

[0079] The heat time series construction module 2 is used to extract change features and indicator refresh frequency features from the data flow log to construct a heat time series.

[0080] The heat spectrum construction module 3 is used to perform joint analysis of the short-term and long-term windows of the heat time series to construct the heat spectrum.

[0081] The dual-spectrum hotspot perception model construction module 4 is used to calculate the sensitivity spectrum of each data node using the causal weight parameters, and fuse the sensitivity spectrum with the heat spectrum to construct a dual-spectrum hotspot perception model.

[0082] The node comprehensive impact score construction module 5 is used to input real-time data features into the bispectral hotspot perception model, use the model output as the node weight input of the multi-level data lineage map, perform attenuation impact propagation calculation, and establish a node comprehensive impact score.

[0083] The node strategy correction management execution module 6 is used to perform node strategy correction management based on the node comprehensive impact score.

[0084] Furthermore, the node comprehensive impact score construction module 5 is used to perform the following steps: The model output is a multidimensional weight vector containing a node heat distribution spectrum and a sensitivity resonance spectrum. The multidimensional weight vector is mapped to the feature dimensions of the multi-level data lineage graph through spectral normalization to establish an initial node weight matrix. The initial node weight matrix is ​​injected into the node feature fields of the multi-level data lineage graph according to the timestamp to form a node weight set with time stamp. The node weight set is used as the zeroth round input, and decay propagation calculation is performed through a discrete iterative propagation mechanism to establish a node comprehensive influence score.

[0085] Furthermore, the node comprehensive impact score construction module 5 is used to perform the following steps: Based on the directed edge causal weights and inter-node hierarchical distances of the aforementioned multi-level data lineage graph, a time-topological composite decay kernel function is defined as follows: ;in, Characterizing the decay kernel, For time indexing, upstream node The time difference between the feature update time and the current propagation calculation time. Representation Nodes To the node The causal weight of the directed edge. The time decay coefficient, The topological attenuation coefficient is... Representation Nodes To the node The hierarchical distance between nodes; based on the time-topology composite decay kernel function, using the node weight set as the zeroth round input, the following recursive relationship is used to calculate the... Wheel node influence vector: ;in, Characterizing the first Nodes during round iteration The propagation impact value, These are the initial weight retention coefficients. Representation Nodes At any moment The initial weight vector, Representation Nodes The set of upstream neighbors, It is a non-linear activation function. Characterizing the first Nodes during round iteration The propagation impact value; perform attenuation propagation calculation based on the node impact vector.

[0086] Furthermore, the node comprehensive impact score construction module 5 is used to perform the following steps: During the discrete iterative propagation process, local energy normalization is performed on each node after each iteration, as follows: ;in, Characterizing the first Nodes during round iteration The propagation impact value, To prevent division by zero constant, Representation by nodes A local subset graph centered on the central point. Characterizing the first Nodes during round iteration The propagation impact value.

[0087] Furthermore, the heat spectrum construction module 3 is used to perform the following steps: The heat index time series is segmented over time to establish a short-term window for capturing sudden changes and a long-term window for extracting stable trends. Within the short-term window, the coupling relationship between indicator refresh frequency, access concentration, and abnormal fluctuations is identified, and transient response intensity is calculated. Within the long-term window, the periodic patterns and baseline drift trends of node activity are monitored, and trend stability and historical inertia factors are extracted. After cross-window correlation mapping of the transient response intensity, trend stability, and historical inertia factors, a two-layer time spectrum structure containing dynamic fluctuation components and equilibrium evolution components is formed. Based on the two-layer time spectrum structure, the energy distribution density and response delay characteristics of each node at different time scales are calculated, and a time-frequency domain verification feature set is extracted to generate a heat index spectrum.

[0088] Furthermore, the node policy correction management execution module 6 is used to perform the following steps: Read the computing resource consumption parameters corresponding to the node, perform a matching analysis of node influence and resource input based on the node's comprehensive influence score and the computing resource consumption parameters, and establish a resource benefit ratio; execute a resource reallocation strategy according to the resource benefit ratio, the resource reallocation strategy includes reducing computing quota, extending task scheduling interval, or migrating computing from low-priority computing pools.

[0089] Furthermore, the node comprehensive impact score construction module 5 is used to perform the following steps: The real-time data features are remapped in the joint frequency domain of the heat spectrum and sensitivity spectrum to establish a real-time response spectrum component. Energy density difference analysis and phase drift matching are performed on the real-time response spectrum component and the reference spectrum component in the dual-spectrum hotspot perception model to identify abnormal resonance regions. When the energy surge magnitude in the abnormal resonance region exceeds a preset spectral threshold and the phase drift trend continues to strengthen, abnormal burst feature weights for the corresponding nodes are generated. The abnormal burst feature weights are added to the model output as supplementary data.

[0090] Furthermore, the node policy correction management execution module 6 is used to perform the following steps: After the calibration management is completed, the calibration evaluation and the corresponding node calibration strategy are read to generate a feedback mapping set; the model parameters of the bispectral hotspot perception model are updated in reverse according to the node state changes and resource allocation results in the feedback mapping set, and adaptive update optimization is performed.

[0091] Furthermore, the node policy correction management execution module 6 is used to perform the following steps: Anomaly scores are accumulated for each node in the global data platform. The accumulated anomaly scores are then used to perform a joint anomaly analysis on the accumulated total value and transient change value, and an anomaly warning is issued.

[0092] The data lineage and hotspot perception-driven full-domain data platform analysis system provided in this embodiment of the invention can execute the data lineage and hotspot perception-driven full-domain data platform analysis method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0093] Although this application makes various references to certain modules in the system according to the embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of this invention.

[0094] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application. In some cases, the actions or steps described in this application can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A comprehensive data platform analysis method driven by data lineage and hotspot perception, characterized in that: The method includes: Read the data flow logs, ETL job metadata, and query dependencies of the full-domain data platform. Based on the reading results, perform dependency path parsing at the field level, table level, and model feature level to form a multi-level data lineage graph. Each directed edge of the multi-level data lineage graph is marked with a causal weight parameter that reflects the strength of data transmission. Change features and indicator refresh frequency features are extracted from the data flow logs to construct a popularity time series; Perform a joint analysis of the short-term and long-term windows of the heat time series to construct a heat spectrum; The sensitivity spectrum of each data node is calculated using the causal weight parameters, and the sensitivity spectrum is fused with the heat spectrum to construct a dual-spectrum hotspot perception model. Real-time data features are used as input data and fed into the bispectral hotspot perception model. The model output is used as the node weight input of the multi-level data lineage map to perform attenuation effect propagation calculation and establish a node comprehensive effect score. Node strategy correction management is carried out based on the comprehensive impact score of the nodes.

2. The data lineage and hotspot perception-driven full-domain data platform analysis method as described in claim 1, characterized in that, The model output is used as the node weight input to a multi-level data lineage graph to perform attenuation effect propagation calculations and establish a comprehensive node impact score, including: The model output is a multi-dimensional weight vector containing the node heat distribution spectrum and the sensitivity resonance spectrum; The multidimensional weight vector is mapped to the feature dimension of the multi-level data lineage graph through spectral normalization to establish the node initial weight matrix; The initial weight matrix of the nodes is injected into the node feature field of the multi-level data lineage graph according to the timestamp, forming a set of node weights with time stamps; Using the set of node weights as the input for the zeroth round, a decay propagation calculation is performed through a discrete iterative propagation mechanism to establish a comprehensive node impact score.

3. The data lineage and hotspot perception-driven full-domain data platform analysis method as described in claim 2, characterized in that, Using the set of node weights as the input for the zeroth round, decay propagation calculation is performed through a discrete iterative propagation mechanism, including: Based on the directed edge causal weights and inter-node hierarchical distances of the aforementioned multi-level data lineage graph, a time-topological composite decay kernel function is defined as follows: ； in, Characterizing the decay kernel, For time indexing, upstream node The time difference between the feature update time and the current propagation calculation time. Representation Nodes To the node The causal weight of the directed edge. The time decay coefficient, The topological attenuation coefficient is... Representation Nodes To the node The hierarchical distance between nodes; Based on the time-topology composite decay kernel function, and using the set of node weights as the input for the zeroth round, the following recursive relationship is followed to calculate the... Wheel node influence vector: ； in, Characterizing the first Nodes during round iteration The propagation impact value, These are the initial weight retention coefficients. Representation Nodes At any moment The initial weight vector, Representation Nodes The set of upstream neighbors, It is a non-linear activation function. Characterizing the first Nodes during round iteration The propagation impact value; Attenuation propagation calculations are performed based on the node influence vector.

4. The data lineage and hotspot perception-driven full-domain data platform analysis method as described in claim 3, characterized in that, During the discrete iterative propagation process, local energy normalization is performed on each node after each iteration, as follows: ； in, Characterizing the first Nodes during round iteration The propagation impact value, To prevent division by zero constant, Representation by nodes A local subset graph centered on the central point. Characterizing the first Nodes during round iteration The propagation impact value.

5. The data lineage and hotspot perception-driven full-domain data platform analysis method as described in claim 1, characterized in that, Perform joint analysis of the short-term and long-term windows of the heat time series to construct a heat spectrum, including: The heat time series is segmented over time to establish a short-term window for capturing sudden changes and a long-term window for extracting stable trends. Within a short window, identify the coupling relationship between indicator refresh frequency, access concentration, and abnormal fluctuations, and calculate the transient response intensity; Within a long-term window, monitor the cyclical patterns and baseline drift trends of node activity, and extract trend stability and historical inertia factors. After performing cross-window correlation mapping on the transient response intensity, trend stability, and historical inertia factor, a two-layer time spectrum structure containing dynamic fluctuation components and equilibrium evolution components is formed. Based on the aforementioned dual-layer time spectrum structure, the energy distribution density and response delay characteristics of each node at different time scales are calculated, and the time-frequency domain complex feature set is extracted to generate a heat spectrum.

6. The data lineage and hotspot perception-driven full-domain data platform analysis method as described in claim 1, characterized in that, Based on the comprehensive impact score of the nodes, node strategy correction management is carried out, including: Read the computing resource consumption parameters corresponding to the node, and perform a matching analysis of node influence and resource input based on the node's comprehensive influence score and the computing resource consumption parameters to establish a resource benefit ratio. Based on the resource efficiency ratio, a resource reallocation strategy is executed, which includes reducing computing quotas, extending task scheduling intervals, or migrating computing from low-priority computing pools.

7. The data lineage and hotspot perception-driven full-domain data platform analysis method as described in claim 1, characterized in that, Using real-time data features as input data, the bispectral hotspot perception model is further included, along with: The real-time data features are remapped in the joint frequency domain space of the heat spectrum and sensitivity spectrum to establish real-time response spectrum components. Energy density difference analysis and phase drift matching are performed on the real-time response spectral components and the reference spectral components in the dual-spectral hotspot sensing model to identify abnormal resonance regions; When the energy surge magnitude in the abnormal resonance region exceeds a preset spectral threshold and the phase drift trend continues to strengthen, an abnormal burst feature weight of the corresponding node is generated. The abnormal burst feature weights are added as supplementary data to the model output.

8. The data lineage and hotspot perception-driven full-domain data platform analysis method as described in claim 1, characterized in that, Based on the comprehensive impact score of the nodes, the node strategy correction management also includes: After the calibration management is completed, read the calibration evaluation and the corresponding node calibration strategy, and generate a feedback mapping set; The model parameters of the bispectral hotspot perception model are updated in reverse based on the node state changes and resource allocation results in the feedback mapping set, and adaptive update optimization is performed.

9. The data lineage and hotspot perception-driven full-domain data platform analysis method as described in claim 1, characterized in that, Anomaly scores are accumulated for each node in the global data platform. The accumulated anomaly scores are then used to perform a joint anomaly analysis on the accumulated total value and transient change value, and an anomaly warning is issued.

10. A full-domain data platform analysis system driven by data lineage and hotspot perception, characterized in that: The system is used to implement the data lineage and hotspot perception-driven full-domain data platform analysis method according to any one of claims 1-9, the system comprising: The multi-level data lineage graph acquisition module is used to read the data flow logs, ETL job metadata and query dependencies of the full-domain data platform. Based on the reading results, it performs field-level, table-level and model feature-level dependency path parsing to form a multi-level data lineage graph. Each directed edge of the multi-level data lineage graph is marked with a causal weight parameter that reflects the strength of data transmission. The heat time series construction module is used to extract change features and indicator refresh frequency features from the data flow log to construct a heat time series. The heat spectrum construction module is used to perform joint analysis of the short-term and long-term windows of the heat time series to construct the heat spectrum; The dual-spectrum hotspot perception model construction module is used to calculate the sensitivity spectrum of each data node using the causal weight parameters, and to fuse the sensitivity spectrum with the heat spectrum to construct the dual-spectrum hotspot perception model. The node comprehensive impact score construction module is used to take real-time data features as input data, input them into the bispectral hotspot perception model, use the model output as the node weight input of the multi-level data lineage map, perform attenuation impact propagation calculation, and establish the node comprehensive impact score. The node strategy correction management execution module is used to perform node strategy correction management based on the node's comprehensive impact score.

Images