A data asset state monitoring method and system

Abstract

The application relates to the technical field of data asset management and intelligent operation and maintenance, and discloses a data asset state monitoring method and system, wherein an edge node arranged on a data source side collects data asset change events, and the data asset change events are de-duplicated and time-sequentially rearranged according to time windows to form an ordered event stream; time-sequential monitoring indexes and asset labels are generated based on the event stream, and a multi-dimensional feature vector is constructed by fusion; the feature vector is input into a reconstructed anomaly detection model, a warning event is generated in combination with a dynamically updated judgment threshold; and corresponding graph elements are located in an asset topology view according to asset identifiers in the warning event, and efficient warning redrawing of target graph elements is realized through a GPU buffer local updating mechanism. The method realizes real-time and accurate perception and visual response to abnormal states of data assets.

Description

Technical Field

[0001] This invention relates to the field of data asset management and intelligent operation and maintenance technology, and more specifically, to a data asset status monitoring method and system. Background Technology

[0002] With the booming development of the digital economy, data has become a core strategic asset for enterprises. To effectively manage and utilize these massive, multi-source, and heterogeneous data assets, various data asset management platforms have emerged. Existing data asset management solutions mainly take the following forms:

[0003] The first category is traditional database monitoring tools, which typically collect database performance metrics (such as QPS and response time) through periodic polling (e.g., SQL queries) and display them using static dashboards. These tools have a simple architecture, but data collection and processing heavily rely on a central node, and the use of scheduled task mechanisms results in data status monitoring latency reaching the minute level, failing to meet the need for real-time, dynamic perception of data assets.

[0004] The second category is open-source data visualization platforms (such as Grafana). While these platforms can integrate multiple time-series data sources and provide rich visualization capabilities, their core functionality focuses on displaying infrastructure metrics, lacking an understanding and management of the business semantics of the data assets themselves. They typically treat data as an undifferentiated stream of numerical values, unable to perform differentiated and intelligent monitoring and early warning based on the business attributes of the data (such as whether it is core transaction data or contains personal privacy information).

[0005] The third category is enterprise-level data asset management platforms (such as Collibra), which provide relatively comprehensive metadata management, data cataloging, and data quality monitoring functions. However, the tagging system of these platforms is often static and predefined, making it difficult to adapt to rapid business changes; their data processing workflows are mostly batch or micro-batch processing modes, which lack real-time performance; at the same time, when facing the visualization of large-scale data asset topologies, rendering performance bottlenecks often lead to interactive lag and a poor user experience.

[0006] In summary, existing technologies generally suffer from core defects such as insufficient real-time performance, rigid tag management, and low performance in large-scale visualization rendering, making it difficult to build a modern data asset status monitoring system that integrates real-time perception, intelligent understanding, accurate early warning, and smooth interaction.

[0007] In view of this, a data asset status monitoring method and system are proposed. Summary of the Invention

[0008] This application proposes a data asset status monitoring method and system to solve one of the aforementioned technical problems.

[0009] The technical solution adopted in this application is as follows:

[0010] In a first aspect, embodiments of this application provide a method for monitoring the status of data assets, including:

[0011] Data asset change events are acquired by edge nodes deployed on the data source side, and the data asset change events are deduplicated and rearranged in time sequence according to time windows to obtain an event stream;

[0012] Based on the event stream, time-series monitoring indicators and asset tags for data assets are generated, and the asset tags are fused with the time-series monitoring indicators as an additional dimension to form a multi-dimensional feature vector.

[0013] The multidimensional feature vector is periodically extracted and input into a pre-trained reconstruction anomaly detection model to calculate the current reconstruction error; historical reconstruction error values ​​within a historical sliding window are obtained, and a clean reconstruction error dataset is generated based on the historical reconstruction error values; the judgment threshold is dynamically updated based on the clean reconstruction error dataset; if the current reconstruction error is greater than the judgment threshold, an early warning event carrying an asset identifier and an alarm level is generated.

[0014] The target element in the asset topology view is located based on the asset identifier carried by the warning event, and the display status of the target element is locally updated.

[0015] Preferably, data asset change events are acquired through edge nodes deployed on the data source side, and the data asset change events are deduplicated and rearranged according to time windows to obtain an event stream, specifically as follows:

[0016] By monitoring and acquiring data asset change events carrying event generation timestamps through edge nodes deployed on the data source side, the data asset change events include data record change events and / or metadata change events;

[0017] The current active time window is maintained based on the event generation timestamp and the preset window advancement rules;

[0018] When the data asset change event includes a data record change event, the data source identifier, data table identifier, and primary key identifier are used to form a deduplication key. Multiple data record change events with the same deduplication key within the current active time window are deduplicated in an overwrite manner, and the state of the last data record within the current active time window is retained.

[0019] When the data asset change event includes metadata change event, the asset identifier or field identifier is used to form a deduplication key, and multiple metadata change events with the same deduplication key within the current active time window are deduplicated and merged.

[0020] When the current active time window meets the window closure condition, the data asset change events within the window after deduplication and merging are rearranged in time according to the event generation timestamp and / or transaction commit order to obtain a cleaned event stream, and the cleaned event stream is pushed to the central message queue.

[0021] If a late event is received, and the event generation timestamp of the late event is earlier than the start timestamp of the current active time window, then the late event is determined to belong to a closed window, the late event is encapsulated as a compensation message and pushed to the late compensation message queue.

[0022] Preferably, during the deduplication and time-series rearrangement process at the edge node, the memory usage of the deduplication hash table used for event deduplication is monitored in real time, and a graded degradation strategy is executed based on a first memory threshold, a second memory threshold, and a third memory threshold, wherein the first memory threshold is less than the second memory threshold, and the second memory threshold is less than the third memory threshold.

[0023] If the memory occupancy rate is greater than or equal to the first memory threshold and less than the second memory threshold, then the window duration of the current active time window is shortened, and deduplication is performed only for data asset change events with preset priority.

[0024] If the memory occupancy rate is greater than or equal to the second memory threshold and less than the third memory threshold, then the writing of the deduplication hash table is turned off, and the arriving data asset change event is assigned a pass-through identifier and output to the central message queue.

[0025] If the memory occupancy rate is greater than or equal to the third memory threshold, a backpressure mechanism is triggered to suspend the consumption of events from the corresponding data source until the memory occupancy rate falls below the preset recovery threshold. Then, the interrupted transmission is resumed based on the saved log offset.

[0026] Preferably, time-series monitoring indicators and asset tags for data assets are generated based on the event stream, and the asset tags are fused with the time-series monitoring indicators as an additional dimension to form a multi-dimensional feature vector, including:

[0027] Consume the cleaned event stream from the central message queue and execute the metric calculation branch and the tag generation branch in parallel;

[0028] In the indicator calculation branch, the time-series monitoring indicators of the data assets are updated based on the cleaned event stream. The time-series monitoring indicators include at least one of data integrity, data timeliness, data availability, and data volume volatility.

[0029] In the tag generation branch, the metadata context corresponding to the cleaned event stream is extracted, candidate tags are generated based on the natural language processing model, and the candidate tags are corrected or conflict arbitration is performed based on the preset rule engine to determine the asset tag.

[0030] The asset tags are injected as additional feature dimensions into the time-series monitoring indicators to form a multi-dimensional feature vector for anomaly detection.

[0031] Preferably, the candidate tags are corrected or conflict arbitration is performed based on a preset rule engine to determine the asset tags, including:

[0032] When the natural language processing model outputs multiple candidate labels for the same data field that match the preset mutual exclusion pair configuration table, and the confidence scores of the multiple candidate labels are all higher than the preset confidence threshold, the branch arbitration logic is triggered.

[0033] If the mutual exclusion label is determined to belong to the security level category of mutual exclusion, then the preset security level weights corresponding to the mutual exclusion labels are compared, and the candidate labels with higher weights are retained;

[0034] If it is determined that the mutual exclusion label belongs to the objective business status type mutual exclusion, then query the objective metadata attributes of the corresponding data table and retain the candidate labels that are consistent with the objective metadata attributes;

[0035] If the mutual exclusion tag is determined to be a time-sensitive description type mutual exclusion, then the update frequency of the corresponding data table is queried. When the update frequency is greater than the set frequency threshold, the real-time candidate tag is retained. When the update frequency is less than or equal to the set frequency threshold, the offline candidate tag is retained.

[0036] When multiple candidate labels conflict with each other but do not meet the judgment conditions of the branch arbitration logic, or when the objective data required to execute the branch arbitration logic is missing, the activation process of the multiple candidate labels is suspended, a temporary label with the highest security level is assigned to the current data field, and a labeling task is generated based on the current data field. The labeling task is then pushed to the manual review queue.

[0037] Preferably, generating a clean reconstruction error dataset based on the historical reconstruction error values ​​includes:

[0038] Obtain multiple historical reconstruction error values ​​within the historical sliding window, and calculate the initial median of the multiple historical reconstruction error values;

[0039] Calculate the absolute difference between each historical reconstruction error value and the initial median to form a deviation set, and calculate the median of the deviation set to obtain the absolute median difference;

[0040] If the absolute median difference is zero, then all historical reconstruction error values ​​within the historical sliding window are used as the clean reconstruction error dataset.

[0041] If the absolute median difference is not zero, a correction score is calculated based on each historical reconstruction error value, the initial median, and the absolute median difference. Historical reconstruction error values ​​whose absolute values ​​of the correction scores are greater than the extreme deviation threshold are removed, and the remaining historical reconstruction error values ​​are used as the clean reconstruction error dataset.

[0042] Preferably, dynamically updating the judgment threshold based on the clean reconstruction error dataset includes:

[0043] Obtain the ratio of the data volume of the clean reconstruction error dataset to the total number of historical reconstruction error values ​​within the historical sliding window;

[0044] If the proportion of the data volume is lower than the set proportion threshold, the threshold update for the current period is abandoned, and the judgment threshold of the previous period is used.

[0045] If the proportion of the data volume is greater than or equal to the set proportion threshold, then the preset quantile of the clean reconstruction error dataset is extracted, and the preset quantile is exponentially smoothed and fused with the judgment threshold of the previous period to obtain the judgment threshold of the current period.

[0046] Preferably, locating the target element in the asset topology view based on the asset identifier carried by the warning event, and partially updating the display state of the target element, includes:

[0047] In response to the view request from the user terminal, extract the asset topology data at the corresponding aggregation granularity for initial rendering;

[0048] When a drill-down request is received from the first display level to a more granular second display level, the target business domain identifier in the drill-down request is extracted;

[0049] Based on the target business domain identifier, query the data source type nodes under the corresponding business domain, the summary indicators of each data source type node, and the connection topology between each data source type node.

[0050] Stop displaying, release, or cache the aggregated view resources of the first display layer, and construct and render the network topology view of the second display layer using the data source type nodes as graph nodes and the connection topology as graph connections;

[0051] During the system initialization phase, a mapping dictionary is established, which records the starting offset position of the primitive color data corresponding to each asset identifier in the graphics processor buffer.

[0052] Upon receiving the warning event containing the target asset identifier and the target color value, the target starting offset position corresponding to the target asset identifier is retrieved from the mapping dictionary;

[0053] The segmented update instruction of the graphics programming interface is invoked to locate the target buffer interval according to the target starting offset position and the preset data length, and only the primitive color data in the target buffer interval is overwritten with the target color value, so that the graphics processor can perform local warning redraw of the target primitive without rebuilding the overall rendering pipeline.

[0054] Secondly, embodiments of this application provide a data asset status monitoring system to implement the method described in any of the above-mentioned embodiments, comprising:

[0055] The event preprocessing module is used to acquire data asset change events through edge nodes deployed on the data source side, and to deduplicate and rearrange the data asset change events according to time windows to obtain an event stream;

[0056] The feature construction module is used to generate time-series monitoring indicators and asset tags for data assets based on the event stream, and to fuse the asset tags as an additional dimension with the time-series monitoring indicators to form a multi-dimensional feature vector.

[0057] The early warning generation module is used to periodically extract the multidimensional feature vector and input it into the pre-trained reconstruction anomaly detection model to calculate the current reconstruction error; obtain the historical reconstruction error value within the historical sliding window, and generate a clean reconstruction error dataset based on the historical reconstruction error value; dynamically update the judgment threshold based on the clean reconstruction error dataset; if the current reconstruction error is greater than the judgment threshold, generate an early warning event carrying an asset identifier and an alarm level.

[0058] The local rendering module is used to locate the target graphic element in the asset topology view based on the asset identifier carried by the warning event, and locally update the display status of the target graphic element.

[0059] This application discloses a data asset status monitoring method and system. Through the synergistic effect of key technologies such as edge nodes deployed on the data source side, multi-dimensional feature vectors fused with asset tags, a reconstructed anomaly detection model, and locally updated topology view rendering, this method achieves the following significant technical effects:

[0060] By acquiring data asset change events at edge nodes on the data source side and performing deduplication and time-series rearrangement by time window, this solution cleans and compresses high-frequency, redundant original change events at the source. This not only significantly reduces the network traffic that needs to be transmitted to the central node, but more importantly, it frees the central node from the heavy burden of raw data cleaning, allowing it to focus on higher-value computing tasks. Thus, end-to-end low-latency monitoring is achieved while ensuring information integrity.

[0061] This solution creatively integrates asset tags generated based on event streams as an additional dimension with traditional time-series monitoring metrics (such as completeness and timeliness) to form a multi-dimensional feature vector. This design enables subsequent anomaly detection models not only to "see" changes in data metrics but also to "understand" the business implications behind the data. For example, if a data table marked as a "core transaction table" experiences a decrease in update frequency, its anomaly weight will be much higher than that of a regular log table, thus making alerts more accurate and more valuable to business.

[0062] Once the anomaly detection model generates an early warning event based on a dynamic threshold, this solution can accurately locate the target element in the asset topology view based on the asset identifier carried by the early warning event, and only locally update the display state of that target element. This incremental and localized rendering strategy avoids triggering a global redraw of the entire complex topology map due to a single point of failure, greatly reducing the computational and data transfer overhead of the GPU. Even in scenarios with millions of data points, it can guarantee a stable frame rate of 60 FPS, providing users with a seamless and smooth interactive experience.

[0063] In summary, by organically integrating the above technical solutions, this invention systematically solves the bottlenecks of existing technologies in terms of real-time performance, intelligence, and interactivity, and constructs a new generation of efficient, intelligent, and smooth data asset status monitoring system. Attached Figure Description

[0064] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0065] Figure 1 A flowchart illustrating a data asset status monitoring method provided in an embodiment of the present invention;

[0066] Figure 2 This is a schematic diagram of the structure of a data asset status monitoring system provided in an embodiment of the present invention. Detailed Implementation

[0067] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0068] like Figure 1 As shown in some embodiments of this application, this embodiment provides a data asset status monitoring method, specifically, the method includes the following steps:

[0069] Step S101: Obtain data asset change events through edge nodes deployed on the data source side, and deduplicate and rearrange the data asset change events according to time windows to obtain an event stream.

[0070] The core of this step is to shift data processing capabilities to the source of data generation in order to solve the high latency and high load problems caused by the traditional centralized collection model.

[0071] Regarding "edge nodes deployed on the data source side," the "edge node" is not a fixed physical device, but a logical computing unit with flexible and diverse forms. It can be a lightweight agent coexisting with a database instance, a sidecar container deployed with business Pods in a containerized environment, or even a physical gateway close to IoT sensors. Its key feature is its proximity to the data source, enabling it to directly and efficiently capture raw change events, avoiding the huge bandwidth consumption caused by transmitting raw high-frequency event streams across the network.

[0072] Regarding "acquiring data asset change events," edge nodes capture changes by adapting their interfaces to different data sources. For relational databases (such as MySQL), it precisely captures data record change events such as INSERT, UPDATE, and DELETE by listening to the database's binary log (binlog). For NoSQL databases or API services, it may obtain metadata change events (such as table structure changes) by subscribing to change callbacks or webhooks. These events form the basic raw material for all subsequent processing.

[0073] The core preprocessing logic executed by edge nodes involves "duplicate removal and time-series rearrangement based on time windows." Specifically:

[0074] Deduplication refers to the process where, in real-world business scenarios, the same data record may be repeatedly modified within a short period (e.g., an order status changes from "pending payment" to "paid," and then to "shipped"). Transmitting every intermediate status to the central node would generate a large amount of redundant data. Edge nodes maintain a time-window-based deduplication hash table to perform overwrite deduplication on multiple changes to the same primary key (composed of the data source identifier, table identifier, and primary key) within the window period, retaining only the last change. This significantly reduces the size of the event stream.

[0075] Timing reordering refers to the situation where, due to network jitter or differences in internal system processing, the order in which events arrive at edge nodes (processing time) may not be consistent with their actual order of occurrence in the data source (event time). To ensure the accuracy of the event stream, edge nodes reorder all events within the window upon window expiration, based on the timestamps (event time) carried by the events themselves or the transaction commit order, ensuring that the output event stream strictly follows the chronological order of business operations.

[0076] Through the above operations, the edge nodes transform the original, redundant, and potentially out-of-order change event stream into a concise, ordered, and accurate event stream, laying a high-quality data foundation for subsequent centralized processing.

[0077] For example, suppose an e-commerce platform has an order table (orders) in its order database (order_db) with the primary key order_id.

[0078] Scenario: Within a 500-millisecond time window, the system performed three consecutive updates on order_123: T=0ms: the status field was updated from “Created” to “Paid”; T=200ms: the status field was updated from “Paid” to “Shipped”; T=400ms: the tracking_number field was updated to “SF123456”.

[0079] Traditional approach: A centralized collector receives these three independent change events separately and processes them sequentially, resulting in unnecessary computation and storage overhead.

[0080] Invention solution:

[0081] The edge node deployed next to order_db listens for these three binlog events in real time. The edge node identifies that these three events have the same deduplication key (order_db_orders_order_123).

[0082] Within a 500ms sliding window, the edge node uses a deduplication hash table to overwrite and merge these three events. Ultimately, the hash table retains only the latest status of order_123, i.e., {status:"Shipped",tracking_number:"SF123456"}.

[0083] When the 500ms window expires, the edge node pushes the merged change event, which represents the final state, along with its correct event timestamp (T=400ms), to the central message queue.

[0084] In this way, the central node only needs to process one event to obtain the complete and final state of order_123 within that time window, which significantly improves processing efficiency and data accuracy.

[0085] Step S102: Generate time-series monitoring indicators and asset tags for data assets based on the event stream, and fuse the asset tags as an additional dimension with the time-series monitoring indicators to form a multi-dimensional feature vector.

[0086] The key to this step is to construct a multidimensional feature vector that can both reflect the objective state of the data and understand its business semantics.

[0087] To improve processing efficiency, the system adopts a parallel pipeline architecture. One pipeline focuses on quantitative computation, which extracts raw data from the event stream and generates objective and measurable time-series indicators such as data integrity (proportion of missing values), timeliness (update latency), and availability (service downtime) through operations such as counting and aggregation. The other pipeline focuses on semantic understanding, which performs in-depth analysis on the metadata context (such as table names, field names, comments, sample values, etc.) carried by the event stream to generate asset tags that describe the business meaning of the data.

[0088] Regarding the generation of asset tags, this solution adopts a two-tiered system of "NLP automatic recognition + rule engine correction." Specifically:

[0089] Automatic NLP recognition refers to using pre-trained natural language processing models (such as BERT) to perform semantic parsing on metadata text and automatically generate initial labels and their confidence levels. For example, the model can predict labels such as "personal privacy" and "identity information" with high confidence from the annotation "ID card number".

[0090] Rule engine correction addresses the issue of inaccurate or conflicting initial labels. To resolve this, the system introduces correction logic based on business rules. The rule engine loads pre-defined compliance rules and business logic (e.g., "Fields containing the keyword 'ID card' must be marked as 'L4-Top Secret'"). When the NLP model's output conflicts with hard rules, the rule engine has the highest priority, ensuring labels comply with regulatory and business requirements. Furthermore, for mutually exclusive labels in the model output (e.g., simultaneously outputting "real-time data" and "offline data"), the rule engine can resolve them based on pre-defined arbitration strategies (e.g., querying the actual update frequency).

[0091] Regarding the "fusion to form a multi-dimensional feature vector": Traditional anomaly detection relies solely on numerical indicators, while this solution transforms qualitative asset labels into quantitative feature dimensions. For example, the "security level" label can be mapped to a numerical value (L1=1, L2=2, ..., L4=4), or a Boolean label such as "whether it is a core transaction table" can be directly used as a one-dimensional feature (yes=1, no=0). Ultimately, these feature dimensions derived from labels are combined with existing time-series monitoring indicators (such as integrity=0.995, timeliness=200ms) to form a multi-dimensional feature vector that integrates objective state and subjective semantics. This vector provides richer and more business-insightful input for subsequent anomaly detection models.

[0092] For example, suppose a bank's data asset table customer_contact_info (customer contact information table) has been successfully labeled with the asset tags "Personal Privacy" and "L4-Top Secret".

[0093] Scenario: During daily operation, the monitoring indicators for this table are as follows: Data integrity: 0.995 (normal); Data timeliness: 200ms (normal); Data volume volatility: 0.01 (normal).

[0094] Traditional approach: The anomaly detection model only sees the above three values ​​and will determine that the table is in a completely normal state.

[0095] The present invention's solution is as follows: The system first obtains the label information of the table from the label system: "Personal Privacy" and "L4-Top Secret". When constructing the feature vector, in addition to the above three indicators, two additional dimensions are added: is_sensitive_data (whether it is sensitive data) = 1 (because of the "Personal Privacy" label); security_level (security level) = 4 (because "L4-Top Secret" is mapped to 4). The final multidimensional feature vector may be: [0.995, 200, 0.01, 1, 4].

[0096] Now, suppose an unusual scenario occurs: the amount of data in the table drops by 50% in a short period of time (the data volume volatility becomes -0.5).

[0097] Traditional approach: The model sees [0.995, 200, -0.5, ...] and may judge that this is an anomaly that needs attention based on historical data, but it cannot distinguish its severity.

[0098] The proposed solution involves a model that observes a value of [0.995, 200, -0.5, 1, 4]. Because the feature vector includes the high-weighted dimensions is_sensitive_data=1 and security_level=4, the model can clearly understand that this sudden drop in data volume did not occur on a regular log table, but rather on a table storing core customer privacy information—a table with the highest security level. Therefore, the model assigns an extremely high reconstruction error to this anomaly, triggering a red emergency alert and immediately notifying the relevant personnel. This alert is not only accurate but also possesses a clear business context, greatly enhancing the value of the alert.

[0099] It should be noted that, in specific implementation scenarios, a multi-granularity tag fusion scheme can be adopted based on the above solution. That is, asset tags can be defined at different granularities, such as table-level tags (e.g., "core transaction table"), field-level tags (e.g., "ID number," "amount"), or even value-level tags (e.g., the business meaning of a specific enumeration value). When constructing feature vectors, the system can flexibly aggregate or weightedly fuse tags of different granularities according to monitoring needs. For example, when performing table-level anomaly detection, all field-level security level tags can be aggregated (e.g., taking the maximum value) to form a comprehensive tag dimension representing the security risk of the entire table. This multi-granularity fusion capability allows the feature vector to more comprehensively depict the state of data assets. All of the above optional solutions fall within the scope of protection of this application.

[0100] Step S103: Periodically extract the multidimensional feature vector and input it into the pre-trained reconstruction anomaly detection model to calculate the current reconstruction error; obtain the historical reconstruction error value within the historical sliding window, and generate a clean reconstruction error dataset based on the historical reconstruction error value; dynamically update the judgment threshold based on the clean reconstruction error dataset; if the current reconstruction error is greater than the judgment threshold, generate an early warning event carrying an asset identifier and an alarm level.

[0101] The core of this step lies in combining the anomaly detection principle based on reconstruction with a dynamic threshold update mechanism.

[0102] Regarding "reconstruction-based anomaly detection models": The core idea of ​​this type of model (such as autoencoders) is to learn the inherent distribution patterns of normal data. During the training phase, the model is required to reconstruct the input multi-dimensional feature vector representing a normal state as accurately as possible. Once the model is trained, it possesses a strong ability to represent "normal" data. During the inference phase, when a completely new multi-dimensional feature vector is input, if the state represented by the vector is normal, the model can reconstruct it well, resulting in a small reconstruction error. Conversely, if the vector contains an abnormal pattern, since the model has not learned this pattern, its reconstruction result will deviate significantly from the original input, resulting in a large reconstruction error. Therefore, the magnitude of the reconstruction error directly reflects the degree of anomaly in the current data state.

[0103] Regarding the "dynamically updated judgment threshold": Traditional anomaly detection methods typically use a fixed threshold to determine whether reconstruction errors constitute an anomaly. However, in real-world data asset environments, normal data behavior patterns drift slowly over time and with business cycles (such as major sales events and holidays). Fixed thresholds are prone to false positives (mistaking normal business fluctuations for anomalies) or false negatives (failing to detect new anomalies). This solution abandons fixed thresholds and instead adopts a dynamic threshold update strategy based on a clean dataset within a historical sliding window. The system periodically reviews recent historical reconstruction errors and uses robust statistical methods (such as Median Absolute Deviation, MAD) to remove known outliers, constructing an error set containing only "clean" normal data. Then, a new judgment threshold is set based on the statistical distribution of this clean set (such as the 95th quantile). This mechanism ensures that the threshold adaptively follows the evolution of normal data patterns, maintaining high sensitivity and a low false positive rate.

[0104] Regarding "Generate Warning Events": When the calculated current reconstruction error exceeds the dynamically updated judgment threshold, the system determines that an anomaly has occurred. At this time, a structured warning event will be generated. This event not only includes the asset identifier that triggered the alarm, used to accurately locate the source of the problem, but may also include the alarm level (classified according to the degree to which the reconstruction error exceeds the threshold, such as minor, severe, and urgent), providing a priority basis for subsequent response processing.

[0105] The system retrieves the multidimensional feature vector of the data asset corresponding to the current moment from the multidimensional feature vector generation module at fixed intervals (e.g., every 5 minutes) and inputs it into a pre-trained reconstructive anomaly detection model (e.g., a neural network model based on an autoencoder or variational autoencoder, VAE). This model attempts to compress and reconstruct the input features, outputting a reconstructed feature vector. The system calculates the difference between the original input vector and the reconstructed vector (typically using mean squared error (MSE) or the L1 norm) to obtain the current reconstruction error, which reflects the degree to which the current data state deviates from the normal pattern learned by the model.

[0106] To avoid a decrease in detection sensitivity due to historical anomalies contaminating the threshold, the system extracts all historical reconstruction error values ​​from a preset-length historical sliding window (e.g., the last 24 hours) before each threshold update. Combined with confirmed anomaly labels or confidence filtering mechanisms, it filters out samples that are not labeled as anomalies or have low confidence, constructing a clean reconstruction error dataset. This dataset contains only error values ​​that the model considers to be normal behavior, representing the normal fluctuation range under the current business scenario.

[0107] Based on this clean reconstruction error dataset, the system uses statistical methods (such as taking the 95th or 99th percentile, or setting a threshold beyond a certain standard deviation after fitting a Gaussian distribution) to dynamically calculate and update the judgment threshold. This threshold is not fixed, but adaptively adjusts according to changes in data asset usage patterns, business cycles, or system load to ensure that the detection benchmark always matches the actual operating conditions.

[0108] When the calculated reconstruction error exceeds the dynamically updated threshold, the system determines that the current data asset status is abnormal and generates an alert event. This alert event contains at least two key pieces of information: first, an asset identifier (such as data source ID, table name, field path, etc., used to accurately locate the abnormal object); and second, an alert level (which can be divided into low, medium, and high levels based on the extent to which the reconstruction error exceeds the threshold, used to guide subsequent response priorities). The generated alert event will be pushed to the monitoring and alarm platform for timely handling by operations and maintenance or data governance personnel.

[0109] For example, suppose the system is monitoring a log table named user_login_log, whose corresponding multidimensional feature vector includes dimensions such as [data volume, update frequency, security level].

[0110] The drawbacks of scenario A - fixed threshold:

[0111] In daily operations, this table generates approximately 10,000 records per minute, with a stable reconstruction error of around 10. The system has a fixed threshold of 20. During peak sales periods, login volume surges to 100,000 records per minute. Although this is a normal business peak as expected, the feature vector undergoes a significant change, causing the reconstruction error to spike to 50.

[0112] Result: The fixed threshold model incorrectly judged this business peak as a serious anomaly, generating a large number of meaningless alarms (false alarms).

[0113] Scenario B - Advantages of the present invention:

[0114] Model Inference: During the same promotional period, the model calculated a reconstruction error of 50. Dynamic Threshold Calculation: Simultaneously, the system's dynamic threshold update module was working. It took the reconstruction error data from the most recent 24 hours, used the MAD algorithm to identify and remove several possible past anomalies (error > 100), resulting in a set of "clean" data. In this set, because it included data from the traffic ramp-up period before the promotion, its 95th percentile error value was automatically adjusted to 60. Early Warning Decision: Current error 50 < dynamic threshold 60. Result: The system correctly identified this traffic surge as normal business behavior, without generating any alarms, effectively avoiding false alarms.

[0115] Scenario C - Capturing Real Exceptions:

[0116] Suppose that on a day outside of a major promotional period, the log table experiences a sudden drop in data volume to 0 due to an upstream service failure. The model calculates a reconstruction error of 80. At this point, the dynamic threshold module, based on recent stable period data, calculates a threshold of 15.

[0117] Result: 80>15, the system immediately generates an emergency warning event, carrying the asset identifier user_login_log, and notifies the operations and maintenance personnel to investigate the upstream service.

[0118] Step S104: Locate the target element in the asset topology view based on the asset identifier carried by the warning event, and locally update the display status of the target element.

[0119] The core of this step lies in accurately and with low overhead linking the anomaly detection results with the visualization rendering, thus avoiding the performance bottleneck caused by global redrawing in traditional solutions.

[0120] Regarding "locating target primitives in the asset topology view": During system initialization or view loading, the front-end assigns a globally unique asset identifier to each data asset (such as a table or a database). Simultaneously, when building the visualization scene, the system establishes a mapping dictionary (or index table). This dictionary precisely records the mapping relationship between each asset identifier and the starting offset position of its corresponding primitive's color data in the graphics processing unit's (GPU) memory buffer (such as a dot or a square). When an alert event occurs, the front-end only needs to perform a quick lookup using this dictionary to instantly locate the exact storage address in the GPU of the target primitive whose color needs to be changed.

[0121] Regarding "partially updating the display state of the target primitive": Traditional web visualization frameworks (such as D3.js) often require manipulating the DOM or rebinding the entire dataset when updating the style of a single element, which is extremely inefficient when dealing with massive amounts of data. This solution uses a low-level graphics API based on WebGL. Specifically, after locating the GPU buffer address of the target primitive, the system calls the segmented update instruction of the graphics programming interface (such as gl.bufferSubData). This instruction allows developers to write new data (such as new RGB color values) only to a specified offset and length range in the GPU memory, without affecting the data of all other primitives in the buffer. When rendering the next frame, the GPU will automatically use these updated local data to draw the scene, thus achieving true "incremental, localized" rendering.

[0122] Overall Synergy: This mechanism forms a perfect closed loop with the aforementioned anomaly detection module. The anomaly detection module is responsible for "discovering" problems and generating structured alert events, while this step is responsible for "presenting" the problems in a way that consumes minimal system resources. This design ensures that even in extreme scenarios with dozens of alerts per second, the user interface maintains a smooth 60FPS frame rate, guaranteeing the availability of the monitoring system and the user experience.

[0123] For example, suppose a large e-commerce platform's data asset management platform is monitoring a global topology graph containing 100,000 data asset nodes.

[0124] Scenario: The system detected that the log table named user_login_log had experienced a sharp drop in data update frequency due to an upstream service interruption, triggering a red emergency alert.

[0125] Traditional approach (e.g., SVG / DOM based): The front-end receives the alert event. It iterates through the entire list containing 100,000 DOM elements, searching for the element with the ID "user_login_log". Once found, its CSS style is modified (e.g., fill:red). The browser's rendering engine needs to recalculate the element's style, potentially triggering a reflow and repaint of the entire SVG container.

[0126] Result: On low-end devices, this process may cause noticeable stuttering (frame drops), especially when there are frequent alarms, resulting in a very poor user experience.

[0127] Invention solution:

[0128] Alert event push: The backend generates a JSON alert event: {"asset_id":"table_user_login_log","alert_level":"red","color":[1.0,0.0,0.0]}, and pushes it to the frontend via WebSocket.

[0129] Precise positioning: After receiving an event, the front-end alert processor immediately queries the pre-built mapping dictionary. Assume the dictionary returns the color data corresponding to `table_user_login_log` at a starting offset of 15200 in the GPU buffer.

[0130] Partial update: The front end calls gl.bufferSubData(gl.ARRAY_BUFFER,15200×4,newFloat32Array([1.0,0.0,0.0])). This instruction tells the GPU: "Please replace the 12 bytes (3 floating-point numbers) starting from byte 15200×4 in the video memory with the red value."

[0131] Efficient rendering: In the next frame, when the GPU executes the rendering pipeline, it reads the color data at position 15200, which is already red. Therefore, the user_login_log node is rendered as red. The data of the remaining 99,999 nodes is not touched, and the rendering process is completely consistent with the previous frame.

[0132] Result: The entire update process was completed within milliseconds, and users saw a smooth and immediate alert effect without any lag. It could handle multiple nodes changing color at the same time without any issues.

[0133] It should be noted that, in specific implementation scenarios, a batch merging optimization scheme for dirty data regions can be adopted based on the above solution. Specifically, in high-frequency warning scenarios, multiple warning events may be received within a short period. If a separate `bufferSubData` call is executed for each event, it will still incur some CPU-GPU communication overhead. Therefore, this solution can introduce a dirty data region merging mechanism. The front-end can maintain a "dirty data list." During the preparation phase before rendering each frame, multiple update requests with consecutive or similar addresses in the list are merged into a larger data block, and then uploaded through a single `bufferSubData` call. This batch processing strategy can further reduce the number of API calls and maximize the utilization of GPU bandwidth. All of the above optional solutions fall within the scope of protection of this application.

[0134] In some embodiments of this application, to clarify the internal processing logic, deduplication strategy, event classification and processing method, and timing guarantee mechanism of the solution, thereby providing clear, complete, and implementable technical support for the entire technical solution, data asset change events are acquired by edge nodes deployed on the data source side, and the data asset change events are deduplicated and rearranged according to time windows to obtain an event stream, specifically as follows:

[0135] By monitoring and acquiring data asset change events carrying event generation timestamps through edge nodes deployed on the data source side, the data asset change events include data record change events and / or metadata change events;

[0136] The current active time window is maintained based on the event generation timestamp and the preset window advancement rules;

[0137] When the data asset change event includes a data record change event, the data source identifier, data table identifier, and primary key identifier are used to form a deduplication key. Multiple data record change events with the same deduplication key within the current active time window are deduplicated in an overwrite manner, and the state of the last data record within the current active time window is retained.

[0138] When the data asset change event includes metadata change event, the asset identifier or field identifier is used to form a deduplication key, and multiple metadata change events with the same deduplication key within the current active time window are deduplicated and merged.

[0139] When the current active time window meets the window closure condition, the data asset change events within the window after deduplication and merging are rearranged in time according to the event generation timestamp and / or transaction commit order to obtain a cleaned event stream, and the cleaned event stream is pushed to the central message queue.

[0140] If a late event is received, and the event generation timestamp of the late event is earlier than the start timestamp of the current active time window, then the late event is determined to belong to a closed window, the late event is encapsulated as a compensation message and pushed to the late compensation message queue.

[0141] As described above, this step begins with edge nodes deployed on the data source side performing a listening operation to capture change events occurring in the data assets in real time. These change events all carry precise event generation timestamps to identify the actual time the event occurred within the data source. The captured event types fall into two main categories: one is data record change events (DML), such as INSERT, UPDATE, and DELETE operations in database tables; the other is metadata change events (DDL), such as the creation, modification, or deletion of table structures.

[0142] Edge nodes dynamically maintain a currently active time window based on preset window advancement rules (such as a sliding window mechanism based on processing time or event time). For incoming change events, the system employs different deduplication strategies depending on their type. When an event is a data record change event, the system combines the data source identifier, data table identifier, and record primary key identifier to form a unique deduplication key. Within the same active time window, if multiple events with the same deduplication key occur, the system performs overwrite deduplication, retaining only the data record state formed by the last change and discarding intermediate states. When an event is a metadata change event, the system uses the asset identifier (for the entire asset) or field identifier (for a specific field) as the deduplication key and merges multiple metadata changes with the same deduplication key within the window, such as retaining the last comment or field definition.

[0143] When the current active time window meets the preset closing conditions (e.g., the window duration has expired), the system will strictly generate timestamps for all change events within the window that have undergone the above deduplication and merging process, or reorder them according to the commit order of database transactions, to ensure the correctness of the event flow's timing. Finally, a cleaned event flow will be formed and pushed to a centralized message queue for downstream systems to consume.

[0144] Furthermore, the system incorporates a fault-tolerant handling mechanism for late events. If an edge node receives an event whose event generation timestamp is earlier than the start timestamp of the current active time window, it is determined that the event belongs to a closed historical window. For such late events, the system does not discard them but encapsulates them into a special compensation message and pushes it to a separate late compensation message queue. This allows the central stream processing engine to perform subsequent compensation processing based on mechanisms such as watermarks, thus ensuring both real-time processing efficiency and data integrity.

[0145] In some embodiments of this application, in order to maintain core data processing capabilities as much as possible while ensuring system stability and availability, and to avoid edge node crashes or data loss due to memory overflow, the memory usage rate of the deduplication hash table used for event deduplication is monitored in real time during the deduplication and time-series rearrangement process performed at the edge nodes. A graded degradation strategy is then implemented based on a first memory threshold, a second memory threshold, and a third memory threshold, wherein the first memory threshold is less than the second memory threshold, and the second memory threshold is less than the third memory threshold.

[0146] If the memory occupancy rate is greater than or equal to the first memory threshold and less than the second memory threshold, then the window duration of the current active time window is shortened, and deduplication is performed only for data asset change events with preset priority.

[0147] If the memory occupancy rate is greater than or equal to the second memory threshold and less than the third memory threshold, then the writing of the deduplication hash table is turned off, and the arriving data asset change event is assigned a pass-through identifier and output to the central message queue.

[0148] If the memory occupancy rate is greater than or equal to the third memory threshold, a backpressure mechanism is triggered to suspend the consumption of events from the corresponding data source until the memory occupancy rate falls below the preset recovery threshold. Then, the interrupted transmission is resumed based on the saved log offset.

[0149] As mentioned above, during the process of deduplicating and reordering data asset change events at edge nodes, the system continuously monitors the proportion of memory used by the deduplication hash table storing deduplication keys (such as primary keys, field identifiers, etc.). This proportion is expressed as the ratio of currently used memory to the preset maximum available memory. The system pre-configures three incremental memory thresholds: a first memory threshold (lower), a second memory threshold (medium), and a third memory threshold (higher), with the first < second < third.

[0150] When the system detects that the memory usage rate is greater than or equal to the first memory threshold but has not yet reached the second memory threshold, it determines that the memory pressure is in a state of mild risk. At this time, the edge node will proactively shorten the window duration of the current active time window (e.g., from 10 seconds to 5 seconds) to speed up window closure and release old keys in the hash table; at the same time, it will only perform complete deduplication logic on change events corresponding to data assets marked as high priority (such as core transaction tables and sensitive data tables involving compliance and supervision), while other low-priority events will skip deduplication and be directly passed through, thereby controlling memory growth while maintaining the processing quality of critical business data.

[0151] When memory usage rises further to a level greater than or equal to the second memory threshold but less than the third, the system determines that memory pressure has entered a medium-risk state. To prevent the hash table from continuing to expand, edge nodes will disable write operations to the deduplication hash table, meaning they will no longer add new deduplication keys. All data asset change events arriving thereafter are assigned a "transparent pass-through identifier" and directly output to the central message queue, without any further deduplication or merging processing. While this stage sacrifices deduplication capabilities, it ensures the continuous forwarding of the event stream, avoiding processing interruptions. In this scheme, the "transparent pass-through identifier" is a metadata tag attached to a data asset change event, indicating that the event has not undergone deduplication or merging processing but is forwarded directly in its original form. When an edge node disables deduplication hash table writes due to medium-risk memory pressure, all newly arriving events are assigned this identifier and immediately output to the central message queue. The purpose of this "pass-through identifier" is to: inform downstream systems that the current event has not been deduplicated and may contain duplicates or out-of-order data, requiring the central system to decide whether to perform secondary cleaning or degradation processing based on business tolerance; preserve event integrity to avoid data loss due to limited edge node resources and ensure the traceability of all change records; and support operational diagnostics, by statistically analyzing the proportion of events with the "pass-through identifier," assessing the load status of edge nodes and system health. Therefore, the "pass-through identifier" does not indicate that the event content has been modified, but rather that its processing path has been simplified to a pass-through mode, representing a controllable degradation strategy under resource-constrained conditions.

[0152] When memory usage continues to deteriorate and reaches or exceeds the third memory threshold, the system determines that memory resources are severely overloaded and at risk of crashing. At this point, the edge node will immediately trigger a backpressure mechanism, pausing the fetching of new change events from the corresponding data source (such as a specific database log) to block the source of memory growth. During the pause, the system continuously monitors memory usage; once memory usage falls below a preset recovery threshold (usually below the first memory threshold), the edge node will resume fetching and processing events from the point of interruption based on the previously saved data consumption log offset (such as a database Binlog offset or Kafka offset), achieving breakpoint resumption and ensuring no data loss and recoverable processing.

[0153] In some embodiments of this application, in order to construct a high-dimensional representation system that includes both dynamic behavioral features and static semantic information, providing a more comprehensive and discriminative input for subsequent reconstructive anomaly detection, time-series monitoring indicators and asset tags for data assets are generated based on the event stream. The asset tags are then fused with the time-series monitoring indicators as an additional dimension to form a multi-dimensional feature vector, including:

[0154] Consume the cleaned event stream from the central message queue and execute the metric calculation branch and the tag generation branch in parallel;

[0155] In the indicator calculation branch, the time-series monitoring indicators of the data assets are updated based on the cleaned event stream. The time-series monitoring indicators include at least one of data integrity, data timeliness, data availability, and data volume volatility.

[0156] In the tag generation branch, the metadata context corresponding to the cleaned event stream is extracted, candidate tags are generated based on the natural language processing model, and the candidate tags are corrected or conflict arbitration is performed based on the preset rule engine to determine the asset tag.

[0157] The asset tags are injected as additional feature dimensions into the time-series monitoring indicators to form a multi-dimensional feature vector for anomaly detection.

[0158] As described above, after consuming the event stream cleaned by the edge nodes from the central message queue, the system simultaneously starts two independent data processing branches in a parallel processing manner: the indicator calculation branch and the label generation branch.

[0159] In the indicator calculation branch, the system updates multiple time-series monitoring indicators for the corresponding data assets in real time based on data record changes or metadata change information carried in the event stream. These indicators are used to quantify the health status of data assets during operation, specifically including but not limited to: data integrity (e.g., primary key missing rate, proportion of non-null fields being empty), data timeliness (e.g., data write latency, difference between the latest record timestamp and the current system time), data availability (e.g., whether a table is queryable, number of access failures), and data volume volatility (e.g., the degree of deviation of the data volume in the current window from the historical average). Each indicator is continuously accumulated and updated according to the time window, forming a numerical sequence that evolves over time. Among them, "data volume volatility" refers to a quantitative indicator used to measure the degree of deviation of the data volume generated by a data asset in the current time window from its historical normal level. Its technical content specifically includes the following elements: Benchmark value determination, whereby the system establishes a time series of data volume for each data asset based on historical event stream data (e.g., the number of new records per hour or day), and calculates a dynamic expected data volume benchmark value using methods such as sliding window, exponentially weighted moving average (EWMA), or periodic average (e.g., the average value for the same period last week); Current value acquisition, whereby within the current processing time window, the actual data volume generated by the data asset (e.g., the total number of inserted records, the number of messages, or the number of bytes) is statistically analyzed from the cleaned event stream; and Volatility measurement calculation, whereby the current data volume is compared with the expected benchmark value using a relative deviation formula (e.g., ...). ,in To prevent the use of methods such as division by zero (small constants) or standardized Z-scores, a dimensionless or normalized volatility value is calculated. The indicator is designed for anomaly sensitivity, meaning it is sensitive to sudden increases (such as due to duplicate writes or web crawler attacks) and sudden decreases (such as due to upstream interruptions or acquisition failures) in data volume, effectively reflecting anomalies in the stability of the data production chain. Therefore, "data volume volatility" does not simply refer to the absolute size of the data volume, but rather is a key time-series monitoring indicator characterizing the self-consistency and stability of data production, used to capture abnormal scenarios that significantly deviate from historical behavior patterns.

[0160] In the tag generation branch, the system extracts metadata context information related to the current data asset from the event stream, such as table comments, field descriptions, business domain, data owner, security level, and other textual or structured attributes. This context information is then input into a pre-trained natural language processing model (such as BERT or its lightweight variant), which understands the semantics and generates a set of candidate tags (such as "user behavior logs," "financially sensitive data," and "core transaction table"). Since automatically generated tags may be ambiguous, redundant, or conflicting, the system further introduces a pre-defined rule engine. Based on a business rule base (such as tag whitelists, mutually exclusive tag pairs, and priority ranking rules), the system corrects, filters, or arbitrates the candidate tags, ultimately determining a set of authoritative, consistent asset tags that conform to the business context.

[0161] After processing the two branches mentioned above, the system transforms the determined asset tags into structured feature representations (e.g., through one-hot encoding or embedding vectors), and uses this as an additional static feature dimension. This representation is then concatenated or fused with the dynamically generated time-series monitoring indicators (as dynamic feature dimensions) to form a unified multi-dimensional feature vector. This vector reflects both the operational status of the data asset within a specific time window (dynamic indicators) and its inherent business semantic attributes (static tags), providing a richer and more context-aware input foundation for subsequent input reconstruction-based anomaly detection models.

[0162] In some embodiments of this application, a structured conflict identification and hierarchical handling process is established to ensure that the final determined asset tags meet system requirements in terms of semantic consistency, business accuracy, and security compliance. The asset tags are determined by modifying or arbitrating the candidate tags based on a preset rule engine, including:

[0163] When the natural language processing model outputs multiple candidate labels for the same data field that match the preset mutual exclusion pair configuration table, and the confidence scores of the multiple candidate labels are all higher than the preset confidence threshold, the branch arbitration logic is triggered.

[0164] If the mutual exclusion label is determined to belong to the security level category of mutual exclusion, then the preset security level weights corresponding to the mutual exclusion labels are compared, and the candidate labels with higher weights are retained;

[0165] If it is determined that the mutual exclusion label belongs to the objective business status type mutual exclusion, then query the objective metadata attributes of the corresponding data table and retain the candidate labels that are consistent with the objective metadata attributes;

[0166] If the mutual exclusion tag is determined to be a time-sensitive description type mutual exclusion, then the update frequency of the corresponding data table is queried. When the update frequency is greater than the set frequency threshold, the real-time candidate tag is retained. When the update frequency is less than or equal to the set frequency threshold, the offline candidate tag is retained.

[0167] When multiple candidate labels conflict with each other but do not meet the judgment conditions of the branch arbitration logic, or when the objective data required to execute the branch arbitration logic is missing, the activation process of the multiple candidate labels is suspended, a temporary label with the highest security level is assigned to the current data field, and a labeling task is generated based on the current data field. The labeling task is then pushed to the manual review queue.

[0168] As mentioned above, when a natural language processing model outputs multiple candidate labels (such as "personal identity information" and "public user identifier") for the same data field (such as the "user_id" field of a table), and these labels simultaneously hit the system's preset mutual exclusion pair configuration table (i.e., label combinations that are explicitly defined as incompatible), and their respective confidence scores are all higher than the preset confidence score threshold, the system will trigger a special branch arbitration logic. Instead of simply retaining high-confidence labels, it will adopt different adjudication methods based on the semantic category to which the labels belong.

[0169] If the mutually exclusive labels are determined to be mutually exclusive in terms of security level (e.g., "Confidential" and "Internal Public"), the system will query the preset security level weight of each label in the security policy (e.g., "Confidential" has a weight of 90, and "Internal Public" has a weight of 50), and retain the candidate labels with higher weights to ensure that the security classification of data assets complies with the principle of least privilege and highest protection.

[0170] If the mutually exclusive labels belong to the objective business status mutual exclusion (such as "real-time transaction data" and "historical archive data"), the system will not rely on the model output, but will actively query the objective attributes of the data table in the metadata management system (such as whether the storage location is an archive library, lifecycle strategy, etc.), and compare the candidate labels with these objective facts, retaining only the labels that are consistent with the actual metadata status.

[0171] If the mutual exclusion label belongs to the time-description category (e.g., "real-time data" and "offline data"), the system will obtain the historical update frequency of the data table (e.g., number of writes per minute or the time interval between the latest records). When the update frequency is higher than the set frequency threshold (e.g., at least once every 5 minutes), it is determined to be an active real-time table, and the "real-time data" label is retained; conversely, if the update frequency is lower than or equal to the threshold, the "offline data" label is retained.

[0172] During the aforementioned arbitration process, if mutually exclusive labels conflict but do not fall into any defined arbitration branch (i.e., it cannot be determined whether they belong to the security, business status, or timeliness category), or if the category can be determined but the objective metadata required for arbitration is lacking (such as missing metadata, unavailable interfaces, etc.), the system will be unable to complete the automatic adjudication. In this case, to ensure data security, the system will suspend the activation process of all candidate labels for that field, temporarily assign it a fallback label with the highest security level (such as "unknown sensitive data"), and automatically generate a labeling task that records the conflict context, candidate labels, and missing information in detail. This task will be pushed to the manual review queue, and the suspension will be lifted and the label will officially take effect only after data governance personnel intervene and confirm it.

[0173] In some embodiments of this application, a robust statistical method based on median and median absolute deviation (MAD) is introduced to effectively identify and remove anomalous reconstruction error points that may exist in the historical window, thereby ensuring that the dataset used for dynamic threshold calculation truly reflects the normal operating state of the system. This method is highly robust to extreme noise and outliers, avoiding threshold drift or decreased detection sensitivity due to contaminated data. A clean reconstruction error dataset is generated based on the historical reconstruction error values, including:

[0174] Obtain multiple historical reconstruction error values ​​within the historical sliding window, and calculate the initial median of the multiple historical reconstruction error values;

[0175] Calculate the absolute difference between each historical reconstruction error value and the initial median to form a deviation set, and calculate the median of the deviation set to obtain the absolute median difference;

[0176] If the absolute median difference is zero, then all historical reconstruction error values ​​within the historical sliding window are used as the clean reconstruction error dataset.

[0177] If the absolute median difference is not zero, a correction score is calculated based on each historical reconstruction error value, the initial median, and the absolute median difference. Historical reconstruction error values ​​whose absolute values ​​of the correction scores are greater than the extreme deviation threshold are removed, and the remaining historical reconstruction error values ​​are used as the clean reconstruction error dataset.

[0178] As described above, the system first obtains all recorded historical reconstruction error values ​​from a preset historical sliding window (e.g., the most recent 100 time periods) to form an original error sequence. Subsequently, the initial median of this sequence is calculated as a robust estimate of the central trend, avoiding the influence of extreme values.

[0179] Next, for each historical reconstruction error value, the absolute difference between it and the initial median is calculated, forming a set of deviations. The median of this set of deviations is then calculated to obtain the absolute median difference (MAD), which measures the dispersion of the error distribution. MAD is a scaling estimator that is insensitive to outliers, making it particularly suitable for non-Gaussian or noisy data.

[0180] If the calculated absolute median difference is zero, it indicates that all historical reconstruction error values ​​are highly concentrated (they may all be equal or have very small differences). In this case, it can be reasonably inferred that there are no significant outliers in the current window. The system directly includes all historical reconstruction error values ​​into the clean reconstruction error dataset for subsequent threshold calculation.

[0181] If the absolute median is not zero, the system further calculates a correction score (also known as the MAD standardized score) for each historical reconstruction error value. The formula is: Correction Score = (Historical Reconstruction Error Value − Initial Median) ÷ (1.4826 × Absolute Median). Here, 1.4826 is a constant factor used to make MAD equivalent to the standard deviation under a normal distribution. This correction score reflects the degree of deviation of each error value from the normal fluctuation range.

[0182] The system presets an extreme deviation threshold (e.g., 3.5 or 4.0, corresponding to approximately 99.9% confidence interval). Historical reconstruction error values ​​with an absolute value greater than this threshold are identified as potential outliers and removed. The remaining error values ​​are considered "clean" samples, collectively forming the final clean reconstruction error dataset. This dataset retains only reconstruction errors that conform to normal behavior patterns, providing a reliable basis for the subsequent generation of dynamic thresholds.

[0183] In some embodiments of this application, by introducing a data volume proportion verification and exponential smoothing fusion strategy, the threshold is avoided from being blindly updated when there are insufficient normal samples, and the threshold is prevented from changing drastically due to short-term fluctuations, thereby improving the robustness and continuity of the anomaly detection system. Dynamically updating the judgment threshold based on the clean reconstruction error dataset includes:

[0184] Obtain the ratio of the data volume of the clean reconstruction error dataset to the total number of historical reconstruction error values ​​within the historical sliding window;

[0185] If the proportion of the data volume is lower than the set proportion threshold, the threshold update for the current period is abandoned, and the judgment threshold of the previous period is used.

[0186] If the proportion of the data volume is greater than or equal to the set proportion threshold, then the preset quantile of the clean reconstruction error dataset is extracted, and the preset quantile is exponentially smoothed and fused with the judgment threshold of the previous period to obtain the judgment threshold of the current period.

[0187] As mentioned above, in each update cycle, the system first calculates the number of valid samples contained in the clean reconstruction error dataset (i.e., the number of reconstruction error values ​​remaining after removing outliers), and compares it with the total number of original historical reconstruction error values ​​in the historical sliding window to obtain a data volume ratio (e.g., if there are 80 clean data and the total window length is 100, then the ratio is 80%).

[0188] If the proportion of this data is lower than the preset threshold (e.g., set to 70%), it indicates that there may be a large amount of abnormal or unreliable data in the current window, resulting in insufficient clean samples that can be used to model normal behavior. In this case, the system determines that the data in the current period is insufficient to support a reliable threshold update, so it abandons the threshold update operation and directly uses the judgment threshold that has been effective in the previous period to maintain the stability of the detection logic and avoid threshold distortion due to data pollution or sparsity.

[0189] If the proportion of the data volume is greater than or equal to a set proportion threshold, the current clean dataset is considered sufficiently representative, and the system extracts a preset quantile (such as the 95th or 99th quantile) as an upper limit estimate of the current normal error distribution. This quantile reflects the maximum acceptable normal reconstruction error under the current business conditions.

[0190] Subsequently, the system does not directly use this quantile as the new threshold, but instead performs exponential smoothing fusion with the judgment threshold of the previous period. Specifically, the judgment threshold for the current period is calculated using the following formula: Current judgment threshold = α × Current preset quantile + (1 − α) × Previous period judgment threshold. Here, α is a smoothing coefficient (e.g., 0.3 or 0.5), ranging from 0 to 1, used to control the degree of influence of new observations on the threshold. This fusion method preserves the responsiveness to the latest normal behavior patterns while suppressing abrupt changes in the threshold, allowing the anomaly detection boundary to evolve smoothly over time, adapting to the gradual changes in data assets, and avoiding false alarms caused by single noise disturbances.

[0191] In some embodiments of this application, on the one hand, on-demand drill-down and resource switching of multi-level topology views support users' exploration of asset structures from macro to micro levels; on the other hand, upon receiving an alert event, a pre-built mapping dictionary and graphics processing unit (GPU) are used to achieve precise, low-overhead local redrawing of only the color attributes of the affected primitives, avoiding the performance loss caused by full-image refresh, thereby improving the real-time performance and interactive smoothness of the large-scale data asset topology visualization system. Locating the target primitive in the asset topology view based on the asset identifier carried by the alert event and locally updating the display state of the target primitive includes:

[0192] In response to the view request from the user terminal, extract the asset topology data at the corresponding aggregation granularity for initial rendering;

[0193] When a drill-down request is received from the first display level to a more granular second display level, the target business domain identifier in the drill-down request is extracted;

[0194] Based on the target business domain identifier, query the data source type nodes under the corresponding business domain, the summary indicators of each data source type node, and the connection topology between each data source type node.

[0195] Stop displaying, release, or cache the aggregated view resources of the first display layer, and construct and render the network topology view of the second display layer using the data source type nodes as graph nodes and the connection topology as graph connections;

[0196] During the system initialization phase, a mapping dictionary is established, which records the starting offset position of the primitive color data corresponding to each asset identifier in the graphics processor buffer.

[0197] Upon receiving the warning event containing the target asset identifier and the target color value, the target starting offset position corresponding to the target asset identifier is retrieved from the mapping dictionary;

[0198] The segmented update instruction of the graphics programming interface is invoked to locate the target buffer interval according to the target starting offset position and the preset data length, and only the primitive color data in the target buffer interval is overwritten with the target color value, so that the graphics processor can perform local warning redraw of the target primitive without rebuilding the overall rendering pipeline.

[0199] As described above, when the system responds to the initial view request initiated by the user terminal, it first extracts the corresponding asset topology data from the backend asset metadata service according to the aggregation granularity specified by the user (such as by business domain, data center or tenant dimension), and completes the initial rendering on the front end to form the aggregated view of the first display layer.

[0200] When a user performs a drill-down operation on the aggregated view (e.g., clicking on a business domain node), the system receives the drill-down request from the first display level to a more granular second display level and parses out the target business domain identifier. Subsequently, based on this identifier, the system queries the data source type nodes (such as MySQL, Kafka, Hive, etc. category nodes) under that business domain, the summary metrics of each type of node (such as the number of instances, anomaly rate, data volume, etc.), and the connection topology relationships between these nodes (indicating data flow or dependencies).

[0201] To optimize memory and rendering performance, the system proactively stops displaying, releases, or caches graphics resources (such as vertex buffers, textures, or DOM elements) used by the aggregated view of the first display layer when switching view layers, preventing resource accumulation. Then, using the data source type nodes obtained from the query as graph nodes and the connection topology relationships as graph connections, a network topology view of the second display layer is constructed and rendered, achieving a fine-grained visualization of the asset structure.

[0202] During system initialization, to support efficient subsequent local updates, the system pre-establishes a mapping dictionary. This dictionary uses asset identifiers (such as "db-prod-01") as keys and their corresponding starting offsets in the graphics processor's memory buffers (such as color buffers or attribute buffers) as values. This mapping is determined during the initial rendering of the primitives and remains consistent with the GPU memory layout.

[0203] When the system receives an alert event containing a target asset identifier and a target color value (e.g., red #FF0000 indicates a high-risk alert), it immediately retrieves the target's starting offset position corresponding to that asset identifier from the mapping dictionary. Subsequently, it calls the segmented update instructions provided by the underlying graphics programming interface (such as WebGL's bufferSubData, Vulkan's vkCmdUpdateBuffer, or DirectX's Map / Unmap) to precisely locate the target region in the GPU buffer based on the starting offset position and the preset data length (e.g., each primitive color occupies 4 bytes of RGBA).

[0204] The system only overwrites the existing color data within the target range with the target color value specified for the warning event. Since this operation does not involve scene reconstruction, shader recompilation, or a complete rendering pipeline refresh, the graphics processor can directly use the updated color data when drawing the next frame, achieving localized warning redrawing of the target primitives. This mechanism significantly reduces the computational and transmission overhead of alarm visualization responses, making it particularly suitable for large-scale topology monitoring scenarios involving tens of thousands of asset nodes.

[0205] Compared with existing technologies, this application discloses a data asset status monitoring method. This method achieves the following significant technical effects through the synergistic effect of key technologies such as edge nodes deployed on the data source side, multi-dimensional feature vectors fused with asset tags, reconstructive anomaly detection models, and locally updated topology view rendering:

[0206] This significantly improves the real-time performance of monitoring and system throughput. By acquiring data asset change events at edge nodes on the data source side and performing deduplication and time-series rearrangement by time window, this solution cleans and compresses high-frequency, redundant raw change events at the source. This not only greatly reduces the network traffic that needs to be transmitted to the central node, but more importantly, it frees the central node from the heavy burden of raw data cleaning, allowing it to focus on higher-value computing tasks. Thus, end-to-end low-latency monitoring is achieved while ensuring information integrity.

[0207] This solution enables intelligent and refined perception of data asset status. It creatively integrates asset tags generated based on event streams as an additional dimension with traditional time-series monitoring indicators (such as completeness and timeliness) to form a multi-dimensional feature vector. This design allows subsequent anomaly detection models to not only "see" changes in data indicator values ​​but also "understand" the business implications behind the data. For example, if a data table marked as a "core transaction table" experiences a decrease in update frequency, its anomaly weight will be far higher than that of a regular log table, making alerts more accurate and more valuable to business operations.

[0208] This ensures a smooth interactive experience in large-scale visualization scenarios. After the anomaly detection model generates an alert event based on a dynamic threshold, this solution can accurately locate the target element in the asset topology view based on the asset identifier carried by the alert event, and only locally update the display state of that target element. This incremental, localized rendering strategy avoids triggering a global redraw of the entire complex topology map due to a single point of failure, greatly reducing the computational and data transfer overhead of the GPU. Even in scenarios with millions of data points, it can guarantee a stable frame rate of 60 FPS, providing users with a seamless and smooth interactive experience.

[0209] In summary, by organically integrating the above technical solutions, this invention systematically solves the bottlenecks of existing technologies in terms of real-time performance, intelligence, and interactivity, and constructs a new generation of efficient, intelligent, and smooth data asset status monitoring system.

[0210] Based on the same inventive concept as the methods described above, this application also proposes a data asset status monitoring system, such as... Figure 2 The diagram shown is a structural schematic of a data asset status monitoring system, which includes:

[0211] The event preprocessing module is used to acquire data asset change events through edge nodes deployed on the data source side, and to deduplicate and rearrange the data asset change events according to time windows to obtain an event stream;

[0212] The feature construction module is used to generate time-series monitoring indicators and asset tags for data assets based on the event stream, and to fuse the asset tags as an additional dimension with the time-series monitoring indicators to form a multi-dimensional feature vector.

[0213] The early warning generation module is used to periodically extract the multidimensional feature vector and input it into the pre-trained reconstruction anomaly detection model to calculate the current reconstruction error; obtain the historical reconstruction error value within the historical sliding window, and generate a clean reconstruction error dataset based on the historical reconstruction error value; dynamically update the judgment threshold based on the clean reconstruction error dataset; if the current reconstruction error is greater than the judgment threshold, generate an early warning event carrying an asset identifier and an alarm level.

[0214] The local rendering module is used to locate the target graphic element in the asset topology view based on the asset identifier carried by the warning event, and locally update the display status of the target graphic element.

[0215] As described above, this data asset status monitoring system aims to achieve real-time anomaly detection and visual alarms for large-scale, multi-source heterogeneous data assets. Its overall architecture is completed by four functional modules working together, and the specific functions of each module are as follows:

[0216] The event preprocessing module is deployed on edge nodes close to the data source (such as database proxies, log collectors, or message middleware bypass components) to capture raw data asset change events in real time (e.g., table structure modifications, field additions, sudden increases in data volume, changes in access permissions, etc.). To address network jitter, duplicate reporting, or out-of-order events, this module aggregates the collected event stream according to a preset time window (e.g., 5 seconds or 10 seconds): First, it deduplicates events based on their unique identifiers (e.g., transaction IDs or event hashes) to eliminate duplicate events; then, it rearranges the events within the window according to their generation timestamps to ensure that the event order conforms to the actual occurrence logic, and finally outputs an ordered, non-redundant, standardized event stream for downstream modules to consume.

[0217] The feature construction module receives the aforementioned event stream and extracts key information reflecting the operational status of data assets. On one hand, based on factors such as event type, frequency, and scope of impact, it aggregates and generates time-series monitoring indicators (such as the number of DDL operations per minute, the number of newly added sensitive fields per day, and data output delay duration). On the other hand, it parses asset tags (such as the business domain, data security level, responsible person, and asset type) from the event metadata. Subsequently, these asset tags are used as additional dimensions and concatenated or embedded with the dynamically changing time-series monitoring indicators to form a unified multi-dimensional feature vector. This vector contains both the dynamic behavioral characteristics of the asset and its static contextual information, providing a more comprehensive input representation for subsequent anomaly detection.

[0218] The early warning generation module inputs the multidimensional feature vector into a reconstruction-based anomaly detection model (such as a neural network model based on an autoencoder or variational autoencoder, VAE). This model reconstructs the input by learning the distribution pattern of the feature vector under normal conditions and calculates the reconstruction error as an anomaly score. The system also maintains a dynamically updated judgment threshold (such as the threshold mentioned above, based on a clean reconstruction error dataset and adjusted exponentially). When the reconstruction error exceeds this threshold at a certain moment, it is judged as an anomaly, triggering the generation of an early warning event. This early warning event includes information such as the asset identifier of the anomalous asset, the anomaly type, the occurrence time, and the confidence level.

[0219] The local rendering module is responsible for quickly mapping alert events to the visualization interface. Upon receiving an alert event, this module locates the corresponding target graph element (such as a graph node representing a data table or a Kafka Topic) in the loaded asset topology view based on the asset identifier it carries. Using the mapping dictionary established during system initialization (which records the color data offset position of each asset identifier in the GPU buffer), it calls the segmented update command of the graphics interface to modify only the color buffer segment corresponding to the target graph element (e.g., highlighting it in red), without needing to rebuild the entire topology rendering pipeline. This achieves low-latency, low-resource-consumption local alert redrawing, enabling operations personnel to intuitively and instantly identify the location and impact range of abnormal assets within the overall topology.

[0220] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program goods. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0221] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program goods according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0222] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0223] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0224] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for monitoring the status of data assets, characterized in that, include: Data asset change events are acquired by edge nodes deployed on the data source side, and the data asset change events are deduplicated and rearranged according to time windows to obtain a cleaned event stream, which is then pushed to the central message queue. The cleaned event stream is consumed from the central message queue, and the indicator calculation branch and the tag generation branch are executed in parallel. In the indicator calculation branch, the time-series monitoring indicators of the data assets are updated based on the cleaned event stream. The time-series monitoring indicators include at least one of data integrity, data timeliness, data availability, and data volume volatility. In the tag generation branch, the metadata context corresponding to the cleaned event stream is extracted, candidate tags are generated based on a natural language processing model, and the candidate tags are corrected or conflict arbitration is performed based on a preset rule engine to determine the asset tags. Specifically, when multiple candidate tags output by the natural language processing model for the same data field hit a preset mutually exclusive pair configuration table, and the confidence levels of all the candidate tags are higher than a preset confidence threshold, branch arbitration logic is triggered. If the mutually exclusive tags are determined to be mutually exclusive based on security level, the preset security level weights corresponding to the mutually exclusive tags are compared, and the candidate tags with higher weights are retained. If the mutually exclusive tags are determined to be mutually exclusive based on objective business status, the objective metadata attributes of the corresponding data table are queried, and candidate tags consistent with the objective metadata attributes are retained. If the mutually exclusive tags are determined to be mutually exclusive based on timeliness description, the update frequency of the corresponding data table is queried. When the update frequency is greater than a set frequency threshold, real-time candidate tags are retained; when the update frequency is less than or equal to the set frequency threshold, offline candidate tags are retained. When multiple candidate tags have mutually exclusive conflicts but do not meet the judgment conditions of the branch arbitration logic, or when the objective data required to execute the branch arbitration logic is missing, the activation process of the multiple candidate tags is suspended, a temporary tag with the highest security level is assigned to the current data field, and a labeling task is generated based on the current data field. The labeling task is then pushed to the manual review queue. The determined asset tags are injected as additional feature dimensions into the time-series monitoring indicators to form a multi-dimensional feature vector for anomaly detection. The multidimensional feature vector is periodically extracted and input into a pre-trained reconstructive anomaly detection model to calculate the current reconstruction error. Multiple historical reconstruction error values ​​within a historical sliding window are obtained, and the initial median of these values ​​is calculated. The absolute difference between each historical reconstruction error value and the initial median is calculated to form a deviation set, and the median deviation of this deviation set is calculated to obtain the absolute median difference. If the absolute median difference is zero, all historical reconstruction error values ​​within the historical sliding window are used as a threshold to update the reconstruction error dataset. If the absolute median difference is not zero, a correction score is calculated based on each historical reconstruction error value, the initial median, and the absolute median difference. Historical reconstruction error values ​​with absolute correction scores greater than the extreme deviation threshold are removed, and the remaining historical reconstruction error values ​​are used as the threshold to update the reconstruction error dataset. The judgment threshold is dynamically updated based on the threshold-updated reconstruction error dataset. If the current reconstruction error is greater than the judgment threshold, a warning event carrying an asset identifier and alert level is generated. The target element in the asset topology view is located based on the asset identifier carried by the warning event, and the display status of the target element is locally updated.

2. The method as described in claim 1, characterized in that, The process involves acquiring data asset change events through edge nodes deployed on the data source side, and then deduplicating and rearranging these events according to time windows to obtain a cleaned event stream. Specifically: By monitoring and acquiring data asset change events carrying event generation timestamps through edge nodes deployed on the data source side, the data asset change events include data record change events and / or metadata change events; The current active time window is maintained based on the event generation timestamp and the preset window advancement rules; When the data asset change event includes a data record change event, the data source identifier, data table identifier, and primary key identifier are used to form a deduplication key. Multiple data record change events with the same deduplication key within the current active time window are deduplicated in an overwrite manner, while retaining the state of the last data record within the current active time window. When the data asset change event includes metadata change event, the asset identifier or field identifier is used to form a deduplication key, and multiple metadata change events with the same deduplication key within the current active time window are deduplicated and merged. When the current active time window meets the window closure condition, the data asset change events within the window after deduplication and merging are rearranged in time according to the event generation timestamp and / or transaction commit order to obtain the cleaned event stream, and the cleaned event stream is pushed to the central message queue. If a late event is received, and the event generation timestamp of the late event is earlier than the start timestamp of the current active time window, then the late event is determined to belong to a closed window, the late event is encapsulated as a compensation message and pushed to the late compensation message queue.

3. The method as described in claim 2, characterized in that, During the deduplication and time-series rearrangement process at the edge node, the memory usage of the deduplication hash table used for event deduplication is monitored in real time, and a graded degradation strategy is executed based on a first memory threshold, a second memory threshold, and a third memory threshold, wherein the first memory threshold is less than the second memory threshold, and the second memory threshold is less than the third memory threshold. If the memory occupancy rate is greater than or equal to the first memory threshold and less than the second memory threshold, then the window duration of the current active time window is shortened, and deduplication is performed only for data asset change events with preset priority. If the memory occupancy rate is greater than or equal to the second memory threshold and less than the third memory threshold, then the writing of the deduplication hash table is turned off, and the arriving data asset change event is assigned a pass-through identifier and output to the central message queue. If the memory occupancy rate is greater than or equal to the third memory threshold, a backpressure mechanism is triggered to suspend the consumption of events from the corresponding data source until the memory occupancy rate falls below the preset recovery threshold. Then, the interrupted transmission is resumed based on the saved log offset.

4. The method as described in claim 1, characterized in that, The step of dynamically updating the judgment threshold based on the threshold-based reconstruction error dataset includes: The ratio of the data volume of the threshold-updated reconstruction error dataset to the total number of historical reconstruction error values ​​within the historical sliding window is obtained. If the proportion of the data volume is lower than the set proportion threshold, the threshold update for the current period is abandoned, and the judgment threshold of the previous period is used. If the proportion of the data volume is greater than or equal to the set proportion threshold, then the preset quantile of the threshold update reconstruction error dataset is extracted, and the preset quantile is exponentially smoothed and fused with the judgment threshold of the previous period to obtain the judgment threshold of the current period.

5. The method as described in claim 1, characterized in that, The step of locating the target element in the asset topology view based on the asset identifier carried by the early warning event, and partially updating the display status of the target element, includes: In response to the view request from the user terminal, extract the asset topology data at the corresponding aggregation granularity for initial rendering; When a drill-down request is received from the first display level to a more granular second display level, the target business domain identifier in the drill-down request is extracted; Based on the target business domain identifier, query the data source type nodes under the corresponding business domain, the summary indicators of each data source type node, and the connection topology between each data source type node. Stop displaying, release, or cache the aggregated view resources of the first display layer, and construct and render the network topology view of the second display layer using the data source type nodes as graph nodes and the connection topology as graph connections; During the system initialization phase, a mapping dictionary is established, which records the starting offset position of the primitive color data corresponding to each asset identifier in the graphics processor buffer. Upon receiving the warning event containing the target asset identifier and the target color value, the target starting offset position corresponding to the target asset identifier is retrieved from the mapping dictionary; The segmented update instruction of the graphics programming interface is invoked to locate the target buffer interval according to the target starting offset position and the preset data length, and only the primitive color data in the target buffer interval is overwritten with the target color value, so that the graphics processor can perform local warning redraw of the target primitive without rebuilding the overall rendering pipeline.

6. A data asset status monitoring system, used to perform the method as described in any one of claims 1 to 5, characterized in that, The system includes: The event preprocessing module is used to acquire data asset change events through edge nodes deployed on the data source side, and to deduplicatize and rearrange the data asset change events according to time windows to obtain a cleaned event stream, and push the cleaned event stream to the central message queue. The feature construction module is used to consume the cleaned event stream from the central message queue and execute the indicator calculation branch and the tag generation branch in parallel. In the indicator calculation branch, the time-series monitoring indicators of the data assets are updated based on the cleaned event stream. The time-series monitoring indicators include at least one of data integrity, data timeliness, data availability, and data volume volatility. In the tag generation branch, the metadata context corresponding to the cleaned event stream is extracted, candidate tags are generated based on a natural language processing model, and the candidate tags are corrected or conflict arbitration is performed based on a preset rule engine to determine the asset tags. When multiple candidate tags for the same data field hit the preset mutual exclusion pair configuration table and the confidence level is higher than the preset confidence threshold, the corresponding branch arbitration is performed according to the security level class, objective business status class, or timeliness description class to which the mutual exclusion tags belong. When there is a mutual exclusion conflict but the corresponding branch arbitration condition is not met, or the objective data required to perform branch arbitration is missing, the candidate tag activation process is suspended, the current data field is assigned a temporary tag with the highest security level, and a manual review task is generated. The determined asset tags are injected as additional feature dimensions into the time-series monitoring indicators to form a multi-dimensional feature vector. The early warning generation module is used to periodically extract the multi-dimensional feature vector and input it into a pre-trained reconstructive anomaly detection model to calculate the current reconstruction error; obtain historical reconstruction error values ​​within a historical sliding window; calculate the initial median and absolute median difference based on the historical reconstruction error values; when the absolute median difference is zero, determine all historical reconstruction error values ​​as the threshold update reconstruction error dataset; when the absolute median difference is not zero, calculate a correction score based on the historical reconstruction error values, the initial median, and the absolute median difference; remove historical reconstruction error values ​​whose absolute value of the correction score is greater than the extreme deviation threshold; determine the remaining historical reconstruction error values ​​as the threshold update reconstruction error dataset; dynamically update the judgment threshold based on the threshold update reconstruction error dataset; and generate an early warning event carrying an asset identifier and an alert level when the current reconstruction error is greater than the judgment threshold. The local rendering module is used to locate the target graphic element in the asset topology view based on the asset identifier carried by the warning event, and locally update the display status of the target graphic element.

Images