A real-time data quality monitoring method and system based on data assets

Abstract

This application relates to the field of data processing technology, and in particular provides a real-time data quality monitoring method and system based on data assets, which aims to provide enterprises with more efficient and accurate data recovery and risk warning services through a series of innovative steps such as simulating data loss scenarios, debugging data disaster recovery models in a targeted manner, monitoring data risks in real time and optimizing monitoring models. The proposal of this technical solution not only solves many problems existing in traditional data disaster recovery and risk monitoring technologies, but also provides strong technical support for enterprise data security and risk control.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a real-time data quality monitoring method and system based on data assets. Background Art

[0002] In the current digital age, data has become the core asset of business or organization operations. However, with the continuous growth of data volume and the increase of data complexity, data security and integrity are facing unprecedented challenges. Once data is lost or damaged, it may bring huge economic losses and reputation risks to the enterprise.

[0003] Although existing data disaster recovery technologies can recover lost data to a certain extent, they often have problems such as slow recovery speed and low recovery accuracy. At the same time, existing data quality monitoring models also have certain lags and false alarm rates in risk warning, making it difficult to meet the urgent needs of enterprises for data security and risk control. Summary of the invention

[0004] In order to improve the above problems, the present application provides a method and system for real-time monitoring of data quality based on data assets.

[0005] The present application provides a data asset-based real-time data quality monitoring method, which is applied to a data asset monitoring system. The method includes:

[0006] Performing data loss simulation on a first time series data stream example including target business asset data to obtain a first partial data stream example from which the target business asset data is purged;

[0007] Debugging the data disaster recovery project of the target business asset data on the first data disaster recovery model according to the first local data flow example to obtain a debugged first data disaster recovery model;

[0008] Performing data disaster recovery on the second local data flow example through the first data disaster recovery model that has been debugged to obtain a first disaster recovery annotated data flow, and performing abnormal risk identification on the second time series data flow example including the target business asset data through the first data quality monitoring model to obtain a first preliminary risk annotated data flow, wherein the second local data flow example is obtained by simulating data loss on the second time series data flow example;

[0009] Determine a first risk monitoring training error corresponding to the first data quality monitoring model according to the first preliminary risk-annotated data stream and the first disaster recovery-annotated data stream, and optimize the first data quality monitoring model according to the first risk monitoring training error to obtain a debugged first data quality monitoring model;

[0010] Among them, the first data quality monitoring model that has completed debugging is used to perform abnormal risk identification on the time series data stream to be monitored including the target business asset data, and obtain the target time series data stream after the risk behavior of the target business asset data is marked.

[0011] Optionally, debugging the data disaster recovery project of the target business asset data on the first data disaster recovery model according to the first local data flow example to obtain the debugged first data disaster recovery model includes:

[0012] Noising the first time series data stream example to obtain a first noisy data stream example;

[0013] Cross-processing the first local data stream example and the first noise-added data stream example to obtain a first cross-data stream example;

[0014] The first data disaster recovery model is debugged for the data disaster recovery project of the target business asset data according to the first cross data flow example to obtain a debugged first data disaster recovery model.

[0015] Optionally, debugging the data disaster recovery project of the target business asset data on the first data disaster recovery model according to the first cross data flow example to obtain the debugged first data disaster recovery model includes:

[0016] Perform data disaster recovery on the first cross data flow example through a first data disaster recovery model with weighted rollback, to obtain a second disaster recovery annotated data flow corresponding to the first cross data flow example;

[0017] Determine a first disaster recovery annotation training error according to the first time series data stream example and the second disaster recovery annotation data stream;

[0018] The first data disaster recovery model of weight rollback is optimized according to the first disaster recovery annotation training error to obtain the first data disaster recovery model that has completed debugging.

[0019] Optionally, the first data disaster recovery model that has been debugged includes a first feature embedding submodel and a first feature mapping submodel, and performing data disaster recovery on the second local data stream example through the first data disaster recovery model that has been debugged to obtain a first disaster recovery annotated data stream includes:

[0020] Noising the second time series data stream example to obtain a second noisy data stream example;

[0021] Cross-processing the second local data stream example and the second noisy data stream example to obtain a second cross-data stream example;

[0022] The state feature of the second cross data flow example is embedded through the first feature embedding submodel to obtain a first state feature embedding vector, and the first state feature embedding vector is mapped by disaster recovery annotation through the first feature mapping submodel to obtain the first disaster recovery annotation data flow.

[0023] Optionally, performing abnormal risk identification on the second time series data stream example including the target business asset data by using the first data quality monitoring model to obtain a first preliminary risk-labeled data stream includes:

[0024] Performing abnormal risk identification on the second time series data flow example through the first data quality monitoring model, obtaining an abnormal time series data interval corresponding to the target business asset data in the second time series data flow example and a second preliminary risk annotation data flow corresponding to the second time series data flow example;

[0025] According to the abnormal time series data interval in the second time series data flow example, the second preliminary risk annotation data flow and the second time series data flow example are linked and annotated and updated to obtain the first preliminary risk annotation data flow.

[0026] Optionally, determining a first risk monitoring training error corresponding to the first data quality monitoring model according to the first preliminary risk annotated data stream and the first disaster recovery annotated data stream includes:

[0027] Obtaining a priori training data interval, and determining a data interval disaster recovery training error according to the abnormal time series data interval corresponding to the target business asset data in the second time series data stream example and the priori training data interval;

[0028] Determine at least one of the following training errors:

[0029] Determining a global disaster recovery annotation training error according to the first disaster recovery annotation data stream and the first preliminary risk annotation data stream;

[0030] Determining an embedded description disaster recovery training error according to the first disaster recovery annotation data stream and the first preliminary risk annotation data stream;

[0031] Determining a stage disaster recovery annotation training error according to the first disaster recovery annotation data stream, the first preliminary risk annotation data stream, and the prior training data interval;

[0032] Determining a risk category prediction training error based on the first disaster recovery annotation data stream and the first preliminary risk annotation data stream;

[0033] Determining an adversarial training error based on the first preliminary risk labeled data stream;

[0034] Determine a first linkage training error according to at least one of the global disaster recovery annotation training error, the embedded description disaster recovery training error, the stage disaster recovery annotation training error, the risk category prediction training error, and the adversarial training error;

[0035] The first linkage training error and the data interval disaster recovery training error are weightedly summed to obtain the first risk monitoring training error.

[0036] Optionally, obtaining a priori training data interval includes:

[0037] Obtain a first disaster recovery annotation word vector for each data distribution index in the first disaster recovery annotation data stream and a second disaster recovery annotation word vector for each data distribution index in the second time series data stream example;

[0038] For each of the data distribution indexes, the following steps are implemented: determining a first characteristic distance between a first disaster recovery annotation word vector of the data distribution index and a second disaster recovery annotation word vector of the data distribution index;

[0039] Performing interval numerical mapping on the first characteristic distance of each of the data distribution indexes to obtain a third disaster recovery annotation word vector of each of the data distribution indexes;

[0040] Based on the third disaster recovery annotation word vector on each of the data distribution indexes, the priori training data interval is generated.

[0041] Optionally, determining a data interval disaster recovery training error according to an abnormal time series data interval corresponding to the target business asset data and a priori training data interval in the second time series data stream example includes:

[0042] Obtain a first query vector for each data distribution index in the abnormal time series data interval and a second query vector for each data distribution index in the prior training data interval;

[0043] For each of the data distribution indexes, the following steps are implemented: determining a second characteristic distance between a first query vector of the data distribution index and a second query vector of the data distribution index;

[0044] The second characteristic distances of the plurality of data distribution indexes are weighted and summed to obtain the data interval disaster recovery training error.

[0045] Optionally, the method further includes:

[0046] Acquire a time series data stream to be monitored including target business asset data;

[0047] The abnormal risk of the time series data stream to be monitored is identified by using the first data quality monitoring model that has been debugged, so as to obtain the abnormal time series data interval corresponding to the target business asset data in the time series data stream to be monitored and the third preliminary risk annotation data stream corresponding to the time series data stream to be monitored;

[0048] According to the abnormal time series data interval corresponding to the target business asset data in the time series data stream to be monitored, the third preliminary risk marking data stream and the time series data stream to be monitored are linked and marked and updated to obtain the target time series data stream after the target business asset data is marked with risk behaviors.

[0049] An embodiment of the present application provides a data asset monitoring system, comprising at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the above method.

[0050] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. The computer program implements the above method when running.

[0051] The embodiment of the present application aims to provide enterprises with more efficient and accurate data recovery and risk warning services through a series of innovative steps such as simulating data loss scenarios, debugging data disaster recovery models in a targeted manner, monitoring data risks in real time and optimizing monitoring models. The proposal of this technical solution not only solves many problems existing in traditional data disaster recovery and risk monitoring technologies, but also provides strong technical support for enterprise data security and risk control. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flowchart of a method for real-time monitoring of data quality based on data assets provided in an embodiment of the present application.

[0053] Figure 2 A structural diagram of a data asset monitoring system 200 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to better understand the above technical scheme, the technical scheme of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical scheme of the present application, rather than limitations on the technical scheme of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0055] Figure 1A real-time data quality monitoring method based on data assets is shown, which is applied to a data asset monitoring system. The method includes the following steps 202 to 208.

[0056] Step 202: The data asset monitoring system performs a data loss simulation on a first time series data stream example including target business asset data to obtain a first partial data stream example from which the target business asset data is cleaned.

[0057] Step 204: The data asset monitoring system debugs the data disaster recovery project of the target business asset data on the first data disaster recovery model according to the first local data flow example to obtain a debugged first data disaster recovery model.

[0058] Step 206: The data asset monitoring system performs data disaster recovery on the second local data flow example through the first data disaster recovery model that has been debugged to obtain a first disaster recovery labeled data flow, and performs abnormal risk identification on the second time series data flow example including the target business asset data through the first data quality monitoring model to obtain a first preliminary risk labeled data flow, wherein the second local data flow example is obtained by simulating data loss of the second time series data flow example.

[0059] Step 208: The data asset monitoring system determines a first risk monitoring training error corresponding to the first data quality monitoring model based on the first preliminary risk-labeled data stream and the first disaster recovery-labeled data stream, and optimizes the first data quality monitoring model based on the first risk monitoring training error to obtain a first data quality monitoring model that has been debugged.

[0060] Among them, the first data quality monitoring model that has completed debugging is used to perform abnormal risk identification on the time series data stream to be monitored including the target business asset data, and obtain the target time series data stream after the risk behavior of the target business asset data is marked.

[0061] Although the technical solutions described in the embodiments of the present application are not limited to the fields mentioned below, in order to illustrate the scope of its application, the technical contents and operational details included in steps 202 to 208 will be explained through examples in four specific fields, namely, Internet finance, supply chain management, digital office, and smart city.

[0062] In an exemplary financial data monitoring scenario, a data asset monitoring system is used to ensure the integrity and security of business data. The data asset monitoring system first selects a first time series data stream example containing target business asset data (e.g., digital financial asset flow data). In order to test data recovery and disaster recovery capabilities, the data asset monitoring system simulates data loss on this data stream, specifically by randomly or according to certain rules to delete part of the data, thereby obtaining a first partial data stream example that cleans out part of the target business asset data. Next, the data asset monitoring system uses this first partial data stream example to debug a preset first data disaster recovery model. The preset first data disaster recovery model is designed to recover data when data is lost or damaged. The data asset monitoring system tests and adjusts model parameters multiple times to ensure that the model can effectively recover the target business asset data in the event of data loss, and finally obtains a first data disaster recovery model that has been debugged and has a certain ability to recover data loss. The data asset monitoring system then uses the debugged first data disaster recovery model to perform data recovery operations on the second partial data stream example (which is obtained by simulating data loss on the second time series data stream example containing the target business asset data) to generate a first disaster recovery annotation data stream. At the same time, the data asset monitoring system also uses the first data quality monitoring model to perform abnormal risk identification on the complete second time series data stream example, such as detecting abnormal values, mutation points, etc. in the data, thereby generating a first preliminary risk annotation data stream. The data asset monitoring system then compares the first preliminary risk annotation data stream with the first disaster recovery annotation data stream and analyzes the difference between the two. This difference reflects the accuracy of the first data quality monitoring model in risk identification. The data asset monitoring system calculates the risk monitoring training error of the first data quality monitoring model based on these differences, and then optimizes the parameters and algorithms of the model based on this error, and finally obtains a more accurate and reliable first data quality monitoring model. Once the debugged first data quality monitoring model is put into use, the debugged first data quality monitoring model can accurately identify abnormal risks for real-time data streams containing target business asset data. For example, it can promptly detect abnormalities in the flow data of digital financial assets, thereby helping financial institutions to quickly respond to potential network attack risks, which can improve the reliability and security of data.

[0063] In a global supply chain management scenario, the data asset monitoring system plays a vital role in ensuring the integrity and security of supply chain data. In a supply chain environment, the target business asset data may include key data such as inventory, order status, and transportation information. The data asset monitoring system first selects a first time series data stream example containing these target business asset data, such as inventory change data of a product over a period of time. In order to test the data recovery and disaster recovery capabilities of the data asset monitoring system, the data asset monitoring system simulates data loss on this time series data stream. In this step, the data asset monitoring system simulates various suspicious data loss situations, such as data transmission errors, storage failures, etc., to obtain a first partial data stream example that cleans up some of the target business asset data. The data asset monitoring system then uses this cleaned first partial data stream example to debug the preset first data disaster recovery model. This disaster recovery model is designed specifically for supply chain data and is used to quickly recover when data is lost or damaged. By simulating data loss multiple times and testing the recovery effect, the data asset monitoring system continuously adjusts the model's parameters and recovery strategy, and finally obtains a first data disaster recovery model that can effectively deal with supply chain data loss situations. Next, the data asset monitoring system uses the debugged first data disaster recovery model to perform data recovery operations on another second local data stream example that has undergone data loss simulation. This data stream may contain supply chain data from another time period. The recovered data is marked as the first disaster recovery annotated data stream. At the same time, the data asset monitoring system also uses the first data quality monitoring model to perform abnormal risk identification on the complete second time series data stream example, such as detecting abnormal fluctuations in inventory, abnormal changes in order status, etc., thereby generating a first preliminary risk annotated data stream. The data asset monitoring system then compares the first preliminary risk annotated data stream with the first disaster recovery annotated data stream and analyzes the differences between the two. These differences reflect the accuracy of the first data quality monitoring model in risk identification. Based on these differences, the data asset monitoring system calculates the risk monitoring training error of the first data quality monitoring model, which reveals the deviation of the model in identifying supply chain data anomalies. Subsequently, the data asset monitoring system adjusts and optimizes the parameters and algorithms of the model based on this error to make it more adaptable to the characteristics of supply chain data, and finally obtains a more accurate and reliable first data quality monitoring model. The first data quality monitoring model that has been debugged can monitor the supply chain data flow containing the target business asset data in real time, and promptly detect and mark abnormal risks. For example, when the inventory of a product suddenly drops sharply while the order volume remains stable, the model will quickly identify this anomaly and issue an early warning. In this way, the data asset monitoring system not only improves the reliability and security of supply chain data, but also provides strong support for the stable operation and risk management of enterprises.

[0064] In the field of digital office, the data asset monitoring system is an important tool to ensure the security and integrity of key business data of an enterprise. The data asset monitoring system first selects the first time series data stream example containing the target business asset data, which can include employee work efficiency data, project completion status, document modification records, etc. In order to test the effectiveness of the data recovery strategy, the data asset monitoring system will simulate data loss for this data stream. Specifically, the data asset monitoring system will simulate various suspicious situations such as data damage and loss, so as to obtain a first partial data stream example that has been cleaned of some target business asset data. Next, the data asset monitoring system uses this cleaned data stream example to start debugging the first data disaster recovery model. This model was originally designed to quickly recover key business data when data is accidentally lost or damaged. The data asset monitoring system simulates data loss scenarios multiple times and continuously adjusts the model's parameters and recovery algorithms to ensure that the model can effectively recover the target business asset data in real data loss events. After multiple tests and adjustments, the data asset monitoring system obtains a debugged and highly reliable first data disaster recovery model. Once the first data disaster recovery model is debugged, the data asset monitoring system will use this model to perform data recovery operations on another second local data stream example that has undergone data loss simulation. This data stream may contain office data from another time period, such as employee time records, project progress, etc. The data recovered by the first data disaster recovery model is marked as the first disaster recovery annotation data stream. At the same time, the data asset monitoring system will also use the first data quality monitoring model to identify abnormal risks for the complete second time series data stream example. This model can detect potential risks such as abnormal values ​​and irregular fluctuations in the data, thereby generating a first preliminary risk annotation data stream. The purpose of this step is to verify the accuracy of the data quality monitoring model and provide a basis for its subsequent optimization. After obtaining the first disaster recovery annotation data stream and the first preliminary risk annotation data stream, the data asset monitoring system will compare and analyze the two. By comparing the two data streams, the data asset monitoring system can calculate the risk monitoring training error of the first data quality monitoring model. This error reflects the accuracy of the model in identifying abnormal risks of office data. Based on this error, the data asset monitoring system will optimize the first data quality monitoring model. The optimization process may include adjusting the parameters of the model, improving the anomaly detection algorithm, etc. After optimization, the data asset monitoring system has obtained a more accurate and reliable first data quality monitoring model. The first data quality monitoring model that has been debugged will be used to monitor the office data flow containing the target business asset data in real time. For example, when an employee's working time record shows abnormal fluctuations, the model can quickly identify and issue an early warning, thereby helping the company to promptly discover and solve potential problems.In this way, the data asset monitoring system not only improves the reliability and security of office data, but also provides strong support for the company's efficient operation and risk management.

[0065] In the construction and management of smart cities, data asset monitoring systems play a vital role, especially in ensuring the integrity and security of urban operation data. In a smart city environment, target business asset data can include key information such as traffic flow data, environmental monitoring data, and public facility usage status. The data asset monitoring system first selects an example of a first time series data stream containing these target business asset data, such as the traffic flow data of a major intersection in the city within a week. In order to test the data recovery capability of the data asset monitoring system, the data asset monitoring system simulates data loss on this data stream. The data asset monitoring system simulates data loss or transmission errors by deleting or damaging part of the data, thereby obtaining an example of a first partial data stream that cleans part of the target business asset data. Next, the data asset monitoring system uses the cleaned first partial data stream example to debug the preset first data disaster recovery model. This disaster recovery model is designed to quickly recover data when data is accidentally lost. The data asset monitoring system tests and adjusts the parameters of the model multiple times to ensure that the model can effectively recover the target business asset data after data loss. After repeated verification and optimization, a first data disaster recovery model that has been debugged for the characteristics of smart city data is finally obtained. The data asset monitoring system then uses the debugged first data disaster recovery model to perform data recovery on another second local data stream example that has been simulated for data loss. This data stream may contain traffic flow data for another time period. After being processed by the first data disaster recovery model, the obtained data is marked as the first disaster recovery annotated data stream. At the same time, the data asset monitoring system also uses the first data quality monitoring model to perform abnormal risk discrimination on the complete second time series data stream example, such as detecting abnormal fluctuations in traffic flow, thereby generating a first preliminary risk annotated data stream. The data asset monitoring system then compares the first preliminary risk annotated data stream with the first disaster recovery annotated data stream and analyzes the differences between the two. These differences reflect the accuracy of the data quality monitoring model in risk discrimination. Based on these differences, the data asset monitoring system calculates the risk monitoring training error of the data quality monitoring model. This error reveals the accuracy of the model in identifying anomalies in smart city data. Subsequently, the data asset monitoring system uses this error to optimize the parameters and algorithms of the data quality monitoring model to make it more adaptable to the characteristics of smart city data. Finally, a more accurate and reliable first data quality monitoring model is obtained. The first data quality monitoring model that has been debugged will be put into practical use to monitor the smart city data stream containing the target business asset data in real time. For example, when the traffic flow at a certain intersection suddenly increases abnormally, the model can quickly identify and issue an early warning to help city managers respond to potential traffic congestion problems in a timely manner. In this way, the data asset monitoring system not only improves the reliability and security of smart city data, but also provides strong support for the efficient operation and risk management of the city.

[0066] The following is a detailed description of steps 202 to 208 in conjunction with the above four application scenarios.

[0067] With respect to step 202, in the field of Internet finance, the target business asset data may be key business information such as investors' transaction records, loan data, and user credit scores; in the field of supply chain, it may include inventory data, logistics information, and transaction history of suppliers and customers; in the field of digital office, such data may involve employee work efficiency, document editing and sharing records, project management data, etc.; and in the field of smart cities, the target business asset data may include real-time data related to urban operations such as traffic flow, air quality monitoring, and the use of public facilities.

[0068] The first time series data stream example is a data stream sample arranged in chronological order for analysis and testing. In Internet finance, it can be a continuous record of stock prices or trading volumes over a period of time; in the field of supply chain, it can be data on cargo transportation and inventory changes over a period of time; in a digital office environment, it can be continuous data on employees' work logs or project progress over a period of time; and in smart cities, the first time series data stream example can be urban traffic flow or environmental monitoring data for several consecutive days.

[0069] Data loss simulation is a testing method that simulates data loss situations that may occur in a real environment by artificially deleting or damaging some data. In Internet finance, this can involve simulating the loss of transaction records; in the supply chain field, it can be simulating the damage of inventory data; in the digital office environment, it can be simulating the loss of project documents; in the smart city field, it can be simulating the loss of environmental monitoring data.

[0070] The first partial data flow example is a data sample obtained after data loss simulation processing, in which part of the target business asset data has been cleaned or lost. This example is used to test the data recovery strategy and the disaster recovery capability of the data asset monitoring system. In various fields, this partial data flow example will help enterprises and city managers evaluate response strategies in the event of data loss.

[0071] In detail, step 202 is an important step in the data asset monitoring system when processing key business data, and its purpose is to test the recovery capability of the data asset monitoring system in the event of data loss or damage.

[0072] In the field of Internet finance, the data asset monitoring system will select a first time series data stream example containing target business asset data such as user transaction records and market dynamics. These data are crucial for risk assessment and investment strategy formulation of financial institutions. The data asset monitoring system will simulate data loss on these data streams, such as randomly deleting certain transaction records or market data points, to obtain a first partial data stream example that has been cleaned of some key information. This is done to verify whether the data asset monitoring system can effectively restore or reconstruct these data in the case of incomplete data, so as to ensure the accuracy of financial analysis and decision-making.

[0073] In the supply chain field, this step can involve similar operations on key data such as inventory changes and logistics information. The data asset monitoring system will simulate data loss, such as deleting inventory data or transportation records at certain points in time, to generate the first partial data flow example. In this way, supply chain managers can test the effectiveness of their data recovery mechanisms and disaster recovery plans to ensure that operations can be quickly restored in the event of a real data loss.

[0074] In the field of digital office, the data asset monitoring system will select time-series data streams containing information such as employee work efficiency and project progress for simulation. By deliberately losing some data, such as deleting work logs or project progress updates for a specific time period, the data asset monitoring system generates the first partial data stream example. Such tests help companies evaluate how to quickly recover and ensure that office processes are not affected when data is accidentally lost.

[0075] In the field of smart cities, this step can be performed for urban operational data such as traffic flow and environmental monitoring. By simulating data loss, such as deleting traffic flow records or environmental monitoring data for a few days, the first local data flow example is generated. City managers can use this example to test their data recovery and disaster recovery strategies to ensure that data can be quickly recovered when lost, ensuring the continuity and effectiveness of urban management.

[0076] With respect to step 204, the first data disaster recovery model is a pre-built data recovery model used to recover the target business asset data in the event of data loss or damage. In the field of Internet finance, this can be a model that can rebuild transaction records or user data; in the field of supply chain, it can focus on recovering inventory or logistics data; in the digital office environment, it can be used to recover lost documents or project data; and in the field of smart cities, this model can be used to recover lost environmental monitoring data or traffic flow information. This model can be based on a series of algorithms and technologies, such as data redundant storage, backup strategies, and data recovery algorithms, to ensure that data can be recovered to the greatest extent possible in the event of a disaster.

[0077] The debugging of the data disaster recovery project for the target business asset data is a process of testing and optimizing the data disaster recovery model to ensure that it can effectively recover the specific target business asset data. The debugging process includes testing the model's recovery capabilities using various data loss scenarios and adjusting the model's parameters and strategies based on the test results. This process is crucial in areas such as Internet finance, supply chain, digital office, and smart cities, as it is directly related to data security and business continuity.

[0078] The first data disaster recovery model that has been debugged is a debugged and optimized data disaster recovery model that has been verified through a series of tests to recover the target business asset data. This model can now be deployed in the production environment to provide protection for actual data loss events. In various fields, this debugged model is a key component to ensure data security and business continuity.

[0079] Furthermore, step 204 is a key step in the data asset monitoring system in building a robust data recovery strategy. In this step, the data asset monitoring system uses the first local data flow example obtained through the data loss simulation to debug the first data disaster recovery model.

[0080] In the field of Internet finance, the data asset monitoring system will use cleaned transaction records or market dynamic data to test the recovery capabilities of the disaster recovery model. For example, if some key transaction data is lost, the data asset monitoring system will try to use the disaster recovery model to recover this data. By comparing the recovered data with the original data, the data asset monitoring system can evaluate the accuracy and efficiency of the model and make adjustments as needed.

[0081] In the supply chain field, the debugging process can involve testing the disaster recovery model using simulated lost inventory or logistics data. The data asset monitoring system observes the performance of the model when restoring this data and adjusts the model's parameters based on the accuracy and completeness of the recovery results.

[0082] In the digital office field, this can be understood as using documents or project progress data that have lost some of their content to test the disaster recovery model. The data asset monitoring system will try to rebuild these lost parts and evaluate the quality of the recovery. Based on the test results, the data asset monitoring system can fine-tune the model to improve its ability to recover lost data.

[0083] In the smart city space, the debugging process can include testing the disaster recovery model using simulated lost environmental monitoring data or traffic flow information. By comparing the recovered data with the original data, the data asset monitoring system can evaluate the performance of the model and make necessary adjustments to ensure that data can be quickly and accurately restored in the event of a real data loss.

[0084] Through the debugging and optimization of this step, the data asset monitoring system will obtain a verified disaster recovery model that can effectively recover the target business asset data. This first data disaster recovery model that has been debugged will provide a solid foundation for subsequent data protection and recovery work.

[0085] For step 206, the second local data stream example is another data sample that has been processed by data loss simulation, which is similar to the first local data stream example, but may include different data points or time periods. In Internet finance, it may be transaction data of another period of time; in the field of supply chain, it may involve logistics information of another batch of goods; in the field of digital office, it may be workflow data of another project; and in the field of smart city, it may be traffic monitoring data of another period of time. These data stream examples are used to further test and optimize the data disaster recovery model and data quality monitoring model.

[0086] The first disaster recovery annotated data stream is the data stream obtained after the debugged first data disaster recovery model recovers the second local data stream example. These data have been processed by the model, and the data are marked which are recovered and which are retained, which helps to evaluate the actual effect of the data disaster recovery model. In various fields, this annotated data stream is crucial to verify the effectiveness of disaster recovery strategies.

[0087] The first data quality monitoring model is a model used to monitor and evaluate data quality. It can identify anomalies, errors or inconsistencies in data and issue alerts in a timely manner. In Internet finance, it can be used to detect abnormal fluctuations in transaction data; in the supply chain field, it can be used to monitor the accuracy of inventory data; in the digital office field, it can be used to ensure the integrity of project data; in the smart city field, it can be used to monitor the rationality of environmental data in real time.

[0088] The second time series data stream example is a complete, unprocessed time series data stream sample, which is used as a benchmark for data loss simulation and subsequent data quality monitoring. In various fields, this data stream example represents the data that should be collected under normal circumstances and is an important basis for evaluating data recovery and data quality.

[0089] The first preliminary risk-annotated data stream is a data stream obtained after the first data quality monitoring model performs abnormal risk identification on the second time series data stream example. The model will mark possible risk points or abnormal data for subsequent analysis and processing. This annotated data stream helps to promptly discover and handle problems in the data, and improve the reliability and accuracy of the data.

[0090] In detail, in step 206, the data asset monitoring system uses the previously debugged first data disaster recovery model to perform data recovery on the second local data flow example, and uses the first data quality monitoring model to perform abnormal risk identification on the complete second time series data flow example.

[0091] In the field of Internet finance, the data asset monitoring system will try to use the data disaster recovery model to recover a time series data stream that may have lost some transaction data, and obtain the first disaster recovery annotation data stream. At the same time, the data quality monitoring model will scan the complete financial market data, mark any abnormal transactions or market fluctuations, and generate the first preliminary risk annotation data stream.

[0092] In the supply chain field, this step can involve using data disaster recovery models to recover simulated lost inventory or logistics data, and using data quality monitoring models to detect anomalies in the data flow of the entire supply chain, such as abnormal delays in goods or sudden changes in inventory.

[0093] In the field of digital office, the data asset monitoring system will use the data disaster recovery model to try to recover lost project data or documents, and use the data quality monitoring model to detect data anomalies in the office process, such as abnormal changes in file modification permissions or unreasonable delays in project progress.

[0094] In the field of smart cities, this step can include using data disaster recovery models to restore simulated lost environmental monitoring data, and using data quality monitoring models to monitor urban operation data in real time to promptly detect and mark any abnormal data points, such as a sudden deterioration in air quality or an abnormal increase in traffic flow.

[0095] Through the implementation of this step, the data asset monitoring system can comprehensively evaluate the effectiveness of its data recovery and data quality monitoring strategies, and provide strong support for subsequent data protection and management.

[0096] With respect to step 208, the first risk monitoring training error refers to the difference between the risk predicted by the model and the actual risk during the training of the data quality monitoring model. Specifically, in the field of Internet finance, it can be manifested as the gap between the transaction risk predicted by the model and the actual transaction loss; in the field of supply chain, it can be the difference between the supply chain disruption risk predicted by the model and the actual delay or damage; in the field of digital office, it can reflect the discrepancy between the project risk predicted by the model and the actual project progress problem; in the field of smart cities, this error can be reflected in the gap between the urban operation risk predicted by the model and the actual traffic congestion, environmental pollution and other problems.

[0097] The first data quality monitoring model that has been debugged is an optimized and adjusted data quality monitoring model, and its accuracy and efficiency have been improved through a series of training and verification processes. In the field of Internet finance, this model can more accurately identify potential transaction risks; in the field of supply chain, it can more effectively predict potential problems in the supply chain; in the field of digital office, the model can more accurately assess the risk points of project management; in the field of smart cities, it can more accurately monitor and warn of various risks in urban operations.

[0098] Finally, in step 208, the data asset monitoring system uses the first preliminary risk annotation data stream and the first disaster recovery annotation data stream generated previously to determine the training error of the first data quality monitoring model in terms of risk monitoring.

[0099] In the field of Internet finance, the data asset monitoring system compares the transaction risk predicted by the model with the risk in the actual transaction, and calculates the error of the model in predicting risk. For example, if the model fails to accurately predict the high risk of a transaction, then this is a training error. The data asset monitoring system will adjust the model's parameters and algorithms based on these errors so that it can more accurately predict transaction risks in the future.

[0100] In the supply chain field, the data asset monitoring system determines the training error by comparing the supply chain risks predicted by the model with the actual supply chain problems that occurred. For example, if the model fails to detect the risk of a supplier's delivery delay in a timely manner, then this is a point that needs to be optimized. By adjusting the model, the data asset monitoring system can improve its ability to predict potential problems in the supply chain.

[0101] In the digital office world, this step can involve evaluating the model’s accuracy in predicting project management risks. If the model fails to accurately identify a project’s key risk points, the data asset monitoring system will adjust the model based on this error to improve the accuracy of its risk predictions.

[0102] In the field of smart cities, the data asset monitoring system will determine the training error by comparing the urban operation risks predicted by the model with actual urban problems, such as traffic congestion and environmental pollution. Through the feedback of these errors, the data asset monitoring system can optimize the model so that it can more accurately monitor and warn of various risks in urban operations in the future.

[0103] Through the optimization and adjustment of this step, the data asset monitoring system will obtain a verified data quality monitoring model that can more accurately predict risks. This first data quality monitoring model that has been debugged will provide more reliable support for subsequent data risk management and early warning work.

[0104] The embodiment of the present application significantly improves the efficiency and accuracy of data disaster recovery and risk monitoring through a series of refined data processing and model optimization processes. First, by simulating data loss on the first time series data stream of the target business asset data, a more realistic data recovery scenario is generated, so that the subsequent data disaster recovery model debugging is closer to the actual situation. This step not only enhances the practicality of the data disaster recovery model, but also improves the reliability of data recovery. Secondly, the first data disaster recovery model is debugged using the first local data stream example after cleaning, ensuring that the model can quickly and accurately restore the target business asset data when facing data loss. This targeted debugging method greatly improves the efficiency and success rate of data disaster recovery, and provides a strong data recovery guarantee for enterprises or organizations when data disasters occur. In addition, the embodiment of the present application also uses the first data quality monitoring model to identify abnormal risks on the second time series data stream, and timely discovers and marks potential data risks. This risk warning mechanism not only helps enterprises to respond to possible data risks in a timely manner, but also effectively avoids huge losses caused by data leakage or damage. Finally, by combining the first preliminary risk annotation data stream and the first disaster recovery annotation data stream, the first data quality monitoring model is accurately optimized. This optimization method based on actual risk-labeled data significantly improves the accuracy and sensitivity of the data quality monitoring model, enabling it to more accurately identify and warn of data risks.

[0105] In summary, the embodiments of the present application achieve efficient integration of data disaster recovery and risk monitoring through a series of innovative data processing and model optimization methods, providing strong technical support for enterprises or organizations in data management and risk control.

[0106] For ease of introduction, the following further introduces the relevant embodiments using the supply chain field as an application scenario. It should be understood that the subsequent relevant embodiments are also applicable to the above-mentioned Internet finance field, digital office field, smart city field, etc.

[0107] In some possible embodiments, the first data disaster recovery model is debugged for the data disaster recovery project of the target business asset data based on the first local data flow example to obtain the debugged first data disaster recovery model, including: adding noise to the first timing data flow example to obtain a first noisy data flow example; cross-processing the first local data flow example and the first noisy data flow example to obtain a first cross-data flow example; debugging the data disaster recovery project of the target business asset data on the first data disaster recovery model based on the first cross-data flow example to obtain the debugged first data disaster recovery model.

[0108] In the field of supply chain, in the implementation of the data asset monitoring system, the process of debugging the first data disaster recovery model is particularly important, because the data mobility and real-time requirements in the supply chain are extremely high, and any data loss or damage may lead to supply chain interruption, thereby causing serious economic losses.

[0109] In order to improve the recovery capability and stability of the data disaster recovery model in actual applications, the data asset monitoring system will first perform noise processing on the first time series data stream example. This step is to add a certain amount of noise to the original data stream by simulating the data interference or damage that may occur in the real environment, thereby generating the first noisy data stream example. The purpose of this is to allow the data disaster recovery model to be exposed to more complex and changeable data situations during the training process, thereby enhancing its robustness in actual applications.

[0110] Next, the data asset monitoring system will cross-process the first local data stream example and the first noise-added data stream example after cleaning the target business asset data. The purpose of this step is to create a more complex and realistic data recovery environment. Through cross-processing, the system combines data from different sources and of different quality to form the first cross-data stream example. Such a data stream contains the characteristics of both the original data and the characteristics of the noise-added data, providing a more comprehensive training set for the data disaster recovery model.

[0111] After completing the above preparations, the data asset monitoring system will conduct targeted debugging of the first data disaster recovery model based on the first cross data flow example. In this process, the system will continuously adjust the parameters and structure of the model to enable it to better adapt to the complex and changing data environment and improve the ability to recover the target business asset data. After repeated training and verification, the system will eventually obtain a debugged first data disaster recovery model.

[0112] This first data disaster recovery model that has been debugged has shown significant superiority in practical applications. Since it has been exposed to and adapted to various complex data situations during the training process, it can quickly and accurately restore the target business asset data when faced with data loss or damage in the actual supply chain. This not only greatly improves the reliability and stability of supply chain data, but also helps enterprises avoid supply chain interruptions and economic losses that may be caused by data loss.

[0113] In this way, by introducing the technical means of noise addition and cross-processing during the debugging process, the data asset monitoring system successfully improved the recovery capability and robustness of the first data disaster recovery model. This innovative implementation not only enhances the security of supply chain data, but also provides enterprises with a more efficient and reliable data recovery solution.

[0114] In the next step, the first data disaster recovery model is debugged for the data disaster recovery project of the target business asset data based on the first cross-data flow example to obtain the first data disaster recovery model that has been debugged, including: performing data disaster recovery on the first cross-data flow example through the first data disaster recovery model with weight rollback to obtain the second disaster recovery annotation data flow corresponding to the first cross-data flow example; determining the first disaster recovery annotation training error based on the first timing data flow example and the second disaster recovery annotation data flow; optimizing the first data disaster recovery model with weight rollback based on the first disaster recovery annotation training error to obtain the first data disaster recovery model that has been debugged.

[0115] In the supply chain field, in order to ensure rapid recovery when a data disaster occurs, the data asset monitoring system needs to fine-tune the data disaster recovery model. Next, we will introduce in detail how the data asset monitoring system debugs the first data disaster recovery model through a series of steps and obtains the debugged first data disaster recovery model.

[0116] First, the data asset monitoring system uses the first cross-data flow example generated previously to train the first data disaster recovery model. The first data disaster recovery model here uses weight rollback technology, which is a common technology in the model training process. By saving the weights of the model during the training process, it can be rolled back to the previous weight state when necessary, thereby avoiding model overfitting or falling into a local optimal solution.

[0117] The first data disaster recovery model with weight rollback (initialized first data disaster recovery model) is used to perform data disaster recovery on the first cross data flow example, and the system will obtain the corresponding second disaster recovery annotated data flow. This process is equivalent to simulating a recovery operation after a data disaster, and the second disaster recovery annotated data flow represents the data state after recovery.

[0118] Next, the data asset monitoring system will compare the original first time series data stream example with the second disaster recovery annotation data stream to determine the first disaster recovery annotation training error. This error reflects the accuracy and efficiency of the model in the data recovery process. The smaller the error, the stronger the model's recovery ability.

[0119] Finally, based on the training error of the first disaster recovery annotation, the data asset monitoring system will optimize the first data disaster recovery model with weight rollback. The optimization process may include adjusting the parameters of the model, changing the structure of the model, or updating the algorithm of the model. After repeated training and optimization, the system will eventually obtain a first data disaster recovery model that has been debugged.

[0120] This first data disaster recovery model that has been debugged can restore data in the supply chain more quickly and accurately in actual applications. In the supply chain field, in other words, when data is lost or damaged, the company can quickly resume normal business processes and reduce losses caused by data interruptions.

[0121] In this way, the data asset monitoring system successfully improved the recovery capability and accuracy of the first data disaster recovery model by introducing weight rollback technology, using the first cross-data stream example for training, and continuously optimizing the model. This embodiment not only provides a strong guarantee for the security of supply chain data, but also brings a more efficient and reliable data recovery solution to the enterprise. Such a technical solution significantly improves the stability of the supply chain and the continuity of business, creating greater value for the enterprise.

[0122] In some replaceable embodiments, the first data disaster recovery model that has been debugged includes a first feature embedding sub-model and a first feature mapping sub-model, and the first data disaster recovery model that has been debugged is used to perform data disaster recovery on the second local data flow example to obtain a first disaster recovery labeled data flow, including: adding noise to the second timing data flow example to obtain a second noisy data flow example; cross-processing the second local data flow example and the second noisy data flow example to obtain a second cross-data flow example; embedding state features of the second cross-data flow example through the first feature embedding sub-model to obtain a first state feature embedding vector, and performing disaster recovery labeling mapping on the first state feature embedding vector through the first feature mapping sub-model to obtain the first disaster recovery labeled data flow.

[0123] Exemplarily, the data disaster recovery function of the data asset monitoring system is particularly important because it is related to the continuous operation of the entire supply chain and the stability of the business. In this alternative embodiment, the data asset monitoring system uses the debugged first data disaster recovery model in an innovative way. The debugged first data disaster recovery model consists of a first feature embedding sub-model and a first feature mapping sub-model, which work together to complete the data disaster recovery task.

[0124] Before performing data disaster recovery, the system first adds noise to the second time series data stream example to simulate data interference or corruption that may be encountered in a real environment. By adding a certain amount of noise, the system generates a second noisy data stream example. The benefit of this is that it makes the data disaster recovery model more robust when processing real data because the model has been exposed to and adapted to noisy data during the training process.

[0125] Next, the data asset monitoring system cross-processes the second local data stream example and the second noisy data stream example to generate a second cross-data stream example. This cross-processing method helps create a more complex data set that is closer to the actual scenario, which is used to test the performance and accuracy of the data disaster recovery model.

[0126] After preparing the second cross-data stream example, the data asset monitoring system will use the first feature embedding sub-model to embed its state features. The purpose of this step is to convert the original data into a high-dimensional feature vector, namely the first state feature embedding vector, which can more comprehensively describe the intrinsic state and characteristics of the data.

[0127] Subsequently, the system will use the first feature mapping sub-model to map the first state feature embedding vector to a disaster recovery annotation. This mapping process maps the high-dimensional feature vector to the corresponding disaster recovery annotation based on the knowledge learned by the model during the training phase, thereby obtaining the first disaster recovery annotation data stream. This data stream not only contains the recovery information of the original data, but also annotates the state of the data during the disaster recovery process, providing an important reference for subsequent data recovery and business continuity.

[0128] In this way, the data asset monitoring system can more effectively handle data disaster recovery issues in the supply chain. First, through noise addition and cross-processing, the system generates training and test data that is closer to the actual scenario, improving the generalization ability of the data disaster recovery model; second, by using feature embedding and mapping technology, the system can more accurately capture the intrinsic state and characteristics of the data, thereby improving the accuracy and efficiency of data recovery; finally, this innovative data disaster recovery method significantly enhances the stability and reliability of supply chain data, providing enterprises with strong data security protection.

[0129] In some alternative embodiments, the method of performing abnormal risk identification on the second time series data flow example including the target business asset data through the first data quality monitoring model to obtain a first preliminary risk labeled data flow includes: performing abnormal risk identification on the second time series data flow example through the first data quality monitoring model to obtain an abnormal time series data interval corresponding to the target business asset data in the second time series data flow example and a second preliminary risk labeled data flow corresponding to the second time series data flow example; and performing linkage labeling and updating on the second preliminary risk labeled data flow and the second time series data flow example based on the abnormal time series data interval in the second time series data flow example to obtain the first preliminary risk labeled data flow.

[0130] In the supply chain field, the data asset monitoring system continuously monitors the quality of the data flow to ensure the accuracy and timeliness of the supply chain information. In some alternative embodiments, the system uses the first data quality monitoring model to perform fine abnormal risk identification on the second time series data flow example containing the target business asset data.

[0131] First, the data asset monitoring system will use the first data quality monitoring model to comprehensively scan and identify the second time series data stream example. This model is specially designed to identify anomalies in data streams. It can automatically detect data intervals that do not conform to normal patterns based on historical data and preset rules. These intervals may represent potential risks or errors.

[0132] During the scanning process, the model will pay special attention to the parts related to the target business asset data. Once an anomaly is detected, the system will record the specific location and range of these abnormal time series data intervals. At the same time, the model will also generate a second preliminary risk annotation data stream, which is an annotated version of the original second time series data stream, marking which data points or intervals are abnormal.

[0133] However, the initial risk annotation may not be completely accurate or comprehensive, so the data asset monitoring system will further update the linkage annotation. Specifically, the system will synchronize the annotation updates of the second preliminary risk annotation data stream and the second time series data stream example based on the previously identified abnormal time series data interval. During this process, the system can adjust the precise location of the annotation, add new annotations, or correct incorrect annotations.

[0134] Finally, the data stream that has been updated through linkage annotation is called the first preliminary risk annotation data stream. This data stream not only accurately reflects the abnormalities in the original data, but also provides important reference information for subsequent data processing and analysis.

[0135] In this way, the data asset monitoring system can more effectively identify potential risks in the supply chain data flow, so as to take timely countermeasures to ensure the integrity and reliability of the data. This refined risk identification method significantly improves the intelligence level of supply chain management, bringing a more robust operating environment and more efficient decision support to enterprises. At the same time, by reducing data errors and delays, this technical solution also helps to improve the overall performance of the supply chain and customer satisfaction.

[0136] Under some other replaceable design ideas, the first risk monitoring training error corresponding to the first data quality monitoring model is determined based on the first preliminary risk labeling data stream and the first disaster recovery labeling data stream, including: (1) obtaining a priori training data interval, and determining a data interval disaster recovery training error based on the abnormal time series data interval and the priori training data interval corresponding to the target business asset data in the second time series data stream example; (2) determining at least one of the following training errors: determining a global disaster recovery labeling training error based on the first disaster recovery labeling data stream and the first preliminary risk labeling data stream; determining an embedded description disaster recovery training error based on the first disaster recovery labeling data stream and the first preliminary risk labeling data stream; and determining a data interval disaster recovery training error based on the first disaster recovery labeling data stream and the first preliminary risk labeling data stream. The first disaster recovery annotation data stream, the first preliminary risk annotation data stream and the prior training data interval are used to determine the stage disaster recovery annotation training error; based on the first disaster recovery annotation data stream and the first preliminary risk annotation data stream, the risk category prediction training error is determined; based on the first preliminary risk annotation data stream, the adversarial training error is determined; (3) based on the global disaster recovery annotation training error, the embedded description disaster recovery training error, the stage disaster recovery annotation training error, the risk category prediction training error, and at least one of the adversarial training errors, the first linkage training error is determined; (4) weighted summation of the first linkage training error and the data interval disaster recovery training error is performed to obtain the first risk monitoring training error.

[0137] In the field of supply chain, one of the core functions of the data asset monitoring system is risk monitoring, which involves continuous monitoring and analysis of supply chain data to ensure the stability and security of the business. Under some alternative design ideas, the system uses complex algorithms to determine the first risk monitoring training error of the first data quality monitoring model. This process is very critical because it is directly related to the accuracy and effectiveness of risk monitoring.

[0138] First, the system obtains the prior training data intervals, which are pre-set based on historical data and expert knowledge, and represent normal or abnormal patterns that may occur in supply chain data under different circumstances. Then, the system combines the abnormal time series data interval corresponding to the target business asset data in the second time series data stream example with the prior training data interval to determine the data interval disaster recovery training error. This error reflects the accuracy of the model in identifying and processing specific data intervals.

[0139] Next, the system calculates multiple training errors, each of which evaluates the performance of the model from a different perspective. The global disaster recovery labeling training error examines the model's disaster recovery labeling capabilities across the entire data stream; the embedded description disaster recovery training error focuses on the model's disaster recovery performance after embedding data into a high-dimensional space; the stage disaster recovery labeling training error evaluates different stages of the data stream; the risk category prediction training error tests the model's prediction accuracy for risk categories; and the adversarial training error tests the robustness of the model by introducing adversarial samples.

[0140] After determining these training errors, the system will determine the first linkage training error based on one or more of them. This linkage training error is a comprehensive indicator that takes into account the performance of the model in multiple aspects.

[0141] Finally, the system will perform a weighted summation of the first linkage training error and the data interval disaster recovery training error to obtain the final first risk monitoring training error. This error value is a comprehensive evaluation indicator that reflects the overall performance of the model in the risk monitoring task.

[0142] Through this method, the data asset monitoring system can accurately evaluate the performance of its risk monitoring model and make corresponding optimizations and adjustments based on the evaluation results. This not only improves the model's ability to identify and respond to supply chain risks, but also provides a strong guarantee for the company's stable operation. At the same time, this refined risk assessment method also helps companies to promptly identify and resolve potential supply chain problems, thereby improving the efficiency and reliability of the entire supply chain.

[0143] In actual implementation, an exemplary calculation method of each of the above training errors is as follows.

[0144] Data Interval Disaster Recovery Training Error (DIDRTE):

[0145] (DIDRTE=frac{1}{N}sum_{i=1}^{N}|y_i^{true}-y_i^{pred}|);

[0146] Where (N) is the number of abnormal time series data intervals, (y_i^{true}) is the true disaster recovery label of the (i)th data interval, and (y_i^{pred}) is the disaster recovery label of the (i)th data interval predicted by the model.

[0147] Global Disaster Recovery Annotation Training Error (GDRATE):

[0148] (GDRATE=frac{1}{M}sum_{j=1}^{M}|z_j^{true}-z_j^{pred}|);

[0149] Among them, (M) is the number of data points in the first disaster recovery annotation data stream, (z_j^{true}) is the true disaster recovery annotation of the (j)th data point, and (z_j^{pred}) is the disaster recovery annotation of the (j)th data point predicted by the model.

[0150] Embedded Description Disaster Recovery Training Error (EDDRTE):

[0151] (EDDRTE=frac{1}{M}sum_{j=1}^{M}||mathbf{e}_j^{true}-mathbf{e}_j^{pred}||_2);

[0152] where (mathbf{e}_j^{true}) and (mathbf{e}_j^{pred}) are the true and predicted embedding vectors of the (j)th data point, respectively, and (||cdot||_2) represents the L2 norm.

[0153] Phased Disaster Recovery Annotation Training Error (PDRATE), for each prior training data interval (k), calculate the phased disaster recovery annotation training error in the interval:

[0154] (PDRATE_k=frac{1}{|I_k|}sum_{linI_k}|r_l^{true}-r_l^{pred}|);

[0155] Where (I_k) is the index set of data points in the (k)th prior training data interval, (r_l^{true}) and (r_l^{pred}) are the true and predicted phase disaster recovery labels of the (l)th data point, respectively. The overall PDRATE can be the average of all (PDRATE_k).

[0156] Risk Category Prediction Training Error (RCPTE): (RCPTE = frac{1}{M}sum_{j=1}^{M}text{Cross Entropy}(c_j^{true}, c_j^{pred}));

[0157] where (c_j^{true}) and (c_j^{pred}) are the true and predicted risk categories of the (j)th data point, respectively, and (text{Cross Entropy}) represents the cross entropy loss function.

[0158] Adversarial Training Error (ATE) can be used to calculate the performance difference of the model under adversarial perturbations by introducing adversarial samples. The specific formula depends on the generation method of adversarial samples and the specific strategy of adversarial training. An example is to use FGSM (FastGradientSignMethod) to generate adversarial samples and calculate the prediction difference between the original sample and the adversarial sample:

[0159] (ATE=frac{1}{M}sum_{j=1}^{M}|f(mathbf{x}_j+eta)-f(mathbf{x}_j)|);

[0160] Among them, (f(cdot)) is the prediction function of the model, (mathbf{x}_j) is the (j)th original data point, and (eta) is the adversarial perturbation.

[0161] The First Joint Training Error (FJTE) can be a weighted sum or other combination of the above training errors, depending on design choices and task requirements. For example:

[0162] (FJTE=w_1cdotGDRATE+w_2cdotEDDRTE+w_3cdotPDRATE+w_4cd otRCPTE+w_5cdotATE);

[0163] Among them, (w_1, w_2, w_3, w_4, w_5) are weight parameters.

[0164] The first risk monitoring training error (FRMTE) may be the weighted sum of the first linkage training error and the data interval disaster recovery training error:

[0165] (FRMTE=alphacdotFJTE+(1-alpha)cdotDIDRTE);

[0166] Among them, (alpha) is the weight parameter that weighs the two error terms.

[0167] In some optional technical schemes, obtaining the prior training data interval includes: obtaining the first disaster recovery annotation word vector for each data distribution index in the first disaster recovery annotation data stream and the second disaster recovery annotation word vector for each data distribution index in the second time series data stream example; implementing the following steps for each data distribution index: determining the first characteristic distance between the first disaster recovery annotation word vector of the data distribution index and the second disaster recovery annotation word vector of the data distribution index; performing interval numerical mapping on the first characteristic distance of each data distribution index to obtain the third disaster recovery annotation word vector of each data distribution index; and generating the prior training data interval based on the third disaster recovery annotation word vector on each data distribution index.

[0168] In the field of supply chain, the application of data asset monitoring system is particularly important. The supply chain involves multiple links, from raw material procurement to final product sales, and the data of each link is crucial. In order to ensure the efficient operation of the supply chain, the data asset monitoring system needs to continuously learn and optimize to adapt to various complex business scenarios.

[0169] Exemplarily, the data asset monitoring system performs a series of sophisticated operations when obtaining the prior training data interval. First, the system obtains the first disaster recovery annotation word vector for each data distribution index from the first disaster recovery annotation data stream. These word vectors are digital representations of the data and can capture the semantic relationship between the data. At the same time, the system also obtains the second disaster recovery annotation word vector for each data distribution index from the second time series data stream example. These data streams contain real-time data from each link in the supply chain, which is crucial for the learning and optimization of the data asset monitoring system.

[0170] Next, for each data distribution index, the system calculates the first characteristic distance between the first disaster recovery annotation word vector and the second disaster recovery annotation word vector. This distance reflects the degree of similarity between the two word vectors in the semantic space, that is, the similarity between the data. By calculating this distance, the system can more accurately understand the associations and differences between the data.

[0171] Then, the system will perform interval numerical mapping on the first characteristic distance of each data distribution index to obtain the third disaster recovery annotation word vector of each data distribution index. This process is equivalent to converting data from one representation to another, making the data more suitable for subsequent machine learning tasks. Through this mapping, the system can more effectively utilize the information in the data and improve the accuracy and efficiency of monitoring.

[0172] Finally, based on the third disaster recovery annotation word vector on each data distribution index, the system generates a priori training data interval. This interval is the basis for the system to conduct subsequent learning, and it contains rich semantic information and data features. Through this interval, the system can have a deeper understanding of the data relationship in the supply chain, thereby more accurately monitoring the status and changes of data assets.

[0173] In summary, by obtaining prior training data intervals, the data asset monitoring system can more effectively utilize data resources in the supply chain and improve the accuracy and efficiency of monitoring. This not only helps to ensure the stable operation of the supply chain, but also provides enterprises with more accurate decision-making support and promotes the continuous optimization and development of supply chain management. In addition, the prior training data intervals obtained in this way also have stronger generalization capabilities and can adapt to various complex and changing situations in the supply chain. In other words, when new data patterns or abnormal situations appear in the supply chain, the data asset monitoring system can still maintain efficient monitoring performance and provide enterprises with timely and accurate early warnings and feedback. This flexibility and adaptability make the data asset monitoring system an indispensable and important tool for supply chain management.

[0174] In some other preferred embodiments, the data interval disaster recovery training error is determined based on the abnormal time series data interval corresponding to the target business asset data in the second time series data stream example and the prior training data interval, including: obtaining the first query vector of each data distribution index in the abnormal time series data interval and the second query vector of each data distribution index in the prior training data interval; implementing the following steps for each of the data distribution indexes: determining the second characteristic distance between the first query vector of the data distribution index and the second query vector of the data distribution index; and weighted summing the second characteristic distances of multiple data distribution indexes to obtain the data interval disaster recovery training error.

[0175] In the supply chain field, one of the core functions of the data asset monitoring system is to detect and respond to abnormal situations to ensure the continued stable operation of the supply chain. To achieve this goal, the system needs to accurately calculate the data interval disaster recovery training error in order to optimize its monitoring and early warning mechanism.

[0176] For example, when calculating the data interval disaster recovery training error, the data asset monitoring system will first focus on two key data intervals: the abnormal time series data interval corresponding to the target business asset data in the second time series data stream example and the prior training data interval. These two intervals represent the actual abnormal situation and the normal situation expected by the system, respectively.

[0177] First, the system obtains the first query vector for each data distribution index in the abnormal time series data interval. These first query vectors are the digital representation of the abnormal data points when the system detects an abnormal situation. At the same time, the system also obtains the second query vector for each data distribution index from the prior training data interval. These second query vectors represent the data representation of the system under normal circumstances.

[0178] Next, for each data distribution index, the system calculates the second characteristic distance between the first query vector and the second query vector. This distance reflects the difference between the actual abnormal situation and the expected normal situation, and is an important indicator for evaluating the disaster recovery capability of the system.

[0179] In order to obtain the overall data interval disaster recovery training error, the system will perform a weighted summation of the second characteristic distances of multiple data distribution indexes. This weight can be set based on factors such as the importance of the data point and the severity of the anomaly. In this way, the system can comprehensively consider the contribution of each data point and obtain a value that fully reflects the data interval disaster recovery training error.

[0180] This process not only helps the system identify abnormal situations more accurately, but also provides an important basis for subsequent early warning and recovery mechanisms. By continuously optimizing this error calculation process, the data asset monitoring system can more efficiently respond to various risks and challenges in the supply chain.

[0181] In this way, the data asset monitoring system can more accurately calculate the data interval disaster recovery training error, thereby improving its monitoring and response capabilities to supply chain anomalies. This not only helps to ensure the stability and reliability of the supply chain, but also provides enterprises with a higher level of data security and promotes the continuous innovation and development of supply chain management.

[0182] On the basis of the above content, the method also includes: obtaining a time series data stream to be monitored including target business asset data; performing abnormal risk identification on the time series data stream to be monitored through the first data quality monitoring model that has been debugged, and obtaining the abnormal time series data interval corresponding to the target business asset data in the time series data stream to be monitored and a third preliminary risk marking data stream corresponding to the time series data stream to be monitored; based on the abnormal time series data interval corresponding to the target business asset data in the time series data stream to be monitored, the third preliminary risk marking data stream and the time series data stream to be monitored are linked and marked and updated to obtain the target time series data stream after the target business asset data is marked with risk behaviors.

[0183] In the supply chain field, the operation of the data asset monitoring system is crucial, especially when conducting risk monitoring on the target business asset data. The following is a detailed explanation of how the monitoring system implements risk monitoring.

[0184] First, the data asset monitoring system will obtain the time series data streams to be monitored, including the target business asset data. These time series data streams record the historical data of various business activities in the supply chain, such as the purchase, transportation, inventory changes, etc. of goods, which are the basis for risk analysis.

[0185] Next, the system will use the debugged first data quality monitoring model to identify abnormal risks in the monitored time series data stream. This model may be built based on a machine learning algorithm, which can learn and identify abnormal patterns from a large amount of data. Through the model's identification, the system can detect abnormal time series data intervals corresponding to the target business asset data in the monitored time series data stream. These intervals may represent potential risk points, such as supply shortages, inventory backlogs, or transportation delays.

[0186] At the same time, the system will also generate a third preliminary risk annotation data stream corresponding to the time series data stream to be monitored. These data streams are preliminary risk annotations for the original data streams, which provide a basis for subsequent in-depth analysis.

[0187] Then, based on the detected abnormal time series data interval, the system will link and update the annotations of the third preliminary risk annotation data stream and the monitored time series data stream. This means that the system will synchronously correct the original data stream and the preliminary risk annotation data stream according to the specific situation of the abnormal interval to more accurately reflect the actual risk situation.

[0188] Finally, after the above steps, the system will obtain the target time series data stream after the target business asset data is labeled with risk behavior. These data streams not only contain the original business data, but also integrate the risk labeling information, providing a comprehensive risk view for supply chain managers.

[0189] Through this method, the data asset monitoring system can monitor the risk points in the supply chain in real time and issue early warnings in a timely manner. This helps enterprises respond to potential problems quickly and ensure the stable operation of the supply chain. At the same time, the system's linkage annotation update mechanism ensures the accuracy and timeliness of risk annotation, providing strong data support for supply chain management. In this way, not only the detection accuracy of supply chain risks is improved, but also the real-time and accuracy of risk annotation are enhanced through the linkage annotation update mechanism. This provides enterprises with more reliable risk management tools, which helps to improve the overall resilience and efficiency of the supply chain.

[0190] In some independent embodiments, after obtaining the target time series data stream after risk behavior annotation of the target business asset data, the method further includes: generating a data asset quality monitoring report according to the target time series data stream.

[0191] In some other independent embodiments, generating a data asset quality monitoring report according to the target time series data stream includes:

[0192] Perform multi-dimensional data analysis on the target time series data stream to extract key risk indicators and abnormal patterns;

[0193] Based on the extracted key risk indicators and abnormal patterns, a data asset quality assessment model is constructed. The data asset quality assessment model is used to quantitatively assess the risk level and abnormality degree of data assets.

[0194] Use the data asset quality assessment model to evaluate the target time series data stream point by point and generate risk annotation keywords including risk level, anomaly type, and occurrence time;

[0195] The risk annotation keywords are integrated with the target time series data stream to form a time series data stream map with risk annotation keywords. The time series data stream map is used to display the risk distribution and change trend of data assets.

[0196] Generate a data asset quality monitoring report based on the time series data flow graph. The data asset quality monitoring report includes the overall risk status of data assets, major risk points and their impact analysis, risk warnings and recommended measures;

[0197] The generated data asset quality monitoring report is presented in a combination of visual charts and text descriptions.

[0198] In the supply chain field, another key function of the data asset monitoring system is to generate a data asset quality monitoring report. This report is crucial for supply chain managers because it provides a comprehensive assessment of data asset quality, helping to identify potential risks in a timely manner and take appropriate countermeasures.

[0199] First, the data asset monitoring system will conduct multi-dimensional data analysis on the target time series data stream. This includes in-depth exploration of the data source, flow direction, change trend and other aspects to extract key risk indicators and abnormal patterns. These indicators and patterns can reflect the overall status of data assets and possible problems.

[0200] Next, based on the extracted key risk indicators and abnormal patterns, the system will build a data asset quality assessment model. This model can quantitatively assess the risk level and degree of abnormality of data assets, thereby providing a more accurate basis for subsequent monitoring and early warning.

[0201] Then, using this assessment model, the data asset monitoring system will conduct a point-by-point assessment of the target time series data stream. This means that the system will carefully check each data point in the data stream, determine its risk level, anomaly type, and occurrence time, and generate corresponding risk annotation keywords.

[0202] Subsequently, the system will merge these risk annotation keywords with the target time series data flow to form a time series data flow map with risk annotation keywords. This map not only shows the flow of data, but also clearly marks each risk point, allowing supply chain managers to clearly understand the risk distribution and change trend of data assets.

[0203] Based on this time series data flow graph, the data asset monitoring system will generate a data asset quality monitoring report. This report includes the overall risk status of data assets, analysis of major risk points and their impact, risk warnings and recommended measures. This information is extremely valuable to supply chain managers because it can help managers fully understand the status of data assets and promptly identify and resolve potential problems.

[0204] Finally, in order to allow supply chain managers to understand the report content more intuitively, the data asset monitoring system will present the generated data asset quality monitoring report in a combination of visual charts and text descriptions. This presentation method not only makes the report easier to understand, but also improves the readability and practicality of the report.

[0205] In summary, through the methods in these embodiments, the data asset monitoring system can generate a comprehensive, accurate and intuitive data asset quality monitoring report, providing powerful decision support for supply chain managers. This not only helps to improve the stability and efficiency of the supply chain, but also helps enterprises better deal with various potential risks.

[0206] In some other independent embodiments, the method further includes: processing the data asset quality monitoring report to obtain a structured text tree corresponding to the data asset quality monitoring report.

[0207] In detail, the processing of the data asset quality monitoring report to obtain the structured text tree corresponding to the data asset quality monitoring report includes: obtaining a first text word frequency distribution relationship chain and a first semantic logic deduction relationship chain of the data asset quality monitoring report, and obtaining a second text word frequency distribution relationship chain and a second semantic logic deduction relationship chain of the quality monitoring report template; determining whether the text word frequency thermal network of the data asset quality monitoring report and the text word frequency thermal network of the quality monitoring report template have commonality according to the first text word frequency distribution relationship chain and the second text word frequency distribution relationship chain, and determining whether the semantic logic of the data asset quality monitoring report and the semantic logic of the quality monitoring report template have commonality according to the first semantic logic deduction relationship chain and the second semantic logic deduction relationship chain; if the text word frequency thermal network of the data asset quality monitoring report and the text word frequency thermal network of the quality monitoring report template have commonality and the semantic logic of the data asset quality monitoring report and the semantic logic of the quality monitoring report template have commonality, then generating a structured text tree corresponding to the data asset quality monitoring report based on the structured mapping features of the quality monitoring report template.

[0208] In the supply chain field, the data asset monitoring system is not only responsible for monitoring the quality of data, but also needs to efficiently process and present this data for analysis and decision-making. One of the important links is to convert the data asset quality monitoring report into a structured text tree for further data processing and analysis.

[0209] First, the data asset monitoring system will obtain the first text word frequency distribution relationship chain and the first semantic logic deduction relationship chain of the data asset quality monitoring report. The text word frequency distribution relationship chain reflects the frequency of occurrence of each word in the report and their mutual relationship, while the semantic logic deduction relationship chain reveals the logical connection between sentences and paragraphs in the report. At the same time, the system will also obtain the second text word frequency distribution relationship chain and the second semantic logic deduction relationship chain of the quality monitoring report template. These templates are usually pre-defined according to business needs and are used to guide the format and content of the report.

[0210] Next, the system will determine whether the text word frequency heat network of the data asset quality monitoring report and the text word frequency heat network of the quality monitoring report template have commonalities based on the first text word frequency distribution relationship chain and the second text word frequency distribution relationship chain. The text word frequency heat network is a visualization tool that can intuitively display the frequency and distribution of each word in the text. By comparing the similarities of the two heat networks, it can be determined whether the content of the monitoring report is consistent with the content of the template in terms of vocabulary usage.

[0211] At the same time, the system will also determine whether the semantic logic of the data asset quality monitoring report and the semantic logic of the quality monitoring report template have commonality based on the first semantic logic deduction relationship chain and the second semantic logic deduction relationship chain. This is mainly done by analyzing the logical relationships between sentences and paragraphs in the report, such as causal relationships, conditional relationships, etc., to determine whether the content of the report conforms to the preset logical structure of the template.

[0212] If the text word frequency heat network of the data asset quality monitoring report and the text word frequency heat network of the quality monitoring report template have commonalities, and the semantic logic of the data asset quality monitoring report and the semantic logic of the quality monitoring report template also have commonalities, then the system can determine that the monitoring report and the template are consistent in content and structure. At this time, the system can generate a structured text tree corresponding to the data asset quality monitoring report based on the structured mapping features of the quality monitoring report template.

[0213] The structured text tree is a hierarchical data structure that can clearly represent the various parts of a report (such as titles, paragraphs, lists, etc.) and the relationships between them. This structured representation not only facilitates automatic processing and analysis by machines, but also improves the efficiency of human users in reading and understanding reports.

[0214] In this way, the data asset monitoring system can efficiently convert data asset quality monitoring reports into structured text trees, further improving the automation and intelligence level of supply chain data analysis. This not only helps companies to more quickly discover and solve problems in the supply chain, but also provides more accurate and comprehensive data support for corporate decision-making.

[0215] In some other independent embodiments, determining whether the text word frequency thermal network of the data asset quality monitoring report and the text word frequency thermal network of the quality monitoring report template have commonalities based on the first text word frequency distribution relationship chain and the second text word frequency distribution relationship chain includes: determining a first commonality weight between the first text word frequency distribution relationship chain and the second text word frequency distribution relationship chain; if the first commonality weight is greater than a first commonality threshold, determining that the text word frequency thermal network of the data asset quality monitoring report and the text word frequency thermal network of the quality monitoring report template have commonalities; determining whether the semantic logic of the data asset quality monitoring report and the semantic logic of the quality monitoring report template have commonalities based on the first semantic logic deduction relationship chain and the second semantic logic deduction relationship chain includes: determining a second commonality weight between the first semantic logic deduction relationship chain and the second semantic logic deduction relationship chain; if the second commonality weight is greater than a second commonality threshold, determining that the semantic logic of the data asset quality monitoring report and the semantic logic of the quality monitoring report template have commonalities.

[0216] In the supply chain, when a data asset monitoring system processes a data asset quality monitoring report, it takes a series of steps to determine the commonality between the report and the preset quality monitoring report template. This process is critical to ensuring the consistency and accuracy of the report.

[0217] First, the data asset monitoring system will generate a first text word frequency distribution relationship chain based on the text content in the data asset quality monitoring report. This relationship chain reflects the frequency of occurrence of each word in the report and the relationship between them. Similarly, the system will also generate a second text word frequency distribution relationship chain for the quality monitoring report template.

[0218] Next, the system calculates the first commonality weight between the two relationship chains. This weight is actually a quantitative indicator used to measure the similarity between the monitoring report and the template in terms of vocabulary usage. If the first commonality weight is greater than the preset first commonality threshold, the system can assume that the text word frequency heat network (i.e., the distribution and frequency graph of vocabulary) of the data asset quality monitoring report has commonalities with the text word frequency heat network of the quality monitoring report template. In other words, this means that the report is consistent with the template in terms of vocabulary usage.

[0219] In addition to the frequency distribution of text words, the data asset monitoring system also pays attention to the semantic logic of the report. The system extracts the first semantic logic deduction relationship chain from the data asset quality monitoring report, which reveals the logical relationship between each sentence and paragraph in the report. Similarly, the system also generates a second semantic logic deduction relationship chain for the quality monitoring report template.

[0220] Then, the system calculates the second commonality weight between the two semantic logic derivation relationship chains. This weight measures the similarity between the monitoring report and the template in logical structure. If the second commonality weight is greater than the preset second commonality threshold, the system can determine that the semantic logic of the data asset quality monitoring report has commonality with the semantic logic of the quality monitoring report template. This means that the report is consistent with the template not only in vocabulary usage, but also in logical structure.

[0221] In this way, the data asset monitoring system can comprehensively evaluate the commonalities between the data asset quality monitoring report and the preset template. This evaluation method not only takes into account the vocabulary used in the report, but also goes deep into the logical structure level of the report, thereby ensuring the consistency and accuracy of the report. This is very valuable for supply chain managers because it can help them understand the quality status of data assets more clearly and make more informed decisions. In other words, this embodiment provides a systematic and quantitative way to evaluate the similarities between data asset quality monitoring reports and templates, making report generation and analysis more efficient and accurate. At the same time, it also enhances the readability and credibility of the report, providing strong decision support for supply chain managers.

[0222] In some other independent embodiments, the first text word frequency distribution relationship chain includes a third text word frequency distribution relationship chain of each first report text paragraph in the data asset quality monitoring report, and the second text word frequency distribution relationship chain includes a fourth text word frequency distribution relationship chain of each second report text paragraph in the quality monitoring report template, and each first report text paragraph corresponds to a second report text paragraph; obtaining the first text word frequency distribution relationship chain of the data asset quality monitoring report and obtaining the second text word frequency distribution relationship chain of the quality monitoring report template include: respectively extracting a first paragraph report set and a second paragraph report set from the data asset quality monitoring report and the quality monitoring report template; respectively decomposing the first paragraph report set and the second paragraph report set into a plurality of first report text paragraphs and a plurality of second report text paragraphs; respectively obtaining the third text word frequency distribution relationship chain of each first report text paragraph and the fourth text word frequency distribution relationship chain of the second report text paragraph corresponding to each first report text paragraph.

[0223] In the supply chain field, when processing data asset quality monitoring reports, the data asset monitoring system will delve into the text details of the report to ensure the accuracy and standardization of the report content. During this process, the system will pay special attention to the word frequency distribution of the text as an important indicator for evaluating the quality of the report.

[0224] Specifically, when processing a data asset quality monitoring report, the data asset monitoring system will first focus on the text paragraphs in the report. The system will split the entire report into multiple text paragraphs, which are called first report text paragraphs, and they constitute the main content of the report. Similarly, the system will also split the preset quality monitoring report template into multiple text paragraphs, which are called second report text paragraphs. Each first report text paragraph will correspond to a second report text paragraph for subsequent comparative analysis.

[0225] After splitting the text paragraphs, the data asset monitoring system will further analyze the word frequency distribution of these paragraphs. For each first report text paragraph, the system will generate a third text word frequency distribution relationship chain. This relationship chain records in detail the frequency of occurrence of each word in the paragraph and the relationship between the words. Similarly, for each corresponding second report text paragraph, the system will also generate a fourth text word frequency distribution relationship chain.

[0226] In order to obtain these text word frequency distribution relationship chains, the data asset monitoring system will first disassemble the first paragraph report set (the paragraph set extracted from the data asset quality monitoring report) and the second paragraph report set (the paragraph set extracted from the quality monitoring report template). During the disassembly process, the system will analyze the words in the paragraphs one by one, count the number of occurrences of each word, and record the association between the words.

[0227] In this way, the data asset monitoring system can establish a comprehensive word frequency distribution network to gain a deep understanding of the text features of the data asset quality monitoring report. This not only helps the system evaluate the quality of the report, but also helps the system identify key information and potential problems in the report. For example, if a certain word appears much more frequently in the first report text paragraph than in the corresponding second report text paragraph, then this may mean that the problem represented by this word is of high importance or urgency in the supply chain. It can be seen that by deeply analyzing the text word frequency distribution relationship chain, the data asset monitoring system can more accurately evaluate the quality and accuracy of the data asset quality monitoring report. This not only improves the level of intelligence in supply chain management, but also provides supply chain managers with more comprehensive and in-depth data support, helping them make more informed decisions.

[0228] In some other independent embodiments, determining the first commonality weight between the first text word frequency distribution relationship chain and the second text word frequency distribution relationship chain includes: respectively determining the commonality weights between the third text word frequency distribution relationship chain of each first report text paragraph and the fourth text word frequency distribution relationship chain of the second report text paragraph corresponding to each first report text paragraph, to obtain multiple fourth commonality weights; determining the average commonality weight of the multiple fourth commonality weights as the first commonality weight between the first text word frequency distribution relationship chain and the second text word frequency distribution relationship chain.

[0229] In the supply chain field, when evaluating the similarity between data asset quality monitoring reports and preset templates, the data asset monitoring system will adopt a refined approach to determine the commonality weights between text word frequency distribution relationship chains.

[0230] Specifically, the system will first focus on each text paragraph in the data asset quality monitoring report, i.e., the first report text paragraph, and the corresponding text paragraph in the preset template, i.e., the second report text paragraph. For each pair of such corresponding paragraphs, the system will calculate their text word frequency distribution relationship chains, i.e., the third text word frequency distribution relationship chain and the fourth text word frequency distribution relationship chain.

[0231] Next, the system determines the commonality weight between the third text word frequency distribution relationship chain of each first report text paragraph and the fourth text word frequency distribution relationship chain of its corresponding second report text paragraph. This commonality weight is actually a quantitative indicator used to measure the similarity of vocabulary usage between two text paragraphs. The system calculates this commonality weight by comparing the frequency, distribution and mutual relationship of vocabulary in the two text paragraphs to obtain a fourth commonality weight.

[0232] Since the data asset quality monitoring report may contain multiple text paragraphs, the system will calculate a fourth commonality weight for each corresponding text paragraph. After obtaining multiple fourth commonality weights, the system will further calculate the average of these weights as the first commonality weight between the text word frequency distribution relationship chain of the entire data asset quality monitoring report and the text word frequency distribution relationship chain of the preset template.

[0233] The reason for this is that it can go deep into every detail of the report, rather than just making a general assessment of the entire report. By calculating the commonality weight of each text paragraph and ultimately determining an average commonality weight, the system can more accurately assess the similarity in vocabulary usage between the data asset quality monitoring report and the preset template. This not only helps to improve the standardization and consistency of the report, but also helps supply chain managers better understand the content of the report and make more informed decisions. In summary, the data asset monitoring system can more comprehensively and accurately evaluate the quality of the data asset quality monitoring report and provide more reliable data support for supply chain management.

[0234] Further, Figure 2 Schematic diagram of the structure of a data asset monitoring system 200 provided in an embodiment of the present application. Figure 2 The data asset monitoring system 200 shown includes a processor 210, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0235] Alternatively, if Figure 2 As shown, the data asset monitoring system 200 may further include a memory 230. The processor 210 may call and run a computer program from the memory 230 to implement the method in the embodiment of the present application.

[0236] The memory 230 may be a separate device independent of the processor 210 , or may be integrated into the processor 210 .

[0237] Alternatively, if Figure 2 As shown, the data asset monitoring system 200 may further include a transceiver 220, and the processor 210 may control the transceiver 220 to interact with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices.

[0238] Optionally, the data asset monitoring system 200 may implement the corresponding processes corresponding to the storage engine or components in the storage engine (such as a processing module) or a device deployed with a storage engine in each method of the embodiments of the present application, which will not be described here for the sake of brevity.

[0239] It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and performed. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0240] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0241] It should be understood that the above-mentioned memory is exemplary but not restrictive. For example, the memory in the embodiments of the present application may also be static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not limited to these and any other suitable types of memory.

[0242] Based on the above, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program implements the above method when running.

[0243] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations.

Claims

1. A real-time data quality monitoring method based on data assets, It is characterized in that Applied to a data asset monitoring system, the method comprises: Performing data loss simulation on a first time series data stream example including target business asset data to obtain a first partial data stream example from which the target business asset data is purged; Debugging the data disaster recovery project of the target business asset data on the first data disaster recovery model according to the first local data flow example to obtain a debugged first data disaster recovery model; Performing data disaster recovery on the second local data flow example through the first data disaster recovery model that has been debugged to obtain a first disaster recovery annotated data flow, and performing abnormal risk identification on the second time series data flow example including the target business asset data through the first data quality monitoring model to obtain a first preliminary risk annotated data flow, wherein the second local data flow example is obtained by simulating data loss on the second time series data flow example; Determine a first risk monitoring training error corresponding to the first data quality monitoring model according to the first preliminary risk-annotated data stream and the first disaster recovery-annotated data stream, and optimize the first data quality monitoring model according to the first risk monitoring training error to obtain a debugged first data quality monitoring model; The first data quality monitoring model that has been debugged is used to identify abnormal risks of the time series data stream to be monitored including the target business asset data, and obtain the target time series data stream after the risk behavior of the target business asset data is marked; The debugging of the data disaster recovery project of the target business asset data on the first data disaster recovery model according to the first local data flow example to obtain the debugged first data disaster recovery model includes: Noising the first time series data stream example to obtain a first noisy data stream example; Cross-processing the first local data stream example and the first noise-added data stream example to obtain a first cross-data stream example; Debugging the data disaster recovery project of the target business asset data on the first data disaster recovery model according to the first cross data flow example to obtain a debugged first data disaster recovery model; The debugging of the data disaster recovery project of the target business asset data on the first data disaster recovery model according to the first cross data flow example to obtain the debugged first data disaster recovery model includes: Perform data disaster recovery on the first cross data flow example through a first data disaster recovery model with weighted rollback, to obtain a second disaster recovery annotated data flow corresponding to the first cross data flow example; Determine a first disaster recovery annotation training error according to the first time series data stream example and the second disaster recovery annotation data stream; The first data disaster recovery model of weight rollback is optimized according to the first disaster recovery annotation training error to obtain the first data disaster recovery model that has completed debugging.

2. The method according to claim 1, It is characterized in that The first data disaster recovery model that has been debugged includes a first feature embedding sub-model and a first feature mapping sub-model, and performing data disaster recovery on the second local data stream example through the first data disaster recovery model that has been debugged to obtain a first disaster recovery annotated data stream includes: Noising the second time series data stream example to obtain a second noisy data stream example; Cross-processing the second local data stream example and the second noisy data stream example to obtain a second cross-data stream example; The state feature of the second cross data flow example is embedded through the first feature embedding submodel to obtain a first state feature embedding vector, and the first state feature embedding vector is mapped by disaster recovery annotation through the first feature mapping submodel to obtain the first disaster recovery annotation data flow.

3. The method according to claim 1, It is characterized in that The abnormal risk identification is performed on the second time series data stream example including the target business asset data by the first data quality monitoring model to obtain a first preliminary risk-labeled data stream, including: Performing abnormal risk identification on the second time series data flow example through the first data quality monitoring model, obtaining an abnormal time series data interval corresponding to the target business asset data in the second time series data flow example and a second preliminary risk annotation data flow corresponding to the second time series data flow example; According to the abnormal time series data interval in the second time series data flow example, the second preliminary risk annotation data flow and the second time series data flow example are linked and annotated and updated to obtain the first preliminary risk annotation data flow.

4. The method according to claim 3, It is characterized in that The determining, according to the first preliminary risk-labeled data stream and the first disaster recovery-labeled data stream, a first risk monitoring training error corresponding to the first data quality monitoring model includes: Obtaining a priori training data interval, and determining a data interval disaster recovery training error according to the abnormal time series data interval corresponding to the target business asset data in the second time series data stream example and the priori training data interval; Determine at least one of the following training errors: Determining a global disaster recovery annotation training error according to the first disaster recovery annotation data stream and the first preliminary risk annotation data stream; Determining an embedded description disaster recovery training error according to the first disaster recovery annotation data stream and the first preliminary risk annotation data stream; Determining a stage disaster recovery annotation training error according to the first disaster recovery annotation data stream, the first preliminary risk annotation data stream, and the prior training data interval; Determining a risk category prediction training error based on the first disaster recovery annotated data stream and the first preliminary risk annotated data stream; Determining an adversarial training error based on the first preliminary risk labeled data stream; Determine a first linkage training error according to at least one of the global disaster recovery annotation training error, the embedded description disaster recovery training error, the stage disaster recovery annotation training error, the risk category prediction training error, and the adversarial training error; The first linkage training error and the data interval disaster recovery training error are weightedly summed to obtain the first risk monitoring training error.

5. The method according to claim 4, It is characterized in that The obtaining of a priori training data interval includes: Obtain a first disaster recovery annotation word vector for each data distribution index in the first disaster recovery annotation data stream and a second disaster recovery annotation word vector for each data distribution index in the second time series data stream example; For each of the data distribution indexes, the following steps are implemented: determining a first characteristic distance between a first disaster recovery annotation word vector of the data distribution index and a second disaster recovery annotation word vector of the data distribution index; Performing interval numerical mapping on the first characteristic distance of each of the data distribution indexes to obtain a third disaster recovery annotation word vector of each of the data distribution indexes; Based on the third disaster recovery annotation word vector on each of the data distribution indexes, the priori training data interval is generated.

6. The method according to claim 4, It is characterized in that The determining of the data interval disaster recovery training error according to the abnormal time series data interval corresponding to the target business asset data and the prior training data interval in the second time series data stream example includes: Obtain a first query vector for each data distribution index in the abnormal time series data interval and a second query vector for each data distribution index in the prior training data interval; For each of the data distribution indexes, the following steps are implemented: determining a second characteristic distance between a first query vector of the data distribution index and a second query vector of the data distribution index; The second characteristic distances of the plurality of data distribution indexes are weighted and summed to obtain the data interval disaster recovery training error.

7. The method according to claim 1, It is characterized in that The method further comprises: Acquire a time series data stream to be monitored including target business asset data; The abnormal risk of the time series data stream to be monitored is identified by using the first data quality monitoring model that has been debugged, so as to obtain the abnormal time series data interval corresponding to the target business asset data in the time series data stream to be monitored and the third preliminary risk annotation data stream corresponding to the time series data stream to be monitored; According to the abnormal time series data interval corresponding to the target business asset data in the time series data stream to be monitored, the third preliminary risk marking data stream and the time series data stream to be monitored are linked and marked and updated to obtain the target time series data stream after the target business asset data is marked with risk behaviors.

8. A data asset monitoring system, It is characterized in that The method comprises at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method according to any one of claims 1 to 7.

Images