Risk data operation monitoring method, device, system, equipment and storage medium

Abstract

The application discloses a risk data operation monitoring method, device and system, equipment and a storage medium, and belongs to the technical field of data processing. The application stores risk data as data assets in a data management database in a pre-constructed data management system, receives a detection request, acquires data assets from the data management database of the pre-constructed data management system according to the detection request, and monitors the flow of the data assets and the change trend of the data assets, so that full-process monitoring of the risk data is realized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to methods, apparatus, systems, equipment and storage media for risk data operation monitoring. Background Technology

[0002] In the credit business process, it is necessary to monitor and identify customers throughout the entire business cycle using a large amount of internal and external data in order to provide customers with more accurate and faster services.

[0003] The collected data is currently divided into two categories: external procurement data and historical business data from within the bank. Due to historical reasons, the access methods for external data vary, and the storage locations also differ. Internal data also comes from different sources, making it impossible to effectively manage and monitor the data and access operations in a unified manner. Summary of the Invention

[0004] The main objective of this invention is to provide a method, system, device, and readable storage medium for risk data operation monitoring, aiming to achieve effective operation monitoring and management of risk data throughout the entire process.

[0005] To achieve the above objectives, the present invention provides a risk data operation monitoring method, which includes the following steps:

[0006] Receive a detection request, and retrieve data assets from a pre-built data governance system based on the detection request;

[0007] The dynamics of the data assets are monitored, including the circulation of the data assets.

[0008] Optionally, the step of dynamically monitoring the data assets includes:

[0009] Monitor the flow of the data assets from one storage node to another through transformation tasks;

[0010] Determine if there are any abnormalities in the described flow;

[0011] If any abnormality is detected in the data transfer process, the abnormality of the data assets will be reported to the relevant management personnel.

[0012] Optionally, the dynamics of the data assets include: the changing trends of the data assets, and the step of monitoring the dynamics of the data assets includes:

[0013] Capture statistical information of the data assets and monitor whether there are any anomalies in the changing trends of the statistical information;

[0014] If an anomaly is detected in the trend, the anomaly will be reported to the relevant management personnel.

[0015] Optionally, the data governance system includes: a scheduled task site, a data governance database, and a source database. Before the step of receiving a detection request and obtaining data assets from the pre-built data governance system based on the detection request, the method further includes:

[0016] First risk data is obtained from the source database, and the first risk data is configured as metadata description in the data governance database;

[0017] Based on the configuration data of the data governance database, the second risk data is obtained from the source database through the scheduled task site;

[0018] The second risk data is stored in a preset database table, and then a data governance database is constructed from the various database tables.

[0019] Optionally, the data governance system further includes: a web site and an online service site, and the step of acquiring data assets from the pre-built data governance system includes:

[0020] The detection request is sent to the web site, and after receiving the detection request, the web site sends a service request to the online service site.

[0021] The data asset corresponding to the service request is obtained from the data governance database through the online service site.

[0022] Optionally, after the step of obtaining the second risk data from the source database, the method further includes:

[0023] Confirm whether the second risk data obtained from the source database meets the inspection rules;

[0024] If the second risk data meets the inspection rules, then the following step is executed: store the second risk data in a preset database table.

[0025] Furthermore, to achieve the above objectives, the present invention also provides a risk data operation monitoring device, the risk data operation monitoring device comprising:

[0026] The acquisition module receives a detection request and, based on the detection request, acquires data assets from a pre-built data governance system.

[0027] The monitoring module is used to monitor the flow status of the data assets, which includes changes in the data asset's link.

[0028] Optionally, the monitoring module is further configured to:

[0029] Monitor the flow of the data assets from one storage node to another through transformation tasks;

[0030] Confirm whether there are any abnormalities in the aforementioned flow;

[0031] If any abnormality is detected in the data transfer process, the abnormality of the data assets will be reported to the relevant management personnel.

[0032] Optionally, the monitoring module is further configured to:

[0033] Capture statistical information of the data assets and monitor whether there are any anomalies in the changing trends of the statistical information;

[0034] If an anomaly is detected in the trend, the anomaly will be reported to the relevant management personnel.

[0035] Optionally, the acquisition module is further configured to:

[0036] First risk data is obtained from the source database, and the first risk data is configured as metadata description in the data governance database;

[0037] Based on the configuration data of the data governance database, the second risk data is obtained from the source database through the scheduled task site;

[0038] The second risk data is stored in a preset database table, and then a data governance database is constructed from the various database tables.

[0039] Optionally, the acquisition module is further configured to:

[0040] The detection request is sent to the web site, and after receiving the detection request, the web site sends a service request to the online service site.

[0041] The data asset corresponding to the service request is obtained from the data governance database through the online service site.

[0042] Optionally, the acquisition module is further configured to:

[0043] Confirm whether the second risk data obtained from the source database meets the inspection rules;

[0044] If the second risk data meets the inspection rules, then the following step is executed: store the second risk data in a preset database table.

[0045] Furthermore, to achieve the above objectives, the present invention also provides a risk data operation monitoring system, the risk data operation monitoring system comprising:

[0046] The link monitoring module is used to receive detection requests, obtain data assets from the pre-built data governance system according to the detection requests, and monitor the dynamics of the data assets, including the flow of the data assets.

[0047] Optionally, the risk data operation monitoring system further includes:

[0048] The data asset module is used to obtain first risk data from the source database, configure the first risk data as metadata description in the data governance database, obtain second risk data from the source database through the scheduled task station based on the configuration data of the data governance database, store the second risk data in a preset database table, and then construct the data governance database from each of the database tables.

[0049] The data quality module is used to confirm whether the second risk data obtained from the source database meets the inspection rules. If the second risk data meets the inspection rules, the data asset module stores the second risk data into a preset database table.

[0050] In addition, to achieve the above objectives, the present invention also provides a risk data operation monitoring device, the risk data operation monitoring device comprising: a memory, a processor, and a risk data operation monitoring program stored in the memory and executable on the processor, wherein the risk data operation monitoring program, when executed by the processor, implements the steps of the risk data operation monitoring method as described above.

[0051] In addition, to achieve the above objectives, the present invention also provides a storage medium storing a risk data operation monitoring program, which, when executed by a processor, implements the steps of the risk data operation monitoring method described above.

[0052] An embodiment of the present invention proposes a risk data operation monitoring method, apparatus, system, device, and storage medium. By storing risk data as data assets in a data governance database and constructing a data governance system, upon receiving a detection request, the system retrieves the dynamics of the data assets from the pre-constructed data governance database according to the detection request, and monitors the flow and statistical information of the data assets to achieve full-process monitoring of risk data. Attached Figure Description

[0053] Figure 1This is a schematic diagram of the structure of the risk data operation monitoring equipment for the hardware operating environment involved in the embodiments of the present invention;

[0054] Figure 2 This is a flowchart illustrating the first embodiment of the risk data operation monitoring method of the present invention;

[0055] Figure 3 This is a database relationship diagram of the data governance database in one embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of the system deployment of a data governance system according to an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of the functional modules of the first embodiment of the risk data operation monitoring device of the present invention.

[0058] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0059] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0060] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of the risk data operation monitoring equipment for the hardware operating environment involved in the embodiments of the present invention.

[0061] like Figure 1 As shown, the risk data operation monitoring device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0062] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the risk data operation monitoring equipment, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0063] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and a risk data operation monitoring program.

[0064] exist Figure 1 In the risk data operation monitoring device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with users; the processor 1001 and memory 1005 in the risk data operation monitoring device of the present invention can be set in the risk data operation monitoring device, and the risk data operation monitoring device calls the risk data operation monitoring program stored in the memory 1005 through the processor 1001 and executes the risk data operation monitoring method provided in the embodiment of the present invention.

[0065] This invention provides a risk data operation monitoring method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the risk data operation monitoring method of the present invention.

[0066] In this embodiment, the risk data operation monitoring method includes:

[0067] Step S10: Receive a detection request, and obtain data assets from a pre-built data governance system based on the detection request;

[0068] Step S20: Monitor the dynamics of the data assets, including the circulation status of the data assets.

[0069] This embodiment provides a risk data operation monitoring method for a risk data operation monitoring system used to centrally manage the data usage across various systems within a bank's risk management department. For ease of description, this system is referred to as the "risk system." Typically, the risk data required by a bank's risk management department falls into two categories: externally procured data and historical business data. Externally procured data is obtained from data providers, such as commercially purchased data from Qixinbao, Wind, and Caihui, or from data obtained through connections with government data departments, such as provincial tax bureaus, State Grid, and Human Resources and Social Security departments. Historical business data consists of customer transactions within the bank, such as deposits, loans, wealth management, and other transactions. The sources and access methods of these data differ; therefore, the risk system needs to provide unified operation and maintenance management and status monitoring for the data and access processes.

[0070] For externally procured data, the risk system accesses the data and then flows to systems such as the risk portal, non-performing asset disposal, credit risk management (CVM), and post-loan management. Taking Qixinbao's business registration data as an example, the risk system accesses the business registration data of all enterprises and then distributes the data according to the corresponding customer lists of each system. For example, the risk portal processes data with a borrower scope, while non-performing asset disposal processes data with a non-performing customer scope. Then, the risk portal also supplies data to other systems in the form of interfaces. Data flows continuously within the risk system. It should be noted that, in this embodiment, data stored in the data governance database is referred to as data assets. Therefore, the risk system receives detection requests and, based on these requests, retrieves data assets from the pre-built data governance system to monitor the dynamic changes of the data assets throughout the entire process.

[0071] The following will provide a detailed explanation of each step:

[0072] Step S10: Receive a detection request, and obtain data assets from a pre-built data governance system based on the detection request;

[0073] In one embodiment, after receiving a detection request, the risk system retrieves the corresponding data assets from a pre-built data governance system based on the request. The detection request can be issued by the risk system administrator or it can be a pre-configured, periodically automatically sent request. Based on this detection request, some data assets can be monitored; for example, if it's necessary to monitor the data assets of a specific branch, the corresponding branch's data assets can be retrieved from the data governance system. Alternatively, all data assets can be monitored, depending on the specific circumstances.

[0074] Step S20: Monitor the dynamics of the data assets, including the circulation status of the data assets.

[0075] In one embodiment, after acquiring data assets from the data governance system, the dynamics of these data assets are monitored. Taking Qixinbao's basic business information data as an example, the main concepts in the database are: Data Asset: Qixinbao's basic business information is considered an asset; Storage Nodes: Databases, Kafka, Kylin, HDFS, HTML, etc., are all considered storage nodes; Transformation Tasks: Transporters that move data from one storage node to another; Link: The configuration description of how assets move from one storage site to another through transformation tasks. In this embodiment, the dynamic monitoring of data assets includes monitoring the flow of data assets in the link, that is, real-time monitoring of storage nodes and transformation tasks to reflect the daily flow and usage of data assets.

[0076] Furthermore, in one embodiment, the step of dynamically monitoring the data assets includes:

[0077] Step S21: Monitor the flow of the data assets from one storage node to another through the conversion task;

[0078] In one embodiment, during the process of monitoring data assets through a transformation task from one storage node to another, the risk system needs to clarify that the data asset's path from one storage node to another through the transformation task is pre-mapped.

[0079] Step S22: Confirm whether there are any abnormalities in the flow situation;

[0080] In one embodiment, the risk system confirms whether there are any anomalies in the data transfer process. Whether the transfer is normal refers to whether any abnormalities occur during the data asset transformation process or at the storage nodes. For example, if the amount of data decreases by more than a certain threshold after data flows from storage node A to storage node B, this could be due to a problem with the data carrier or the transformation task. This situation is considered an anomaly. Alternatively, if the data asset does not follow the pre-defined path, and is not detected at one storage node but is detected at another, this is also considered an anomaly. Specific anomalies can be set according to actual conditions. Furthermore, by setting thresholds, when certain characteristic values ​​of the data asset in the path, such as quantity, exceed the threshold, this is determined to be an abnormal state.

[0081] Step S23: If an abnormality is detected in the flow, the abnormality of the data asset is reported to the relevant management personnel.

[0082] In one embodiment, if an anomaly occurs during the data asset transfer process (i.e., during the transfer itself), the anomaly is reported to the relevant personnel. Understandably, if an anomaly occurs in the data asset's journey from one storage site to another via a transformation task, this anomaly must be reported. In this embodiment, the relevant personnel can be the data manager or the risk system administrator. That is, the anomaly can be directly reported to the person in charge of the data asset's source, who will then handle it; or it can be reported to the risk system administrator for direct handling; or the risk system administrator can assess the situation and then report the anomaly to the relevant personnel for further processing.

[0083] Furthermore, in one embodiment, the dynamics of the data asset include: the changing trend of the data asset, and the step of monitoring the dynamics of the data asset includes:

[0084] Step S24: Capture statistical information of the data assets and monitor whether there are any abnormalities in the changing trend of the statistical information;

[0085] In one embodiment, the risk system captures statistical information from data assets and monitors whether there are any anomalies in the statistical information within a predetermined time. If anomalies are found, the system reports the anomalies to relevant personnel. It should be noted that the data governance system aggregates data from the source database into datasets, which are then stored as data assets in the data governance database. Each dataset contains a large amount of statistical information, such as basic customer information, business volume, customer statistical values, and other dynamic statistical information—that is, information that changes over time. The risk system captures this statistical information, compiles it into tables, and then creates trend charts to characterize the changing trends of this data. These trends are then monitored to confirm whether any anomalies exist.

[0086] Step S25: If an abnormality is detected in the trend of change, the abnormality of the trend of change is reported to the relevant management personnel.

[0087] In one embodiment, the risk system captures statistical information from data assets, monitors the trends of these statistical information for anomalies, and if anomalies are found, reports the abnormal trends to relevant personnel. It should be noted that...

[0088] This embodiment receives a detection request and, based on the request, retrieves data assets from a pre-built data governance system. It then monitors the changes in these data assets and the trends in their statistical information. This allows viewing the same data asset across different risk systems. For example, basic business registration information imported from external data A might enter subsystem 01 via task X, and then enter subsystem 02 via task Y. This allows for monitoring of data asset flow and confirming whether there are any anomalies in the changes and trends of the data assets, enabling faster detection of data issues and timely feedback to relevant management personnel for processing.

[0089] Furthermore, based on the first embodiment of the risk data operation monitoring method of the present invention, a second embodiment of the risk data operation monitoring method of the present invention is proposed.

[0090] The second embodiment of the risk data operation monitoring method differs from the first embodiment in that, prior to the step of receiving a detection request and, based on the detection request, obtaining data assets from a pre-built data governance system, the method further includes:

[0091] Step a: Obtain first risk data from the source database and configure the first risk data as metadata description in the data governance database;

[0092] Step b: Based on the configuration data of the data governance database, obtain the second risk data from the source database through the scheduled task site;

[0093] Step c: Store the second risk data into a preset database table, and then construct a data governance database from each of the database tables.

[0094] This embodiment obtains first risk data from a source database, configures this first risk data as metadata in the data governance database, and then, based on the configuration data in the data governance database, obtains second risk data from the source database through the scheduled task station. After obtaining the second risk data, it is stored in a preset table, and the data governance database is constructed from these tables. In the data governance database of this embodiment, the overall picture and details of all risk data can be seen, such as what types of data there are, the source of the data, the storage location of the data, and what fields are present, while also providing search functionality.

[0095] The following provides a detailed explanation of each step:

[0096] Step a: Obtain first risk data from the source database and configure the first risk data as metadata description in the data governance database;

[0097] In one embodiment, before acquiring data assets from a pre-built data governance system, first risk data is obtained from a source database, and this first risk data is configured as metadata description in the data governance database. Here, the source database is a database storing internal and external risk data; the first risk data refers to all data stored in the source database; and the metadata description is a description of the data stored in the data governance database. Because the scheduled task station needs to know what data to acquire and where it is located during the process of acquiring risk data and updating it in the data governance database, the data governance database contains metadata description configurations, enabling the scheduled task station to collect the necessary data from the source database.

[0098] Step b: Based on the configuration data of the data governance database, obtain the second risk data from the source database through the scheduled task site;

[0099] In one embodiment, the scheduled task station retrieves second risk data from the source database based on the configuration data in the data governance database. The second risk data includes job information sent by the task station, internal and external risk data, etc. The difference between the second and first risk data is that the second risk data includes risk data from the task stations. Correspondingly, the source database includes various task stations. The scheduled task station retrieves the second risk data according to the configuration data in the data governance database, such as the execution location and periodicity attributes, and updates the data governance database accordingly.

[0100] Specifically, the scheduled task site provides an SDK package for each job site to import. During the lifecycle of their tasks, job status is sent through a unified internal log collection platform. Taking file download as an example, if a task is required to successfully download a file once within a period (e.g., once per day), the scheduled task will attempt to download the file. If the file is found, it will be imported; otherwise, the status will be set to L (L-LACK, dependency not met). The next time the scheduled task finds the latest status of the day is L, it will continue downloading until success (S-SUCCESS). Therefore, the task lifecycle is: Start (B-BEGIN), Execution (L-LACK, dependency not met; F-current failure), End (S-final success; F-final failure).

[0101] Step c: Store the second risk data into a preset database table, and then construct a data governance database from each of the database tables.

[0102] In one embodiment, after obtaining the second risk data from the source database, the second risk data is stored in a preset database table, and a data governance database is constructed from the preset database tables.

[0103] Combination Figure 3 The data governance database of this embodiment will be described. Figure 3This is a database relationship diagram of the data governance database in one embodiment of the present invention. The data governance database is constructed from multiple two-dimensional tables, which are connected by connecting lines. These two-dimensional tables are the aforementioned preset database tables. The tables in the data governance database include: a data asset definition table, a data classification definition table, a data source definition table, a task-data chain definition table, a chain definition detail table, a task definition table, a task execution status table, a task site table, a data entity table basic information table, a data node table, and a data asset storage table - field definition table. The data asset storage table - field definition table is the main table, and other tables are connected to the data asset definition table according to foreign key relationships. The main table records the codes, table names, Chinese table names, schema names, data asset codes, and data node codes of each data table in the database. The specific data asset information in the data asset code is obtained from the associated data asset definition table. For example, by inputting a specific data asset code, the name, description, and data source definition code of the asset can be obtained by calling the data asset definition table. The data source definition code is associated with the data source definition table, and the source name, description, and other records of the data can be found through the data source definition code corresponding to the data asset code. When an anomaly is detected in a data asset, the IT manager or source of the data asset can be found through the data governance database, and the anomaly can be reported to that IT manager.

[0104] Furthermore, in one embodiment, the data governance system further includes: a web site and an online service site, and the step of acquiring data assets from the pre-built data governance system includes:

[0105] Step d: The detection request is sent to the web site, and after receiving the detection request, the web site sends a service request to the online service site.

[0106] In one embodiment, after receiving a user's request to detect data, the detection request is sent to a web site. The web site then sends a service request to an online service site based on the detection request. In other words, when an external request is received, the data is not obtained directly from the internal business network, but rather through this web site.

[0107] Step e: Obtain the data asset corresponding to the service request from the data governance database through the online service site.

[0108] In one embodiment, based on a service request, the online service site retrieves the corresponding data assets from the data governance database. It is understood that the online service site provides services based on this data governance database; therefore, a service request is sent to the online service site through the DMZ zone's web site, and the online service site retrieves the corresponding data assets from the data governance database and provides them to the risk system.

[0109] Reference Figure 4 , Figure 4 This is a schematic diagram illustrating the system deployment of a data governance system according to an embodiment of the present invention. To further explain this embodiment, the data governance system is divided into three sub-sites: a web site, an online service site, and a data governance scheduled task site. The scheduled task site mentioned herein is... Figure 4 The data governance scheduled task site in this document refers to the same site. The scheduled task site provides an SDK package for each job site to import, and sends job status updates through a unified log platform. Additionally, each database is configured as metadata in the data governance database, and the scheduled tasks use this to connect to each database to obtain daily data information. The web site is located in the DMZ (Demilitarized Zone, the area between internal and external firewalls), a small network area between the enterprise's internal and external networks. It should be noted that the job sites and databases listed in the diagram are for illustrative purposes only and do not limit the number of job sites and databases.

[0110] This embodiment pre-constructs a data governance system, in which a web site is associated with an online service site, which in turn is associated with a data governance database. The source database stores risk data from various sources, while the data governance database stores the risk data required by the risk system. This embodiment obtains first risk data from the source database and configures it in the data governance database. A scheduled task site obtains second risk data from the source database and stores it in a pre-defined database table, thus constructing the data governance database to classify and store risk data. The web site provides a management interface. When data assets need to be retrieved from the pre-constructed data governance system—that is, upon receiving an external detection request—a service request is sent to the online service site. Data interaction is conducted via HTTPS protocol, and the required data assets are obtained from the data governance database, thus more effectively protecting the internal network.

[0111] Furthermore, based on the second embodiment of the risk data operation monitoring method of the present invention, a third embodiment of the risk data operation monitoring method of the present invention is proposed.

[0112] The third embodiment of the risk data operation monitoring method differs from the second embodiment of the drone scheduling method in that, after the step of obtaining the second risk data from the source database, the method further includes:

[0113] Step S31: Confirm whether the second risk data obtained from the source database meets the inspection rules;

[0114] Step S32: If the second risk data meets the inspection rules, then execute the step: store the second risk data in a preset database table.

[0115] In this embodiment, the second risk data obtained from the source database is verified. If the second risk data meets the inspection rules, it is stored in a preset database table. It is understandable that after obtaining the second risk data from the source database, it needs to be screened to identify data quality issues earlier. For example, if the pre-update quantity of a certain data point in the second risk data is normally 5000, but today's pre-update quantity is less than 1000, this data does not meet the inspection rules and is therefore problematic. In this case, the data is temporarily not stored in the preset database table for use. Due to the different data stored in the source database, the second risk data may involve external or internal data. If it does not meet the inspection rules, i.e., if there is a problem, the relevant data personnel need to be contacted for processing. For example, if the data obtained from a branch has a problem, the issue should be reported to the branch manager for handling.

[0116] The inspection rules are used to describe the abnormal data. Whether the second-risk data meets the inspection rules can be confirmed through single-field checks, structured query statements (SQL), or other methods. It should be noted that due to the diversity of storage nodes, the inspection rules must also mask these differences in storage nodes to provide a consistent interface to the application.

[0117] This embodiment performs quality control on the second risk data by confirming whether it meets the inspection rules, and does not store the second risk data that does not meet the inspection rules into the preset database table, thereby screening out bad data and preventing it from causing other problems after flowing into other business systems.

[0118] The implementation of the risk data operation monitoring method of this invention is based on a risk data operation monitoring system architecture, which includes:

[0119] The link monitoring module is used to receive detection requests, obtain data assets from the pre-built data governance system according to the detection requests, and monitor the dynamics of the data assets, including the flow of the data assets.

[0120] Furthermore, the risk data operation monitoring system also includes:

[0121] The data asset module is used to obtain first risk data from the source database, configure the first risk data as metadata description in the data governance database, obtain second risk data from the source database through the scheduled task station based on the configuration data of the data governance database, store the second risk data in a preset database table, and then construct the data governance database from each of the database tables.

[0122] The data quality module is used to confirm whether the second risk data obtained from the source database meets the inspection rules. If the second risk data meets the inspection rules, the data asset module stores the second risk data into a preset database table.

[0123] In this embodiment, the risk data operation monitoring method flow is as follows: When the risk system receives a detection request, it sends a detection request to the web site through the link monitoring module, and the web site sends a service request to the service site. The service site then obtains the corresponding data assets. The link monitoring module monitors the flow of the data assets from one storage node to another through a conversion task, confirming whether there are any anomalies in the flow. When an anomaly is detected, the anomaly is reported to the relevant management personnel. Simultaneously, the link monitoring module captures the statistical information of the data assets and monitors whether there are any anomalies in the trend of the statistical information. If an anomaly is detected in the trend, the anomaly is reported to the relevant management personnel.

[0124] The data assets are provided by the data governance system managed by the data asset module. The data asset module obtains the first risk data from the source database and configures the first risk data as metadata description in the data governance database. Based on the configuration data of the data governance database, the second risk data is obtained from the source database through the scheduled task station. At this time, the data quality module checks the obtained second risk data. If the second risk data meets the check rules, the data asset module stores the second risk data in a preset database table, and then the data governance database is constructed from the various database tables.

[0125] Reference Figure 5 , Figure 5This is a functional module diagram of the first embodiment of the risk data operation monitoring device of the present invention. The present invention also provides a risk data operation monitoring device. The risk data operation monitoring device of the present invention includes:

[0126] The acquisition module 10 receives a detection request and acquires data assets from a pre-built data governance system based on the detection request.

[0127] The monitoring module 20 is used to monitor the dynamics of the data assets, including changes in the data assets.

[0128] Optionally, the monitoring module is further configured to:

[0129] Monitor the flow of the data assets from one storage node to another through transformation tasks;

[0130] Confirm whether there are any abnormalities in the aforementioned flow;

[0131] If any abnormality is detected in the data transfer process, the abnormality of the data assets will be reported to the relevant management personnel.

[0132] Optionally, the monitoring module is further configured to:

[0133] Capture statistical information of the data assets and monitor whether there are any anomalies in the changing trends of the statistical information;

[0134] If an anomaly is detected in the trend, the anomaly will be reported to the relevant management personnel.

[0135] Optionally, the acquisition module is further configured to:

[0136] First risk data is obtained from the source database, and the first risk data is configured as metadata description in the data governance database;

[0137] Based on the configuration data of the data governance database, the second risk data is obtained from the source database through the scheduled task site;

[0138] The second risk data is stored in a preset database table, and then a data governance database is constructed from the various database tables.

[0139] Optionally, the acquisition module is further configured to:

[0140] The detection request is sent to the web site, and after receiving the detection request, the web site sends a service request to the online service site.

[0141] The data asset corresponding to the service request is obtained from the data governance database through the online service site.

[0142] Optionally, the acquisition module is further configured to:

[0143] Confirm whether the second risk data obtained from the source database meets the inspection rules;

[0144] If the second risk data meets the inspection rules, then the following step is executed: store the second risk data in a preset database table.

[0145] Furthermore, embodiments of the present invention also propose a readable storage medium storing a risk data operation monitoring program, wherein the risk data operation monitoring program, when executed by a processor, implements the steps of the risk data operation monitoring method described above.

[0146] The method implemented when the risk data operation monitoring program running on the processor is executed can be referred to in various embodiments of the risk data operation monitoring method of the present invention, and will not be repeated here.

[0147] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0148] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0150] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for monitoring risk data operations, characterized in that, The risk data operation monitoring method includes the following steps: Receive a detection request, and obtain data assets from a pre-built data governance system based on the detection request. The data governance system includes a scheduled task site, a data governance database, a source database, a web site, and an online service site. The web site is located in the internal and external network firewall zones. Prior to the step of receiving a detection request and obtaining data assets from a pre-built data governance system based on the detection request, the method further includes: First risk data is obtained from the source database, and the first risk data is configured as metadata description in the data governance database; Based on the configuration data of the data governance database, the second risk data is obtained from the source database through the scheduled task site; After the step of obtaining the second risk data from the source database, the method further includes: Confirm whether the second risk data obtained from the source database meets the inspection rules; If the second risk data meets the inspection rules, then the following step is executed: store the second risk data in a preset database table; The step of obtaining second risk data from the source database based on the configuration data of the data governance database through the scheduled task site includes: Based on the configuration data of the data governance database, the SDK package is provided to the job site corresponding to the source database through the scheduled task site; Based on the SDK package, the job information of the job site is collected through the unified log platform; The task information is sent to the scheduled task station to obtain the second risk data; The second risk data is stored in a preset database table, and then a data governance database is constructed from each of the database tables; The step of acquiring data assets from a pre-built data governance system includes: The detection request is sent to the web site, and after receiving the detection request, the web site sends a service request to the online service site. The data asset corresponding to the service request is obtained from the data governance database through the online service site; The dynamics of the data assets are monitored, including the circulation of the data assets. The step of dynamically monitoring the data assets includes: Monitor the flow of the data assets from one storage node to another through transformation tasks; Confirm whether there are any abnormalities in the aforementioned flow; If any abnormality is detected in the flow of data assets, the abnormality will be reported to the relevant management personnel. The dynamics of the data assets include: the changing trends of the data assets, and the step of monitoring the dynamics of the data assets further includes: Capture statistical information of the data assets and monitor whether there are any anomalies in the changing trends of the statistical information; If an anomaly is detected in the trend, the anomaly will be reported to the relevant management personnel.

2. A risk data operation monitoring device, characterized in that, The risk data operation monitoring device includes: The acquisition module receives a detection request and, based on the detection request, acquires data assets from a pre-built data governance system. The data governance system includes a scheduled task site, a data governance database, a source database, a web site, and an online service site. The web site is located in the internal and external network firewall zones. The step of receiving a detection request and obtaining data assets from a pre-built data governance system based on the detection request further includes: First risk data is obtained from the source database, and the first risk data is configured as metadata description in the data governance database; Based on the configuration data of the data governance database, the second risk data is obtained from the source database through the scheduled task site; Confirm whether the second risk data obtained from the source database meets the inspection rules; If the second risk data meets the inspection rules, then the following step is executed: store the second risk data in a preset database table; The step of obtaining second risk data from the source database based on the configuration data of the data governance database through the scheduled task site includes: Based on the configuration data of the data governance database, the SDK package is provided to the job site corresponding to the source database through the scheduled task site; Based on the SDK package, the job information of the job site is collected through the unified log platform; The task information is sent to the scheduled task station to obtain the second risk data; The second risk data is stored in a preset database table, and then a data governance database is constructed from each of the database tables; The step of acquiring data assets from a pre-built data governance system includes: The detection request is sent to the web site, and after receiving the detection request, the web site sends a service request to the online service site. The data asset corresponding to the service request is obtained from the data governance database through the online service site; The monitoring module monitors the dynamics of the data assets, including the flow of data assets; wherein the steps for monitoring the dynamics of the data assets include: Monitor the flow of the data assets from one storage node to another through transformation tasks; Confirm whether there are any abnormalities in the aforementioned flow; If any abnormality is detected in the flow of data assets, the abnormality will be reported to the relevant management personnel. Capture statistical information of the data assets and monitor whether there are any anomalies in the changing trends of the statistical information; If an anomaly is detected in the trend, the anomaly will be reported to the relevant management personnel.

3. A risk data operation monitoring system, characterized in that, The risk data operation monitoring system includes: The link monitoring module is used to receive detection requests, obtain data assets from a pre-built data governance system based on the detection requests, and monitor the dynamics of the data assets. The dynamics of the data assets include the flow of data assets. The data governance system includes a scheduled task site, a data governance database, a source database, a web site, and an online service site. The web site is located in the internal and external network firewall zones. The link monitoring module is also used to send the detection request to the WEB site, and the WEB site sends a service request to the online service site after receiving the detection request. The data asset corresponding to the service request is obtained from the data governance database through the online service site; Monitor the flow of the data assets from one storage node to another through transformation tasks; Confirm whether there are any abnormalities in the aforementioned flow; If any abnormality is detected in the flow of data assets, the abnormality will be reported to the relevant management personnel. The dynamics of the data assets include: the changing trends of the data assets, and the link monitoring module is also used for: Capture statistical information of the data assets and monitor whether there are any anomalies in the changing trends of the statistical information; If an anomaly is detected in the trend of change, the anomaly will be reported to the relevant management personnel. The risk data operation monitoring system also includes a data asset module, which is used to obtain first risk data from the source database, configure the first risk data as metadata description in the data governance database, obtain second risk data from the source database through the scheduled task station based on the configuration data of the data governance database, store the second risk data in a preset database table, and then construct the data governance database from each of the database tables. The data asset module is also used to confirm whether the second risk data obtained from the source database meets the inspection rules. If the second risk data meets the inspection rules, then the following step is executed: store the second risk data in a preset database table; The data asset module is also used to provide an SDK package to the job site corresponding to the source database through the scheduled task site based on the configuration data of the data governance database. Based on the SDK package, the job information of the job site is collected through the unified log platform; The task information is sent to the scheduled task station to obtain the second risk data; The second risk data is stored in a preset database table, and then a data governance database is constructed from each of the database tables; The data quality module is used to confirm whether the second risk data obtained from the source database meets the inspection rules. If the second risk data meets the inspection rules, the data asset module stores the second risk data into a preset database table.

4. A risk data operation monitoring device, characterized in that, The risk data operation monitoring device includes: a memory, a processor, and a page generation program stored in the memory and executable on the processor. When the risk data operation monitoring program is executed by the processor, it implements the steps of the risk data management method as described in claim 1.

5. A storage medium, characterized in that, The storage medium stores a risk data operation monitoring program, which, when executed by a processor, implements the steps of the risk data operation monitoring method as described in claim 1.

Images