A real-time monitoring method and system for data kinship change

The real-time monitoring method and system address the inability of traditional methods to track data lineage changes by constructing diagrams and using a big language analysis model to detect and alert on anomalies, ensuring timely detection and response.

CN118689945BActive Publication Date: 2024-12-03HANGZHOU JUYI ZHILIAN TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410871572.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-01
Publication Date
2024-12-03
Estimated Expiration
2044-07-01

AI Technical Summary

Technical Problem

Traditional data lineage management methods cannot track changes in data lineage in real time, failing to detect anomalies promptly.

Method used

A real-time monitoring method and system that constructs data lineage relationship diagrams, calculates similarities, and uses a big language analysis model to identify and alert on data lineage anomalies.

Benefits of technology

Enables real-time tracking and quick response to data lineage anomalies, improving data quality and security by performing two-level analysis on data lineage changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118689945B_ABST
    Figure CN118689945B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of data management technology, and provides a real-time monitoring method and system for data lineage relationship changes. After the real-time operation data is acquired, the scheme of the present invention can perform a two-level analysis on whether there is a data lineage abnormality, and judge whether the current real-time operation data has a data lineage abnormality through a pre-generated data lineage graph. Compared with the existing static data lineage management method, it can track the changes in data lineage in real time and respond quickly to abnormal situations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a real-time monitoring method and system for data lineage relationship changes. Background Art

[0002] With the rapid development of big data technology, the amount of data has exploded, and data lineage relationships have become increasingly complex. Data lineage refers to the generation and dependency relationship between data, and is an important guarantee for data quality and data security. However, traditional static data lineage management methods cannot track changes in data lineage in real time. They can only check data lineage regularly, but cannot detect data lineage anomalies in a timely manner.

[0003] It can be seen that the existing management of data lineage relationships cannot meet the real-time requirements and needs to be improved. Summary of the invention

[0004] In this regard, the present invention provides a real-time monitoring method, system, electronic device, computer storage medium and computer program product for data lineage relationship changes to solve the above-mentioned technical problems.

[0005] The present invention provides a real-time monitoring method for data kinship change, the method comprising the following steps:

[0006] Collecting first source data of a target application scenario, processing the first source data, constructing a first data lineage relationship diagram based on the processed first source data, and storing the first data lineage relationship diagram and the first source data together in a data warehouse;

[0007] Monitor the real-time operation data, extract the associated source data associated with the real-time operation data from the first source data stored in the data warehouse, and construct a second data lineage relationship diagram based on the real-time operation data and the associated source data;

[0008] Calculating the similarity between the first data lineage relationship graph and the second data lineage relationship graph in the trunk level mapping relationship;

[0009] If the similarity is lower than a preset value, the first data lineage relationship diagram, the second data lineage relationship diagram and the third data lineage relationship diagram are input into the big language analysis model to obtain a conclusion that the real-time operation data predicted by the big language analysis model has data lineage anomalies; wherein the third data lineage relationship diagram is constructed by the big language analysis model based on second source data of several other application scenarios belonging to the same field as the target application scenario;

[0010] If the conclusion indicates that there is an abnormal data blood relationship in the real-time operation data, an alarm message is output.

[0011] Optionally, processing the first source data and constructing a first data blood relationship graph according to the processed first source data includes:

[0012] Preprocessing the first source data, grouping each data in the first source data according to the fields associated with each business type; wherein each business type includes multiple fields, and each field is configured with different hierarchical attributes, and the hierarchical attributes include a trunk level, an Nth branch level, and N=1,…,n;

[0013] The mapping relationships between the groups are determined according to the hierarchical attributes, and the mapping relationships form a mapping relationship table. The first data blood relationship diagram is obtained according to the mapping relationship table and the grouped data.

[0014] Optionally, monitoring the real-time operation data, and extracting associated source data associated with the real-time operation data from the first source data stored in the data warehouse, includes:

[0015] Convert the real-time operation data into natural language text, perform semantic analysis on the natural language text using a BERT model, and predict a corresponding first business type;

[0016] According to the first business type, a plurality of second business types are matched from the mapping relationship table of the first data lineage relationship diagram, and the data under each field belonging to the second business type in the data warehouse is determined as associated source data.

[0017] Optionally, the matching of the first business type from a mapping relationship table of the first data lineage relationship graph to obtain a plurality of second business types includes:

[0018] According to the first business type, a plurality of the second business types having a business relevance higher than a relevance threshold are matched from a mapping relationship table of the first data lineage relationship diagram, and are sorted from high to low according to the relevance;

[0019] Acquire abnormal records of recent data lineage abnormalities in the target application scenario, draw an abnormal record trend graph based on each abnormal record, numerically represent the abnormal trend of the abnormal record trend graph, and obtain the recent abnormality degree;

[0020] The associated quantity is determined according to the recent abnormality, and the second business type with the associated quantity that is at the top after sorting is filtered.

[0021] Optionally, the calculating the similarity between the first data lineage relationship graph and the second data lineage relationship graph in a trunk-level mapping relationship includes:

[0022] Determine a plurality of trunk-level fields according to the hierarchical attributes, and simplify the mapping relationship tables of the first data lineage relationship diagram and the second data lineage relationship diagram into a first mapping relationship table and a second mapping relationship table according to the trunk-level fields;

[0023] A first mapping feature matrix and a second mapping feature matrix are extracted from the first mapping relationship table and the second mapping relationship table respectively, and a similarity between the first mapping feature matrix and the second mapping feature matrix is ​​calculated.

[0024] Optionally, the step of inputting the first data blood relationship graph, the second data blood relationship graph, and the third data blood relationship graph into a large language analysis model to obtain a conclusion that the real-time operation data has data blood relationship anomaly predicted by the large language analysis model includes:

[0025] The first model of the large language analysis model is a convolutional network model, and the first model extracts features from the first data blood relationship graph, the second data blood relationship graph, and the third data blood relationship graph to obtain a first blood relationship feature, a second blood relationship feature, and a third blood relationship feature;

[0026] The second model of the large language analysis model is a Transformer model, and the second model performs deep processing on the first blood relationship feature and the second blood relationship feature to obtain a first abnormality evaluation value of the data blood relationship abnormality; and performs deep processing on the second blood relationship feature and the third blood relationship feature to obtain a second abnormality evaluation value of the data blood relationship abnormality; and performs deep processing on the first blood relationship feature and the third blood relationship feature to obtain a third abnormality evaluation value of the data blood relationship abnormality;

[0027] The third model of the large language analysis model calculates a correction coefficient based on the second abnormality evaluation value and the third abnormality evaluation value, and multiplies the first abnormality evaluation value by the correction coefficient to obtain a fourth abnormality evaluation value. If the fourth abnormality evaluation value is higher than the abnormality threshold, a first conclusion that the data blood relationship is abnormal is generated; otherwise, a second conclusion that the data blood relationship is not abnormal is generated.

[0028] The present invention also provides a real-time monitoring system for data kinship change, the system comprising a first construction module, a second construction module, a first abnormality analysis module, a second abnormality analysis module, and an alarm module;

[0029] The first construction module is used to collect first source data of a target application scenario, process the first source data, construct a first data lineage relationship diagram based on the processed first source data, and store the first data lineage relationship diagram and the first source data together in a data warehouse;

[0030] The second construction module is used to monitor the real-time operation data, extract the associated source data associated with the real-time operation data from the first source data stored in the data warehouse, and construct a second data lineage relationship diagram based on the real-time operation data and the associated source data;

[0031] The first anomaly analysis module is used to calculate the similarity between the first data blood relationship graph and the second data blood relationship graph in the trunk level mapping relationship;

[0032] The second abnormality analysis module is used for inputting the first data lineage relationship diagram, the second data lineage relationship diagram and the third data lineage relationship diagram into the big language analysis model if the similarity is lower than a preset value, so as to obtain the conclusion that the real-time operation data predicted by the big language analysis model has data lineage abnormality; wherein the third data lineage relationship diagram is constructed by the big language analysis model based on the second source data of several other application scenarios belonging to the same field as the target application scenario;

[0033] The alarm module is used to output an alarm message if the conclusion indicates that there is an abnormal data blood relationship in the real-time operation data.

[0034] The present invention also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the computer program implements the method described in any of the preceding items when executed by the processor.

[0035] The present invention also provides a computer storage medium storing a computer program executable by a processor to implement any of the methods described above.

[0036] The present invention also provides a computer program product, which includes a computer program that can be executed by a processor to implement any of the methods described above.

[0037] The beneficial effect of the present invention is that after the real-time operation data is acquired, the scheme of the present invention can perform a two-level analysis on whether there is any data lineage anomaly, and judge whether the data lineage anomaly is present in the current real-time operation data through a pre-generated data lineage graph. Compared with the existing static data lineage management method, it can track the changes in data lineage in real time and respond quickly to abnormal situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a flow chart of a method for real-time monitoring of data lineage relationship changes disclosed in an embodiment of the present invention;

[0040] Figure 2 is an example diagram of a mapping relationship table disclosed in an embodiment of the present invention;

[0041] Figure 3 It is a structural schematic diagram of a real-time monitoring system for data lineage relationship changes disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following is a description of the implementation of the present application by specific specific embodiments. People familiar with the technology can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0043] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0044] like Figure 1 As shown, the embodiment of the present invention discloses a real-time monitoring method for data lineage relationship changes, the method comprising the following steps:

[0045] Collecting first source data of a target application scenario, processing the first source data, constructing a first data lineage relationship diagram based on the processed first source data, and storing the first data lineage relationship diagram and the first source data together in a data warehouse;

[0046] Monitor the real-time operation data, extract the associated source data associated with the real-time operation data from the first source data stored in the data warehouse, and construct a second data lineage relationship diagram based on the real-time operation data and the associated source data;

[0047] Calculating the similarity between the first data lineage relationship graph and the second data lineage relationship graph in the trunk level mapping relationship;

[0048] If the similarity is lower than a preset value, the first data lineage relationship diagram, the second data lineage relationship diagram and the third data lineage relationship diagram are input into the big language analysis model to obtain a conclusion that the real-time operation data predicted by the big language analysis model has data lineage anomalies; wherein the third data lineage relationship diagram is constructed by the big language analysis model based on second source data of several other application scenarios belonging to the same field as the target application scenario;

[0049] If the conclusion indicates that there is an abnormal data blood relationship in the real-time operation data, an alarm message is output.

[0050] First, collect the historical operation data (involving a variety of different businesses) of the target application scenario (such as banking, transportation, shopping platforms, etc.), use it as the source data for data lineage change analysis, analyze and obtain the first data lineage diagram of these source data, and store the first data lineage diagram and the processed source data in the data warehouse. After completing the above preparation stage, monitor the real-time operation data. These data are the new operation data generated by the above target application scenario in the current period, such as bank loan approval data, etc. Through specific association relationships, some associated source data with a large association relationship with the real-time operation data can be extracted from the source data stored in the data warehouse, such as bank loan approval certification materials, etc. The same analysis method is used to process the real-time operation data and the associated source data to construct a second data lineage diagram. Next, the similarity between the second data kinship graph and the first data kinship graph is calculated at the trunk-level mapping relationship level. When the similarity is large enough, it means that the probability of data kinship anomaly in the real-time operation data is small and no further analysis is required. When the similarity is not large enough, it means that the probability of data kinship anomaly in the real-time operation data is large. The first data kinship graph, the second data kinship graph and the third data kinship graph pre-constructed from the source data of several other application scenarios in the same field are input into the big language analysis model, and the big language analysis model conducts a more in-depth analysis of whether the second data kinship graph really has data kinship anomaly (i.e., whether the real-time operation data is abnormal). If the big language analysis model also determines that the real-time operation data has data kinship anomaly, it will promptly output an alarm message to notify relevant personnel to handle it.

[0051] It can be seen that after the real-time operation data is acquired, the solution of the present invention can perform a two-level analysis on whether there is any data lineage anomaly. The pre-generated data lineage graph can be used to determine whether the current real-time operation data has a data lineage anomaly. Compared with the existing static data lineage management method, it can track the changes in data lineage in real time and respond quickly to abnormal situations.

[0052] The source data can be collected by proxy, log analysis, code analysis, etc., which can be determined according to the characteristics of the target application scenario, and the types and sources of collected data should be as many as possible. In addition, for abnormal alarms of data lineage, emails, text messages, WeChat, etc. can be used, and the present invention does not limit this.

[0053] The target application scenario may be a certain cat shopping platform, and other application scenarios may be a certain Dong shopping platform, a certain Hui shopping platform, and other platforms belonging to the electronic shopping field. The above is only an example, and is not used to limit the specific types of application scenarios of the present invention.

[0054] Optionally, processing the first source data and constructing a first data blood relationship graph according to the processed first source data includes:

[0055] Preprocessing the first source data, grouping each data in the first source data according to the fields associated with each business type; wherein each business type includes multiple fields, and each field is configured with different hierarchical attributes, and the hierarchical attributes include a trunk level, an Nth branch level, and N=1,…,n;

[0056] The mapping relationships between the groups are determined according to the hierarchical attributes, and each of the mapping relationships constitutes a mapping relationship table (such as Figure 2 As shown), the first data blood relationship diagram is constructed based on the mapping relationship table and the grouped data.

[0057] In this embodiment, the collected first source data may contain abnormal data such as errors and missing data, which need to be processed by deletion, replacement, etc., and also include normalization and conversion of data of different structures. The collected data are all marked with their own business attributes, which include both business types and specific business names under the business types, i.e., fields. For example, a certain data is loan data, and its data format is [A, B1, B2, C], where B1 is "loan application", B2 is "first-level approval", C is "second-level approval", and C is "fund transfer", and each item is configured with corresponding operation data. According to the above fields, each operation data can be divided into corresponding groups, and each group corresponds to the corresponding field. At the same time, the fields corresponding to each business are pre-configured with hierarchical attributes, for example, A, B, and C are trunk levels, B1 and B2 are branch levels, and the branch level can be configured with n levels. According to the hierarchical attributes, the mapping relationship between the groups can be obtained, and all the mapping relationships can be integrated to obtain a mapping relationship table, and then the data of each group is mapped to the corresponding node position in the mapping relationship table, that is, the first data lineage relationship diagram is obtained. Since the construction method of the second data lineage relationship diagram and the third data lineage relationship diagram is the same as that of the first data lineage relationship diagram, the present invention will not repeat the construction process of the two here.

[0058] Optionally, monitoring the real-time operation data, and extracting associated source data associated with the real-time operation data from the first source data stored in the data warehouse, includes:

[0059] Convert the real-time operation data into natural language text, perform semantic analysis on the natural language text using a BERT model, and predict a corresponding first business type;

[0060] According to the first business type, a plurality of second business types are matched from the mapping relationship table of the first data lineage relationship diagram, and the data under each field belonging to the second business type in the data warehouse is determined as associated source data.

[0061] In this embodiment, after monitoring the newly generated real-time operation data of the system, the NLG (Natural Language Generation) technology is first used to convert it into natural language text, and the BERT model is used to perform semantic understanding on the natural language text, so that the business type (even including the corresponding fields) to which the real-time operation data belongs can be predicted. Several second business types that have an association relationship with the first business type are selected from the data, and then the data corresponding to these second business types are determined as the associated source data.

[0062] Optionally, the matching of the first business type from a mapping relationship table of the first data lineage relationship graph to obtain a plurality of second business types includes:

[0063] According to the first business type, a plurality of the second business types having a business relevance higher than a relevance threshold are matched from a mapping relationship table of the first data lineage relationship diagram, and are sorted from high to low according to the relevance;

[0064] Acquire abnormal records of recent data lineage abnormalities in the target application scenario, draw an abnormal record trend graph based on each abnormal record, numerically represent the abnormal trend of the abnormal record trend graph, and obtain the recent abnormality degree;

[0065] The associated quantity is determined according to the recent abnormality, and the second business type with the associated quantity that is at the top after sorting is filtered.

[0066] In this embodiment, although the amount of associated source data is appropriately larger, it is beneficial to construct a more accurate second data lineage relationship diagram, but it should not be too much, otherwise it will affect the real-time analysis of data lineage anomaly. In this regard, the present invention first screens out several second business types whose business relevance is higher than the association threshold, and sorts them from high to low according to the relevance; then, according to the abnormal records of data lineage anomalies in the target application scenario in the recent period (for example, one week, one month), an abnormal record trend chart is drawn. The abnormal record trend chart represents the recent abnormal trend, such as growth, significant growth, stability, decrease, significant decrease, etc. The abnormal trend in the figure is numerically represented to obtain the recent abnormal degree. Finally, when the recent abnormality is greater, the larger the number of associations is set, that is, the data volume of the associated source data is increased by increasing the number of second business types, so as to improve the accuracy of the second data lineage relationship diagram, and then make a more accurate analysis of recent data lineage anomalies; conversely, the smaller the number of associations is set, that is, the data volume of the associated source data is reduced by reducing the number of second business types, so as to increase the construction rate of the second data lineage relationship diagram, thereby improving the timeliness of the analysis of recent data lineage anomalies.

[0067] For a specific target application scenario, the correlation between all business types involved in the target application scenario can be established in advance, for example, the correlation between loan business and savings business is high, and the correlation between loan business and social security business is low. In addition, the numerical representation of abnormal trends can be calculated by calculating the cumulative value of the difference between adjacent data points in the abnormal record trend chart, where the horizontal axis of the abnormal record trend chart is a number of time periods and the vertical axis is the number of abnormal records in the corresponding time period.

[0068] Optionally, the calculating the similarity between the first data lineage relationship graph and the second data lineage relationship graph in a trunk-level mapping relationship includes:

[0069] Determine a plurality of trunk-level fields according to the hierarchical attributes, and simplify the mapping relationship tables of the first data lineage relationship diagram and the second data lineage relationship diagram into a first mapping relationship table and a second mapping relationship table according to the trunk-level fields;

[0070] A first mapping feature matrix and a second mapping feature matrix are extracted from the first mapping relationship table and the second mapping relationship table respectively, and a similarity between the first mapping feature matrix and the second mapping feature matrix is ​​calculated.

[0071] In this embodiment, the present invention adopts a two-level analysis method, that is, by analyzing the similarity between the first data kinship graph and the second data kinship graph, it is used to identify those real-time operation data with obvious data kinship abnormalities. Therefore, the present invention adopts the method of identifying real-time operation data with obvious data kinship abnormalities based on the similarity in the trunk-level mapping relationship. Specifically, according to the aforementioned hierarchical attributes, it is determined that the fields belonging to several trunk levels are obtained, and the trunk of the aforementioned mapping relationship table is extracted based on this to obtain a simplified first mapping relationship table and a second mapping relationship table, and the simplified table no longer contains branch-level fields. The first mapping relationship table and the second mapping relationship table are subjected to feature transformation to obtain a mapping feature matrix, so that the similarity of the mapping feature matrices of the two can be directly calculated.

[0072] The similarity involved in the present invention can be calculated by formulas such as Euclidean distance, Cosine similarity, and Jaccard distance, but the present invention does not make any limitation thereto.

[0073] Optionally, the step of inputting the first data blood relationship graph, the second data blood relationship graph, and the third data blood relationship graph into a large language analysis model to obtain a conclusion that the real-time operation data has data blood relationship anomaly predicted by the large language analysis model includes:

[0074] The first model of the large language analysis model is a convolutional network model, and the first model extracts features from the first data blood relationship graph, the second data blood relationship graph, and the third data blood relationship graph to obtain a first blood relationship feature, a second blood relationship feature, and a third blood relationship feature;

[0075] The second model of the large language analysis model is a Transformer model, and the second model performs deep processing on the first blood relationship feature and the second blood relationship feature to obtain a first abnormality evaluation value of the data blood relationship abnormality; and performs deep processing on the second blood relationship feature and the third blood relationship feature to obtain a second abnormality evaluation value of the data blood relationship abnormality; and performs deep processing on the first blood relationship feature and the third blood relationship feature to obtain a third abnormality evaluation value of the data blood relationship abnormality;

[0076] The third model of the large language analysis model calculates a correction coefficient based on the second abnormality evaluation value and the third abnormality evaluation value, and multiplies the first abnormality evaluation value by the correction coefficient to obtain a fourth abnormality evaluation value. If the fourth abnormality evaluation value is higher than the abnormality threshold, a first conclusion that the data blood relationship is abnormal is generated; otherwise, a second conclusion that the data blood relationship is not abnormal is generated.

[0077] In this embodiment, the large language analysis model includes three sub-models: a first model, a second model, and a third model. Among them, the first model is a convolutional network model, which is used to extract features from the first data blood relationship graph, the second data blood relationship graph, and the third data blood relationship graph respectively; the second model is a Transformer model, which is the core of the large language model and is used to perform pairwise analysis on the blood relationship features extracted by the first model to obtain the corresponding abnormal evaluation value; the third model is a common calculation and processing model, which is used to calculate the correction coefficient calculated according to the second abnormal evaluation value and the third abnormal evaluation value, and use the correction coefficient to multiply the first abnormal evaluation value to obtain the fourth abnormal evaluation value, and then generate a conclusion on whether it is abnormal or not according to the above relationship.

[0078] Preferably, the correction coefficient = 1-|third abnormal evaluation value-second abnormal evaluation value| / (second abnormal evaluation value+third abnormal evaluation value). This formula is used to characterize the degree of deviation between the first data kinship diagram and the third data kinship diagram in the same field. The greater the degree of deviation, the higher the inaccuracy of the first data kinship diagram, and the correction coefficient is correspondingly adjusted to a smaller value, thereby appropriately reducing the first abnormal evaluation value; conversely, the smaller the degree of deviation, the lower the inaccuracy of the first data kinship diagram, and the correction coefficient is correspondingly adjusted to a larger value, thereby appropriately increasing the first abnormal evaluation value. The above-mentioned correction coefficient is determined by drawing on the general situation of data kinship in other application scenarios in the same field, and is used as a reference to intervene in the conclusion of whether there is an abnormal data kinship in real-time operation data. It should be noted that the correction coefficient will appropriately increase the first abnormal evaluation value when the "difference is smaller", which may cause "allergy" to abnormal data kinship, but this is generally within an acceptable range.

[0079] like Figure 3 As shown, the embodiment of the present invention also discloses a real-time monitoring system for data lineage relationship changes, the system comprising a first construction module, a second construction module, a first abnormality analysis module, a second abnormality analysis module, and an alarm module;

[0080] The first construction module is used to collect first source data of a target application scenario, process the first source data, construct a first data lineage relationship diagram based on the processed first source data, and store the first data lineage relationship diagram and the first source data together in a data warehouse;

[0081] The second construction module is used to monitor the real-time operation data, extract the associated source data associated with the real-time operation data from the first source data stored in the data warehouse, and construct a second data lineage relationship diagram based on the real-time operation data and the associated source data;

[0082] The first anomaly analysis module is used to calculate the similarity between the first data blood relationship graph and the second data blood relationship graph in the trunk level mapping relationship;

[0083] The second abnormality analysis module is used for inputting the first data lineage relationship diagram, the second data lineage relationship diagram and the third data lineage relationship diagram into the big language analysis model if the similarity is lower than a preset value, so as to obtain the conclusion that the real-time operation data predicted by the big language analysis model has data lineage abnormality; wherein the third data lineage relationship diagram is constructed by the big language analysis model based on the second source data of several other application scenarios belonging to the same field as the target application scenario;

[0084] The alarm module is used to output an alarm message if the conclusion indicates that there is an abnormal data blood relationship in the real-time operation data.

[0085] An embodiment of the present invention further discloses an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the computer program implements the method described in any of the preceding items when executed by the processor.

[0086] An embodiment of the present invention further discloses a computer storage medium, which stores a computer program that can be executed by a processor to implement any of the methods described in the preceding items.

[0087] An embodiment of the present invention further discloses a computer program product, which includes a computer program that can be executed by a processor to implement any of the methods described in the preceding items.

[0088] It should be understood that the terms used in the present application are only for the purpose of describing specific embodiments, but are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "having", "may have", "include" and "include", or "may include" and "may include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but the presence of additional features is not excluded.

[0089] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. The technical solutions recorded in the embodiments of the present application can be arbitrarily combined without conflict. In several embodiments provided in this application, it should be understood that the disclosed methods, devices and clothing processing equipment can be implemented in other ways. The embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the clothing processing equipment or units can be electrical, mechanical or other forms.

[0090] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0091] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units. The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. A real-time monitoring method for data lineage relationship changes, characterized by: The method comprises the following steps: Collecting first source data of a target application scenario, processing the first source data, constructing a first data lineage relationship diagram based on the processed first source data, and storing the first data lineage relationship diagram and the first source data together in a data warehouse; Monitor the real-time operation data, extract the associated source data associated with the real-time operation data from the first source data stored in the data warehouse, and construct a second data lineage relationship diagram based on the real-time operation data and the associated source data; Calculating the similarity between the first data lineage relationship graph and the second data lineage relationship graph in the trunk level mapping relationship; If the similarity is lower than a preset value, the first data lineage relationship diagram, the second data lineage relationship diagram and the third data lineage relationship diagram are input into a large language analysis model to obtain a conclusion on whether the real-time operation data predicted by the large language analysis model has data lineage abnormality; wherein the third data lineage relationship diagram is constructed by the large language analysis model based on second source data of several other application scenarios belonging to the same field as the target application scenario; If the conclusion indicates that the real-time operation data has abnormal data blood relationship, outputting an alarm message; The step of inputting the first data blood relationship graph, the second data blood relationship graph, and the third data blood relationship graph into a large language analysis model to obtain a conclusion of whether the real-time operation data predicted by the large language analysis model has data blood relationship anomalies includes: The first model of the large language analysis model is a convolutional network model, and the first model extracts features from the first data blood relationship graph, the second data blood relationship graph, and the third data blood relationship graph to obtain a first blood relationship feature, a second blood relationship feature, and a third blood relationship feature; The second model of the large language analysis model is a Transformer model, and the second model performs deep processing on the first blood relationship feature and the second blood relationship feature to obtain a first abnormality evaluation value of the data blood relationship abnormality; and performs deep processing on the second blood relationship feature and the third blood relationship feature to obtain a second abnormality evaluation value of the data blood relationship abnormality; and performs deep processing on the first blood relationship feature and the third blood relationship feature to obtain a third abnormality evaluation value of the data blood relationship abnormality; The third model of the large language analysis model calculates a correction coefficient based on the second abnormality evaluation value and the third abnormality evaluation value, and multiplies the first abnormality evaluation value by the correction coefficient to obtain a fourth abnormality evaluation value. If the fourth abnormality evaluation value is higher than the abnormality threshold, a first conclusion that the data blood relationship is abnormal is generated; otherwise, a second conclusion that the data blood relationship is not abnormal is generated.

2. A real-time monitoring method for data lineage relationship changes according to claim 1, characterized in that: The processing of the first source data and constructing a first data blood relationship diagram according to the processed first source data includes: Preprocessing the first source data, grouping each data in the first source data according to the fields associated with each business type; wherein each business type includes multiple fields, and each field is configured with different hierarchical attributes, and the hierarchical attributes include a trunk level, an Nth branch level, and N=1,…,n; The mapping relationships between the groups are determined according to the hierarchical attributes, and the mapping relationships form a mapping relationship table. The first data blood relationship diagram is obtained according to the mapping relationship table and the grouped data.

3. A real-time monitoring method for data lineage relationship changes according to claim 2, characterized in that: Monitoring the real-time operation data, and extracting associated source data associated with the real-time operation data from the first source data stored in the data warehouse, comprising: Convert the real-time operation data into natural language text, perform semantic analysis on the natural language text using a BERT model, and predict a corresponding first business type; According to the first business type, a plurality of second business types are matched from the mapping relationship table of the first data lineage relationship diagram, and the data under each field belonging to the second business type in the data warehouse is determined as associated source data.

4. A real-time monitoring method for data lineage relationship changes according to claim 3, characterized in that: The matching of obtaining a plurality of second business types from a mapping relationship table of the first data lineage relationship diagram according to the first business type includes: According to the first business type, a plurality of the second business types having a business relevance higher than a relevance threshold are matched from a mapping relationship table of the first data lineage relationship diagram, and are sorted from high to low according to the relevance; Acquire abnormal records of recent data lineage abnormalities in the target application scenario, draw an abnormal record trend graph based on each abnormal record, numerically represent the abnormal trend of the abnormal record trend graph, and obtain the recent abnormality degree; The associated quantity is determined according to the recent abnormality, and the second business type with the associated quantity that is at the top after sorting is filtered.

5. A real-time monitoring method for data lineage relationship changes according to claim 4, characterized in that: The calculating the similarity between the first data lineage relationship graph and the second data lineage relationship graph in the trunk-level mapping relationship includes: Determine a plurality of trunk-level fields according to the hierarchical attributes, and simplify the mapping relationship tables of the first data lineage relationship diagram and the second data lineage relationship diagram into a first mapping relationship table and a second mapping relationship table according to the trunk-level fields; A first mapping feature matrix and a second mapping feature matrix are extracted from the first mapping relationship table and the second mapping relationship table respectively, and a similarity between the first mapping feature matrix and the second mapping feature matrix is ​​calculated.

6. A real-time monitoring system for data lineage relationship changes, the system is based on the method according to any one of claims 1 to 5, the system comprises a first construction module, a second construction module, a first abnormality analysis module, a second abnormality analysis module, and an alarm module; characterized in that: The first construction module is used to collect first source data of a target application scenario, process the first source data, construct a first data lineage relationship diagram based on the processed first source data, and store the first data lineage relationship diagram and the first source data together in a data warehouse; The second construction module is used to monitor the real-time operation data, extract the associated source data associated with the real-time operation data from the first source data stored in the data warehouse, and construct a second data lineage relationship diagram based on the real-time operation data and the associated source data; The first anomaly analysis module is used to calculate the similarity between the first data blood relationship graph and the second data blood relationship graph in the trunk level mapping relationship; The second abnormality analysis module is used to input the first data lineage relationship diagram, the second data lineage relationship diagram and the third data lineage relationship diagram into the big language analysis model if the similarity is lower than a preset value, to obtain a conclusion as predicted by the big language analysis model whether the real-time operation data has data lineage abnormality; wherein the third data lineage relationship diagram is constructed by the big language analysis model based on second source data of several other application scenarios belonging to the same field as the target application scenario; The alarm module is used to output an alarm message if the conclusion indicates that there is an abnormal data blood relationship in the real-time operation data.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the computer program implements the method according to any one of claims 1 to 5 when executed by the processor.

8. A computer storage medium, characterized in that: The computer storage medium stores a computer program that can be executed by a processor to implement the method according to any one of claims 1 to 5.

9. A computer program product, characterized in that: The computer program product comprises a computer program executable by a processor to implement the method according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Data blood relationship display method and device for data warehouse and electronic equipment

    CN111639143A

  • Data blood relationship analysis method and device, equipment and storage medium

    CN113486008A