Big data blood relationship data dynamic management method, system, device and medium

Abstract

The present disclosure relates to a big data blood relationship data dynamic management method, system, medium and equipment, and relates to the technical field of big data, wherein the method comprises: collecting big data blood relationship data and storing them according to the collected blood relationship metadata types respectively; performing data processing on the point attribute change metadata and the new metadata written in kafka respectively; wherein the new metadata comprises: new point attribute metadata and new edge attribute metadata; updating the big data blood relationship data in the graph database according to the results of data processing; and displaying the big data blood relationship graph according to the updated big data blood relationship data. The technical scheme of the present disclosure supports global blood relationship acquisition and display of the big data platform, has strong expansibility, low degree of customization, and low development cost for new component access. The end scheme of the present disclosure may have the case of mis-collection for the new blood relationship, and the system supports AI and manual verification double-checking mechanism to ensure accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of big data technology, and more specifically, to methods, systems, devices, and media for dynamic management of big data lineage data. Background Technology

[0002] In the big data field, lineage management typically employs open-source technologies such as Apache Atlas. However, these solutions focus primarily on metadata collection and management, offering very limited support for data lineage. Current open-source solutions generally only support lineage parsing for Hive data warehouses. As business scenarios and requirements become increasingly complex, big data increasingly utilizes various storage engines to address these issues, such as Hive, HBase, Kafka, Elasticsearch, Redis, and MySQL. At this point, simply displaying Hive's data lineage reveals significant limitations: it fails to reveal the global dependencies across the entire big data platform's upstream and downstream systems, and it hinders rapid data tracing and anomaly localization. Lineage changes are also a common scenario, requiring users to understand the upstream and downstream relationships of the changed objects and assess their impact. Furthermore, the accuracy of collected lineage data is questionable, lacking effective solutions for handling erroneous lineage relationships or metadata. Summary of the Invention

[0003] To address the technical problem that existing big data lineage management systems cannot meet users' data management needs.

[0004] To achieve the above technical objectives, this disclosure provides a method for dynamic management of big data lineage data, including:

[0005] Collect large-scale kinship data and store it according to the type of kinship metadata collected;

[0006] Data processing is performed based on the node attribute change metadata and new metadata written in Kafka; wherein, the new metadata includes: new node attribute metadata and new edge attribute metadata;

[0007] Update the big data lineage data in the graph database based on the results of data processing;

[0008] A big data kinship diagram is displayed based on the updated big data kinship data.

[0009] Furthermore, the data processing based on the point attribute change metadata and newly added metadata written in Kafka specifically includes:

[0010] Change parsing and offline correction processing are performed on the point attribute change metadata.

[0011] Perform real-time accuracy assessment on newly added metadata.

[0012] Furthermore, the real-time accuracy assessment of the newly added metadata specifically includes:

[0013] The accuracy of newly added metadata is judged in real time based on a preset threshold. If it is greater than or equal to the preset threshold, the subsequent steps of updating the big data lineage data in the graph database will continue directly.

[0014] If the data is less than the preset threshold, manual review is required, and the data that passes the manual review will be updated in the big data lineage data in the graph database.

[0015] Furthermore, the real-time judgment of the accuracy of newly added metadata based on a preset threshold specifically involves:

[0016] Using the initial and accurate kinship data in the graph database as the training set, the accuracy of newly added metadata is judged in real time.

[0017] Furthermore, the change parsing process specifically includes:

[0018] Retrieve the upstream and downstream relationships of changed point attribute metadata from the graph database, issue data change warnings for the entities corresponding to the changed point attribute metadata upstream and downstream relationships, and update the changed point attribute metadata information to the graph database after the warning ends.

[0019] Furthermore, the offline correction process specifically includes:

[0020] The modified kinship metadata is collected from the data acquisition component, and inaccurate kinship metadata in the graph database is corrected to ensure that the metadata displayed by the big data kinship data dynamic management system is completely consistent with the kinship metadata obtained by the component itself.

[0021] Furthermore, the collection of big data kinship data specifically includes:

[0022] Collect point attribute metadata via HBase, Hive, and / or Kafka, and collect edge attribute metadata via Spark, Hive, Kafka, and / or Flume.

[0023] To achieve the above technical objectives, this disclosure also provides a big data lineage data dynamic management system, including:

[0024] The bloodline data acquisition module is used to collect large amounts of bloodline data and store them according to the type of bloodline metadata collected.

[0025] The lineage data processing module is used to process data based on the node attribute change metadata and new metadata written in Kafka; wherein, the new metadata includes: new node attribute metadata and new edge attribute metadata;

[0026] The bloodline data update module is used to update the big data bloodline data in the graph database based on the results of data processing.

[0027] The bloodline display module is used to display a big data bloodline diagram based on the updated big data bloodline data.

[0028] To achieve the above technical objectives, this disclosure also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, is used to implement the steps of the above-described method for dynamic management of big data lineage data.

[0029] To achieve the above-mentioned technical objectives, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-mentioned big data lineage data dynamic management method.

[0030] The beneficial effects of this disclosure are as follows:

[0031] 1. The technical solution disclosed herein supports the global lineage acquisition and display of big data platforms, has strong scalability, low degree of customization, and low development cost for integrating new components.

[0032] 2. The termination solution disclosed herein supports real-time early warning and dynamic real-time updating of the bloodline view for upstream and downstream related bloodlines in scenarios with dynamic changes in bloodlines, and has an offline correction mechanism in cases of abnormal dynamic updates.

[0033] 3. The termination scheme disclosed herein may result in erroneous data collection for newly added blood relationships. The system supports a dual verification mechanism of AI and manual verification to ensure accuracy. Attached Figure Description

[0034] Figure 1 A flowchart illustrating the method of Embodiment 1 of this disclosure is shown;

[0035] Figure 2 A schematic diagram of the system structure of Embodiment 2 of this disclosure is shown;

[0036] Figure 3 A schematic diagram of the system structure of Embodiment 2 of this disclosure is shown;

[0037] Figure 4 A schematic diagram of the structure of Embodiment 4 of this disclosure is shown. Detailed Implementation

[0038] Embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0039] The accompanying drawings illustrate various structural schematics according to embodiments of the present disclosure. These drawings are not to scale, and some details have been enlarged and may have been omitted for clarity. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0040] In human society, kinship refers to interpersonal relationships arising from marriage or procreation, and is the earliest form of social relationship. Today, humanity has entered the era of big data. Every day, massive amounts of data of various types and complex relationships are rapidly generated worldwide. This vast and complex data converges to generate new data. In the process of data generation, fusion, flow, and disappearance, a logical relationship is formed. We borrow the concept of kinship in human society to express this relationship between data, calling it the kinship of data.

[0041] The lineage of data includes some unique characteristics: data attribution (the organization or individual to which the data belongs), data source diversity (the same data comes from more than one source), data traceability (the entire process of data from its creation to its destruction), and data hierarchy (the classification, induction, and summarization of data form data hierarchies), etc.

[0042] The existing technical solution has the following drawbacks:

[0043] 1. It only supports Hive lineage, cannot cover other storage components for big data, and cannot form a global lineage view. It has poor scalability, low customizability, and high development costs for integrating new components.

[0044] 2. In scenarios where blood relations change, it is impossible to dynamically update the bloodline view in real time. Furthermore, if data loss occurs during dynamic updates, it will lead to inconsistent metadata.

[0045] 3. Blood relation data is often collected by mistake, and there is no efficient and effective mechanism to ensure the accuracy of blood relation data.

[0046] In big data, the data lineage graph is divided into two parts: vertex data and edge data. "Vertex" represents metadata, including database table information, field information, responsible parties, descriptions, etc., while "edge" represents the dependencies between metadata, containing upstream and downstream storage type information, database table information, dependent fields, etc. For different storage engines, this system provides different customizable plugins to collect and aggregate vertex and edge data, ensuring that metadata information is sent to the lineage system during data processing, guaranteeing the timeliness of system data.

[0047] Example 1:

[0048] like Figure 1 As shown:

[0049] This disclosure provides a big data lineage data dynamic management system, including:

[0050] S101: Collect big data on bloodline data and store it according to the type of bloodline metadata collected.

[0051] Furthermore, the collection of big data kinship data specifically includes:

[0052] Collect point attribute metadata via HBase, Hive, and / or Kafka;

[0053] 1. HBase Hook is implemented using a coprocessor. Every time data in the table structure is added, deleted, or modified, the operation instructions in the coprocessor to collect metadata are triggered.

[0054] 2. Hive Hook uses the Hive interface provided by Hive itself. The Hook is triggered after Hive parses the SQL, so compared to manually parsing SQL or using open-source SQL parsing tools, Hook has higher parsing accuracy. Hook can collect all CRUD operations on the Hive table structure (i.e., the metadata information required by this system).

[0055] 3. For storage engines that cannot integrate Hooks into the system to trigger real-time metadata reporting, use a timed polling method to obtain metadata, such as Kafka, ElasticSearch, MySQL, etc.

[0056] And collect edge attribute metadata via Spark, Hive, Kafka and / or Flume.

[0057] 1. Hive Hook is a special case in this system. It has the attributes of both data storage and data processing, so it can collect vertex and edge information at the same time. Both DDL and DML SQL will trigger Hook. DDL triggers vertex metadata collection, and DML triggers edge dependency collection.

[0058] 2. Hook code embedded in the Spark user-developed code to record and report the data source of Spark, the target storage to which it is written, and field information, etc.

[0059] 3. Flume uses interceptors to obtain and report the data source and target storage.

[0060] 4. Datax, Canal, Sqoop, and other similar software obtain upstream and downstream storage dependencies by parsing configuration files.

[0061] S102: Perform data processing based on the node attribute change metadata and new metadata written in Kafka; wherein, the new metadata includes: new node attribute metadata and new edge attribute metadata.

[0062] Kafka, developed by the Apache Software Foundation, is an open-source stream processing platform written in Scala and Java. It's a high-throughput, distributed publish-subscribe messaging system capable of handling streaming data from all consumer actions on a website. These actions (web browsing, searching, and other user actions) are a key element of many social functions on the modern web. This data is typically handled through log processing and log aggregation due to throughput requirements. Kafka offers a viable solution for log data and offline analytics systems like Hadoop, but with the constraint of real-time processing. Kafka aims to unify online and offline message processing through Hadoop's parallel loading mechanism, and also to provide real-time messaging via clustering.

[0063] Furthermore, the data processing based on the point attribute change metadata and newly added metadata written in Kafka specifically includes:

[0064] Change parsing and offline correction processing are performed on the point attribute change metadata.

[0065] Perform real-time accuracy assessment on newly added metadata.

[0066] Specifically, the real-time accuracy assessment of the newly added metadata includes:

[0067] The accuracy of newly added metadata is judged in real time based on a preset threshold. If it is greater than or equal to the preset threshold, the subsequent steps of updating the big data lineage data in the graph database will continue directly.

[0068] If the data is less than the preset threshold, manual review is required, and the data that passes the manual review will be updated in the big data lineage data in the graph database.

[0069] The preferred value of the preset threshold is 0.6.

[0070] Specifically, the offline correction process includes:

[0071] The modified kinship metadata is collected from the data acquisition component, and inaccurate kinship metadata in the graph database is corrected to ensure that the metadata displayed by the big data kinship data dynamic management system is completely consistent with the kinship metadata obtained by the component itself.

[0072] Specifically, the change parsing process includes:

[0073] Retrieve the upstream and downstream relationships of changed point attribute metadata from the graph database, issue data change warnings for the entities corresponding to the changed point attribute metadata upstream and downstream relationships, and update the changed point attribute metadata information to the graph database after the warning ends.

[0074] S103: Update the big data lineage data in the graph database based on the results of data processing;

[0075] S104: Display the big data kinship diagram based on the updated big data kinship data.

[0076] Furthermore, the real-time judgment of the accuracy of newly added metadata based on a preset threshold specifically involves:

[0077] Using the initial and accurate kinship data in the graph database as the training set, the accuracy of newly added metadata is judged in real time.

[0078] Example 2:

[0079] like Figure 2 As shown,

[0080] This disclosure also provides a big data lineage data dynamic management system, including:

[0081] The bloodline data acquisition module 201 is used to collect big data bloodline data and store it according to the type of bloodline metadata collected;

[0082] The lineage data processing module 202 is used to process data based on the node attribute change metadata and new metadata written in Kafka; wherein, the new metadata includes: new node attribute metadata and new edge attribute metadata;

[0083] Bloodline data update module 203 is used to update the big data bloodline data in the graph database based on the results of data processing;

[0084] The bloodline data display module 204 is used to display a big data bloodline relationship diagram based on the updated big data bloodline data.

[0085] The bloodline data acquisition module 201 is sequentially connected to the bloodline data processing module 202, the bloodline data update module 203, and the bloodline data display module 204.

[0086] like Figure 3 As shown, this is a preferred embodiment of Example 2:

[0087] A big data lineage dynamic management system based on heterogeneous storage can display metadata and lineage diagrams of the entire chain from data generation to data application of the big data platform; it can also provide real-time accuracy prediction of newly added point-edge relationship lineages, and if the accuracy meets the standard, it will be stored and displayed in the database, and if it does not meet the standard, it will require manual review; in addition, for dynamically changed data, it will parse and obtain the upstream and downstream relationships of the changed object in real time and issue change warnings. For possible anomalies or data loss in the real-time change parsing process, it will provide an offline correction module, which will directly obtain the changed metadata to perform offline correction on the original data.

[0088] The lineage data acquisition module (01) corresponds to the function of the lineage data acquisition module 201. It is divided into the collection of vertex metadata and edge metadata. For different types of storage and computing components, the system provides different plugins for collecting and reporting metadata.

[0089] Furthermore, the system disclosed herein specifies a unified data reporting standard, and third-party components can customize plugins to implement metadata collection and reporting functions, thereby improving scalability.

[0090] The Vertex metadata (02) mainly collects data storage engines, and the specific content collected includes storage type, database name, table name, field information, description information, etc.

[0091] Before edge metadata was collected, vertex metadata was just a series of independent and unrelated points.

[0092] Here are a few examples of plugin implementations:

[0093] 1. HBase Hook is implemented using a coprocessor. Every time data in the table structure is added, deleted, or modified, the operation instructions in the coprocessor to collect metadata are triggered.

[0094] 2. Hive Hook uses the Hive interface provided by Hive itself. The Hook is triggered after Hive parses the SQL, so compared to manually parsing SQL or using open-source SQL parsing tools, Hook has higher parsing accuracy. Hook can collect all CRUD operations on the Hive table structure (i.e., the metadata information required by this system).

[0095] 3. For storage engines that cannot integrate Hooks into the system to trigger real-time metadata reporting, use a timed polling method to obtain metadata, such as Kafka, ElasticSearch, MySQL, etc.

[0096] The target of edge metadata collection (03) is mainly the data processing engine. The specific collection content includes upstream and downstream storage type information, database table information, dependent fields, etc.

[0097] The purpose of collecting edge metadata is to link vertex metadata together to present a complete data lineage map.

[0098] 1. Hive Hook is a special case in this system. It has the attributes of both data storage and data processing, so it can collect vertex and edge information at the same time. Both DDL and DML SQL will trigger Hook. DDL triggers vertex metadata collection, and DML triggers edge dependency collection.

[0099] 2. Hook code embedded in the Spark user-developed code to record and report the data source of Spark, the target storage to which it is written, and field information, etc.

[0100] 3. Flume uses interceptors to obtain and report the data source and target storage.

[0101] 4. Datax, Canal, Sqoop, and other similar software obtain upstream and downstream storage dependencies by parsing configuration files.

[0102] From the perspective of controlling the accuracy of bloodline data:

[0103] For the newly added data (04) of point attribute metadata and edge attribute metadata, the lineage data acquisition module (01) first writes the data into Kafka, and the AI ​​module (05) pulls the data from Kafka and performs the prediction of the accuracy of the lineage relationship. The initialized and confirmed accurate original lineage data is used as the training set (06) to predict the accuracy of the lineage relationship. If the accuracy is greater than 0.6, the data is stored in the graph database (08) for the front end (09) to display. If the accuracy is less than 0.6, the accuracy of the lineage needs to be manually reviewed (07) to ensure the accuracy of the lineage.

[0104] For data with dynamically changing point attributes:

[0105] The system disclosed herein will also write lineage data into Kafka through the lineage data acquisition module (01), and the change parsing module (11) will read lineage data from Kafka and parse the change object.

[0106] Then, the upstream and downstream relationships of the changed object are obtained from the graph database (08), and change warnings (12) are issued to the entities corresponding to these relationships. The responsible persons are notified in a timely manner to reduce the risks brought about by data changes.

[0107] After the warning ends, the change information will be updated in the graph database (08) for display by the front end (09).

[0108] Since there may be abnormal scenarios when changing data, such as data loss during the transfer process, or multiple update operations being executed concurrently at the same time, which may cause problems with the timing of data updates, the offline correction module (13) provides a metadata correction function. It will directly connect to the component to obtain the changed metadata and ensure that the metadata displayed by the lineage system is completely consistent with the metadata of the component itself.

[0109] The system disclosed herein supports global lineage acquisition and display of big data platforms, has strong scalability, low degree of customization, and low development cost for integrating new components.

[0110] The system disclosed herein supports real-time early warning and dynamic real-time updating of the bloodline view for upstream and downstream related bloodlines in scenarios with dynamic changes in bloodlines, and has an offline correction mechanism for abnormal dynamic update scenarios.

[0111] The system disclosed herein may erroneously collect newly added blood relations. The system supports a dual verification mechanism of AI and manual verification to ensure accuracy.

[0112] Example 3:

[0113] This disclosure also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, is used to implement the steps of the above-described big data lineage data dynamic management system.

[0114] The computer storage medium disclosed herein can be implemented using semiconductor memory, magnetic core memory, magnetic drum memory, or disk memory.

[0115] Semiconductor memory, primarily used in computers, mainly consists of two types of semiconductor storage elements: MOSFETs and bipolar transistors. MOSFETs offer high integration density and simple manufacturing processes but are relatively slow. Bipolar transistors have complex manufacturing processes, high power consumption, and low integration density but are fast. The advent of NMOS and CMOS technologies led to MOSFETs becoming the dominant type of semiconductor memory. NMOS is fast; for example, Intel's 1K-bit static random access memory (SRAM) has an access time of 45ns. CMOS, on the other hand, consumes less power; a 4K-bit CMOS SRAM has an access time of 300ns. The semiconductor memories mentioned above are all random access memories (RAM), meaning they can be randomly read from and written to during operation. Semiconductor read-only memories (ROMs), however, can be randomly read from but not written to during operation; they are used to store pre-programmed programs and data. ROMs are further divided into two types: non-rewritable fuse-type ROMs (PROMs) and rewritable EPROMs (EPROMs).

[0116] Magnetic core memory is characterized by low cost and high reliability, and has over 20 years of practical application experience. Before the mid-1970s, magnetic core memory was widely used as main memory. Its storage capacity could reach 10 bits or more, with the fastest access time being 300 ns. Typical international magnetic core memory capacities ranged from 4 MS to 8 MB, with access cycles of 1.0 to 1.5 μs. Even after the rapid development of semiconductor memory replaced magnetic core memory as the main memory, magnetic core memory can still be used as a large-capacity expansion memory.

[0117] Magnetic drum memory is a type of external storage device that records magnetic data. Due to its fast data access speed and stable, reliable operation, although its capacity is relatively small and it is gradually being replaced by disk storage, it is still used as external storage for real-time process control computers and medium- to large-scale computers. To meet the needs of small and microcomputers, ultra-miniature magnetic drums have emerged, which are small in size, lightweight, highly reliable, and easy to use.

[0118] Disk storage is a type of external storage device that records magnetic data. It combines the advantages of magnetic drums and magnetic tapes: its storage capacity is larger than that of magnetic drums, its access speed is faster than that of magnetic tapes, and it can be stored offline. Therefore, disks are widely used as high-capacity external storage in various computer systems. Disks are generally divided into two main categories: hard disks and floppy disks.

[0119] There are many types of hard disk storage devices. Structurally, they are divided into two types: replaceable and fixed. Replaceable disks have interchangeable platters, while fixed disks have fixed platters. Both replaceable and fixed disks have multi-platter and single-platter structures, and can be further divided into fixed-head and movable-head types. Fixed-head disks have smaller capacities, lower recording densities, and higher access speeds, but are more expensive. Movable-head disks have higher recording densities (up to 1000-6250 bits / inch), resulting in larger capacities, but their access speeds are relatively lower than fixed-head disks. Disk products can have storage capacities of several hundred megabytes, with a bit density of 6250 bits / inch and a track density of 475 tracks / inch. Multi-platter replaceable disk storage devices, due to their replaceable platters, offer very large independent capacity, and with high speeds, can store large amounts of information and are widely used in online information retrieval systems and database management systems.

[0120] Example 4:

[0121] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described big data lineage data dynamic management system.

[0122] Figure 4 This is a schematic diagram of the internal structure of an electronic device in one embodiment. For example... Figure 4 As shown, the electronic device includes a processor, storage medium, memory, and network interface connected via a system bus. The storage medium stores an operating system, a database, and computer-readable instructions. The database stores control information sequences. When the computer-readable instructions are executed by the processor, they enable the processor to implement a large-scale lineage data dynamic management system. The processor provides computing and control capabilities to support the operation of the entire computer device. The memory stores computer-readable instructions, which, when executed by the processor, enable the processor to implement a large-scale lineage data dynamic management system. The network interface of the computer device is used for communication with terminals. Those skilled in the art will understand that… Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0123] This electronic device includes, but is not limited to, smartphones, computers, tablets, wearable smart devices, artificial intelligence devices, and power banks.

[0124] In some embodiments, the processor may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory (e.g., executing remote data read / write programs) and calls data stored in the memory to perform various functions of the electronic device and process data.

[0125] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory and at least one processor, etc.

[0126] Figure 4 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 4 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0127] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor via a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be elaborated further here.

[0128] Furthermore, the electronic device may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device and other electronic devices.

[0129] Optionally, the electronic device may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0130] Furthermore, the computer's usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system, applications required for at least one function, etc.; and the data storage area may store data created based on the use of blockchain nodes, etc.

[0131] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0132] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0133] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0134] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for dynamic management of big data lineage data, characterized in that, include: Collect large-scale kinship data and store it according to the type of kinship metadata collected; Data processing is performed based on the node attribute change metadata and new metadata written in Kafka; wherein, the new metadata includes: new node attribute metadata and new edge attribute metadata; Update the big data lineage data in the graph database based on the results of data processing; A big data kinship diagram is displayed based on the updated big data kinship data; The specific data processing based on the point attribute change metadata and newly added metadata written in Kafka includes: Change parsing and offline correction processing are performed on the point attribute change metadata. The newly added metadata is subjected to real-time accuracy assessment; the offline correction process is used to resolve timing issues caused by data loss or concurrent updates during data flow. The change parsing process specifically includes: The graph database retrieves the upstream and downstream relationships of the changed point attribute metadata, issues data change warnings for the entities corresponding to the upstream and downstream relationships of the changed point attribute metadata, and updates the changed point attribute metadata information to the graph database after the warning ends. Specifically, the offline correction process includes: The modified kinship metadata is collected from the data acquisition component, and inaccurate kinship metadata in the graph database is corrected to ensure that the metadata displayed by the big data kinship data dynamic management system is completely consistent with the kinship metadata obtained by the component itself.

2. The method according to claim 1, characterized in that, The process of performing real-time accuracy assessment on newly added metadata specifically includes: The accuracy of newly added metadata is judged in real time based on a preset threshold. If it is greater than or equal to the preset threshold, the subsequent steps of updating the big data lineage data in the graph database will continue directly. If the data is less than the preset threshold, manual review is required, and the data that passes the manual review will be updated in the big data lineage data in the graph database.

3. The method according to claim 2, characterized in that, The real-time accuracy judgment of newly added metadata based on a preset threshold specifically involves: Using the initial and accurate kinship data in the graph database as the training set, the accuracy of newly added metadata is judged in real time.

4. The method according to any one of claims 1 to 3, characterized in that, The collection of big data on bloodlines specifically includes: Collect point attribute metadata via HBase, Hive, and / or Kafka, and collect edge attribute metadata via Spark, Hive, Kafka, and / or Flume.

5. A big data lineage data dynamic management system, characterized in that, include: The bloodline data acquisition module is used to collect large amounts of bloodline data and store them according to the type of bloodline metadata collected. The lineage data processing module is used to process data based on the node attribute change metadata and new metadata written in Kafka; wherein, the new metadata includes: new node attribute metadata and new edge attribute metadata; The bloodline data update module is used to update the big data bloodline data in the graph database based on the results of data processing. The bloodline display module is used to display a big data bloodline diagram based on the updated big data bloodline data. The specific data processing based on the point attribute change metadata and newly added metadata written in Kafka includes: Change parsing and offline correction processing are performed on the point attribute change metadata. The newly added metadata is subjected to real-time accuracy assessment; the offline correction process is used to resolve timing issues caused by data loss or concurrent updates during data flow. The change parsing process specifically includes: The graph database retrieves the upstream and downstream relationships of the changed point attribute metadata, issues data change warnings for the entities corresponding to the upstream and downstream relationships of the changed point attribute metadata, and updates the changed point attribute metadata information to the graph database after the warning ends. Specifically, the offline correction process includes: The modified kinship metadata is collected from the data acquisition component, and inaccurate kinship metadata in the graph database is corrected to ensure that the metadata displayed by the big data kinship data dynamic management system is completely consistent with the kinship metadata obtained by the component itself.

6. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps corresponding to the big data lineage data dynamic management method described in any one of claims 1 to 4.

7. A computer storage medium storing computer program instructions thereon, characterized in that, When the program instructions are executed by the processor, they are used to implement the steps corresponding to the big data lineage data dynamic management method described in any one of claims 1 to 4.

Images