Blood relationship data processing method and device

Abstract

The invention provides a blood relationship data processing method and device, and relates to the technical field of big data. The method comprises: receiving a general message reported by a component in a big data platform, the general message carrying blood relationship data, source data information, data processing information and target data information, the source data information being information of source data received by the component in the big data platform; the target data information is information of target data obtained by processing the source data by a component in the big data platform; according to the data processing information, performing compliance verification on the blood relationship data; performing element extraction on the blood relationship data according to the source data information, the data processing information and the target data information in response to passing of the compliance verification to obtain nodes corresponding to the blood relationship data, and forming edges according to a relationship between the nodes; and converting the node and the edge into a saved message, and storing the saved message into a graph database. According to the method, the processing efficiency can be improved when the blood relationship data is processed.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a method and device for processing bloodline data. Background Art

[0002] As one of the core links of data governance, lineage data processing aims to ensure the accuracy, consistency and security of data by tracking and recording the process of data throughout its life cycle from generation, processing, processing, fusion, flow to final disappearance.

[0003] At present, the big data platform provides various development components to support the development and use of data, such as data collection components, integration / replication components, and data service components. When comprehensively recording the lineage data of the big data platform, the lineage data processing device receives the lineage data according to different reporting messages based on the data characteristics of each component, and then extracts the lineage relationship between the data reported by each component and the lineage relationship between the data reported by different components based on these messages and saves them.

[0004] However, when a new component is added to the big data platform, the efficiency of processing the lineage data using the above lineage data processing method is low. Summary of the invention

[0005] The present application provides a method and device for processing bloodline data, so as to solve the problem of low efficiency in processing bloodline data in the prior art.

[0006] In a first aspect, the present application provides a method for processing bloodline data, comprising:

[0007] Receive general messages reported by components in the big data platform, which carry lineage data, source data information, data processing information, and target data information. The source data information is the information of the source data received by the components in the big data platform, and the target data information is the information of the target data obtained by the components in the big data platform processing the source data;

[0008] Conduct compliance verification on bloodline data based on data processing information;

[0009] In response to the compliance check being passed, elements of the lineage data are extracted according to the source data information, the data processing information, and the target data information, the nodes corresponding to the lineage data are obtained, and edges are formed according to the relationships between the nodes;

[0010] Convert nodes and edges into save messages, and store the save messages in the graph database.

[0011] In some embodiments, the general message also carries a data source component identifier;

[0012] In response to the compliance check being passed, elements of the lineage data are extracted according to the source data information, data processing information, and target data information to obtain nodes corresponding to the lineage data, and edges are formed according to the relationships between the nodes, including:

[0013] Extracting a first basic element and a composite element from the lineage data, the first basic element including a source component identifier and an operation, and the composite element including a table;

[0014] The composite element is split into second base elements, the second base elements including fields in the table.

[0015] In some embodiments, converting nodes and edges into save messages and storing the save messages in a graph database includes:

[0016] Generate node identifiers corresponding to the elements according to element rules;

[0017] The edge identifier is determined based on the start node identifier and the end node identifier of the edge.

[0018] In some embodiments, the general message also carries an environment code, where the environment includes a test environment and a production environment, and the test environment and the production environment correspond to different graph databases;

[0019] Convert nodes and edges into saved messages and store the saved messages in the graph database, including:

[0020] Determine the target graph database according to the environment coding;

[0021] The node identifiers and edge identifiers are converted into save messages suitable for the target graph database and stored in the target graph database.

[0022] In some embodiments, after receiving the general message reported by the component in the big data platform, the method further includes:

[0023] Verify the data format of the general message;

[0024] If the response data format verification is passed, a tracking mark is added to the lineage data and the lineage data is uploaded to the message queue;

[0025] Accordingly, based on the data processing information, the compliance check of the bloodline data is performed, including:

[0026] Pull the lineage data from the message queue in sequence.

[0027] In some embodiments, it also includes:

[0028] receiving a query request from a user, the query request including query request content;

[0029] When the query request content is a job, obtain the source component identifier and job address corresponding to the job;

[0030] Generate a query identifier for the job based on the source component identifier and the job address;

[0031] Convert the query identifier of the job into a query message and query the corresponding saved message from the graph database according to the query message;

[0032] Parse the saved message obtained by the query to obtain the node information and edge information that are related to the job.

[0033] In a second aspect, the present application provides a bloodline data processing device, comprising:

[0034] A receiving module is used to receive general messages reported by components in the big data platform. The general messages carry lineage data, source data information, data processing information and target data information. The source data information is the information of the source data received by the components in the big data platform, and the target data information is the information of the target data obtained by the components in the big data platform processing the source data.

[0035] A verification module is used to verify the compliance of bloodline data based on data processing information;

[0036] An extraction module is used to extract elements of the lineage data in response to the compliance check being passed, according to the source data information, the data processing information and the target data information, obtain the nodes corresponding to the lineage data, and form edges according to the relationship between the nodes;

[0037] The storage module is used to convert nodes and edges into saved messages and store the saved messages in the graph database.

[0038] In a third aspect, the present application provides an electronic device, including: a memory and a processor;

[0039] The memory is used to store computer programs; the processor is used to execute the computer programs stored in the memory to implement the bloodline data processing method in the first aspect and any one of the embodiments of the first aspect.

[0040] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the bloodline data processing method in the first aspect and any one of the embodiments of the first aspect is implemented.

[0041] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the bloodline data processing method in the first aspect and any one of the embodiments of the first aspect.

[0042] The bloodline data processing method and device provided in the present application receive the general message reported by the component in the big data platform, and the general message carries bloodline data, source data information, data processing information and target data information; perform compliance verification on the bloodline data according to the data processing information; in response to the compliance verification passing, extract elements of the bloodline data according to the source data information, data processing information and target data information, obtain the nodes corresponding to the bloodline data, and form edges according to the relationship between the nodes; convert the nodes and edges into saved messages, and store the saved messages in the graph database. Since each component reports the data bloodline information according to the general message, when a new component is added to the big data platform, the bloodline data processing device can also directly receive the general message reported by the new component, without the need to design a new reporting message for the new component, thereby improving the processing efficiency of the bloodline data. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A schematic diagram of the structure of a blood relationship data processing device provided in one embodiment of the present application;

[0045] Figure 2 A flow chart of a method for processing bloodline data provided in one embodiment of the present application;

[0046] Figure 3 A schematic diagram of element extraction provided in one embodiment of the present application;

[0047] Figure 4 A flowchart of another method for processing bloodline data provided in one embodiment of the present application;

[0048] Figure 5 A schematic diagram of the structure of another bloodline data processing device provided in one embodiment of the present application;

[0049] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0052] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0053] Terminology explanation:

[0054] Data Lineage: The source, transformation, and destination of data in the entire processing and calculation process, forming the path of data flow, including the source and destination of each data element.

[0055] Data Flow: The path by which data is transmitted during processing and computing, including processing operations, authorization, and data services.

[0056] Processing Job: A job or task that processes, converts, and calculates data, such as Extract-Transform-Load (ETL) jobs.

[0057] Data Service: Provides services for data, such as query and analysis, such as Application Programming Interface (API) and World Wide Web (Web) services.

[0058] Strategy Pattern: A design pattern used to define a family of algorithms that can be replaced independently without changing the client code that uses the algorithms.

[0059] Graph Database: A database specifically used to store and query graph-structured data, suitable for representing and querying data lineage relationships.

[0060] Element: A node in a graph database that represents a specific element in the data lineage.

[0061] Edge: A line in a graph database that represents the relationship between two elements.

[0062] The present application relates to the field of big data technology. The big data platform provides various development components to support the development and use of data, such as data acquisition components for connecting data outside the big data platform to the big data platform; data integration / replication components for processing related tables in big data to meet the needs of specific business scenarios; data service components for exposing the use of database tables in the form of services (restful interfaces).

[0063] When comprehensively recording the data lineage of a big data platform, it is necessary to process the working mode of each component of the big data platform. Specifically, each component can be further subdivided. For example, acquisition includes batch acquisition, stream acquisition, acquisition from files, and acquisition from external data sources. Data integration can be divided into stream computing processing and offline computing processing. Stream processing and offline computing processing can be further subdivided into a variety of operators. Data services include real-time processing and batch processing, which can be divided into multiple versions. In other words, the data lineage information of a big data platform is usually scattered among various components.

[0064] In the related art, when each component reports data lineage information to the lineage data processing device, it uses different reporting messages, and the lineage data processing device can extract the lineage relationship of the lineage data from the reporting message. When a new component is added to the big data platform, the lineage data processing device needs to design a new reporting message based on the characteristics of the new component, so that the new component uses the new reporting message to report the data lineage information. Correspondingly, the lineage data processing device needs to add a big data type and a corresponding big data node definition for the data processed by the new component. If the new component receives an external system file, the lineage data processing device needs to add a node definition of the external system file. In addition, a new parsing method needs to be configured inside the lineage data processing device so that the lineage relationship of the lineage data can be extracted from the reporting message of the new component.

[0065] The above-mentioned bloodline data processing method adapts to the data characteristics of each component. However, due to the inconsistency of the reporting messages of each component, a new reporting message needs to be designed when a new component is added, which makes the bloodline data processing efficiency low.

[0066] In addition, in lineage data processing devices, data tracking tools are dedicated to recording the source and destination of data to track the flow of data. However, these tools are usually limited to surface-level tracking and lack a deep understanding of the data processing process.

[0067] Metadata management tools are used to capture and maintain data metadata, including its source, structure, and transformation rules. However, when processing large-scale data and dynamic data processing flows, there are data quality and data synchronization issues due to the different reporting messages of different components, resulting in low timeliness of data processing.

[0068] In response to the above problems, the present application proposes a method and device for processing lineage data. In this method, each component of the big data platform uses a general message to report data lineage information, including source data information received by the component, data processing information performed by the component on the source data, and target data information after the component processes the source data. The lineage data processing device uses the above information to express the relationship between the source data and the target data, without the need to develop special nodes or design new reporting messages based on the characteristics of each component. When adding new components, the lineage data processing device can process lineage data more efficiently and uniformly.

[0069] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0070] Figure 1 FIG. 1 is a schematic diagram showing the structure of a blood relationship data processing device provided by an embodiment of the present application. Figure 1 As shown, the big data platform includes a data acquisition component, a data integration component and a data service component. The lineage data processing device includes a receiving module, a parsing module, a processing module and a query module. The receiving module is used to receive the lineage data reported by each component of the big data platform and upload the lineage data to the message queue. The parsing module is used to actively pull data from the message queue to realize asynchronous consumption. The processing module is used to save the lineage data to the graph database after processing. The query module is used to receive user query requests, and query the required data from the graph database according to the query request, and then feed back the query results to the user terminal. That is, the lineage data processing device can obtain the lineage relationship between data and operations and services from the big data platform components, save the lineage data to the graph database through the internal processing mechanism, and provide external query capabilities.

[0071] In this application, the bloodline data processing device is used as the execution subject to execute the bloodline data processing method of the following embodiments. Specifically, the execution subject can be a hardware device of the bloodline data processing device, or a software application that implements the following embodiments in the bloodline data processing device, or a computer-readable storage medium installed with the software application that implements the following embodiments, or a code that implements the software application of the following embodiments.

[0072] Figure 2FIG. 1 is a flow chart showing a method for processing blood relationship data provided by an embodiment of the present application. Figure 2 As shown, with the bloodline data processing device as the execution subject, the method of this embodiment may include the following steps:

[0073] S101. Receive a general message reported by a component in a big data platform.

[0074] Among them, the general message carries lineage data, source data information, data processing information and target data information. The source data information is the information of the source data received by the components in the big data platform, and the target data information is the information of the target data obtained by the components in the big data platform processing the source data.

[0075] For example, external data becomes a table of the big data platform through the collection component operation. The information of the external data is the source data information reported by the collection component, the table information is the target data information reported by the collection component, and the collection component operation information is the data processing information.

[0076] For example, the table obtained by the collection component job becomes a new table after the integration component job. The information of the table obtained by the collection component job is the source data information reported by the integration component, the information of the new table is the target data information reported by the integration component, and the integration component job information is the data processing information.

[0077] For another example, the data service component encapsulates certain tables into a specific service, then the table information is the source data information reported by the data service component, the specific service information is the target data information reported by the data service component, and the data service encapsulation information is the data processing information.

[0078] The data lineage relationship can be determined based on the source data information and target data information reported by two components that process the same data successively.

[0079] Data processing information refers to an "action" of data processing and processing, which can be a job, query, proxy query, authorization, or other relationship mapping types. In addition to the source data and target data being lineage information, different types of data processing information will also have other mapping information.

[0080] The jobs include collection jobs, integration jobs, and stream computing jobs. The lineage information related to the jobs includes data source, data target, table-level lineage, and field-level lineage. The query corresponds to the definition of service, interactive query, and other definitions. The relevant lineage information includes table-level lineage and field-level lineage. The relevant lineage information of the proxy query only includes the data source. Authorization is used to report lineage information in the service authorization application scenario.

[0081] Relationship mapping types include table-level lineage mapping, field-level lineage mapping, and authorization relationships. Table-level lineage mapping and field-level lineage mapping support one-to-one or many-to-one relationship reporting, and authorization relationships support one-to-many relationship reporting. For example, if the source data is multiple tables and is processed by one job, the target data generated is one table, which forms a many-to-one mapping.

[0082] Optionally, the general message can also carry the data types of the source data and the target data. The data types include external data sources, industry cloud data sets, industry cloud data tables, field arrays of industry cloud data tables, field information of industry cloud data tables, output field arrays, and output fields. Among them, the external data sources include the hierarchical relationship between logical subsystems and physical subsystems. The industry cloud data tables correspond to the metadata of data management.

[0083] S102. Perform compliance verification on the bloodline data based on the data processing information.

[0084] In this embodiment, the data processing information includes the data flow type, and the policy object for compliance verification can be matched according to the data flow type.

[0085] For example, if the data flow type is a job, you can check whether the version requirements of the job are compliant. Among them, the job version of the integrated job does not allow the new version to directly overwrite the old version, but needs to actively call the delete operation and then save the new version to avoid data loss.

[0086] If the data flow type is service, you can check whether the number of service association tables is compliant. Among them, there is a limit on the number of data service access tables.

[0087] If the data stream type is a table, this report is skipped because tables are not currently supported as data streams for lineage reporting.

[0088] S103. In response to compliance verification being passed, elements of the bloodline data are extracted according to the source data information, data processing information and target data information, nodes corresponding to the bloodline data are obtained, and edges are formed according to the relationships between the nodes.

[0089] In this embodiment, elements are extracted from lineage data, and the extracted elements are nodes of lineage data. For example, the source component of lineage data can be used as a node, the job of component processing lineage data can be used as a node, and the table where lineage data is located can be used as a node. Edges can be formed when there is a relationship between nodes. For example, if a table is obtained through a job to obtain another table, an edge can be formed between the two tables.

[0090] After processing the blood relationship data, several nodes and edges can be obtained. After determining the nodes and edges, there is no need to pay attention to the source of the nodes and edges, just pay attention to them themselves.

[0091] S104: Convert the nodes and edges into save messages, and store the save messages in the graph database.

[0092] It should be understood that the data of a graph database is stored in the form of a graph, which mainly includes two basic elements: nodes and edges. Nodes represent specific entities or objects, while edges represent the connections or relationships between nodes. When converting nodes and edges into messages, it is necessary to design a suitable message structure to carry the information of nodes and edges. The message structure should be able to clearly represent the properties of nodes and edges, as well as the connection relationship between them.

[0093] When converting nodes and edges into a saved message, it is necessary to fill the node and edge information into the corresponding fields of the message according to the defined message format, and finally obtain a complete saved message.

[0094] When storing the saved message in the graph database, redundant storage can be performed to ensure successful storage of the saved message. In addition, the bloodline data processing device can also monitor the health status of the graph database and issue an early warning. For example, when an abnormality occurs when the bloodline data processing device calls the calling interface of the graph database, a warning message is issued.

[0095] The bloodline data processing method provided in this embodiment unifies the data information reported by the big data platform components into source data information, data processing information, and target data information, and maps the data elements to the nodes and edges of the graph database, thereby achieving fine control over the data bloodline relationship and improving the flexibility, consistency, and manageability of data processing. When a new component is added to the big data platform, the bloodline data processing device can use a universal message to receive the bloodline data reported by the new component and the processing information of the bloodline data in the new component, without having to design a new reporting message based on the new component, so that each component of the big data platform reports data information according to a unified standard, improving the efficiency of the bloodline data processing device in processing data and the comprehensiveness of data records.

[0096] In some embodiments, the generic message also carries a data source component identifier.

[0097] On this basis, Figure 2 The specific implementation of step S103 in the embodiment includes:

[0098] S1031. Extract the first basic element and the composite element in the lineage data, the first basic element including the source component identifier and the job, and the composite element including the table.

[0099] S1032. Split the composite element into second basic elements, where the second basic elements include fields in the table.

[0100] Optionally, the general message may also carry application information to which the data belongs, to indicate which application processes the data in the source component.

[0101] Optionally, the source components include data management (identified as dm), data acquisition (identified as dg), data service (identified as ds), data integration (identified as di), stream computing (identified as sc), etc.

[0102] Application types include big data cloud applications (identified as bdcp_app), external applications (identified as ext_app), external source systems (identified as ext_sys), and data service applications (identified as ds_app). Among them, the external source system contains the hierarchical relationship between logical subsystems and physical subsystems, and the data service application is an application unique to data services.

[0103] In this embodiment, taking the data lineage information reported by the collection component as an example, the collection component, the operation of the collection component, and the application used to process the data can all be used as the first basic element.

[0104] The table in the message is a composite element with n fields. This composite element can be decomposed into n field nodes and a table node, and can also be decomposed into n edges, that is, the link relationship between the n fields and the table.

[0105] For example, service authorization is also a composite element, which can be decomposed into two nodes: service and application, and the downstream edge between the service and application.

[0106] Figure 3 A schematic diagram of element extraction provided in one embodiment of the present application. Figure 3 As shown, the report message of the data acquisition component includes data processing information, source data information and target data information, wherein the data processing information is a job, the source data information is an ext-table, and the target data information is a table. The report message of the integration component includes data processing information, source data information, target data information and relationship mapping type, wherein the data processing information is a job, the source data information is a table, the target data information is a table, and the relationship mapping type is a table-level lineage mapping, a field-level lineage mapping or an authorization relationship. The data lineage processing device parses the message to obtain a node list and an edge list, and the node list includes jobs, tables, ext-tables, access proxy queries, fields, etc.; the edge list includes authorization relationships, downstream edges between services and applications, etc. The above is only used to help understand element extraction and is not a limitation of this embodiment.

[0107] In some embodiments, Figure 2 The specific implementation of step S104 in the embodiment includes:

[0108] S1041. Generate a node identifier corresponding to the element according to the element rule.

[0109] For example, for a job, the source component identifier of the job is concatenated with the address of the job and hashed to obtain the node identifier of the job.

[0110] For a table, the source database table address of the table is hashed to obtain the node identifier of the table.

[0111] For the fields in the table, the source database table address of the table is concatenated with the address of the field and hashed to obtain the node identifier of the field.

[0112] S1042: Determine an edge identifier according to the start node identifier and the end node identifier of the edge.

[0113] Specifically, an edge is formed according to the relationship between two nodes, and the identifiers of the two nodes are used to form an edge identifier. For example, the start node identifier and the end node identifier of the edge are concatenated and hashed to obtain the edge identifier.

[0114] Optionally, after determining the node identifiers and edge identifiers, the nodes and edges may be deduplicated based on the uniqueness of the identifiers.

[0115] In some embodiments, the general message also carries environment code, and the environment includes a test environment and a production environment, and the test environment and the production environment correspond to different graph databases.

[0116] On this basis, Figure 2 The specific implementation method of step S104 in the embodiment includes: determining the target graph database according to the environment code; converting the node identifier and the edge identifier into a save message applicable to the target graph database and storing it in the target graph database.

[0117] It should be understood that the message formats and extensions of different graph databases are different, and the saved messages need to adapt to the saving requirements of the graph database. The blood relationship data processing device stores the saved messages in the corresponding graph database to improve the query efficiency.

[0118] In some embodiments, after receiving the general message reported by the component in the big data platform, the bloodline data processing device can also verify the data format of the general message, and in response to the data format verification passing, add a tracking mark to the bloodline data and upload the bloodline data to the message queue.

[0119] Accordingly, the bloodline data processing device executes Figure 2 In step S102 of the embodiment, it is necessary to pull the blood relationship data from the message queue in sequence.

[0120] In this embodiment, the verification of the bloodline data by the bloodline data processing device may include format verification and content verification. Among them, the format verification is to verify whether it is in JSON format. The content verification includes verifying whether the general format contains data processing information, etc.

[0121] This embodiment can reduce the server pressure caused by excessive data by using a message queue, reduce the risk of data loss, and improve the security of data processing. By adding a tracking mark to the lineage data, the cause of the abnormality can be quickly determined when an abnormality occurs.

[0122] Figure 4 This is a flow chart of another method for processing blood relationship data provided by an embodiment of the present application. Figure 4 As shown, in some embodiments, the bloodline data processing device may further perform the following steps:

[0123] S201. Receive a query request from a user, where the query request includes query request content.

[0124] S202: When the query request content is a job, obtain a source component identifier and a job address corresponding to the job.

[0125] For example, if the query request content is a request to query the collection component job, the bloodline data processing device obtains the identifier of the collection component and the address of the job.

[0126] S203: Generate a query identifier for the job according to the source component identifier and the job address.

[0127] Similar to generating the job node identifier in the aforementioned embodiment, the lineage data processing device concatenates the source component identifier and the job address and performs hashing to obtain the query identifier of the job.

[0128] S204: Convert the query identifier of the job into a query message and query the corresponding saved message from the graph database according to the query message.

[0129] Optionally, when there are multiple graph databases, the lineage data processing device can also obtain the environment code corresponding to the job to determine the target graph database.

[0130] When the bloodline data processing device converts the query identifier into a query message, it needs to adapt to the query requirements of the target graph database.

[0131] Since the query identifier and the node identifier are generated in the same way, the corresponding query message and the saved message are also generated in the same way. Therefore, when the bloodline data processing device queries the saved message from the target graph database, it can find the saved message that is consistent with the query message.

[0132] S205. parse the saved message obtained by the query to obtain node information and edge information that are related to the job.

[0133] In this embodiment, parsing and saving the message may be to obtain node information and edge information associated with the operation node according to the operation node, and feed back all information related to the operation node to the user.

[0134] Similarly, when the query request content is a table, the source library table address of the table is obtained, and the source library table address is hashed to obtain a query identifier.

[0135] The lineage data processing method provided in the embodiment of the present application adopts a universal data lineage reporting method, which is convenient for newly added components to report data lineage information, can retain the complete data lineage relationship of the big data platform, and has excellent performance and high processing efficiency when processing large quantities of data.

[0136] Figure 5 FIG. 1 is a schematic diagram showing the structure of a blood relationship data processing device provided by an embodiment of the present application. Figure 5 As shown, the bloodline data processing device 10 of this embodiment is used to implement the operations corresponding to the bloodline data processing device in any of the above method embodiments. The bloodline data processing device 10 of this embodiment includes:

[0137] The receiving module 11 is used to receive the general message reported by the components in the big data platform, and the general message carries the lineage data, source data information, data processing information and target data information. The source data information is the information of the source data received by the components in the big data platform, and the target data information is the information of the target data obtained by the components in the big data platform processing the source data;

[0138] A verification module 12, used to verify the compliance of the blood relationship data according to the data processing information;

[0139] The extraction module 13 is used to extract elements of the lineage data in response to the compliance check being passed, according to the source data information, the data processing information and the target data information, obtain the nodes corresponding to the lineage data, and form edges according to the relationship between the nodes;

[0140] The storage module 14 is used to convert the nodes and edges into save messages and store the save messages in the graph database.

[0141] The bloodline data processing device 10 provided in the embodiment of the present application can execute the above method embodiment. Its specific implementation principles and technical effects can be found in the above method embodiment, and this embodiment will not be repeated here.

[0142] Figure 6 FIG. 1 shows a hardware structure diagram of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device 20 is used to implement the operations corresponding to the bloodline data processing device in any of the above method embodiments. The electronic device 20 of this embodiment may include: a memory 21, a processor 22 and a communication interface 24.

[0143] The memory 21 is used to store computer programs. The memory 21 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk.

[0144] The processor 22 is used to execute the computer program stored in the memory to implement the method in the above embodiment. For details, please refer to the relevant description in the above method embodiment. The processor 22 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the invention can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0145] Optionally, the memory 21 may be independent or integrated with the processor 22 .

[0146] When the memory 21 is a device independent of the processor 22, the electronic device 20 may further include a bus 23. The bus 23 is used to connect the memory 21 and the processor 22. The bus 23 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0147] The communication interface 24 may be connected to the processor 22 via the bus 23. The processor 22 may control the communication interface 24 to implement the functions of receiving and sending signals.

[0148] The electronic device 20 provided in this embodiment can be used to execute the above-mentioned bloodline data processing method, and its implementation method and technical effect are similar, which will not be repeated in this embodiment.

[0149] The present application also provides a computer-readable storage medium, in which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, it is used to implement the methods provided in the various embodiments described above.

[0150] Among them, the computer-readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a computer-readable storage medium is coupled to a processor so that the processor can read information from the computer-readable storage medium and write information to the computer-readable storage medium. Of course, the computer-readable storage medium can also be a component of the processor. The processor and the computer-readable storage medium can be located in an application-specific integrated circuit (ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the computer-readable storage medium can also exist in a communication device as discrete components.

[0151] Specifically, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0152] The present application also provides a computer program product, which includes a computer program / instruction, and the computer program / instruction is stored in a computer-readable storage medium. At least one processor of the device can read the computer program / instruction from the computer-readable storage medium, and at least one processor executes the computer program / instruction so that the device implements the methods provided in the various embodiments described above.

[0153] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0154] Among them, each module can be physically separated, for example, installed in different locations of a device, or installed on different devices, or distributed on multiple network units, or distributed on multiple processors. Each module can also be integrated together, for example, installed in the same device, or integrated in a set of codes. Each module can exist in the form of hardware, or can also exist in the form of software, or can also be implemented in the form of software plus hardware. The present application can select some or all of the modules according to actual needs to achieve the purpose of the present embodiment.

[0155] It should be understood that, although the various steps in the flowcharts in the above-described embodiments are sequentially displayed according to the indications of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily to be carried out sequentially, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above embodiments can still be modified, or some or all of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for processing blood relationship data, characterized in that: The method comprises: Receive a general message reported by a component in the big data platform, wherein the general message carries lineage data, source data information, data processing information, and target data information. The source data information is information about source data received by the component in the big data platform, and the target data information is information about target data obtained by processing source data by the component in the big data platform; Performing compliance verification on the bloodline data according to the data processing information; In response to the compliance check being passed, extracting elements from the lineage data according to the source data information, the data processing information and the target data information, obtaining nodes corresponding to the lineage data, and forming edges according to the relationships between the nodes; The nodes and the edges are converted into save messages, and the save messages are stored in a graph database.

2. The method according to claim 1, characterized in that The general message also carries the data source component identifier; In response to the compliance check being passed, elements of the lineage data are extracted according to the source data information, the data processing information and the target data information to obtain nodes corresponding to the lineage data, and edges are formed according to the relationship between the nodes, including: Extracting a first basic element and a composite element from the lineage data, wherein the first basic element includes a source component identifier and a job, and the composite element includes a table; The composite element is split into second base elements, the second base elements including fields in a table.

3. The method according to claim 2, characterized in that Converting the node and the edge into a save message, and storing the save message in a graph database, includes: Generate a node identifier corresponding to the element according to the element rule; The edge identifier is determined based on the start node identifier and the end node identifier of the edge.

4. The method according to claim 3, characterized in that The general message also carries an environment code, where the environment includes a test environment and a production environment, and the test environment and the production environment correspond to different graph databases; Converting the node and the edge into a save message, and storing the save message in a graph database, includes: Determining a target graph database according to the environment code; The node identifier and the edge identifier are converted into a save message applicable to the target graph database and stored in the target graph database.

5. The method according to any one of claims 1 to 4, characterized in that: After receiving the general message reported by the components in the big data platform, it also includes: Verifying the data format of the general message; If the response data format verification is passed, a tracking mark is added to the bloodline data and the bloodline data is uploaded to the message queue; Accordingly, the compliance check of the bloodline data is performed according to the data processing information, including: Pull the bloodline data from the message queue in sequence.

6. The method according to any one of claims 1 to 4, characterized in that: Also includes: receiving a query request from a user, wherein the query request includes query request content; When the query request content is a job, obtaining a source component identifier and a job address corresponding to the job; Generate a query identifier for the job according to the source component identifier and the job address; Convert the query identifier of the job into a query message and query the corresponding saved message from the graph database according to the query message; The saved message obtained by the query is parsed to obtain node information and edge information that are related to the job.

7. A blood relationship data processing device, characterized in that: include: A receiving module, used to receive a general message reported by a component in the big data platform, wherein the general message carries lineage data, source data information, data processing information and target data information. The source data information is information about source data received by the component in the big data platform, and the target data information is information about target data obtained by processing source data by the component in the big data platform; A verification module, used for performing compliance verification on the blood relationship data according to the data processing information; An extraction module, for extracting elements of the lineage data in response to the compliance check being passed, according to the source data information, the data processing information and the target data information, obtaining nodes corresponding to the lineage data, and forming edges according to the relationships between the nodes; The storage module is used to convert the nodes and the edges into save messages and store the save messages in a graph database.

8. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory to implement the bloodline data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the bloodline data processing method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the bloodline data processing method according to any one of claims 1 to 6 is implemented.

Images