Data blood relationship model determination method and device, and blood relationship determination method and device

Abstract

The invention provides a data blood relationship model determination method and device and a blood relationship determination method and device. Relates to the technical field of data management. The data blood relationship model determination method comprises the following steps: receiving message data which is transmitted by each request component and meets a preset condition; the preset conditions comprise a data source condition and a data content condition; analyzing the message data to obtain basic blood relationship elements corresponding to the message data, and obtaining basic graph elements according to the basic blood relationship elements; under the condition that the analysis result of the message data further comprises mapping blood relationship elements, virtual mapping graph elements are obtained according to the mapping blood relationship elements and the basic blood relationship elements; and according to the basic graph elements and the virtual mapping graph elements, generating a data blood relationship model corresponding to each request component. According to the method, the technical problem that in the prior art, when a data blood relationship model carries out blood relationship query, the query efficiency is low is solved.

Description

Technical Field

[0001] The present application relates to the field of big data governance technology, and in particular to a data lineage relationship model determination method, a lineage relationship determination method and a device. Background Art

[0002] In the field of data governance, data lineage refers to the record of the flow path and change process of data from the source to the end use. It helps to establish a comprehensive data governance system, improve the quality, traceability and manageability of data, thereby improving the transparency and credibility of data.

[0003] In the process of determining data lineage relationships, although graph models and graph database technologies provide powerful tools and methods, there are still some technical problems. For example, there is a high time complexity when performing lineage relationship queries, which will affect the system performance and user experience.

[0004] Therefore, how to reduce the query complexity of blood relationship in the data blood relationship model and improve the query efficiency is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a data blood relationship model determination method, a blood relationship determination method and a device, so as to solve the technical problem that the data blood relationship model in the prior art has low query efficiency when performing blood relationship query.

[0006] In a first aspect, the present application provides a method for determining a data lineage relationship model, comprising:

[0007] Receiving message data transmitted by each request component that meets preset conditions; the preset conditions include data source conditions and data content conditions;

[0008] Parsing the message data to obtain basic blood relationship elements corresponding to the message data, and obtaining basic graph elements according to the basic blood relationship elements;

[0009] In the case where the parsing result of the message data also includes a mapping blood relationship element, obtaining a virtual mapping graph element according to the mapping blood relationship element and the basic blood relationship element;

[0010] A data lineage relationship model corresponding to each of the request components is generated according to the basic graph elements and the virtual mapping graph elements.

[0011] In a possible design, receiving message data transmitted by each request component and meeting a preset condition includes:

[0012] For message data transmitted by any request component, verify the request component corresponding to the received message data;

[0013] If the request component passes the verification, verify whether the message data contains all preset data fields;

[0014] If the message data includes all preset data fields, receiving the message data;

[0015] If the request component fails the verification or the message data does not contain all preset data fields, the message data is discarded, and a discard prompt message corresponding to discarding the message data is generated and fed back to the request component.

[0016] In one possible design, before receiving the message data transmitted by each request component and satisfying the preset condition, the method further includes:

[0017] For any request component, upon receiving a data transmission request sent by the request component, sending a data transmission request to the request component;

[0018] The data transmission requirement includes a preset data field included in the request component when performing data transmission, so that each preset component generates and transmits message data according to the data transmission requirement;

[0019] The preset data fields include component source information, component environment information, component node information, component application, component source data table information, component target data table information and component mapping information; wherein the component mapping information may be empty.

[0020] In a possible design, parsing the message data to obtain basic blood relationship elements corresponding to the message data, and obtaining basic graph elements according to the basic blood relationship elements include:

[0021] Extracting and verifying the message data based on preset parsing rules to obtain an element set corresponding to the message data; the element set includes a plurality of basic blood relationship elements corresponding to preset data fields;

[0022] According to the preset element conversion rules, each of the basic blood relationship elements is converted to obtain the basic graph elements corresponding to the generated data blood relationship model; the basic graph elements include basic node elements and basic edge elements.

[0023] In a possible design, when the parsing result of the message data also includes a mapping blood relationship element, obtaining a virtual mapping element according to the mapping blood relationship element and the basic blood relationship element includes:

[0024] When it is determined that there is a composite element corresponding to the component mapping information in the element set, the composite element is disassembled to obtain a plurality of mapping lineage relationship elements; the mapping lineage relationship elements include table-level mapping lineage relationship elements and field-level mapping lineage relationship elements;

[0025] According to each of the mapped blood relationship elements and each of the basic blood relationship elements, a virtual mapping element corresponding to the generated data blood relationship model is obtained.

[0026] In one possible design, the virtual mapping graph element includes a virtual mapping node element and a virtual mapping edge element;

[0027] According to each of the mapped blood relationship elements and each of the basic blood relationship elements, a virtual mapping element corresponding to the generated data blood relationship model is obtained, including:

[0028] Determine and generate at least one virtual mapping node element corresponding to the data lineage relationship model according to each of the mapping lineage relationship elements; the virtual mapping node element includes a table-level virtual mapping node element and a field-level virtual mapping node element;

[0029] According to each of the basic blood relationship elements and each of the mapping virtual component nodes, a plurality of virtual mapping edge elements corresponding to the generated data blood relationship model are determined.

[0030] In a second aspect, the present application provides a method for determining blood relationship, comprising:

[0031] Receive the node to be analyzed;

[0032] Determine, according to the node type of the node to be analyzed, a basic node element and a virtual mapping node element of the same type as the node in a pre-generated data lineage relationship model; the data lineage relationship model is generated according to the data lineage relationship model determination method described in the first aspect and various possible designs of the first aspect;

[0033] A blood relationship search is performed based on each of the basic node elements and each of the virtual mapping node elements to obtain the blood relationship corresponding to the node to be analyzed.

[0034] In a third aspect, the present application provides a data lineage relationship model determination device, comprising:

[0035] A message data receiving module, used to receive message data transmitted by each request component that meets preset conditions; the preset conditions include data source conditions and data content conditions;

[0036] A basic graph element generation module, used for parsing the message data, obtaining basic blood relationship elements corresponding to the message data, and obtaining basic graph elements according to the basic blood relationship elements;

[0037] A virtual mapping element generating module, configured to obtain a virtual mapping element according to the mapped blood relationship element and the basic blood relationship element when the parsing result of the message data also includes a mapped blood relationship element;

[0038] The data lineage relationship model generation module is used to generate a data lineage relationship model corresponding to each of the request components according to the basic graph elements and the virtual mapping graph elements.

[0039] In a fourth aspect, the present application provides a blood relationship determination device, comprising:

[0040] A node receiving module, used for receiving nodes to be analyzed;

[0041] A node element determination module, used to determine, according to the node type of the node to be analyzed, the basic node elements and virtual mapping node elements of the same node type in the pre-generated data lineage relationship model; the data lineage relationship model is generated according to the data lineage relationship model determination method described in the first aspect and various possible designs of the first aspect;

[0042] The blood relationship determination module is used to perform blood relationship search based on each of the basic node elements and each of the virtual mapping node elements to obtain the blood relationship corresponding to the node to be analyzed.

[0043] In a fifth aspect, an embodiment of the present application provides an electronic device comprising: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the data blood relationship model determination method described in the first aspect and various possible designs of the first aspect, and / or the blood relationship determination method described in the second aspect.

[0044] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the data blood relationship model determination method described in the first aspect and various possible designs of the first aspect, and / or the blood relationship determination method described in the second aspect are implemented.

[0045] In the seventh aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the data blood relationship model determination method described in the first aspect and various possible designs of the first aspect, and / or the blood relationship determination method described in the second aspect.

[0046] The data kinship model determination method, kinship determination method and device provided by the present application receive message data that meets preset conditions transmitted by each request component; the preset conditions include data source conditions and data content conditions; parse the message data to obtain the basic kinship elements corresponding to the message data, and obtain the basic graph elements according to the basic kinship elements; when the parsing result of the message data also includes mapping kinship elements, obtain virtual mapping graph elements according to the mapping kinship elements and the basic kinship elements; generate the data kinship model corresponding to each of the request components according to the basic graph elements and the virtual mapping graph elements. Since the basic graph elements formed by the kinship between data, i.e., the relationship between the base and the elements, are complex and intertwined, if the query path is generated by the above-mentioned basic graph elements when performing kinship queries, the query efficiency will be low; and the virtual mapping elements are only connected through the mapping relationship, so when performing kinship queries, the kinship is quickly queried through the virtual mapping graph elements, which can improve the query efficiency when performing kinship queries. Therefore, the technical problem of low query efficiency in the data blood relationship model in the prior art when performing blood relationship query can be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] Figure 1 A schematic diagram of a blood relationship table provided by the prior art;

[0049] Figure 2 A schematic diagram of a blood relationship graph model provided by the prior art;

[0050] Figure 3 An application scenario diagram of the technical solution provided in the embodiment of the present application;

[0051] Figure 4 A flow chart of a method for determining a data blood relationship model provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of a preset data field provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of a process for receiving message data provided in an embodiment of the present application;

[0054] Figure 7 A schematic diagram of a data kinship model provided in an embodiment of the present application;

[0055] Figure 8A flow chart of a method for determining blood relationship provided in an embodiment of the present application;

[0056] Fig. 9 A schematic diagram of the structure of a data blood relationship model determination device provided in an embodiment of the present application;

[0057] Fig.10 A schematic diagram of the structure of a blood relationship determination device provided in an embodiment of the present application;

[0058] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0059] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0060] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0061] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0062] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0063] In the current era of big data, data processing and management are becoming increasingly complex. Data lineage refers to all operations and transformation paths of data from its source to its final use, which is crucial to understanding the source, quality and impact of data. In enterprise-level data architecture, tracking data lineage can help organizations better manage data quality, understand the impact of data changes, and meet compliance and audit requirements.

[0064] In an optional implementation, an open source framework is used to construct the kinship relationship in the form of a table; for example, the open source framework Marquez is used to construct the table-level kinship relationship between the task execution node and the task data; Figure 1 An optional embodiment provides a table-level kinship relationship, through Figure 1 From the blood relationship table shown in , we can see that the blood relationship (first-level relationship) between the task node and the task data can be directly obtained from the connecting edge between the task node and the task data.

[0065] In another optional implementation, graph data can be used to store and display data lineage relationships. Figure 2 A kinship graph model is provided for another optional implementation. Since data kinship is usually hierarchical, multi-layered and complex, the kinship modeled by the graph database can be clearer, that is, the graph structure of nodes and edges in the graph data can be intuitively mapped to the actual data relationship, and the relationship can be efficiently queried and retrieved; and the attributes of nodes and relationships can be expanded as needed without complex schema changes.

[0066] However, the blood relationship models constructed in the above two optional implementations have the technical problem of high time complexity when performing blood relationship queries. For example, in the first optional implementation: Marquez uses the relational database Postgres to store the blood relationship model, which cannot well express the complex many-to-many relationship between entity objects. The relationship is a mesh structure, and the query performance is poor; the traversal performance is weak, and when it is necessary to search for multi-level relationships, the SQL statement needs to be very complex, and the search time complexity is high; in table-level blood relationship, the cross-level relationship between tables can only be obtained through complex association queries, and the query efficiency is low; for example Figure 1 In the example, the upstream tables of public.orders_7_days are public.categories, public.menu_items, public.menus and public.ordes, and the downstream table of public.orders_7_days is public.dell_ery_7_days. When querying the upstream table of this table, two associated queries are required. When querying the downstream table, two associated queries are also required. If you want to query the parent table of the parent table of a certain table (4 layers), you may have to write a more complex SQL query statement. If you want to query the parent table of the parent table of the parent table of a certain table (6 layers), etc., it will be even more complicated.

[0067] In the second optional implementation, for a given blood relationship, to find the blood relationship of a node in the graph, the commonly used search methods are depth-first search (DFS) and breadth-first search (BFS). Among them, DFS is an algorithm for traversing the graph, which can be used to cross multiple levels. Through recursion or the use of an explicit stack, DFS can go deep into the graph and backtrack after finding the target node or reaching a certain depth. DFS can be used to search between multiple levels of the graph. However, for cross-level searches, DFS will search in a depth-first manner. If all solutions are to be found, it will cause an algorithm dimensional disaster and a surge in time complexity. BFS is an algorithm that traverses the graph layer by layer, which is very suitable for cross-level searches. However, for cross-level search scenarios, pruning strategies are not used for acceleration, and the time complexity is also high. The BFS algorithm using the pruning strategy needs to be customized for specific problems, but it is necessary to design efficient pruning optimization for the graph structure, which requires certain manpower and conditions.

[0068] In response to the above technical problems, the present application proposes the following technical concept: since the relationship between the basic graph elements, that is, the relationship between the basis and the elements, formed by the blood relationship between the data is complex and intertwined, if the above basic graph elements are used to generate the query path when performing blood relationship queries, it will lead to low query efficiency; therefore, when generating blood relationship, the message data reported by each component can be parsed to obtain the basic blood relationship elements, and the basic graph elements corresponding to the basic blood relationship elements can be generated according to the element conversion rules; and the mapped blood relationship elements obtained by parsing the message data are element-converted to obtain the corresponding virtual mapping graph elements, and then a data blood relationship model is generated according to the basic graph elements and the virtual mapping graph elements; since the virtual mapping elements in the above data blood relationship model are only connected through mapping relationships, blood relationship queries can be performed through virtual mapping graph elements when performing blood relationship queries, which can improve the query efficiency when performing blood relationship queries.

[0069] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0070] Figure 3An application scenario diagram of the technical solution provided in this application. The application scenario of the technical solution provided in this application can be: in industries such as finance, data governance can be carried out through a data lineage model to help companies meet regulatory requirements; specifically, the data lineage model can provide transparency and traceability of data flow; the data lineage model can also be used to track the source and change process of data, identify and correct data quality issues, and ensure data accuracy and consistency.

[0071] Optionally, the technical solution provided in this application can also be applied to machine learning projects; specifically, the data lineage relationship in the data lineage relationship model can help manage and track the source, processing process and version of the model training data to ensure the repeatability and reliability of the model training; the data lineage relationship can also be used to better explain and understand the decision-making process of the machine learning model, thereby improving the transparency and credibility of the model.

[0072] In summary, the data lineage relationship model determination method and lineage relationship determination method provided in this application have broad application value in data management, analysis, governance, security and other aspects, and can help enterprises improve the transparency, accuracy and efficiency of data processing.

[0073] The application scenario of data governance of data generated by enterprises in the financial industry is used as an example for illustrative description. In this application scenario, the technical solution can be implemented in an electronic device, such as a server. Optionally, in some other implementations, the technical solution of the current scenario can also be applied to a terminal device with big data processing capabilities, without specific limitation. Figure 3 , the scenario includes: various components 31 in the big data platform and a data lineage relationship model management system 32; specifically, each component 31 generates message data according to the lineage relationship generated during the operation process, and transmits the message data to the data lineage relationship model management system 32, so as to generate a data lineage relationship model corresponding to each component in the data lineage relationship model management system 32, and realize subsequent lineage relationship query through the data lineage relationship model, thereby realizing data traceability based on lineage relationship.

[0074] The data lineage relationship model management system 32 includes a data receiving module 321, a data processing module 322, a model generation module 323 and a data query module 324. Specifically, when receiving the message data transmitted by the component, first, the data receiving module 321 determines whether the message data meets the preset conditions; optionally, if so, the message data is received; secondly, the data processing module 322 is used to parse the message data to obtain the basic lineage relationship elements corresponding to the message data, and the basic graph elements are obtained based on the basic lineage relationship elements; and when the parsing result of the message data also includes the mapping lineage relationship elements, the virtual mapping graph elements are obtained based on the mapping lineage relationship elements and the basic lineage relationship elements; finally, the data lineage relationship model corresponding to each of the request components is generated based on the basic graph elements and the virtual mapping graph elements; on this basis The model generation module 323 has also developed a corresponding blood relationship query algorithm based on the data blood relationship model. The algorithm can use the virtual mapping nodes and virtual mapping edges in the data blood relationship model to perform blood relationship query when the data query module 324 inputs content for blood relationship query. Since the basic nodes and basic edges formed by the blood relationship between the data are complex and intertwined, if the above-mentioned basic nodes and basic edges are used to generate the query path when performing blood relationship query, it will lead to low query efficiency; and the virtual mapping nodes and virtual mapping edges only connect the nodes with the mapping relationship. Therefore, when performing blood relationship query, the query is performed through the virtual mapping nodes and virtual mapping edges, which can improve the query efficiency of blood relationship query.

[0075] based on Figure 3 In the application scenario shown, an embodiment of the present application provides a method for determining a data lineage relationship model. Figure 4 Flow chart of the method for determining the data kinship model provided in the embodiment of the present application. Figure 4 As shown, the method for determining the data blood relationship model includes:

[0076] S401, receiving message data that meets preset conditions transmitted by each request component; the preset conditions include data source conditions and data content conditions.

[0077] The execution subject of the method of this embodiment may be Figure 3 The data lineage model management system shown in.

[0078] In actual applications, multiple components are set up in the server, and each component can be used as the underlying component of the big data platform to support the big data platform for business processing. Optionally, the above components can also open usage permissions to third-party applications, receive data processing tasks uploaded by applications with usage permissions, and provide feedback after generating the processing results corresponding to the tasks. Exemplarily, the components in this application include but are not limited to: data management components, data acquisition components, data service components, data integration components, and stream computing components.

[0079] When running, components will be accompanied by data flow, and the data flow will generate blood relations between the data. To facilitate subsequent data traceability, the component can generate message data based on the generated data blood relations and transmit it to the current data blood relations model management system. After system processing, a data blood relations model is generated, which is convenient for subsequent big data platforms and business modules to perform blood relations queries through the data blood relations model to achieve data traceability.

[0080] In an embodiment of the present application, the component to upload the blood relationship can be called a request component; on this basis, the request component can generate corresponding message data for the blood relationship generated by the data flow generated by the component, and transmit the above message data to the data blood relationship model relationship system through the http interface for processing to generate a data blood relationship model.

[0081] In order to make the generated data lineage relationship model more accurate and complete, the corresponding data validation rules for data reception can be set in advance in the data lineage relationship model management system, and then when the message data uploaded by the request component is received, it is verified based on the data validation rules, and when it meets the preset conditions, the message is received into the current system for model generation, otherwise the message data is discarded. Furthermore, in order to reduce the probability of the message data transmitted by the request component being discarded, data transmission requirements can be issued to each request component in advance, even if each request component processes and transmits data based on the received data transmission requirements.

[0082] S402, parsing the message data to obtain basic blood relationship elements corresponding to the message data, and obtaining basic graph elements according to the basic blood relationship elements.

[0083] In this embodiment, when receiving message data that meets the preset conditions, the message data can be parsed based on the preset parsing rules to obtain the basic blood relationship elements uploaded by the request component. The basic blood relationship elements may include but are not limited to: applications, components, data services, external data, data sets, tables, fields, and output fields.

[0084] Furthermore, the preset element conversion rules may be used to perform element conversion on the basic blood relationship elements to obtain basic graph elements for subsequently generating a blood relationship model. Specifically, the basic graph elements include basic node elements and basic edge elements.

[0085] S403: When the parsing result of the message data also includes a mapping blood relationship element, a virtual mapping graph element is obtained according to the mapping blood relationship element and the basic blood relationship element.

[0086] In the embodiment of the present application, the mapping lineage relationship can be understood as the corresponding relationship between the various data flowing in the components; in the present application, the granularity of the mapping lineage relationship includes table-level mapping lineage relationship and field-level mapping lineage relationship.

[0087] Specifically, whether the data uploaded by the request component contains the mapped blood relationship element can be determined based on whether the parsing result of the message data contains a judgment result of a specific field in the preset data field.

[0088] Optionally, if a specific field exists, it means that the data uploaded by the request component includes mapping blood relationship elements; on this basis, the mapping blood relationship elements can be element-processed to obtain virtual mapping graph elements for generating data blood relationship models; optionally, when performing graph element conversion, basic blood relationship elements can be used to assist in element conversion processing of the mapping blood relationship elements to generate virtual mapping graph elements.

[0089] S404: Generate a data lineage relationship model corresponding to each of the request components according to the basic graph elements and the virtual mapping graph elements.

[0090] In the present application, when basic graph elements and virtual mapping graph elements are generated according to the above-mentioned implementation mode, the basic graph elements and the virtual graph elements can be stored together to realize the generation of a data lineage relationship model.

[0091] Exemplarily, the generated basic graph elements and virtual mapping graph elements (including nodes and edges) can be saved in one or more csv or json files according to the data writing requirements of the preset location, and then the files are stored in the preset location of the data lineage relationship model management system to obtain the generated data lineage relationship model.

[0092] In the above technical scheme, by receiving the message data that meets the preset conditions transmitted by each request component; the preset conditions include data source conditions and data content conditions; parsing the message data, obtaining the basic blood relationship elements corresponding to the message data, and obtaining the basic graph elements according to the basic blood relationship elements; in the case where the parsing result of the message data also includes the mapping blood relationship elements, obtaining the virtual mapping graph elements according to the mapping blood relationship elements and the basic blood relationship elements; generating the data blood relationship model corresponding to each of the request components according to the basic graph elements and the virtual mapping graph elements. Since the relationship between the basic graph elements, i.e., the relationship between the base and the elements, formed by the blood relationship between the data is complex and intertwined, if the query path is generated by the above basic graph elements when performing blood relationship query, the query efficiency will be low; and the virtual mapping elements are only connected through the mapping relationship, so when performing blood relationship query, the blood relationship is quickly queried through the virtual mapping graph elements, which can improve the query efficiency when performing blood relationship query.

[0093] Optionally, in an embodiment of the present application, before the requesting component transmits the blood relationship generated by the data flow, the technical solution provided by the present application also includes: for any requesting component, upon receiving the data transmission request sent by the requesting component, sending a data transmission requirement to the requesting component; wherein the data transmission requirement includes a preset data field included in the data transmission by the requesting component, so that each of the preset components generates and transmits message data according to the data transmission requirement; the preset data field includes component source information, component environment information, component node information, component application, component source data table information, component target data table information and component mapping information; wherein the component mapping information may be empty.

[0094] Specifically, in order to reduce the probability of data being discarded when the request component transmits data to the data lineage relationship model management system due to failure to meet the system's data reception conditions, a data transmission request can be sent to the data lineage relationship model management system in advance before the request component transmits the data to obtain the data transmission requirements fed back by the data lineage relationship model management system; then the data request component processes the lineage relationship according to the data transmission requirements, generates message data and uploads it.

[0095] Optionally, the request component may periodically send data transmission requests to the data lineage model management system at preset intervals to prevent the data lineage model management system from updating the data transmission requirements; optionally, in order to further reduce the probability of data discard, a data transmission request may be sent before each data upload, without limiting the probability of sending the request.

[0096] Of course, in some optional embodiments, the data lineage relationship model management system may also periodically and proactively send the latest data transmission request to the request component, so that the request component can timely transmit data that meets the latest requirements to the data lineage relationship model management system.

[0097] In this application, data transmission requirements may include but are not limited to preset data fields contained in the message data uploaded by the request component. On this basis, the data kinship model management system can generate more complete kinship information according to the data content corresponding to each preset data field uploaded, thereby improving the accuracy of the kinship in the model.

[0098] For example, see Figure 5 , the preset data fields may include: component source information (component), component environment information (env), component node information (dataflow), component application (app), component source data table information (source), component target data table information (target) and component mapping information (mapping). Among them, the component source information represents the request component that sends the message data; illustratively, the component source information may be a data management component, a data acquisition component, a data service component, a data integration component, and a stream computing component. Component environment information represents the environment in which the components are located when the blood relationship is generated; exemplarily, the component environment information can be a test environment or a production environment; component node information represents the task data processing information corresponding to the current component; exemplarily, the component node information includes data processing type, business primary key, node name, and custom attributes; the application to which the component belongs represents the application to which the current component belongs; exemplarily, the application to which the component belongs can be a big data platform or a third-party application; component source data table information represents the source table of the component node (dataflow); component target data table information (target) represents the target table of the component node (dataflow); component mapping information (mapping) represents the correspondence between the source table and the target table of the component; exemplarily, the mapping information includes mapping type, mapping source table, and mapping target table; specific mapping types include table-level blood relationship and field blood relationship, and the relationship supports one-to-one, many-to-one, and many-to-many; the mapping source table and the mapping target table can be internal tables and external tables; it should be noted that the above-mentioned source table, target table, mapping source table, and mapping target table contain the following fields: element type, data source name, database name, data table name, field name set, and extended attributes, etc.

[0099] Optionally, after the request component generates and transmits a data message according to the data transmission rules, the technical solution of the present application also includes: for message data transmitted by any request component, verifying the request component corresponding to the received message data; if the request component passes the verification, verifying whether the message data contains all preset data fields; if the message data contains all preset data fields, receiving the message data; if the request component fails the verification or the message data does not contain all preset data fields, discarding the message data, generating a discard prompt message corresponding to discarding the message data and feeding it back to the request component.

[0100] Figure 6 A flow chart of message data reception provided in an embodiment of the present application. When the data lineage relationship model management system receives message data transmitted by a request component, take the message data transmitted by any request component as an example: Figure 6 , determine the request component corresponding to the received message data; further, verify whether the request component has pre-enabled the blood relationship reporting function; that is, verify whether the request component has the authority to transmit the message data; optionally, if so, it means that the request component has passed the verification; otherwise, the request component has not passed the verification, and the message data corresponding to the request component is directly discarded, and a discard prompt message corresponding to the message data is generated and fed back to the request component; further, verify whether the received message data contains the preset data field in the data transmission requirement; optionally, if it does, it means that the message data has passed the verification and the message data has been received; otherwise, it means that the message data has not passed the verification, and the message data can be directly discarded, and a discard prompt message corresponding to the message data is generated and fed back to the request component.

[0101] It should be noted that if the requesting component is a data management component, the component mapping information field in the message data uploaded by the device may be empty; the reason is that the data management component only generates a target table during the task execution process, and there is no corresponding source table, so a component mapping relationship cannot be formed.

[0102] On the basis of receiving the message data that meets the conditions, the message data can be processed to generate a data lineage relationship model.

[0103] Optionally, in an embodiment of the present application, parsing the message data to obtain a basic blood relationship element corresponding to the message data, and obtaining a basic graph element according to the basic blood relationship element includes:

[0104] Based on preset parsing rules, elements of the message data are extracted and verified to obtain an element set corresponding to the message data; the element set includes multiple basic blood relationship elements corresponding to preset data fields; according to preset element conversion rules, each of the basic blood relationship elements is converted to obtain basic graph elements corresponding to the generated data blood relationship model; the basic graph elements include basic node elements and basic edge elements.

[0105] Specifically, for the message data transmitted by the request component, the preset data fields in the message data can be extracted and verified based on the preset parsing rules, such as the JSONSchema parsing rules, to obtain an element set. The element set contains multiple basic lineage elements; wherein the basic lineage elements are other elements in the element set except the elements corresponding to the component mapping information field. Exemplarily, the basic lineage elements include but are not limited to: application, component, data service, external data, data set, table, field, output field.

[0106] Furthermore, each entity object (such as node / edge type, id, display name, attribute value, etc.) in the above-mentioned basic blood relationship elements is converted into basic graph elements, namely basic node elements and node edge elements, according to predefined element conversion rules. Specifically, each entity class object may be matched one-to-one with a graph element, and the format of the graph element may be converted and verified, and finally basic node elements and basic edge elements that meet the storage requirements of the graph model may be generated. Exemplarily, the basic node elements in the generated basic graph elements may include: application information nodes, component nodes, data service query definition nodes, external data source nodes, data set nodes, data table nodes, field nodes, and output field nodes; basic edge elements include belonging relationship edges and upstream and downstream relationship edges.

[0107] On this basis, the technical solution of the present application also includes determining whether the parsing results of the message data contain mapped blood relationship elements; if so, performing element conversion and generating a data blood relationship model based on the converted graph elements.

[0108] Optionally, when the parsing result of the message data also includes mapping blood relationship elements, a virtual mapping graph element is obtained according to the mapping blood relationship elements and the basic blood relationship elements, including: when it is determined that there is a composite element corresponding to the component mapping information in the element set, the composite element is disassembled to obtain multiple mapping blood relationship elements; the mapping blood relationship elements include table-level mapping blood relationship elements and field-level mapping blood relationship elements; according to each of the mapping blood relationship elements and each of the basic blood relationship elements, a virtual mapping graph element corresponding to the generated data blood relationship model is obtained.

[0109] Specifically, in the element set obtained after parsing the message data, find out whether there is an element corresponding to the component mapping information field, that is, a composite element; it should be explained that a composite element can be understood as an element that can be further disassembled into basic elements.

[0110] Optionally, if there are composite elements in the element combination, the composite elements are disassembled according to predefined element disassembly rules to obtain multiple mapping lineage elements; optionally, according to different mapping types, the mapping lineage elements include table-level mapping lineage elements and field-level mapping lineage elements.

[0111] Furthermore, element conversion is performed based on the mapped blood relationship elements obtained by the disassembly and the basic blood relationship elements obtained by the above analysis to obtain virtual mapping elements.

[0112] Optionally, in an embodiment of the present application, the virtual mapping graph elements include virtual mapping node elements and virtual mapping edge elements; at least one virtual mapping node element corresponding to the data lineage relationship model is determined to be generated according to each of the mapping lineage relationship elements; the virtual mapping node elements include table-level virtual mapping node elements and field-level virtual mapping node elements; based on each of the basic lineage relationship elements and each of the mapping virtual component nodes, multiple virtual mapping edge elements corresponding to the data lineage relationship model are determined to be generated.

[0113] Exemplarily, if the virtual blood relationship element is a table-level mapping, a table-level virtual mapping node element can be constructed based on the node information between the mapping source table element and the mapping target table element in the virtual blood relationship element; further, a virtual mapping relationship, i.e., a virtual mapping edge element, is constructed based on the source node element and the target node element in the basic blood relationship element of the request component and the generated virtual mapping node element; exemplary, the constructed virtual mapping edge elements include but are not limited to: virtual mapping edge elements between source data table node elements and virtual mapping node elements, virtual mapping variable elements between virtual mapping node elements and target data table node elements, and virtual mapping edges between component nodes and virtual mapping nodes.

[0114] Furthermore, based on the above generated basic node elements, basic variable elements, virtual mapping node elements and basic mapping variable elements, they are stored together to generate a data lineage relationship model. Exemplarily, the generated lineage relationship model can be as follows: Figure 7 shown.

[0115] In this way, when performing a blood relationship query in the generated data blood relationship model, a query path is generated and a query is performed through virtual mapping nodes and virtual mapping edges, which can improve the query efficiency during blood relationship query.

[0116] Figure 8This is a flow chart of the method for determining blood relationship provided in the embodiment of the present application. Figure 8 As shown, the method for determining blood relationship comprises the following steps:

[0117] S801: Receive a node to be analyzed.

[0118] In the present application, the node to be analyzed may be a node input by a user through a preset blood relationship query interface. For example, a user may input a unique node identifier corresponding to the node to be analyzed to facilitate subsequent accurate query.

[0119] It should also be noted that since the bloodline relationship uploaded by the request component when generating the data bloodline relationship model contains elements corresponding to multiple data fields, the generated data bloodline relationship model contains multiple types of node elements; for example, table-level nodes, field-level nodes, etc.

[0120] To ensure accurate and fast query of subsequent blood relationship, the user can determine the unique identifier of the node to be analyzed based on the type of node to be queried and the corresponding node code under this type, and input the unique identifier. In this way, the data blood relationship model can determine the node to be analyzed based on the unique identifier, and perform blood relationship query with the node to be analyzed as the starting query node.

[0121] S802: Determine, according to the node type of the node to be analyzed, basic node elements and virtual mapping node elements of the same node type as the node in a pre-generated data lineage relationship model.

[0122] In this application, the node type of the node to be analyzed is a table-level node as an example for exemplary introduction. Specifically, the node type of the node to be analyzed is determined according to the unique identifier of the node to be analyzed received. Figure 7 When it is determined that the input node to be analyzed is a table-level node, that is, a TableA node, the basic node elements and virtual node elements of the same type as the TableA node are determined in the data lineage model, exemplarily including: TableB, TableC, TableD, Dummy_Info1, Dummy_Info2, Dummy_Info3, and Dummy_Info4.

[0123] S803: Perform a blood relationship search based on each of the basic node elements and each of the virtual mapping node elements to obtain the blood relationship corresponding to the node to be analyzed.

[0124] In this application, when performing a blood relationship query in a data blood relationship model, it is also necessary to know the query depth. For example, the query depth can be pre-set in the embodiment of this application. Of course, in some other embodiments, the query depth can also be input by the user according to the needs, and there is no specific limitation on this. Among them, the query depth can be understood as querying from a node in the data blood relationship model to the next level node corresponding to the node; for example, as the depth increases, the number of node levels queried will also increase accordingly; in some other implementations, the query depth can also be referred to as the cross-level query level.

[0125] Furthermore, based on the pre-set query depth, for example, the depth is 2, that is, the number of cross-levels of the cross-layer query is 2, and then the node to be analyzed is used as the starting query node, and according to the query depth, the basic edge elements and virtual edge elements between the basic node elements and the virtual mapping node elements are used as the query path to perform a blood relationship query and obtain the query result.

[0126] Exemplarily, the obtained query path includes:

[0127] 1. TableA->Dummy_Info1->TableB->Dummy_Info4->TableD;

[0128] 2. TableA->Dummy_Info2->TableB->Dummy_Info4->TableD;

[0129] 3. TableA->Dummy_Info3->TableB->Dummy_Info4->TableD;

[0130] In summary, we find that the nodes starting from TableA and with a cross-layer depth of 2 include: TableB, TableC, and TableD. Therefore, the kinship relationships obtained by the query are TableA->TableB->TableD and TableA->TableC->TableD.

[0131] It should be noted that, still taking the above-mentioned query of blood relationship as an example, that is, given the query starting point TableA, the cross-layer search depth is 2, if the blood relationship graph model constructed in the existing implementation method is adopted, that is Figure 2 The blood relationship graph model in the query is used to query blood relationship. The specific query results may include:

[0132] 1. TableA-> Job1-> TableB, find the node TableB with a cross-layer depth of 1;

[0133] 2. TableA-> Job2-> TableB, find the node TableB with a cross-layer depth of 1;

[0134] 3. TableA-> Job3-> TableB, find the node TableB with a cross-layer depth of 1;

[0135] 4. TableA-> Job3-> TableC, find the node TableC with a cross-layer depth of 1;

[0136] Next, starting from the node with a cross-layer depth of 1 (TableB and TableC), find the node with a cross-layer depth of 1:

[0137] 1. TableB-> Job4-> TableE, find the node TableD with a cross-layer depth of 1;

[0138] 2. TableC-> Job4-> TableE, find the node TableD with a cross-layer depth of 1;

[0139] In summary, we find that the nodes with TableA as the starting point and a cross-layer depth of 2 are: TableB, TableC, and TableD. Therefore, the blood relationship obtained by the query is TableA->TableB->TableD and TableA->TableC->TableD.

[0140] By comparison, it can be found that the number of traversals is less when using virtual mapping nodes for path query, and there is no need to enter more complex query conditions during the query process; and because Figure 2 There are some edge elements between the midpoints that are not shown. Figure 2 There are other edge elements between the nodes, which causes some invalid paths to be traversed during the path query process, resulting in reduced query efficiency; while in the technical solution of the embodiment of the present application, that is, in the data lineage relationship model constructed in the embodiment of the present application, virtual mapping nodes are used for path query, and invalid paths will not be traversed during the path traversal process, thereby accelerating the query speed and improving the query efficiency.

[0141] Fig. 9 This is a schematic diagram of the structure of the data kinship model determination device provided in the embodiment of the present application. Fig. 9 As shown, the data lineage relationship model determination device 90 includes: a message data receiving module 901, a basic graph element generation module 902, a virtual mapping graph element generation module 903 and a data lineage relationship model generation module 904; wherein,

[0142] The message data receiving module 901 is used to receive the message data transmitted by each request component and meeting the preset conditions; the preset conditions include data source conditions and data content conditions;

[0143] A basic graph element generation module 902 is used to parse the message data, obtain the basic blood relationship elements corresponding to the message data, and obtain the basic graph elements according to the basic blood relationship elements;

[0144] A virtual mapping element generating module 903 is used to obtain a virtual mapping element according to the mapped blood relationship element and the basic blood relationship element when the parsing result of the message data also includes a mapped blood relationship element;

[0145] The data lineage relationship model generation module 904 is used to generate a data lineage relationship model corresponding to each of the request components according to the basic graph elements and the virtual mapping graph elements.

[0146] In an optional implementation manner, the message data receiving module 901 includes:

[0147] A first verification submodule, for verifying the request component corresponding to the received message data transmitted by any request component;

[0148] A second verification submodule, configured to verify whether the message data contains all preset data fields if the request component passes the verification;

[0149] A message data receiving submodule, configured to receive the message data if the message data contains all preset data fields;

[0150] The message data discarding submodule is used to discard the message data if the request component fails to pass the verification or the message data does not contain all preset data fields, generate a discard prompt message corresponding to discarding the message data and feed it back to the request component.

[0151] In an optional embodiment, the device further comprises:

[0152] A data sending module, for sending a data transmission request to any request component upon receiving a data transmission request sent by the request component;

[0153] The data transmission requirement includes a preset data field included in the request component when performing data transmission, so that each preset component generates and transmits message data according to the data transmission requirement;

[0154] The preset data fields include component source information, component environment information, component node information, component application, component source data table information, component target data table information and component mapping information; wherein the component mapping information may be empty.

[0155] In an optional implementation, the basic graph element generation module 902 includes:

[0156] An element parsing submodule, used to extract and verify the message data based on preset parsing rules to obtain an element set corresponding to the message data; the element set includes a plurality of basic blood relationship elements corresponding to preset data fields;

[0157] The element conversion submodule is used to convert each of the basic blood relationship elements according to the preset element conversion rules to obtain the basic graph elements corresponding to the generated data blood relationship model; the basic graph elements include basic node elements and basic edge elements.

[0158] In an optional implementation, the virtual map element generation module 903 includes:

[0159] A mapping blood relationship element acquisition submodule is used to disassemble the composite element to obtain a plurality of mapping blood relationship elements when it is determined that there is a composite element corresponding to the component mapping information in the element set; the mapping blood relationship elements include table-level mapping blood relationship elements and field-level mapping blood relationship elements;

[0160] The virtual mapping diagram element generation submodule is used to obtain the virtual mapping diagram element corresponding to the generated data blood relationship model according to each of the mapped blood relationship elements and each of the basic blood relationship elements.

[0161] In an optional implementation, the virtual mapping graph element includes a virtual mapping node element and a virtual mapping edge element;

[0162] The virtual map element generation submodule includes:

[0163] A virtual mapping node element generating unit, used for determining and generating at least one virtual mapping node element corresponding to a data lineage relationship model according to each of the mapping lineage relationship elements; the virtual mapping node element includes a table-level virtual mapping node element and a field-level virtual mapping node element;

[0164] The virtual mapping edge element generation submodule is used to determine and generate multiple virtual mapping edge elements corresponding to the data blood relationship model according to each of the basic blood relationship elements and each of the mapping virtual component nodes.

[0165] The data lineage relationship model determination device provided in the embodiment of the present application can be used to execute the technical solution of the data lineage relationship model determination method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0166] Fig.10 This is a schematic diagram of the structure of the blood relationship determination device provided in the embodiment of the present application. Fig.10 As shown, the blood relationship determination device 100 includes: a node receiving module 101, a node element determination module 102 and a blood relationship determination module 103; wherein,

[0167] The node receiving module 101 is used to receive the node to be analyzed;

[0168] A node element determination module 102 is used to determine, according to the node type of the node to be analyzed, the basic node elements and virtual mapping node elements of the same node type in the pre-generated data lineage relationship model; the data lineage relationship model is generated according to the data lineage relationship model determination method described in the above embodiment;

[0169] The blood relationship determination module 103 is used to perform blood relationship search based on each of the basic node elements and each of the virtual mapping node elements to obtain the blood relationship corresponding to the node to be analyzed.

[0170] The blood relationship determination device provided in the embodiment of the present application can be used to execute the technical solution of the blood relationship determination method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0171] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the first verification submodule can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the first verification submodule. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0172] Fig.11This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.11 As shown, the electronic device may include: a transceiver 111 , a processor 112 , and a memory 113 .

[0173] The processor 112 executes the computer-executable instructions stored in the memory, so that the processor 112 executes the scheme in the above embodiment. The processor 112 can be a general-purpose processor, including a central processing unit CPU, a network processor (network processor, NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0174] The memory 113 is connected to the processor 112 via a system bus and completes communication between them. The memory 113 is used to store computer program instructions.

[0175] The transceiver 111 may be used to obtain tasks to be executed and configuration information of the tasks to be executed.

[0176] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0177] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0178] An embodiment of the present application also provides a chip for running instructions, which is used to execute the technical solutions of the data blood relationship model determination method and the blood relationship determination method in the above embodiments.

[0179] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the technical solutions of the data kinship relationship model determination method and the kinship determination method of the above-mentioned embodiment.

[0180] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solutions of the data blood relationship model determination method and the blood relationship determination method in the above-mentioned embodiments.

[0181] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0182] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0183] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0184] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0185] It should be understood that the above processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0186] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0187] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0188] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0189] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0190] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0191] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for determining a data kinship model, characterized in that: include: Receive message data that meets preset conditions transmitted by each request component; The preset conditions include data source conditions and data content conditions; Parsing the message data to obtain basic blood relationship elements corresponding to the message data, and obtaining basic graph elements according to the basic blood relationship elements; In the case where the parsing result of the message data also includes a mapping blood relationship element, obtaining a virtual mapping graph element according to the mapping blood relationship element and the basic blood relationship element; A data lineage relationship model corresponding to each of the request components is generated according to the basic graph elements and the virtual mapping graph elements.

2. The method according to claim 1, characterized in that Receive message data that meets preset conditions transmitted by each request component, including: For message data transmitted by any request component, verify the request component corresponding to the received message data; If the request component passes the verification, verify whether the message data contains all preset data fields; If the message data includes all preset data fields, receiving the message data; If the request component fails the verification or the message data does not contain all preset data fields, the message data is discarded, and a discard prompt message corresponding to discarding the message data is generated and fed back to the request component.

3. The method according to claim 1 or 2, characterized in that: Before receiving the message data transmitted by each request component and meeting the preset conditions, the method further includes: For any request component, upon receiving a data transmission request sent by the request component, sending a data transmission request to the request component; The data transmission requirement includes a preset data field included in the request component when performing data transmission, so that each preset component generates and transmits message data according to the data transmission requirement; The preset data fields include component source information, component environment information, component node information, component application, component source data table information, component target data table information and component mapping information; wherein the component mapping information may be empty.

4. The method according to claim 1, characterized in that: Parsing the message data to obtain basic blood relationship elements corresponding to the message data, and obtaining basic graph elements according to the basic blood relationship elements, including: Extracting and verifying the message data based on preset parsing rules to obtain an element set corresponding to the message data; the element set includes a plurality of basic blood relationship elements corresponding to preset data fields; According to the preset element conversion rules, each of the basic blood relationship elements is converted to obtain the basic graph elements corresponding to the generated data blood relationship model; the basic graph elements include basic node elements and basic edge elements.

5. The method according to claim 1, characterized in that: In the case that the parsing result of the message data also includes a mapping blood relationship element, obtaining a virtual mapping element according to the mapping blood relationship element and the basic blood relationship element includes: When it is determined that there is a composite element corresponding to the component mapping information in the element set, the composite element is disassembled to obtain a plurality of mapping lineage relationship elements; the mapping lineage relationship elements include table-level mapping lineage relationship elements and field-level mapping lineage relationship elements; According to each of the mapped blood relationship elements and each of the basic blood relationship elements, a virtual mapping element corresponding to the generated data blood relationship model is obtained.

6. The method according to claim 5, characterized in that The virtual mapping graph elements include virtual mapping node elements and virtual mapping edge elements; According to each of the mapped blood relationship elements and each of the basic blood relationship elements, a virtual mapping element corresponding to the generated data blood relationship model is obtained, including: Determine and generate at least one virtual mapping node element corresponding to the data lineage relationship model according to each of the mapping lineage relationship elements; the virtual mapping node element includes a table-level virtual mapping node element and a field-level virtual mapping node element; According to each of the basic blood relationship elements and each of the mapping virtual component nodes, a plurality of virtual mapping edge elements corresponding to the generated data blood relationship model are determined.

7. A method for determining blood relationship, characterized in that: include: Receive the node to be analyzed; Determine, according to the node type of the node to be analyzed, the basic node elements and virtual mapping node elements of the same node type in the pre-generated data lineage relationship model; The data lineage relationship model is generated according to the data lineage relationship model determination method according to any one of claims 1 to 6; A blood relationship search is performed based on each of the basic node elements and each of the virtual mapping node elements to obtain the blood relationship corresponding to the node to be analyzed.

8. A data kinship model determination device, characterized in that: include: A message data receiving module, used to receive message data that meets preset conditions and is transmitted by each request component; The preset conditions include data source conditions and data content conditions; A basic graph element generation module, used for parsing the message data, obtaining basic blood relationship elements corresponding to the message data, and obtaining basic graph elements according to the basic blood relationship elements; A virtual mapping element generating module, configured to obtain a virtual mapping element according to the mapped blood relationship element and the basic blood relationship element when the parsing result of the message data also includes a mapped blood relationship element; The data lineage relationship model generation module is used to generate a data lineage relationship model corresponding to each of the request components according to the basic graph elements and the virtual mapping graph elements.

9. A blood relationship determination device, characterized in that: include: A node receiving module, used for receiving nodes to be analyzed; A node element determination module, used to determine, according to the node type of the node to be analyzed, the basic node elements and virtual mapping node elements of the same node type in the pre-generated data kinship model; The data lineage relationship model is generated according to the data lineage relationship model determination method according to any one of claims 1 to 6; The blood relationship determination module is used to perform blood relationship search based on each of the basic node elements and each of the virtual mapping node elements to obtain the blood relationship corresponding to the node to be analyzed.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the data blood relationship model determination method as described in any one of claims 1-6, and / or the blood relationship determination method as described in claim 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data blood relationship model determination method as described in any one of claims 1 to 6, and / or the blood relationship determination method as described in claim 7.

12. A computer program product, characterized in that It comprises a computer program which, when executed by a processor, implements the data blood relationship model determination method described in any one of claims 1 to 6, and / or the blood relationship determination method described in claim 7.

Images