Data processing method and device

Abstract

The invention provides a data processing method and device, and the method comprises the steps: carrying out the metadata collection in a data circulation process in response to a data processing instruction, and obtaining each piece of metadata; each metadata comprises data source information, data conversion configuration information, data storage information and a data access record of a data entity to which the metadata belongs; storing the metadata in a preset graph database, and analyzing the metadata stored in the graph database by using a preset blood relationship analysis technology to obtain a blood relationship between data entities to which the metadata belong; and carrying out visual display on the blood relationship among the data entities. By applying the method provided by the invention, the blood relationship of the data can be clearly displayed, and data tracing is facilitated.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method and device. Background Art

[0002] In the modern enterprise environment, data processing processes are becoming increasingly complex, involving interactions between multiple systems, platforms, and tools. With the expansion of business scale and technological advancement, data is not only exploding in quantity, but also its processing methods are becoming more diverse. Data usually needs to be converted, processed, and integrated multiple times before it can be finally transformed into valuable business insights or decision support information.

[0003] However, this complex processing process brings significant challenges: data flows in from multiple sources, goes through a series of ETL (Extract, Transform, Load) jobs and other data processing steps, and is then stored in different target systems. Due to the frequent and diverse operations, the source and flow of data in its life cycle become extremely unclear, making it difficult to trace and locate data problems. Summary of the invention

[0004] The technical problem to be solved by this application is to provide a data processing method that can clearly display the blood relationship of data and facilitate data tracing.

[0005] The present application also provides a data processing device to ensure the implementation and application of the above method in practice.

[0006] A data processing method, the method comprising:

[0007] In response to the data processing instruction, metadata collection is performed during the data flow process to obtain various metadata; each of the metadata includes data source information, data conversion configuration information, data storage information and data access records of the data entity to which it belongs;

[0008] The metadata are stored in a preset graph database, and the metadata stored in the graph database are analyzed using a preset lineage analysis technology to obtain the lineage relationship between the data entities to which the metadata belong;

[0009] The blood relationship between each of the data entities is visualized.

[0010] Optionally, the above method, after visually displaying the blood relationship between the data entities, further comprises:

[0011] When an operation on the data entity is detected, metadata of the data entity stored in the graph database is updated according to the operation to update the data entity blood relationship.

[0012] Optionally, the above method, after visually displaying the blood relationship between the data entities, further comprises:

[0013] Determine the starting node selected by the user;

[0014] Querying a preset graph database according to the starting node to determine a data link of the starting node; the data link includes at least one of an upstream link and a downstream link of the starting node;

[0015] The data link is displayed.

[0016] In the above method, optionally, the metadata collection is performed during the data flow process to obtain various metadata, including:

[0017] Parsing the data processing instruction to obtain collection requirement information;

[0018] Metadata collection is performed during data flow according to the collection requirement information to obtain metadata corresponding to the collection requirement information.

[0019] Optionally, the above method uses a preset lineage analysis technology to analyze the metadata stored in the graph database to obtain the lineage relationship between the data entities to which each metadata belongs, including:

[0020] Analyze the metadata stored in the graph database using a preset lineage analysis technology to obtain the dependency relationship between the data entities to which each metadata belongs;

[0021] Each of the data entities is taken as a node, and the dependency relationship between the data entities to which each metadata belongs is taken as an edge, so as to construct a blood relationship graph of each of the data entities and obtain the blood relationship between the data entities to which each metadata belongs.

[0022] A data processing device, comprising:

[0023] An execution unit, configured to collect metadata in response to a data processing instruction during data flow to obtain various metadata; each of the metadata includes data source information, data conversion configuration information, data storage information, and data access records of the data entity to which it belongs;

[0024] An analysis unit, used to store each of the metadata in a preset graph database, and analyze the metadata stored in the graph database using a preset lineage analysis technology to obtain the lineage relationship between the data entities to which each of the metadata belongs;

[0025] The display unit is used to visually display the blood relationship between each of the data entities.

[0026] The above device may optionally further include:

[0027] A detection unit is used to update the metadata of the data entity stored in the graph database according to the operation when an operation on the data entity is detected, so as to update the blood relationship of the data entity.

[0028] The above device may optionally further include:

[0029] A determination unit, used to determine a starting node selected by a user;

[0030] A query unit, configured to query a preset graph database according to the starting node to determine a data link of the starting node; the data link includes at least one of an upstream link and a downstream link of the starting node;

[0031] A display unit is used to display the data link.

[0032] In the above device, optionally, the execution unit includes:

[0033] A parsing subunit, used to parse the data processing instruction to obtain collection requirement information;

[0034] The collection subunit is used to collect metadata during data flow according to the collection requirement information, and obtain metadata corresponding to the collection requirement information.

[0035] In the above device, optionally, the analysis unit comprises:

[0036] An analysis subunit, used to analyze the metadata stored in the graph database using a preset lineage analysis technology to obtain dependency relationships between data entities to which each metadata belongs;

[0037] The execution subunit is used to take each of the data entities as a node and the dependency relationship between the data entities to which each metadata belongs as an edge, so as to construct a blood relationship map of each of the data entities and obtain the blood relationship between the data entities to which each metadata belongs.

[0038] A storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the data processing method as described above.

[0039] An electronic device includes a memory and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to be used by one or more processors to execute the data processing method described above.

[0040] Compared with the prior art, this application has the following advantages:

[0041] The present application provides a data processing method and device, the method comprising: in response to a data processing instruction, performing metadata collection during data flow to obtain various metadata; each metadata includes data source information, data conversion configuration information, data storage information, and data access records of the data entity to which it belongs; storing each metadata in a preset graph database, and using a preset lineage analysis technology to analyze the metadata stored in the graph database to obtain the lineage relationship between the data entities to which each metadata belongs; visually displaying the lineage relationship between each data entity. In the embodiments provided in the present application, the lineage relationship of the data can be clearly displayed, which is convenient for data tracing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0043] Figure 1 A flowchart of a data processing method provided for this application;

[0044] Figure 2 A flowchart of a process for obtaining various metadata provided by this application;

[0045] Figure 3 A flowchart of another data processing method provided by the present application;

[0046] Figure 4 A schematic diagram of the structure of a data processing device provided in this application;

[0047] Figure 5 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0050] The present application embodiment provides a data processing method, which can be applied to electronic equipment. The method flow chart of the method is as follows: Figure 1 As shown, specifically including:

[0051] S101: In response to a data processing instruction, metadata collection is performed during data flow to obtain various metadata; each of the metadata includes data source information, data conversion configuration information, data storage information and data access records of the corresponding data entity.

[0052] In this embodiment, the data flow process may include at least one of a data generation process, a data processing process, and a data use process.

[0053] Optionally, each metadata contains key information of the data entity, such as data source information, data conversion configuration information, data storage information, and data access records. Specifically, when a new data operation occurs, the system automatically triggers the metadata collection mechanism. These metadata not only describe the static properties of the data, but also record its dynamic behavior and historical changes.

[0054] Optionally, data source information describes the original source of the data, including the system or platform where the data is generated, the specific file or database location; data transformation configuration information records the transformation rules and logic that the data undergoes during processing, such as cleaning and aggregation operations in ETL jobs; data storage information indicates the final location of the data, such as database table name, file path or cloud storage bucket; and data access records include access information such as the user who accessed the data, time, and access method, to ensure transparency and traceability of data use. Together, these metadata provide a comprehensive view of the data life cycle, supporting more effective data management and governance.

[0055] In an embodiment provided in the present application, based on the above solution, optionally, the process of collecting metadata during data flow to obtain various metadata is as follows: Figure 2 As shown, including:

[0056] S201: Parse the data processing instruction to obtain collection requirement information.

[0057] In this embodiment, by parsing the instructions, a specific data processing task (such as an ETL job, an API call, etc.) can be identified, and the required metadata type can be determined accordingly. For example, for an ETL job, detailed information such as input and output mapping, conversion logic, and execution timestamp may need to be recorded.

[0058] S202: performing metadata collection during data flow according to the collection requirement information to obtain metadata corresponding to the collection requirement information.

[0059] In this embodiment, metadata collection can be performed during the execution of the data processing task according to the collection requirement information to obtain metadata corresponding to the collection requirement information.

[0060] S102: storing each of the metadata in a preset graph database, and analyzing the metadata stored in the graph database using a preset lineage analysis technology to obtain the lineage relationship between the data entities to which each of the metadata belongs.

[0061] In an embodiment provided by the present application, based on the above solution, optionally, the metadata stored in the graph database is analyzed using a preset lineage analysis technology to obtain the lineage relationship between the data entities to which each metadata belongs, including:

[0062] Analyze the metadata stored in the graph database using a preset lineage analysis technology to obtain the dependency relationship between the data entities to which each metadata belongs;

[0063] Each of the data entities is taken as a node, and the dependency relationship between the data entities to which each metadata belongs is taken as an edge, so as to construct a blood relationship graph of each of the data entities and obtain the blood relationship between the data entities to which each metadata belongs.

[0064] In this embodiment, all collected metadata can be stored in a preset graph database, such as Neo4j. Subsequently, the metadata is deeply analyzed using the preset lineage analysis technology to reveal the dependencies between various data entities. Specifically, the system uses each data entity as a node and the dependencies between them as edges to construct a comprehensive lineage map. This graph model intuitively shows the flow path of data and their mutual influence, helping users better understand and manage complex data relationship networks.

[0065] Optionally, the blood relationship analysis technology may be a traversal algorithm, rule matching or other technology.

[0066] S103: Visually display the blood relationship between the data entities.

[0067] In this embodiment, the constructed blood relationship map is visualized so that users can intuitively see the connection mode and influence scope between different data entities. The graphical interface not only improves the user experience, but also enhances the interpretability and transparency of the data. Users can view more detailed blood relationship information by clicking on different nodes or links, and support two-way tracing from any data point - tracing the source of data upstream and exploring the scope of influence downstream. In addition, the system also provides interactive tools that allow users to customize query conditions and flexibly explore data links.

[0068] In an embodiment provided in the present application, based on the above solution, optionally, see Figure 3 After visually displaying the blood relationship between the data entities, the following steps are also included:

[0069] S104: When an operation on the data entity is detected, metadata of the data entity stored in the graph database is updated according to the operation to update the data entity blood relationship.

[0070] In this embodiment, when a new operation on a data entity is detected, the relevant metadata in the graph database is automatically updated to ensure that the blood relationship is always kept up to date. At the same time, the system supports users to select a specific starting node and query the data link of the node based on it, including the upstream link (data source) and the downstream link (scope of influence). This function greatly facilitates problem troubleshooting and decision support, allowing users to quickly locate the root cause of the problem and evaluate its potential impact, thereby improving the overall data governance efficiency.

[0071] In an embodiment provided by the present application, based on the above solution, optionally, after visually displaying the blood relationship between the data entities, the method further includes:

[0072] Determine the starting node selected by the user;

[0073] Querying a preset graph database according to the starting node to determine a data link of the starting node; the data link includes at least one of an upstream link and a downstream link of the starting node;

[0074] In this embodiment, a user interface may be displayed to allow the user to select a specific data entity of interest as a starting node. The starting node may be selected by clicking a node on the graphical interface or inputting a specific identifier.

[0075] Optionally, after the user selects the starting node, a preset graph database (such as Neo4j) is queried to find all metadata records associated with the node. This includes directly connected edges (indicating dependencies) as well as other indirectly related information. Using a traversal algorithm, starting from the starting node, its upstream and downstream links are explored along the direction of the edges. For upstream links, the system traces backward to all possible data sources; for downstream links, it traces forward to all affected target nodes. The query results are presented to the user in an intuitive manner, such as displaying the complete path of the data flow through a graphical interface. The user can see each node and the edges connected to it, and can further click to view detailed metadata information. In addition, filtering and sorting functions can be provided to allow users to more easily browse complex link structures.

[0076] and Figure 1 Corresponding to the method described above, the embodiment of the present application further provides a data processing device for Figure 1 The specific implementation of the method is shown in the following diagram: Figure 4 As shown, specifically including:

[0077] The execution unit 401 is used to collect metadata in the process of data flow in response to the data processing instruction to obtain various metadata; each of the metadata includes data source information, data conversion configuration information, data storage information and data access records of the data entity to which it belongs;

[0078] The analysis unit 402 is used to store each metadata in a preset graph database, and analyze the metadata stored in the graph database using a preset lineage analysis technology to obtain the lineage relationship between the data entities to which each metadata belongs;

[0079] The display unit 403 is used to visually display the blood relationship between each of the data entities.

[0080] In an embodiment provided in the present application, based on the above solution, optionally, it further includes:

[0081] A detection unit is used to update the metadata of the data entity stored in the graph database according to the operation when an operation on the data entity is detected, so as to update the blood relationship of the data entity.

[0082] In an embodiment provided in the present application, based on the above solution, optionally, it further includes:

[0083] A determination unit, used to determine a starting node selected by a user;

[0084] A query unit, configured to query a preset graph database according to the starting node to determine a data link of the starting node; the data link includes at least one of an upstream link and a downstream link of the starting node;

[0085] A display unit is used to display the data link.

[0086] In an embodiment provided in the present application, based on the above solution, optionally, the execution unit 401 includes:

[0087] A parsing subunit, used to parse the data processing instruction to obtain collection requirement information;

[0088] The collection subunit is used to collect metadata during data flow according to the collection requirement information, and obtain metadata corresponding to the collection requirement information.

[0089] In an embodiment provided in the present application, based on the above solution, optionally, the analysis unit includes:

[0090] An analysis subunit, used to analyze the metadata stored in the graph database using a preset lineage analysis technology to obtain dependency relationships between data entities to which each metadata belongs;

[0091] The execution subunit is used to take each of the data entities as a node and the dependency relationship between the data entities to which each metadata belongs as an edge, so as to construct a blood relationship map of each of the data entities and obtain the blood relationship between the data entities to which each metadata belongs.

[0092] The specific principles and execution processes of each unit and module in the data processing device disclosed in the above-mentioned embodiment of the present application are the same as the data processing method disclosed in the above-mentioned embodiment of the present application. Please refer to the corresponding parts of the data processing method provided in the above-mentioned embodiment of the present application, and will not be repeated here.

[0093] An embodiment of the present application further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned data processing method.

[0094] The present application also provides an electronic device, whose structural diagram is as follows: Figure 5 As shown, it specifically includes a memory 501 and one or more instructions 502, wherein the one or more instructions 502 are stored in the memory 501 and are configured to be executed by one or more processors 503 to perform the following operations:

[0095] In response to the data processing instruction, metadata collection is performed during the data flow process to obtain various metadata; each of the metadata includes data source information, data conversion configuration information, data storage information and data access records of the data entity to which it belongs;

[0096] The metadata are stored in a preset graph database, and the metadata stored in the graph database are analyzed using a preset lineage analysis technology to obtain the lineage relationship between the data entities to which the metadata belong;

[0097] The blood relationship between each of the data entities is visualized.

[0098] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0099] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0100] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0101] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0102] The above is a detailed introduction to a data processing method provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method, characterized in that: include: In response to the data processing instruction, metadata collection is performed during the data flow process to obtain various metadata; Each of the metadata includes data source information, data conversion configuration information, data storage information and data access records of the data entity to which it belongs; The metadata are stored in a preset graph database, and the metadata stored in the graph database are analyzed using a preset lineage analysis technology to obtain the lineage relationship between the data entities to which the metadata belong; The blood relationship between each of the data entities is visualized.

2. The method according to claim 1, characterized in that After visually displaying the blood relationship between the data entities, the method further includes: When an operation on the data entity is detected, metadata of the data entity stored in the graph database is updated according to the operation to update the data entity blood relationship.

3. The method according to claim 1, characterized in that After visually displaying the blood relationship between the data entities, the method further includes: Determine the starting node selected by the user; Querying a preset graph database according to the starting node to determine a data link of the starting node; the data link includes at least one of an upstream link and a downstream link of the starting node; The data link is displayed.

4. The method according to claim 1, characterized in that: The metadata collection is performed during the data flow process to obtain various metadata, including: Parsing the data processing instruction to obtain collection requirement information; Metadata collection is performed during data flow according to the collection requirement information to obtain metadata corresponding to the collection requirement information.

5. The method according to claim 1, characterized in that The method of analyzing the metadata stored in the graph database by using a preset lineage analysis technology to obtain the lineage relationship between the data entities to which each metadata belongs includes: Analyze the metadata stored in the graph database using a preset lineage analysis technology to obtain the dependency relationship between the data entities to which each metadata belongs; Each of the data entities is taken as a node, and the dependency relationship between the data entities to which each metadata belongs is taken as an edge, so as to construct a blood relationship graph of each of the data entities and obtain the blood relationship between the data entities to which each metadata belongs.

6. A data processing device, characterized in that: include: An execution unit, for responding to a data processing instruction, collecting metadata during data flow, and obtaining various metadata; Each of the metadata includes data source information, data conversion configuration information, data storage information and data access records of the data entity to which it belongs; An analysis unit, used to store each of the metadata in a preset graph database, and analyze the metadata stored in the graph database using a preset lineage analysis technology to obtain the lineage relationship between the data entities to which each of the metadata belongs; The display unit is used to visually display the blood relationship between each of the data entities.

7. The device according to claim 6, characterized in that Also includes: A detection unit is used to update the metadata of the data entity stored in the graph database according to the operation when an operation on the data entity is detected, so as to update the blood relationship of the data entity.

8. The device according to claim 6, characterized in that Also includes: A determination unit, used to determine a starting node selected by a user; A query unit, configured to query a preset graph database according to the starting node to determine a data link of the starting node; the data link includes at least one of an upstream link and a downstream link of the starting node; A display unit is used to display the data link.

9. The device according to claim 6, characterized in that The execution unit comprises: A parsing subunit, used to parse the data processing instruction to obtain collection requirement information; The collection subunit is used to collect metadata during data flow according to the collection requirement information, and obtain metadata corresponding to the collection requirement information.

10. The device according to claim 6, characterized in that The analysis unit comprises: An analysis subunit, used to analyze the metadata stored in the graph database using a preset lineage analysis technology to obtain dependency relationships between data entities to which each metadata belongs; The execution subunit is used to take each of the data entities as a node and the dependency relationship between the data entities to which each metadata belongs as an edge, so as to construct a blood relationship map of each of the data entities and obtain the blood relationship between the data entities to which each metadata belongs.

Images