A data processing method, device and storage medium

Abstract

The application discloses a data processing method and device and a storage medium. The method comprises the following steps: collecting target data from a target database; in the case that the target data comprises a structured query statement, obtaining a target cluster matched with the version of the target database from at least one parsing client cluster registered on a registration center, and selecting a target parsing client available from the target cluster; the parsing clients of different clusters in the at least one parsing client cluster have the function of parsing various database languages of different versions; the target parsing client is used to parse the structured query statement to obtain target blood relationship information; unknown parts in the target blood relationship information are supplemented to obtain complete blood relationship information, and the complete blood relationship information is stored in a database. Through the above scheme, the compatibility of the structured query statement parsing is improved, so that the blood relationship information of data of different sources can be obtained.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and in particular to a data processing method, apparatus and storage medium. Background Technology

[0002] The era of big data has brought about massive data volumes and enormous data value, and data has been regarded as an important strategic asset by enterprises. Data lineage analysis is particularly important in scenarios such as identifying and analyzing data anomalies, determining the correctness of dependencies between data tasks, and locating task bottlenecks to improve scheduling performance.

[0003] Currently, the structured query parsing tool built into the analysis system is usually used to parse the data into structured query statements to obtain lineage information. However, this structured query parsing tool can only parse data from a certain type of database, has poor compatibility, and is difficult to obtain lineage information from data from different sources. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention aims to provide a data processing method, apparatus, and storage medium. This invention can acquire corresponding parsing clients for structured statement parsing of data from various databases of different versions, improving the compatibility of structured query statement parsing and thereby enabling the acquisition of lineage information for data from different sources.

[0005] The technical solution of this invention is implemented as follows:

[0006] This invention provides a data processing method, the method comprising:

[0007] Collect target data from the target database;

[0008] When the target data includes structured query statements, a target cluster matching the version of the target database is obtained from at least one parsing client cluster registered on the registry center, and an available target parsing client is selected from the target cluster; in the at least one parsing client cluster, the parsing clients of different clusters have the function of parsing various database languages ​​of different versions;

[0009] Using the target parsing client, the structured query statement is parsed to obtain the target lineage information;

[0010] The unknown parts of the target bloodline information are supplemented to obtain complete bloodline information, and the complete bloodline information is stored in the database.

[0011] In the above method, before obtaining the target cluster matching the version of the target database from at least one parsing client cluster registered on the registry center when the target data includes a structured query statement, the method further includes:

[0012] Register the identification information of the target cluster on the registration center.

[0013] In the above method, obtaining the target cluster whose version matches the target database from at least one parsing client cluster registered on the registry center includes:

[0014] Obtain the identification information corresponding to the version of the target database from the registration center;

[0015] The cluster in the at least one parsing client cluster that matches the identification information is identified as the target cluster.

[0016] In the above method, the step of using the target parsing client to parse the structured query statement to obtain the target lineage information includes:

[0017] In the target parsing client, a lexical file and a syntax file that correspond to the target database and conform to a preset format are set;

[0018] The lexical file and the syntax file are used to generate a parsing file, and the parsing file is used to parse the structured query statements in the target data to generate an abstract syntax tree;

[0019] The abstract syntax tree is traversed and merged using a depth-first search algorithm to obtain the merged traversal result, and the target lineage information is generated based on the merged traversal result.

[0020] In the above method, supplementing the unknown portion of the target bloodline information to obtain complete bloodline information includes:

[0021] Query the unknown portion from the data management database;

[0022] The information obtained from the query is added to the target bloodline information to obtain the complete bloodline information.

[0023] This invention provides a data processing apparatus, comprising:

[0024] The data acquisition module is used to collect target data from the target database;

[0025] The information determination module is used to, when the target data includes a structured query statement, obtain a target cluster matching the version of the target database from at least one parsing client cluster registered on the registry center, and select an available target parsing client from the target cluster; in the at least one parsing client cluster, the parsing clients of different clusters have the function of parsing various database languages ​​of different versions; using the target parsing client, the structured query statement is parsed to obtain the target lineage information;

[0026] The bloodline service module is used to supplement the unknown parts of the target bloodline information to obtain complete bloodline information, and store the complete bloodline information in the database.

[0027] In the above-described device, the information determination module is further configured to register the identification information of the target cluster on the registration center.

[0028] In the above-described device, the information determination module is specifically used to obtain identification information corresponding to the version of the target database from the registration center;

[0029] The cluster in the at least one parsing client cluster that matches the identification information is identified as the target cluster.

[0030] In the above-mentioned device, the information determination module is specifically used to set a lexical file and a syntax file that correspond to the target database and conform to a preset format in the target parsing client;

[0031] The lexical file and the syntax file are used to generate a parsing file, and the parsing file is used to parse the structured query statements in the target data to generate an abstract syntax tree;

[0032] The abstract syntax tree is traversed and merged using a depth-first search algorithm to obtain the merged traversal result, and the target lineage information is generated based on the merged traversal result.

[0033] In the aforementioned device, the bloodline service module is specifically used to query the unknown portion from the data management database; and to supplement the target bloodline information with the queried information to obtain the complete bloodline information.

[0034] This invention provides a data processing apparatus, comprising: a processor, a memory, and a communication bus;

[0035] The communication bus is used to realize the communication connection between the processor and the memory;

[0036] The processor is used to execute the data processing program stored in the memory to implement the above-described data processing method.

[0037] The present invention provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the above-described data processing method.

[0038] This invention provides a data processing method, apparatus, and storage medium. The method includes: collecting target data from a target database; when the target data includes structured query statements, obtaining a target cluster matching the version of the target database from at least one parsing client cluster registered on a registry center, and selecting an available target parsing client from the target cluster; in the at least one parsing client cluster, parsing clients in different clusters have the function of parsing various database languages ​​of different versions; using the target parsing client to parse the structured query statements to obtain target lineage information; supplementing the unknown parts of the target lineage information to obtain complete lineage information, and storing the complete lineage information in the database. The technical solution provided by this invention can obtain corresponding parsing clients for structured statement parsing for data from various databases of different versions, improving the compatibility of structured query statement parsing, thereby enabling the acquisition of lineage information of data from different sources. Attached Figure Description

[0039] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of the present invention;

[0040] Figure 2 A schematic diagram illustrating communication between an exemplary data integration module and a parsing client cluster, provided as an embodiment of the present invention;

[0041] Figure 3 This is an exemplary flowchart illustrating the process of determining bloodline information, provided as an embodiment of the present invention.

[0042] Figure 4 This is a schematic diagram of an exemplary data processing flow provided in an embodiment of the present invention;

[0043] Figure 5 A schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. Figure 1 ;

[0044] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. Figure 2 . Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0046] This invention provides a data processing method. Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention. Figure 1 As shown, the main steps include:

[0047] S101. Collect target data from the target database.

[0048] In embodiments of the present invention, the data processing device can directly collect target data from the target database.

[0049] It should be noted that, in the embodiments of the present invention, the target database can be a conventional database, such as a data warehouse tool (Hive), a relational database (PostgreSQL, Oracle SQL, etc.), or other types of databases, such as a distributed file system (HDFS) file or a file transfer protocol (FTP) file, which are databases for storing data. The specific target database can be determined according to actual needs and application scenarios.

[0050] It should be noted that, in the embodiments of the present invention, the target data is any data stored in the target database that can determine bloodline information, that is, the data processing device can process the data bloodline relationship of the target data in the target database to obtain bloodline information.

[0051] S102. When the target data includes structured query statements, obtain the target cluster that matches the version of the target database from at least one parsing client cluster registered on the registry center, and select an available target parsing client from the target cluster; in at least one parsing client cluster, the parsing clients of different clusters have the function of parsing various database languages ​​of different versions.

[0052] In an embodiment of the present invention, when the target data includes a structured query statement, the data processing device obtains a target cluster that matches the version of the target database from at least one parsing client cluster registered on the registry center, and selects an available target parsing client from the target cluster; in the at least one parsing client cluster, the parsing clients of different clusters have the function of parsing various database languages ​​of different versions.

[0053] It should be noted that, in the embodiments of the present invention, each parsing client cluster is registered in the registry center and is a cluster that parses structured query statements for different versions and types of databases.

[0054] It should be noted that, in the embodiments of the present invention, before the data processing device obtains the target cluster matching the version of the target database from at least one parsing client cluster registered on the registration center, it can first preprocess the collected target data. Preprocessing refers to cleaning and transforming the target data. Data cleaning is to filter out data that does not meet the requirements, and data transformation mainly involves converting inconsistent data, converting data granularity, and calculating some business rules. The specific preprocessing method can be set according to actual needs and application scenarios.

[0055] It should be noted that, in the embodiments of the present invention, after the data processing device preprocesses the target data collected from the target database, it determines whether the collected target data contains a structured query statement. If the target data does not contain a structured query statement, the data processing device directly uses the preprocessed target data to determine the data lineage relationship and obtain the lineage information.

[0056] Specifically, in an embodiment of the present invention, before the data processing device obtains a target cluster matching the version of the target database from at least one parsing client cluster registered on the registry center when the target data includes a structured query statement, the data processing device further includes the following step: registering the identification information of the target cluster on the registry center.

[0057] It should be noted that, in the embodiments of the present invention, the identification information may be the Internet protocol address and port number of the client cluster, or it may be a universally unique identifier of the client cluster, or an identity identifier. The specific identification information may be determined according to the actual scenario and requirements, and the present invention does not limit it.

[0058] It should be noted that, in the embodiments of the present invention, when the data processing device deploys each parsing client cluster, it registers its respective identification information in the registration center in a cluster manner, and stores the identification information corresponding to each parsing client cluster in the registration center, so as to facilitate the registration center to manage multiple parsing client clusters.

[0059] Specifically, in an embodiment of the present invention, the data processing device obtains a target cluster that matches the version of the target database from at least one parsing client cluster registered on the registration center, including: obtaining identification information corresponding to the version of the target database from the registration center; and determining the cluster in the at least one parsing client cluster that matches the identification information as the target cluster.

[0060] It should be noted that, in the embodiments of the present invention, when the target data includes a structured query statement, the data processing device will first obtain the identification information of the target cluster from the registry center according to the version of the database, and then determine the parsing client cluster that matches the identification information as the target cluster.

[0061] It should be noted that, in the embodiments of the present invention, the parsing client cluster of the data processing device includes multiple parsing clients. Each parsing client is deployed on a server and is referred to as an instance. When an instance starts, each parsing client registers with the registration center using its unique identifier name. The registration center assigns a client address and a client port number to each parsing client and stores the mapping relationship between the unique identifier name, client address, and client port number of each parsing client and the identifier information of the parsing client cluster it belongs to in the registration center. This allows the registration center to manage the client addresses and client port numbers of each instance. In other words, in the mapping relationship of the registration center, the identifier information of each parsing client cluster corresponds to the client address and client port number of the parsing clients included in the cluster. Furthermore, the parsing clients use the HAProxy+Keepalived method to achieve dual-machine hot standby high availability to ensure the normal operation of the target cluster. The HAProxy+Keepalived method is a way in which other parsing clients continue to perform their work when one parsing client fails.

[0062] It should be noted that, in the embodiments of the present invention, the client address is the Internet Protocol address of the client.

[0063] It should be noted that, in the embodiments of the present invention, the registration center will periodically perform health checks and heartbeat detection on each instance. When the target parsing client is unavailable, the registration center will mark it as failed. Then, when the data processing device selects an available target parsing client from the target cluster, the target parsing client marked as failed will not be selected, thereby ensuring that the parsing of structured query statements can be achieved.

[0064] Figure 2 This is a schematic diagram illustrating communication between an exemplary data integration module and a parsing client cluster, provided as an embodiment of the present invention. Figure 2 As shown, the data processing device includes a data integration module. When the data processing device deploys a parsing client cluster, each parsing client cluster registers its own identification information in the registration center. When data parsing is required, the data integration module establishes a connection with the corresponding parsing client cluster based on the identification information of the parsing client cluster to achieve communication.

[0065] It should be noted that, in the embodiments of the present invention, when the data processing device establishes communication with the target cluster, it also determines the target parsing clients available in the target cluster. The specific determination steps are as follows: first, from at least one parsing client cluster registered on the registration center, obtain the target cluster that matches the version of the target database, obtain the identification information corresponding to the target cluster, and then determine the available target parsing clients from the client address and client port number of the parsing client corresponding to the identification information.

[0066] S103. Using the target parsing client, parse the structured query statement to obtain the target lineage information.

[0067] In an embodiment of the present invention, the data processing device uses a target parsing client to parse a structured query statement to obtain target lineage information.

[0068] It should be noted that, in the embodiments of the present invention, after determining the available target parsing client, the data processing device will automatically call the corresponding target parsing client to parse the structured query statement and obtain the target lineage information.

[0069] Figure 3 This is a schematic diagram illustrating an exemplary process for determining bloodline information, provided as an embodiment of the present invention. Figure 3 As shown, the specific steps include:

[0070] S301. In the target parsing client, set the lexical file and syntax file that correspond to the target database and conform to the preset format.

[0071] In an embodiment of the present invention, the data processing device sets up a lexical file and a syntax file in the target parsing client that correspond to the target database and conform to a preset format.

[0072] It should be noted that, in the embodiments of the present invention, the data processing device sets a lexical file and a syntax file in a preset format corresponding to the target database in the target parsing client. The preset format can be antlr4 format or other preset formats. The specific preset format can be preset according to actual needs and application scenarios.

[0073] S302. Use the lexical file and syntax file to generate a parsing file, and use the parsing file to parse the structured query statements in the target data to generate an abstract syntax tree.

[0074] In embodiments of the present invention, the data processing device uses a lexical file and a syntax file to generate a parsing file, and uses the parsing file to parse the structured query statements in the target data to generate an abstract syntax tree.

[0075] It should be noted that, in the embodiments of the present invention, the data processing device uses a lexical file and a syntax file in a preset format to generate a parsing file. The parsing file can be a Java parsing file or any other parsing file that can be parsed. The specific parsing file can be set according to actual needs and application scenarios.

[0076] S303. Use the depth-first search algorithm to traverse and merge the abstract syntax tree to obtain the merge traversal result, and generate the target lineage information based on the merge traversal result.

[0077] In an embodiment of the present invention, the data processing device uses a depth-first algorithm to traverse and merge the abstract syntax tree to obtain the merge traversal result, and generates the target lineage information based on the merge traversal result.

[0078] It should be noted that, in the embodiments of the present invention, the data processing device uses a depth-first search algorithm to perform a depth-first traversal of the abstract syntax tree. Specifically, when performing a depth-first traversal of the abstract syntax tree, if TOK_TAB or TOK_TABREF is encountered, the table being operated on is determined, and the information corresponding to the table being operated on is stored in the lineage information. When a select*from clause is encountered, it enters the stack for data processing. After processing the select*from clause, it jumps out of the stack and obtains the operation type of the lineage information. When it is pushed onto the stack, it needs to determine whether it is a join. If it is a join, the join condition is determined. When encountering WHERE, SELECT, or JOIN statements, the corresponding fields and conditions need to be obtained and encapsulated into the corresponding tables. When encountering TOK_SUBQUERY, the current subquery information is saved for use by the parent query. When the processed table has aliases, a mapping TABLE_ALIAS_MAP is established between the table alias and the actual table name, and a mapping COLUMN_ALIAS_MAP is established between the column alias and the actual column name, where the key is the actual name and the value is the alias. When the TOK_UNION statement ends, the current column information is merged and truncated, and then the table and field mapping relationship is obtained through merge traversal to generate lineage information.

[0079] It should be noted that, in the embodiments of the present invention, after the data processing device generates kinship information, it can encapsulate the kinship information into an interface, and then the data integration module can directly call the interface to obtain the kinship information.

[0080] For example, the data processing device can call the corresponding structured query parser (SQLParser agent) interface according to the structured query statement and the data type of the database to parse different database languages, thereby obtaining lineage information, and then encapsulate the obtained lineage information into a JSON string and temporarily store it on the Kafka cluster.

[0081] It should be noted that, in the embodiments of the present invention, after the data processing device uses the target parsing client to parse the structured query statement and obtain the target lineage information, considering the scenarios such as downtime, upgrades and service interruptions that may occur during the data processing process, the data processing device encapsulates the target lineage data into a JSON (JavaScript Object Notation) string, that is, encapsulates a lineage data into a JSON string and temporarily stores it on the Kafka cluster to prevent data loss.

[0082] It should be noted that, in the embodiments of the present invention, Kafka is a distributed system that runs in a cluster. The Kafka cluster includes one or more servers for temporarily storing JSON strings encapsulated with lineage data.

[0083] It should be noted that, in the embodiments of the present invention, the data processing device consumes the lineage information on the Kafka cluster, obtains a JSON string, and performs data parsing on the JSON string to obtain the JSON content, i.e., the target lineage information.

[0084] S104. Supplement the unknown parts of the target bloodline information to obtain complete bloodline information, and store the complete bloodline information in the database.

[0085] In an embodiment of the present invention, the data processing device supplements the unknown parts of the target bloodline information to obtain complete bloodline information, and stores the complete bloodline information in the database.

[0086] It should be noted that, in the embodiments of the present invention, after the data processing device obtains the target lineage information, since the SQL statement contains ambiguous fields such as select*, the unknown fields need to be filled in to obtain complete lineage information.

[0087] Specifically, in an embodiment of the present invention, the data processing device supplements the unknown part of the target kinship information to obtain complete kinship information, including: querying the unknown part from the data management database; and supplementing the queried information into the target kinship information to obtain complete kinship information.

[0088] It should be noted that, in the embodiments of the present invention, the data management module includes a data management library. The data management library is a data management library for storing metadata information set up by technicians according to specific application scenarios and actual needs. When an unknown part appears in the target lineage information, the corresponding table field information can be obtained by querying the metadata information in the data management library. The obtained table field information is then added to the target lineage information to obtain complete single lineage flow information.

[0089] It should be noted that, in the embodiments of the present invention, the target lineage information obtained by the data processing device is a collection of one or more table-to-table level information and field-to-field flow relationships within the tables. When the SQL statement in a single lineage flow information contains ambiguous fields such as `select *`, it is supplemented to obtain complete single lineage flow information. It should also be noted that, in the embodiments of the present invention, after obtaining complete single lineage information, the data processing device sends the complete single lineage information to a database. This database can be a Neo4j graph database. Specifically, a Neo4j graph database contains rich functions and procedures, fully supports complete transaction management features, provides a distributed high-availability mode, and can supplement complex graph algorithms and data operation functions, thereby improving storage and query capabilities and efficiency.

[0090] For example, in the Neo4j graph database, if you need to initialize and load a large amount of data, you can use the neo4j-import function; if you need to ensure database service while synchronously loading incremental data, you can use the CSV loading function (load comma separated values, load csv); if you can accept short pauses in the Neo4j database service, you can use the batch insert function; if you only want to insert a small amount of data and real-time requirements are not high, you can directly use the Cypher language; if you want to meet the conditions of short processing time, no conversion of CSV files, high efficiency, and no interruption of database service, you can choose a Neo4j apoc. A specific Neo4j apoc is a collection of functions and procedures that can be used in Cypher to dynamically create labels and relations. Using labels and indexes can improve the storage efficiency and query performance of the database.

[0091] It should be noted that, in the embodiments of the present invention, the complete lineage information stored in the neo4j graph database by the data processing device is a single lineage flow information. When it is necessary to query the lineage flow information of a certain table, the neo4j graph database will summarize all the lineage information related to the table in the single lineage flow information into a set of lineage information related to the table.

[0092] This invention discloses a data processing method, comprising: collecting target data from a target database; when the target data includes structured query statements, obtaining a target cluster matching the version of the target database from at least one parsing client cluster registered on a registry center, and selecting an available target parsing client from the target cluster; each parsing client in the at least one parsing client cluster having the function of parsing different types of database languages; using the target parsing client to parse the structured query statements to obtain target lineage information; supplementing the unknown parts in the target lineage information to obtain complete lineage information, and storing the complete lineage information in a database. The data processing method provided by this invention can obtain corresponding parsing clients for structured query parsing of data from various databases of different versions, improving the compatibility of structured query parsing and thus enabling the acquisition of lineage information from data from different sources.

[0093] Figure 4 This is a schematic diagram illustrating an exemplary data processing flow according to an embodiment of the present invention. Figure 4 As shown, the data processing device includes: a data acquisition module, an information determination module, a lineage service module, a Kafka cluster, a data management module, and a database. The information determination module includes a data integration module and a parsing client cluster. The specific steps of data processing are as follows: After the data acquisition module obtains the target data from the target database, the data integration module performs a series of preprocessing steps on the target data. Users can select relevant components, database parameters, table transition fields, or write structured query language (SQL) scripts. For target data including structured query statements, the parsing client interface is called to obtain lineage information. The data integration module encapsulates the obtained lineage information into a data string and sends it to the Kafka cluster. The lineage service module consumes the string from the Kafka cluster, obtains JSON content, and parses it. If the field relationship in the JSON content is *, the data management module is queried to obtain the metadata information of the relevant table fields to obtain complete lineage information. The complete lineage information is then stored in the database for users to query.

[0094] This invention provides a data processing device. Figure 5 A schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. Figure 1 .like Figure 5 As shown, the data processing device includes:

[0095] Data acquisition module 501 is used to acquire target data from the target database;

[0096] The information determination module 502 is used to, when the target data includes a structured query statement, obtain a target cluster matching the version of the target database from at least one parsing client cluster registered on the registry center, and select an available target parsing client from the target cluster; each parsing client in the at least one parsing client cluster has the function of parsing different types of database languages; using the target parsing client, the structured query statement is parsed to obtain the target lineage information;

[0097] The bloodline service module 503 is used to supplement the unknown parts of the target bloodline information to obtain complete bloodline information, and store the complete bloodline information in the database.

[0098] Optionally, the information determination module 502 is further configured to register the identification information of the target cluster on the registration center.

[0099] Optionally, the information determination module 502 is specifically used to obtain identification information corresponding to the version of the target database from the registration center; and to determine the cluster in the at least one parsing client cluster that matches the identification information as the target cluster.

[0100] Optionally, the information determination module 502 is specifically used in the target parsing client to set a lexical file and a syntax file that correspond to the target database and conform to a preset format; to generate a parsing file using the lexical file and the syntax file, and to parse the structured query statements in the target data using the parsing file to generate an abstract syntax tree; to traverse and merge the abstract syntax tree using a depth-first search algorithm to obtain a merge traversal result, and to generate the target lineage information based on the merge traversal result.

[0101] Optionally, the bloodline service module 503 is specifically used to query the unknown part from the data management database; and to supplement the target bloodline information with the queried information to obtain the target bloodline data.

[0102] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. Figure 2 .like Figure 6 As shown, the data processing device includes: a processor 601, a memory 602, and a communication bus 603;

[0103] The communication bus 603 is used to realize the communication connection between the processor 601 and the memory 602;

[0104] The processor 601 is used to execute the data processing program stored in the memory 602 to implement the above-mentioned data processing method.

[0105] This invention provides a data processing apparatus that collects target data from a target database; when the target data includes structured query statements, it obtains a target cluster matching the version of the target database from at least one parsing client cluster registered on a registry center, and selects an available target parsing client from the target cluster; each parsing client in the at least one parsing client cluster has the function of parsing different types of database languages; using the target parsing client, it parses the structured query statements to obtain target lineage information; it supplements the unknown parts of the target lineage information to obtain complete lineage information, and stores the complete lineage information in the database. The apparatus provided by this invention can obtain corresponding parsing clients for structured statement parsing for data from various databases of different versions, improving the compatibility of structured query statement parsing, thereby enabling the acquisition of lineage information from data from different sources.

[0106] The present invention provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the above-described data processing method.

[0107] This invention provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the data processing method described above. Those skilled in the art will understand that embodiments of this invention can be provided as methods, systems, or computer program products. Therefore, this invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this invention can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage and optical storage) containing computer-readable program code.

[0108] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, The method includes: Collect target data from the target database; When the target data includes structured query statements, a target cluster matching the version of the target database is obtained from at least one parsing client cluster registered on the registry center, and an available target parsing client is selected from the target cluster; in the at least one parsing client cluster, the parsing clients of different clusters have the function of parsing various database languages ​​of different versions; Using the target parsing client, the structured query statement is parsed to obtain the target lineage information; Supplement the unknown parts of the target bloodline information to obtain complete bloodline information, and store the complete bloodline information in the database; The step of using the target parsing client to parse the structured query statement to obtain the target lineage information includes: In the target parsing client, a lexical file and a syntax file that correspond to the target database and conform to a preset format are set; The lexical file and the syntax file are used to generate a parsing file, and the parsing file is used to parse the structured query statements in the target data to generate an abstract syntax tree; The abstract syntax tree is traversed and merged using a depth-first search algorithm to obtain the merged traversal result, and the target lineage information is generated based on the merged traversal result.

2. The method according to claim 1, characterized in that, Before obtaining a target cluster matching the version of the target database from at least one parsing client cluster registered on the registry center when the target data includes a structured query statement, the method further includes: Register the identification information of the target cluster on the registration center.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining a target cluster matching the version of the target database from at least one parsing client cluster registered on the registry center includes: Obtain the identification information corresponding to the version of the target database from the registration center; The cluster in the at least one parsing client cluster that matches the identification information is identified as the target cluster.

4. The method according to claim 1, characterized in that, The process of supplementing the unknown parts of the target bloodline information to obtain complete bloodline information includes: Query the unknown portion from the data management database; The information obtained from the query is added to the target bloodline information to obtain the complete bloodline information.

5. A data processing apparatus, characterized in that, include: The data acquisition module is used to collect target data from the target database; The information determination module is used to, when the target data includes a structured query statement, obtain a target cluster that matches the version of the target database from at least one parsing client cluster registered on the registry center, and select an available target parsing client from the target cluster; In the at least one parsing client cluster, the parsing clients in different clusters have the function of parsing various database languages ​​of different versions; using the target parsing client, the structured query statement is parsed to obtain the target lineage information; The bloodline service module is used to supplement the unknown parts of the target bloodline information to obtain complete bloodline information, and store the complete bloodline information in the database; The information determination module is also used to set a lexical file and a syntax file that correspond to the target database and conform to a preset format in the target parsing client; The lexical file and the syntax file are used to generate a parsing file, and the parsing file is used to parse the structured query statements in the target data to generate an abstract syntax tree; The abstract syntax tree is traversed and merged using a depth-first search algorithm to obtain the merged traversal result, and the target lineage information is generated based on the merged traversal result.

6. The apparatus according to claim 5, characterized in that, The information determination module is specifically used to obtain identification information corresponding to the version of the target database from the registration center; and to determine the cluster in the at least one parsing client cluster that matches the identification information as the target cluster.

7. A data processing apparatus, characterized in that, include: Processor, memory, and communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute a data processing program stored in the memory to implement the data processing method according to any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the data processing method according to any one of claims 1-4.

Images