Data set blood relationship determination method and device based on computing power of intelligent computing center

Abstract

The invention provides a data set blood relationship determination method and device based on the computing power of an intelligent computing center. The method comprises the steps of obtaining a to-be-processed SQL statement in an SQL workflow; analyzing the data to obtain an analysis result; determining a data flow direction between first database tables indicated by the to-be-processed SQL statement according to an analysis result, and determining a blood relationship between the first database tables; and respectively replacing the first database table indicated by the to-be-processed SQL statement with a self-associated target data set according to a preset association relationship between all the database tables and the data set in the blood relationship to obtain the blood relationship of the target data set. Therefore, the integrity of blood relationship display can be guaranteed, it is ensured that when an intelligent computing center processes large-scale data, the data flow direction is accurately grasped through the accurate blood relationship, computing power resources are dispatched more effectively, waste of the computing power resources is avoided, the computing power of the intelligent computing center is improved, and the computing efficiency of the intelligent computing center is improved. And the efficiency and reliability of links such as data processing, model training and reasoning are improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of computing power infrastructure, and in particular, to a method and device for determining the blood relationship of a data set based on the computing power of an intelligent computing center. Background Art

[0002] With the development of artificial intelligence technology and computing power technology, the concept of intelligent computing center has emerged. "Intelligent computing center" refers to the use of large-scale heterogeneous computing power resources, including general computing power and intelligent computing power, mainly for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model reasoning and other scenarios) to provide the required computing power, data and algorithms. Intelligent computing center covers facilities, hardware, software, and can provide full-stack capabilities from bottom-level computing power to top-level application enablement.

[0003] In the database, a view table is a virtual table that is generated based on the query results of one or more base tables or other view tables. The view table does not actually store data, but dynamically generates results when it is queried based on the defined query statement. This dynamism enables the view table to provide different logical views of the underlying data, simplifying data access and operation. However, due to its characteristics, the view table also has certain disadvantages. For example: in SQL real-time jobs, since the view table does not store actual data, but obtains data from its base table or other view tables in real time, in some cases, the data set information of the view table may not be fully collected, resulting in the lack of corresponding data set information about the lineage relationship of the view table in the job lineage display (displaying the relationship between the source and destination of the data in the job, through the lineage display, the data transmission and transformation process in the entire job process can be tracked).

[0004] The above situation makes it impossible for users to directly obtain the data set information of the view table from the blood relationship, which will lead to incomplete blood relationship display, difficulty in data traceability, and incomplete data impact analysis. The existence of this problem limits the computing power of the intelligent computing center. In the application of the intelligent computing center, since the view table does not store actual data, its information often relies on real-time acquisition from the basic table. This dynamic mechanism leads to unclear display of blood relationship, which in turn affects the comprehensiveness of data analysis. Specifically, when the data set information of the view table is missing in real-time operation, it is difficult for users to accurately track the source and destination of the data, which not only leads to incomplete blood relationship, but also limits the computing power effect of the intelligent computing center, directly affecting the efficiency and reliability of the intelligent computing center in important links such as data processing, model training and reasoning. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for determining the blood relationship of a data set based on the computing power of an intelligent computing center, so as to solve the technical problem in the related technology that a user cannot directly obtain data set information about a view table from the blood relationship, which will lead to incomplete blood relationship display, limit the computing power of the intelligent computing center, and further affect the efficiency and reliability of the intelligent computing center in important links such as data processing, model training and reasoning.

[0006] In a first aspect, an embodiment of the present invention provides a method for determining a blood relationship of a data set based on the computing power of an intelligent computing center, the method comprising:

[0007] Step S1: in the SQL job flow, obtaining the SQL statement to be processed;

[0008] Step S2: parsing the SQL statement to be processed to obtain a parsing result;

[0009] Step S3: determining the data flow direction between the first database tables indicated by the SQL statement to be processed according to the analysis result;

[0010] Step S4: determining the blood relationship between the first database tables based on the data flow direction;

[0011] Step S5: In the blood relationship, according to the preset association relationship between all database tables and data sets, the first database table indicated by the SQL statement to be processed is replaced with the target data set associated with it, so as to obtain the blood relationship of the target data set.

[0012] Optionally, the parsing result is: a reference field in the SQL statement to be processed and a first database table indicated by the SQL statement to be processed; wherein the reference field includes at least one of the following: a query field and a condition field;

[0013] The step S3 comprises:

[0014] Step S31: determining the data flow direction between the first database table indicated by the SQL statement to be processed according to the reference field and the first database table indicated by the SQL statement to be processed.

[0015] Optionally, after step S4, the method further includes:

[0016] Step S6: storing the blood relationship in the first database table into a blood relationship pool;

[0017] Step S7: receiving a query instruction from a user; wherein the query instruction is used to query all blood relationships of the target data set;

[0018] Step S8: determining at least one second database table associated with the target data set according to the query instruction;

[0019] Step S9: searching for the blood relationship of each second database table in the blood relationship pool according to the unique identifier of the at least one second database table associated with the target data set;

[0020] Step S10: respectively searching for the other database tables associated with the blood relationship of each second database table; wherein the other database tables are database tables other than the second database table involved in the blood relationship of each second database table;

[0021] Step S11: searching for data sets associated with each database table in the remaining database tables;

[0022] Step S12: According to the search result, the strategy corresponding to the search result is executed on the remaining database tables, and all blood relationships of the target data set are determined according to the strategy.

[0023] Optionally, the step S12 includes:

[0024] Step S121: Replacement step: replace the remaining database tables for which the associated data set is found in the search result with the associated data set in the blood relationship of the corresponding second database table;

[0025] Step S122: retaining step: retaining the remaining database tables for which the search result shows that no associated data set is found in the blood relationship of the corresponding second database table;

[0026] Step S123: performing the replacement step of step S121 and the retention step of step S122 on each blood relationship in the second database table to obtain the processed blood relationship of each second database table;

[0027] Step S124: Determine the processed blood relationship of each of the second database tables as all blood relationships of the target data set.

[0028] Optionally, after step S5, the method further includes:

[0029] Step S13: determining a target node in the blood relationship;

[0030] Step S14: performing data monitoring on the target node, including: determining whether data anomalies occur in the target node according to the data operation log corresponding to the target node and the execution status of the stream job associated with the target node;

[0031] Step S15: If yes, determining the source and trend of the abnormal data according to the blood relationship;

[0032] Step S16: Generate an abnormality analysis report based on the source and trend of the abnormal data, wherein the abnormality analysis report is used to indicate the impact of the abnormal data on the remaining nodes in the blood relationship except the target node.

[0033] Optionally, after step S5, the method further includes:

[0034] Step S17: Display the blood relationship of the target data set, including:

[0035] Step S18: receiving a selection instruction from a user, wherein the selection instruction is used to indicate a display mode, and the display mode includes at least one of the following: display by job granularity, display by data set granularity;

[0036] Step S19: displaying the blood relationship of the target data set according to the display method.

[0037] Optionally, before step S1, the method further includes:

[0038] Step S01: Create a data source;

[0039] Step S02: creating all database tables, and setting associations between all created database tables and data sets in the data source; generating the SQL job flow based on all created database tables.

[0040] In a second aspect, an embodiment of the present invention provides a device for determining a blood relationship of a data set based on the computing power of an intelligent computing center, the device comprising:

[0041] An acquisition module is used to acquire SQL statements to be processed in the SQL job flow;

[0042] An execution module is used to parse the SQL statement to be processed and obtain a parsing result;

[0043] Determine the data flow between the first database tables indicated by the SQL statement to be processed according to the analysis result;

[0044] Based on the data flow direction, determining the blood relationship between the first database tables;

[0045] In the blood relationship, according to the preset association relationship between all database tables and data sets, the first database table indicated by the SQL statement to be processed is replaced with the target data set associated with it, so as to obtain the blood relationship of the target data set.

[0046] Optionally, the parsing result is: a reference field in the SQL statement to be processed and a first database table indicated by the SQL statement to be processed; wherein the reference field includes at least one of the following: a query field and a condition field;

[0047] The execution module is further used to determine the data flow direction between the first database table indicated by the SQL statement to be processed according to the reference field and the first database table indicated by the SQL statement to be processed.

[0048] Optionally, the execution module is further configured to store the blood relationship of the first database tables in a blood relationship pool after determining the blood relationship between the first database tables;

[0049] Receive a query instruction from a user; wherein the query instruction is used to query all blood relationships of the target data set;

[0050] Determining, according to the query instruction, at least one second database table associated with the target data set;

[0051] According to the unique identifier of the at least one second database table associated with the target data set, searching for the blood relationship of each second database table in the blood relationship pool;

[0052] Respectively search for the remaining database tables associated with the blood relationship of each second database table,

[0053] Find the data sets associated with each database table in the remaining database tables;

[0054] According to the search result, the strategy corresponding to the search result is executed on the remaining database tables, and all the blood relations of the target data set are determined according to the strategy.

[0055] Optionally, the execution module is further used to execute a replacement step and a retention step, wherein the replacement step is: replacing the remaining database tables for which the associated data set is found as a result of the search with the associated data set in the blood relationship of the corresponding second database table;

[0056] Retention step: retaining the remaining database tables for which the search result shows that no associated data set is found in the blood relationship of the corresponding second database table;

[0057] The replacing step and the retaining step are performed on each blood relationship of the second database table to obtain the processed blood relationship of each second database table;

[0058] The processed blood relationship of each of the second database tables is determined as all blood relationships of the target data set.

[0059] Optionally, the execution module is further configured to, after obtaining the blood relationship of the target data set,

[0060] In the blood relationship, a target node is determined; data monitoring is performed on the target node, including: determining whether data anomalies occur in the target node based on the data operation log corresponding to the target node and the execution status of the stream job associated with the target node; if so, determining the source and trend of the abnormal data based on the blood relationship; generating an abnormality analysis report based on the source and trend of the abnormal data, wherein the abnormality analysis report is used to indicate the impact of the abnormal data on the remaining nodes in the blood relationship except the target node.

[0061] Optionally, the execution module is further used to display the lineage relationship of the target data set after obtaining the lineage relationship of the target data set, which includes: receiving a selection instruction from a user, wherein the selection instruction is used to indicate a display method, and the display method includes at least one of the following: display by job granularity, display by data set granularity;

[0062] According to the display method, the blood relationship of the target data set is displayed.

[0063] Optionally, the execution module is further used to create a data source before obtaining the SQL statement to be processed;

[0064] All database tables are created, and association relationships between all created database tables and data sets in the data source are set; based on all created database tables, the SQL job flow is generated.

[0065] In a third aspect, an embodiment of the present invention provides an electronic device comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method for determining the blood relationship of a data set based on the computing power of an intelligent computing center as described in the first aspect.

[0066] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for determining the blood relationship of a data set based on the computing power of an intelligent computing center as described in the first aspect are implemented.

[0067] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method for determining the blood relationship of a data set based on the computing power of an intelligent computing center as described in the first aspect.

[0068] In an embodiment of the present invention, the lineage relationship of the target data set can be obtained through the mapping relationship between SQL statements and database tables, and between database tables and data sets, so as to understand in detail the source and conversion history of the data in the data set in the entire processing flow, and the generation and change process of the data set, thereby ensuring the integrity of the lineage display and avoiding the problems of difficult data tracing and incomplete data impact analysis.

[0069] Through accurate blood relationship, the intelligent computing center can more effectively dispatch computing resources to meet the data processing needs at different stages and avoid waste or shortage of computing resources. The optimized blood relationship provides a more comprehensive and accurate data foundation for the model of the intelligent computing center. The model can refer to the complete data set information during training and reasoning, improve the accuracy of data analysis, and make business decisions more scientific and effective. In the intelligent computing center, through the blood relationship information obtained in real time, the computing power can quickly respond to changing data sources and data requirements. This real-time monitoring capability, combined with powerful computing resources, enables the intelligent computing center to efficiently handle complex real-time computing tasks and improve the overall response speed of the intelligent computing center. In summary, the computing power of the intelligent computing center can be improved, and the efficiency and reliability of the intelligent computing center in important links such as data processing, model training and reasoning can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0071] Figure 1 A flowchart of a method for determining the blood relationship of a data set based on the computing power of an intelligent computing center provided by an embodiment of the present invention;

[0072] Figure 2 A flowchart of a method for determining the blood relationship of a data set based on the computing power of an intelligent computing center provided by an embodiment of the present invention;

[0073] Figure 3 A flowchart of a method for determining the blood relationship of a data set based on the computing power of an intelligent computing center provided by an embodiment of the present invention;

[0074] Figure 4 A schematic diagram of an SQL job lineage display provided by an embodiment of the present invention;

[0075] Figure 5 A schematic diagram of a data set lineage display provided by an embodiment of the present invention;

[0076] Figure 6A schematic diagram of a data set lineage display provided by an embodiment of the present invention;

[0077] Figure 7 A flowchart of a method for determining the blood relationship of a data set based on the computing power of an intelligent computing center provided by an embodiment of the present invention;

[0078] Figure 8 A schematic diagram of a data set lineage display provided by an embodiment of the present invention;

[0079] Fig. 9 A schematic diagram of a data set lineage display provided by an embodiment of the present invention;

[0080] Fig.10 A structural block diagram of a device for determining the blood relationship of a data set based on the computing power of an intelligent computing center provided in an embodiment of the present invention;

[0081] Fig.11 A structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] The "computing power" mentioned in the present invention is the ability of computer equipment or computing / data center to process information. It is the ability of computer hardware and software to work together to execute certain computing requirements. It is the computing power to achieve target result output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity. It mainly provides services to the society through computing power infrastructure.

[0084] The "computing power (CP)" mentioned in the present invention is the ability of a data center server to process data and output results. It is a comprehensive indicator to measure the computing power of a data center, including general computing power, super computing power and intelligent computing power. The commonly used unit of measurement is the number of floating point operations per second (Floating Point Operations Per Second, FLOPS, where 1EFLOPS=10^18FLOPS). The larger the value, the stronger the comprehensive computing power. According to calculations, 1EFLOPS is approximately the computing power output of 5 Tianhe-2A or 500,000 mainstream server CPUs (Central Processing Unit) or 2 million mainstream notebooks. The calculation formula is: CP=CP 通用 +CP 智能 +CP 超级 In the embodiment of the present application, the computing power adopts the half-precision floating point computing power (FP16) of the graphics card.

[0085] The "carrying capacity" (Network Power, NP) described in the present invention is a manifestation of the data transmission capability of computing facilities, including comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, etc. Carrying capacity involves network transmission within and between data centers, and is a comprehensive indicator for measuring network transmission scheduling capabilities. In the embodiment of the present application, carrying capacity uses video memory bandwidth. In the embodiment of the present application, storage capacity uses video memory bandwidth.

[0086] The "storage power" (SP) described in the present invention refers to the comprehensive capabilities of a data center in terms of data storage capacity, performance, safety and reliability, and green and low-carbon. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and server built-in storage devices. The commonly used unit of measurement for storage capacity is exabyte (EB, 1EB = 2^60bytes), and the commonly used unit of measurement for performance is the number of reads and writes per second per unit capacity (IOPS / TB, Input / Output Operations Per Second / TB). The disaster recovery ratio is an important manifestation of safety and reliability. In the embodiment of the present application, the storage power adopts the video memory capacity.

[0087] The "computing power infrastructure" described in the present invention is a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity. It can realize centralized computing, storage, transmission and application of information, and presents characteristics such as diversity and ubiquity, intelligence and agility, safety and reliability, green and low-carbon. It is of great significance to promote industrial transformation and upgrading, enable my country's scientific and technological innovation, meet people's better life and realize high-efficiency social governance.

[0088] The “computing power” mentioned in the present invention includes general computing power CP 通用 、Intelligent computing power CP 智能 And super computing power CP 超级 .

[0089] The "general computing power" mentioned in the present invention refers to the computing power provided by servers based on CPU chips, which is used to support basic general computing such as cloud computing and edge computing.

[0090] The "intelligent computing power" mentioned in the present invention refers to the large-scale deployment of intelligent computing centers based on special chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit) for various innovative artificial intelligence applications, such as natural language processing, machine vision, etc.

[0091] The "super computing power" mentioned in the present invention is mainly the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for calculations in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0092] The "intelligent computing center" mentioned in the present invention refers to a facility that provides the required computing power, data and algorithms for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model reasoning) by using large-scale heterogeneous computing resources, including general computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.). The intelligent computing center covers facilities, hardware, and software, and can provide full-stack capabilities from bottom-level computing power to top-level application enablement.

[0093] The "computing resources" mentioned in the present invention refer to the technologies and facilities with information calculation, transmission, storage and application capabilities required for the development of the digital society, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guarantee resources such as wind, fire, water and electricity.

[0094] The data source described in the present invention: data source refers to a place or system that stores and provides data. It can be various types of databases, files, API interfaces, sensors or other data acquisition devices. For example, a kafka cluster can be defined as a data source, and each data source has a unique dataSourceName (the name or identifier of the data source, which is used to uniquely identify a specific data source).

[0095] The dataset described in the present invention: A dataset refers to a way of collecting, organizing and storing related data in a unified data set. It is usually composed of a group of related data records or data tables, and is used to represent data in a specific field, theme or problem. Each dataset has a unique dataSourceCode (the name or identifier of the dataset). For example, a topic in a kafka cluster can be called a dataset. Specifically, if there is a topic: student in the devKafka cluster, its dataSourceCode is devKafka_student.

[0096] Table information in the present invention refers to metadata information used to describe a data table in a relational database, which includes information about the table structure, column definition, constraints, indexes, etc. Creating a table requires associating a data set, and each table has a catalogCode (the name or identifier of the table). For example, if a student table is created, its catalogCode is catalogName_database_student.

[0097] SQL statement lineage analysis described in the present invention: SQL statement lineage analysis refers to tracking and analyzing SQL statements to understand the relationship and impact between the statement and other tables, views, and columns in the database. For example: define an SQL statement insert into student select*from student_score where student Code=1, parse the SQL statement, and analyze the data flow between the database tables involved in the SQL statement. It can be seen that the SQL statement is to insert the data in the student_score table into the student table, then student_score is from, student is to, and the parsed data lineage relationship will be saved in mysql, and in specific application scenarios, the SQL lineage relationship can be saved in real time or in time periods.

[0098] Figure 1 A method for determining the blood relationship of a data set based on the computing power of an intelligent computing center according to an embodiment of the present invention is shown, and the method includes:

[0099] Step S1: in the SQL job flow, obtain the SQL statement to be processed;

[0100] Step S2, parsing the SQL statement to be processed to obtain the parsing result;

[0101] Step S3: Determine the data flow direction between the first database tables indicated by the SQL statement to be processed according to the analysis result;

[0102] Step S4: determining the blood relationship between the first database tables based on the data flow direction;

[0103] Step S5: In the blood relationship, according to the preset association relationship between all database tables and data sets, the first database table indicated by the SQL statement to be processed is replaced with the target data set associated with it, so as to obtain the blood relationship of the target data set.

[0104] In a possible implementation, before step S1, obtaining the SQL statement to be processed, the method further includes: step S01, creating a data source; step S02, creating all database tables and setting the association relationship between the created database tables and the data sets in the data source; and generating an SQL job flow based on the created database tables.

[0105] The data source includes at least one of the following: kafka, mysql, hbase, elasticserach or clickhouse, and the corresponding data set in the data source includes at least one of the following: topic in kafka, table in mysql, table in hbase, index in elasticsearch or table in clickhouse. A data set can be associated with multiple database tables, and one database table is associated with one data set.

[0106] It should be noted that SQL jobs can be divided into three parts: data source creation: refers to the process of creating a data source table, which can be loading data from an external data source or manually creating a table; table creation: refers to the table structure involved in the job creation process, including defining table names, column names, data types, constraints, etc.; SQL job logic: refers to the writing and logical processing of SQL statements, including operations such as query, update, and insert. And the association between the created database table and the data set in the data source needs to be set.

[0107] It is understandable that SQL jobs are composed of multiple or single SQL statements. SQL statements process data in one table and store it in another table, and the table needs to be associated with a dataset. The dataset lineage relationship is obtained through SQL data lineage relationship, mainly by mapping table information to dataset information, thereby obtaining dataset lineage relationship.

[0108] Based on this, in steps S1 to S4, the data flow of the database tables in the SQL statements, i.e., the SQL lineage relationship, needs to be analyzed through the information of the database tables and the granularity of the SQL statements. In a possible implementation, the parsing result in step S2 is: the reference field in the SQL statement to be processed and the first database table indicated by the SQL statement to be processed; wherein the reference field includes at least one of the following: the query field and the condition field; then Figure 2 As shown, step S3 determines the data flow between the first database tables indicated by the SQL statement to be processed according to the parsing result, including:

[0109] Step S31: Determine the data flow direction between the first database tables indicated by the SQL statement to be processed according to the reference field and the first database table indicated by the SQL statement to be processed.

[0110] Further, in step S5, if the database table is associated with a data set, the lineage relationship of the SQL-related data set can be obtained through the SQL lineage relationship. That is: in the lineage relationship, according to the pre-set association relationship between all database tables and data sets, the first database table indicated by the SQL statement to be processed is replaced with the target data set associated with itself, and the lineage relationship of the target data set is obtained. In a specific application scenario, the input data source of each table can be determined according to the SQL lineage relationship, and the source of each input data source can be traced, including the upstream table, data set or data source. Repeat this step until the original source of the data set is found. Therefore, the lineage relationship of the target data set can be obtained through the mapping relationship between the SQL statement and the database table, and the database table and the data set, so as to understand in detail the source and conversion history of the data in the data set in the entire processing flow, and the generation and change process of the data set, to ensure the integrity of the lineage display, and avoid the problems of difficult data tracing and incomplete data impact analysis. This ensures that when the intelligent computing center efficiently processes large-scale data, it can accurately grasp the flow of data through accurate blood relationships, and more effectively dispatch computing resources to meet data processing needs at different stages, avoid waste or shortage of computing resources, and improve the computing power of the intelligent computing center. Improve the efficiency and reliability of the intelligent computing center in important links such as data processing, model training and reasoning.

[0111] Specifically, after obtaining the blood relationship of data sets, the blood relationship of data sets can be used in the following scenarios. In the scenario of data quality management of the intelligent computing center, by understanding the blood relationship of data sets, the source and flow of data can be traced to help discover data quality problems. For example, if the input data source of a data set has changed or a data set is used by multiple jobs, potential data quality risks can be discovered in time and corresponding corrective measures can be taken. In the scenario of data tracing and source tracing in the intelligent computing center, the blood relationship of data sets can help with data tracing. By tracing the blood relationship of data sets, it is possible to understand how data is generated, converted, and transmitted, and effectively locate and troubleshoot the root cause when data problems occur. In the scenario of data integration in the intelligent computing center, the blood relationship of data sets can reveal the dependencies between different data sets to improve the efficiency and accuracy of data integration. By analyzing the blood relationship, it is possible to determine which data sets are important and critical, so as to better plan and optimize the data integration process. In the scenario of data security maintenance in the intelligent computing center, understanding the lineage of data sets can help ensure data security and compliance. By tracking the lineage of data sets, data access rights can be controlled to prevent unauthorized data use and leakage.

[0112] In a possible implementation, after obtaining the blood relationship of the target data set in step S5, the method further includes: step S17, displaying the blood relationship of the target data set, such as Figure 3 As shown, the blood relationship of the target dataset includes:

[0113] Step S171, receiving a user's selection instruction;

[0114] Step S172: Display the blood relationship of the target data set according to the display method.

[0115] It should be noted that the selection instruction is used to indicate the display method, and the display method includes at least one of the following: display by job granularity and display by data set granularity. Among them, displaying by job granularity can be to display the data sets used by each job process and their lineage relationships according to the execution order of the SQL job. Therefore, by displaying the data set information by job granularity, the data sets used by each job and their dependencies can be understood, and job scheduling and optimization can be better performed. Displaying by data set granularity can be to display the lineage relationship of each data set according to the hierarchical relationship of the data set, including upstream data sets and downstream data sets. Displaying the lineage relationship by data set granularity can assist in data quality management by understanding the data source, flow and usage of each data set, and more accurately track and locate data quality problems, and timely discover potential data quality risks. In general, displaying the lineage relationship of data sets helps to understand the source and flow of data, and provides support for data management and data traceability of the intelligent computing center. This can better understand and analyze data, ensure data quality and data compliance.

[0116] The above implementation is now described with reference to a specific application scenario.

[0117] SQL homework lineage display: Parse the SQL statement: insert into Student select*from student_score where studentCode=1, then the student_score table belongs to from, student belongs to to, and the data flow is to insert the data of studentCode=1 in the student_score table into the student table. That is, select the record with student number 1 from the source table student_score and insert it into the target table student, such as Figure 4 shown.

[0118] It should be noted that, when storing, the student_score table and the student table are uniquely represented by catalogCode (catalogName.database.tableName), that is, a table is uniquely identified by a combination of catalog name, database name, and table name (tableName), thereby ensuring accurate identification and positioning of the table in the entire data management system.

[0119] Dataset lineage display: By querying the SQL job lineage relationship and the association between the table and the dataset (when creating a database table, the dataset unique identifier dataSourceCode will be associated with the database table), the dataset information can be mapped through the database table information, such as Figure 5 shown.

[0120] It should be noted that if a new SQL logic is added, for example: insert into score select*fromStudent, then in this SQL statement, the student table is from, the score table is to, and the data direction is to insert the data in the student table into the score table. Then the parsed lineage data can be saved in MySQL, and the lineage display of this data set is as follows: Figure 6 As shown, that is, the method shown in the embodiment of the present invention can be directed to any SQL statement and perform real-time or time-division processing.

[0121] In one possible implementation, Figure 7 As shown, after step S4, determining the blood relationship between database tables, the method further includes:

[0122] Step S6, storing the blood relationship in the first database table into the blood relationship pool;

[0123] Step S7, receiving a user's query instruction;

[0124] Among them, the query instruction is used to query all the blood relations of the target data set;

[0125] Step S8: determining at least one second database table associated with the target data set according to the query instruction;

[0126] Step S9: searching for the blood relationship with each second database table in the blood relationship pool according to the unique identifier of at least one second database table associated with the target data set;

[0127] Step S10, respectively searching for the remaining database tables associated with the blood relationship of each second database table;

[0128] The remaining database tables are database tables other than the second database table involved in the blood relationship of each second database table;

[0129] Step S11, searching for the data sets associated with each database table in the remaining database tables;

[0130] Step S12: Based on the search results, execute the strategy corresponding to the search results on the remaining database tables, and determine all the blood relationships of the target data set according to the strategy.

[0131] And in a possible implementation, step S12 includes: step S121 replacement step: replacing the remaining database tables whose search results are that the associated data set is found with the associated data set in the blood relationship of the corresponding second database table; step S122 retention step: retaining the remaining database tables whose search results are that the associated data set is not found in the blood relationship of the corresponding second database table; step S123: executing the step S121 replacement step and the step S122 retention step for each blood relationship of the second database table to obtain the processed blood relationship of each second database table; step S124: determining the processed blood relationship of each second database table as all the blood relationships of the target data set.

[0132] The implementation method is now explained with a specific application scenario. To query the lineage relationship of a data set, all tables mapped by the dataSourceCode of the data set (one data set corresponds to multiple database tables) can be queried. In specific implementation, the unique identifier catalogCode of the database table can be used to obtain all lineage relationship records ( Figure 7 The blood relationship pool in the implementation shown in the figure), and the blood relationship records obtained include direct or indirect blood relationships. Figure 8 As shown, two records can be obtained by querying the test data set. After querying the lineage graph of the SQL job, the table information in the data lineage graph can be replaced with the data set information through the mapping relationship between the data set and the database table information, which corresponds to the above replacement step.

[0133] It should be noted that if the method shown in the embodiment of the present invention is applied to the view table, that is, the view table is created by SQL statements in the SQL job, but the data set information related to the view table is not saved, so there is a situation where the data set information cannot be queried, that is, corresponding to the above-mentioned retention step (retaining catalogCode), and the remaining database tables that find the associated data set are replaced with their respective associated data sets (replacing catalogCode with dataSourCode), and the blood relationship of the target data set is obtained. Fig. 9 As shown, the black-marked Blood_rtcatalog.default.people represents the second database table that is retained when the associated data set is not found.

[0134] In one possible implementation, after obtaining the blood relationship of the target data set in step S5, the method also includes: step S13: determining the target node in the blood relationship; step S14: performing data monitoring on the target node, including: determining whether data anomalies occur in the target node based on the data operation log corresponding to the target node and the execution status of the flow job associated with the target node; step S15: if so, determining the source and direction of the abnormal data based on the blood relationship; step S16: generating an abnormality analysis report based on the source and direction of the abnormal data, wherein the abnormality analysis report is used to indicate the impact of the abnormal data on the remaining nodes in the blood relationship except the target node.

[0135] It should be noted that in data lineage, key nodes refer to those data sets or database tables that play an important role in the data flow. For example, a data set that is commonly referenced by multiple reports, or a common input point for multiple data processing processes, these nodes are particularly critical to data quality because their data quality directly affects multiple downstream processes. The operation log records all the operation details of the data set in the entire process, such as when the data was updated, who updated it, and what was updated. These logs are essential for tracking data changes, monitoring data quality issues, and subsequent problem analysis. A stream job refers to a series of data processing operations that are automatically performed in a certain order to transform, aggregate, or analyze data. For example, a stream job may include extracting data from one data set, cleaning the data and storing it in another data set, and then aggregating and reporting the data.

[0136] When using data lineage for data quality control, we first focus on identifying and monitoring these key nodes. By analyzing the operation logs and the execution of stream jobs, we can monitor data changes and automatically discover potential data quality issues. Once a problem is found, we can use the lineage graph to trace the source of the problem and evaluate its impact on other data sets and stream jobs.

[0137] It should be noted that the method described in the embodiment of the present invention can be applied to the operation and maintenance of real-time data processing systems, such as in banking scenarios, where banking business is complex and changeable, upstream and downstream systems are independent of each other, and data errors need to be detected in a timely manner. By applying the method shown in the embodiment of the present invention, the upstream and downstream blood relationship related to the data set can be queried through the data set to clearly view which operation processes are affected by the current data set, which is convenient for rapid error detection to minimize risks and losses. And from an overall perspective, the overall business logic can be better understood.

[0138] Therefore, through the mapping relationship between SQL statements and database tables, and between database tables and data sets, the lineage relationship of the target data set can be obtained, so as to understand in detail the source and conversion history of the data in the data set in the entire processing flow, and the generation and change process of the data set, to ensure the integrity of the lineage display and avoid the problems of difficult data tracing and incomplete data impact analysis.

[0139] Through accurate blood relationship, the intelligent computing center can more effectively dispatch computing resources to meet the data processing needs at different stages and avoid waste or shortage of computing resources. The optimized blood relationship provides a more comprehensive and accurate data foundation for the model of the intelligent computing center. In this way, the model can refer to the complete data set information during training and reasoning, improve the accuracy of data analysis, and make business decisions more scientific and effective. In the intelligent computing center, through the blood relationship information obtained in real time, the computing power can quickly respond to changing data sources and data requirements. This real-time monitoring capability, combined with powerful computing resources, enables the intelligent computing center to efficiently handle complex real-time computing tasks and improve the overall response speed of the system. In summary, the computing power of the intelligent computing center can be improved, and the efficiency and reliability of the intelligent computing center in important links such as data processing, model training and reasoning can be improved.

[0140] The embodiment of the present invention also provides a device 10 for determining the blood relationship of a data set based on the computing power of an intelligent computing center, such as Fig.10 As shown, the device 10 includes:

[0141] The acquisition module 1001 is used to acquire the SQL statement to be processed in the SQL job flow;

[0142] An execution module 1002 is used to parse the SQL statement to be processed and obtain a parsing result;

[0143] Determine the data flow between the first database tables indicated by the SQL statement to be processed according to the parsing result;

[0144] Based on the data flow direction, determine the blood relationship between the first database tables;

[0145] In the blood relationship, according to the preset association relationship between all database tables and data sets, the first database table indicated by the SQL statement to be processed is replaced with the target data set associated with itself, so as to obtain the blood relationship of the target data set.

[0146] In a possible implementation, the parsing result is: a reference field in the SQL statement to be processed and a first database table indicated by the SQL statement to be processed; wherein the reference field includes at least one of the following: a query field and a condition field;

[0147] The execution module 1002 is further configured to determine the data flow direction between the first database tables indicated by the SQL statement to be processed according to the reference field and the first database table indicated by the SQL statement to be processed.

[0148] In a possible implementation, the execution module 1002 is further configured to store the blood relationship of the first database table in a blood relationship pool after determining the blood relationship between the first database tables;

[0149] Receive a query instruction from a user; wherein the query instruction is used to query all blood relationships of a target data set;

[0150] Determining at least one second database table associated with the target data set according to the query instruction;

[0151] According to the unique identifier of the at least one second database table associated with the target data set, searching for the blood relationship of each second database table in the blood relationship pool;

[0152] Respectively searching for the remaining database tables associated in the blood relationship of each second database table; wherein the remaining database tables are database tables other than the second database table involved in the blood relationship of each second database table;

[0153] Find the data sets associated with each database table in the remaining database tables;

[0154] According to the search result, the strategy corresponding to the search result is executed on the remaining database tables, and all the blood relations of the target data set are determined according to the strategy.

[0155] In a possible implementation, the execution module 1002 is further configured to execute a replacement step and a retention step, wherein the replacement step is: replacing the remaining database tables for which the associated data set is found as a result of the search with the associated data set in the blood relationship of the corresponding second database table;

[0156] Retention step: retaining the remaining database tables for which the search result shows that no associated data set is found in the blood relationship of the corresponding second database table;

[0157] The replacing step and the retaining step are performed on each blood relationship of the second database table to obtain the processed blood relationship of each second database table;

[0158] The processed blood relationship of each of the second database tables is determined as all blood relationships of the target data set.

[0159] In one possible implementation, the execution module 1002 is also used to determine the target node in the blood relationship after obtaining the blood relationship of the target data set; perform data monitoring on the target node, including: determining whether data anomalies occur in the target node based on the data operation log corresponding to the target node and the execution status of the flow job associated with the target node; if so, determining the source and direction of the abnormal data based on the blood relationship; generating an abnormality analysis report based on the source and direction of the abnormal data, wherein the abnormality analysis report is used to indicate the impact of the abnormal data on the remaining nodes in the blood relationship except the target node.

[0160] In a possible implementation, the execution module 1002 is further used to display the lineage relationship of the target data set after obtaining the lineage relationship of the target data set, which includes: receiving a selection instruction from a user, wherein the selection instruction is used to indicate a display method, and the display method includes at least one of the following: display by job granularity, display by data set granularity;

[0161] Display the lineage relationship of the target dataset according to the display method.

[0162] In a possible implementation, the execution module 1002 is further configured to create a data source before obtaining the SQL statement to be processed;

[0163] Create all database tables and set the association between all created database tables and data sets in the data source; generate SQL job flows based on all created database tables.

[0164] Therefore, through the mapping relationship between SQL statements and database tables, and between database tables and data sets, the lineage relationship of the target data set can be obtained, so as to understand in detail the source and conversion history of the data in the data set in the entire processing flow, and the generation and change process of the data set, to ensure the integrity of the lineage display and avoid the problems of difficult data tracing and incomplete data impact analysis.

[0165] Through accurate blood relationship, the intelligent computing center can more effectively dispatch computing resources to meet the data processing needs at different stages and avoid waste or shortage of computing resources. The optimized blood relationship provides a more comprehensive and accurate data foundation for the model of the intelligent computing center. In this way, the model can refer to the complete data set information during training and reasoning, improve the accuracy of data analysis, and make business decisions more scientific and effective. In the intelligent computing center, through the blood relationship information obtained in real time, the computing power can quickly respond to changing data sources and data requirements. This real-time monitoring capability, combined with powerful computing resources, enables the intelligent computing center to efficiently handle complex real-time computing tasks and improve the overall response speed of the system. In summary, the computing power of the intelligent computing center can be improved, and the efficiency and reliability of the intelligent computing center in important links such as data processing, model training and reasoning can be improved.

[0166] The embodiment of the present invention further provides an electronic device 110, such as Fig.11 As shown, the system includes: a processor 1101, a memory 1102, and a program stored in the memory 1102 and executable on the processor 1101. When the program is executed by the processor, the following is achieved: Figure 1 The steps of the method for determining the blood relationship of data sets based on the computing power of the intelligent computing center are shown.

[0167] The embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above Figure 1 The various processes of the embodiment of the method for determining the blood relationship of a data set based on the computing power of an intelligent computing center shown in the figure can achieve the same technical effect. To avoid repetition, they are not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0168] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the various processes of the above-mentioned method embodiment for determining the blood relationship of data sets based on the computing power of an intelligent computing center, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0169] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0170] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0171] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for determining the blood relationship of a data set based on the computing power of an intelligent computing center, characterized in that: The method comprises: Step S1: in the SQL job flow, obtaining the SQL statement to be processed; Step S2: parsing the SQL statement to be processed to obtain a parsing result; Step S3: determining the data flow direction between the first database tables indicated by the SQL statement to be processed according to the analysis result; Step S4: determining the blood relationship between the first database tables based on the data flow direction; Step S5: In the blood relationship, according to the preset association relationship between all database tables and data sets, the first database table indicated by the SQL statement to be processed is replaced with the target data set associated with it, so as to obtain the blood relationship of the target data set.

2. The method according to claim 1, characterized in that: The parsing result is: the reference field in the SQL statement to be processed and the first database table indicated by the SQL statement to be processed; wherein the reference field includes at least one of the following: a query field and a condition field; The step S3 comprises: Step S31: determining the data flow direction between the first database table indicated by the SQL statement to be processed according to the reference field and the first database table indicated by the SQL statement to be processed.

3. The method according to claim 1, characterized in that: After step S4, the method further includes: Step S6: storing the blood relationship in the first database table into a blood relationship pool; Step S7: receiving a query instruction from a user; wherein the query instruction is used to query all blood relationships of the target data set; Step S8: determining at least one second database table associated with the target data set according to the query instruction; Step S9: searching for the blood relationship of each second database table in the blood relationship pool according to the unique identifier of the at least one second database table associated with the target data set; Step S10: respectively searching for the other database tables associated with the blood relationship of each second database table; wherein the other database tables are database tables other than the second database table involved in the blood relationship of each second database table; Step S11: searching for data sets associated with each database table in the remaining database tables; Step S12: According to the search result, the strategy corresponding to the search result is executed on the remaining database tables, and all blood relationships of the target data set are determined according to the strategy.

4. The method according to claim 3, characterized in that The step S12 comprises: Step S121: Replacement step: replace the remaining database tables for which the associated data set is found in the search result with the associated data set in the blood relationship of the corresponding second database table; Step S122: retaining step: retaining the remaining database tables for which the search result shows that no associated data set is found in the blood relationship of the corresponding second database table; Step S123: performing the replacement step of step S121 and the retention step of step S122 on each blood relationship in the second database table to obtain the processed blood relationship of each second database table; Step S124: Determine the processed blood relationship of each of the second database tables as all blood relationships of the target data set.

5. The method according to claim 1, characterized in that After step S5, the method further includes: Step S13: determining a target node in the blood relationship; Step S14: performing data monitoring on the target node, including: determining whether data anomalies occur in the target node according to the data operation log corresponding to the target node and the execution status of the stream job associated with the target node; Step S15: If yes, determining the source and trend of the abnormal data according to the blood relationship; Step S16: Generate an abnormality analysis report based on the source and trend of the abnormal data, wherein the abnormality analysis report is used to indicate the impact of the abnormal data on the remaining nodes in the blood relationship except the target node.

6. The method according to claim 1, characterized in that After step S5, the method further includes: Step S17: Display the blood relationship of the target data set, including: Step S171: receiving a selection instruction from a user, wherein the selection instruction is used to indicate a display mode, and the display mode includes at least one of the following: display by job granularity, display by data set granularity; Step S172: displaying the blood relationship of the target data set according to the display method.

7. The method according to claim 1, characterized in that Before step S1, the method further includes: Step S01: Create a data source; Step S02: creating all database tables, and setting associations between all created database tables and data sets in the data source; generating the SQL job flow based on all created database tables.

8. A device for determining blood relationship of data sets based on the computing power of an intelligent computing center, characterized in that: The device comprises: An acquisition module is used to acquire SQL statements to be processed in the SQL job flow; An execution module is used to parse the SQL statement to be processed and obtain a parsing result; Determine the data flow between the first database tables indicated by the SQL statement to be processed according to the analysis result; Based on the data flow direction, determining the blood relationship between the first database tables; In the blood relationship, according to the preset association relationship between all database tables and data sets, the first database table indicated by the SQL statement to be processed is replaced with the target data set associated with it, so as to obtain the blood relationship of the target data set.

9. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method for determining the blood relationship of a data set based on the computing power of an intelligent computing center as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for determining the blood relationship of a data set based on the computing power of an intelligent computing center as described in any one of claims 1 to 7.

11. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps of the method for determining the blood relationship of a data set based on the computing power of an intelligent computing center as described in any one of claims 1 to 7.

Images