Data flow information generation method, storage medium and electronic equipment

Abstract

The invention relates to the field of information processing, in particular to a data flow information generation method, a storage medium and electronic equipment. Comprising the following steps: acquiring node information and data channel information of each data set circulation by monitoring a preset operation behavior in a multi-source data processing system; according to all the node information and the data channel information, a graph data structure corresponding to a data set dependency relationship in the multi-source data processing system is created, and in response to a query request for dependency relationship information of a target data set, information in a dependency relationship chain to which the target data set in the HBase database belongs is obtained. And a graph data structure between the nodes can be constructed through relevant information such as a configuration file corresponding to the circulation behavior. And the more detailed data circulation process and the upstream and downstream dependency blood relationship are displayed through the dependency relationship chain, so that the related personnel can more intuitively and accurately position the point location with the abnormal data.

Description

Background Art

[0002] With the development of information technology and the increase of various demands in application scenarios, in order to meet the corresponding demands, the corresponding data processing system needs to be able to connect to multiple data source sending platforms to collect and obtain various types of data. In addition, in order to be able to distribute the processed data to various downstream applications for direct use, the data processing system is usually required to connect to the data management platforms of various applications. Usually, these data include not only traditional structured data (such as tabular data in the database), but also a large amount of unstructured or semi-structured data (such as text, pictures, videos, etc.). For the above reasons, for some data processing systems with high comprehensive service capabilities, the data flow relationship is usually very complicated, which will also lead to difficulties in accurately and quickly determining the problem points where data anomalies may occur during data maintenance and governance in the later stage. Summary of the invention

[0003] In view of the above technical problems, the technical solution adopted by the present invention is:

[0004] According to one aspect of the present invention, a method for generating data flow information is provided, which is used to generate dependency information of data sets in a multi-source data processing system, wherein the multi-source data processing system includes a Kafka message queue, HDFS, a Hive data warehouse, and data management platforms on multiple application sides, wherein the Kafka message queue is connected to the Hive data warehouse through HDFS, and the Hive data warehouse is connected to the data management platforms on multiple application sides respectively;

[0005] The method for generating data flow information includes the following steps:

[0006] By monitoring the preset operation behaviors in the multi-source data processing system, the node information and data channel information of each data set flow are obtained; the node information is the storage node information during the data set flow process; the data channel information is the flow dependency relationship information between the data sending node and the data receiving node during the data set flow process; the preset operation behaviors include data reading and data transmission behaviors;

[0007] Based on all node information and data channel information, a graph data structure corresponding to the data set dependency in the multi-source data processing system is created and stored in the HBase database;

[0008] In response to a query request for target data set dependency information, obtain information in a dependency chain to which the target data set belongs in an HBase database;

[0009] According to the hierarchical order in the dependency chain, the node information of each level is written into the corresponding hierarchical information window in sequence; if the length of all node information in the level is greater than the length of the corresponding hierarchical information window, a scroll bar is used to display different node information in the hierarchical information window;

[0010] According to the data channel information in the dependency chain, a flow indicator line between each node information in the current level information window is generated.

[0011] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method for generating data flow information is implemented.

[0012] According to a third aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for generating data flow information when executing the computer program.

[0013] The present invention has at least the following beneficial effects:

[0014] Since, in the process of data flow in the system, data sets will inevitably involve operations such as data reading and / or data transmission task establishment. Therefore, by monitoring the operation behaviors that will inevitably occur in the data flow process, each data flow behavior can be discovered. Then, through the configuration file and other related information corresponding to the flow behavior, the information of the sender and receiver of the data set in each flow process can be determined, and the corresponding node information and data channel information can be established from this information. Among them, two node information are directed connected by corresponding data channel information, and the data structure is the same as the graph data structure. Then, using all the collected node information and data channel information, a relationship diagram of the data flow relationship in the multi-source data processing system can be established. The corresponding graph data information is stored in the HBase database, so that when a certain data node is queried later, the corresponding data node and its dependency chain can be queried more quickly, and a more detailed data flow process and upstream and downstream dependency blood relationship can be displayed through the dependency chain, so that relevant personnel can more intuitively and accurately locate the data abnormality point.

[0015] In addition, since a multi-source data processing system usually has very complex business data, the corresponding data node dependencies are also extremely rich, so it is possible that a large amount of node information will appear in a certain level. When the present invention displays the relevant information in the dependency chain, the size specifications of the window corresponding to each level are consistent. When the length of all node information in the level is greater than the length of the corresponding level information window, only the information in the window is displayed, and the remaining node information is displayed using a scroll bar. This ensures that the overall layout of the relationship display interface remains stable, and also ensures that the font size of the displayed information remains consistent, which is convenient for viewing and use. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A flowchart of a method for generating data flow information provided by an embodiment of the present invention;

[0018] Figure 2 A flowchart of a method for determining data node information modification authority provided by an embodiment of the present invention;

[0019] Figure 3 A schematic diagram of displaying relevant information in a dependency chain in each level of information windows provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0021] As a possible embodiment of the present invention, Figure 1 As shown, a method for generating data flow information is provided, which is used to generate dependency information of data sets in a multi-source data processing system. The multi-source data processing system includes a Kafka message queue, HDFS, a Hive data warehouse and data management platforms on multiple application sides. The Kafka message queue is connected to the Hive data warehouse through HDFS, and the Hive data warehouse is connected to the data management platforms on multiple application sides respectively.

[0022] The way of obtaining the upstream and downstream dependencies of data in this embodiment belongs to the scenario of big data development, so the method for generating data flow information in this embodiment is run in the ecological environment of Hadoop. Hadoop is an open source framework for processing and analyzing large-scale data sets, which provides a powerful environment for big data development. Hadoop is not just a single technology, but a collection of a whole set of tools and services, called the Hadoop ecosystem. This ecosystem includes many other projects, such as Hive (data warehouse), Pig (data flow language), HBase (NoSQL database), HDFS (distributed file system), Spark (general computing engine), etc. It can provide an efficient, reliable and flexible environment for big data development, which is particularly suitable for processing and analyzing PB-level data sets.

[0023] In the multi-source data processing system of this embodiment, in order to realize certain business functions, it is inevitable to obtain relevant business data from other data sources. Usually when obtaining external data, there will be a high burst of data that needs to be processed in real time. Therefore, in order to smoothly access these data, this embodiment sets a Kafka message queue to obtain the above data. After obtaining the data, it enters the Hive data warehouse after a series of data cleaning, verification and standardization. Then, the data management platform of different downstream application sides requests the business data of the corresponding format from the Hive data warehouse for use by the corresponding business. In this embodiment, the data management platform on the application side can include Kylin analysis engine, Elasticsearch (ES) distributed search and analysis engine and MySQL relational database. In the above-mentioned multi-source data processing system, due to the diversity of business, there will be very complex blood dependency between data sets. At the same time, with the daily maintenance and upgrade operations, the upstream and downstream blood dependency between some data sets will also change. Therefore, by extracting and combing the flow information (blood dependency) of the complex data sets in the above-mentioned multi-source data processing system, the relevant use value can be excavated to facilitate the subsequent system maintenance and the formulation of relevant decisions.

[0024] Specifically, the method for generating data flow information includes the following steps:

[0025] S100: By monitoring the preset operation behaviors in the multi-source data processing system, the node information and data channel information of each data set flow are obtained. Node information is the storage node information during the data set flow process. Data channel information is the flow dependency information between the data sending node and the data receiving node during the data set flow process. The preset operation behavior includes data reading and data transmission behavior. The preset operation behavior is mainly some operations to be performed when data reference dependencies are performed between different data sources (such as between the Kafka message queue and the Hive data warehouse or between the Hive data warehouse and the Kylin analysis engine or the Elasticsearch (ES) distributed search or analysis engine and the MySQL relational database). And some operations to be performed when data reference dependencies are performed within the Hive data warehouse. Thus, the operation triggers the corresponding information acquisition task, and then obtains the data flow relationship and node name between any two nodes.

[0026] Specifically, S100 includes:

[0027] S110: Obtain node information and data channel information of each data set flow by monitoring preset operation behaviors between different types of data platforms in the multi-source data processing system.

[0028] In this embodiment, S110 is to obtain the data flow relationship outside the Hive data warehouse, including the flow relationship between the Kafka message queue and the Hive data warehouse or between the Hive data warehouse and the Kylin analysis engine or the Elasticsearch (ES) distributed search or analysis engine and the MySQL relational database.

[0029] When the data platform is Kafka message queue and Hive data warehouse, S110 includes:

[0030] S1101: When a newly created Kafka connection is detected, the current topic name in the Kafka message queue and the HDFS (Hadoop Distributed File System) storage path corresponding to each topic are obtained.

[0031] As a distributed stream processing platform, Kafka message queue is often used to process real-time data in the data processing process, such as data access. In this process, the topic in Kafka is the classification identifier of the data. The producer sends the message to the topic, and the consumer obtains the message from the topic for processing.

[0032] Therefore, when the creation of a Kafka connection (preset operation behavior) is detected, the existing capture tool can be triggered to automatically collect topic information, producer information, and consumer information. The topic information may include the topic name, the name of the cluster to which it belongs, and so on.

[0033] In this embodiment, since the Kafka message queue is used to directly receive external data, it is necessary to go through preset processing steps after data access to clean, verify and standardize the data. Therefore, the consumers at this time are usually some pre-set data processing projects, and after these projects complete the data processing, the corresponding data will be stored in the specified HDFS storage path. Thus, the advantages of HDFS such as high availability, high performance, scalability and cost-effectiveness can be fully utilized to ensure the security and reliability of data, while providing convenience for subsequent data processing and analysis. This is particularly important for systems that process large-scale data sets. In this embodiment, Kafka and Hive often play a role in different links. Kafka is used to process real-time streaming data, while Hive is a data warehouse for storing and analyzing large-scale structured data. Creating a data connection channel from Kafka to Hive can achieve effective connection from real-time data processing to data warehouse storage and analysis.

[0034] Correspondingly, in this scenario, the sending node may be a producer or topic in Kafka. The sending node writes the data in the topic to the HDFS path, and the receiving node obtains the data in the HDFS path by creating a Hive table or querying the statement.

[0035] S1102: Obtain the table name of each newly created Hive table and the HDFS source path of the table data through the configuration information of the newly created Hive table in the Hive data warehouse.

[0036] If a Hive table in a Hive data warehouse is to reference data in a related topic, the HDFS source path to which it refers must be specified in its configuration information when the Hive table is created.

[0037] S1103: Generate corresponding node information according to the subject name and the table name.

[0038] Node information is mainly used to represent some attribute information of the corresponding data node. For example, the topic information (topic name, cluster name, etc.) collected by the toolkit is created as a data node with a data type of 'ume_Kafka_topic', and the corresponding topic entity in Kafka is clearly identified in the relevant data dependency diagram.

[0039] S1104: Determine the correspondence between each topic name and table name by matching the HDFS source path with the HDFS storage path, so as to generate data channel information between corresponding node information.

[0040] By matching the HDFS source path with the HDFS storage path, topics and Hive tables with blood dependencies can be connected, that is, a mapping relationship between "topic-hdfs disk path-Hive table" is established (that is, data channel information).

[0041] For example, if the topic named "sales_events" in Kafka is written to the HDFS path of "log / data", and the table named "sales_data" in Hive obtains the data in the HDFS path, the corresponding data channel information is: sales_events-HDFS disk path-sales_data. This data channel information clearly shows the lineage path of data from Kafka to Hive, thereby achieving orderly flow and conversion of data between different systems. The creation of this data connection channel not only facilitates data transmission and integration, but also provides better visibility and controllability for data management. From the perspective of data lineage, the data link from Kafka to Hive is improved, making the source and destination of data clearer, and facilitating data tracing, data quality assessment, and data security management.

[0042] When the data platform is ES and Hive data warehouse, S110 includes:

[0043] Elasticsearch (ES for short) is a distributed search and analysis engine that is widely used in scenarios such as full-text search, log analysis, and real-time data processing.

[0044] S1111: When a newly created data transmission task between the Es and Hive data warehouses is detected, the datax-json configuration file corresponding to the data transmission task is obtained through the datax tool.

[0045] Datax is a data synchronization tool that is mainly used to solve data migration and synchronization problems between different data sources. It supports data transmission between multiple data sources, including but not limited to relational databases (such as MySQL, Oracle), NoSQL databases (such as MongoDB, HBase), file systems (such as HDFS, local files), etc.

[0046] S1112: Obtain the origin and destination information of the data set through the datax-json configuration file.

[0047] S1113: Generate two corresponding node information according to the name of the origin and destination of the data set.

[0048] S1114: Generate a mapping relationship between the origin and the destination according to the location information of the origin and the destination of the data set, so as to serve as data channel information corresponding to the two node information.

[0049] When a data transfer task is created between ES and the Hive data warehouse, a datax-json configuration file is generated, which records in detail the various parameters and configurations of the data transfer task. During the data transfer process from Hive to ES, this configuration file contains the key information of the transfer task between the data source (Hive table) and the data destination (Elasticsearch index).

[0050] By parsing the datax-json configuration file, we can obtain the data source (Hive table)-data destination (Elasticsearch index) related information, build the relationship between the two, and extract the node information to create a data node with the data type of 'es'. The node information can include the index name, es cluster name, etc., and can track the specific es destination of the Hive table flow. In addition, the bloodline path of data flowing from Hive to es can be clearly displayed. When a problem occurs with the data, it can be determined based on the dependency relationship whether it is a problem generated on the Hive table side, or a problem that occurs during the transmission process or on the Elasticsearch index side.

[0051] When the data platform is MySQL and Hive data warehouse, S110 includes:

[0052] S1121: When a newly created data transmission task for importing MySQL data into a Hive data warehouse is detected, a sqoop configuration file corresponding to the data transmission task is obtained through a sqoop component.

[0053] S1122: Obtain the source and destination information of the data set through the sqoop configuration file.

[0054] S1123: Generate two corresponding node information according to the name of the origin and destination of the data set.

[0055] S1124: Generate a mapping relationship between the origin and the destination according to the location information of the origin and the destination of the data set, as data channel information corresponding to the two node information.

[0056] The sqoop component is a tool for efficiently transferring data between the Hadoop ecosystem (such as Hive) and relational databases (such as MySQL). It can extract data from relational databases to Hadoop-related storage systems (such as Hive) for further analysis and processing, and can also write processed data back from Hadoop to relational databases. When creating a data transfer task to import MySQL data into Hive, a sqoop script is automatically generated, which contains information about the data source and target table.

[0057] Therefore, by parsing the automatically collected sqoop configuration file for transmitting data from MySQL to Hive, MySQL related information (data table, database, person in charge, update time, creation time, etc.) can be extracted, and a data node with a data type of 'MySQL' can be created based on this information.

[0058] At the same time, the corresponding data connection channel (Hive-to-es) can be determined through the parsed information to clarify which tables or data ranges in the MySQL database are the data source, and which database and table structure in Hive is the target, and to establish a mapping relationship between the MySQL table and the Hive table as the MySQL-Hive lineage path, that is, the data channel information.

[0059] When the data platform is Kylin and Hive data warehouse, S110 includes:

[0060] Kylin is an open source distributed analysis engine that builds data into multidimensional data sets (cubes) through pre-computation technology, thereby achieving sub-second query response speed. Kylin-API is a set of interfaces provided by Kylin that allows external systems to interact with Kylin and obtain various information inside Kylin. Through Kylin-API, you can obtain key information in Kylin, such as cubes and their source information, to understand how a specific cube is built and which Hive tables it depends on, so as to establish related node dependencies.

[0061] S1131: When the preset node information acquisition instruction is obtained, the cube list query interface in Kylin-API is called to obtain the basic information of all cubes in the target Kylin, including the cube name and the dependent Hive table.

[0062] S1132: Generate two corresponding node information respectively according to the cube name and the name information of the dependent Hive table.

[0063] S1133: Generate a mapping relationship between the source and destination of the data set according to the Hive table to which the cube depends, as data channel information corresponding to the two node information.

[0064] Specifically, you can use the interface of querying cube list in Kylin-API, and parse and process the data in JSON format returned by the interface to obtain the basic information of all cubes in the Kylin instance, including cube name, project, dimension, metric, creation time, update time, dependent Hive table and other key attributes. And based on the above node information, you can construct a data node with data type 'kylin_cube_data'.

[0065] At the same time, you can also obtain the Hive table name or identifier based on the Hive table information of the data source recorded in the basic information of the above cube, and interact with the metadata of the Hive table to further obtain detailed information of the Hive table, and finally establish a connection relationship between the two nodes.

[0066] S120: Obtain node information and data channel information for each data set flow by monitoring preset operation behaviors in the Hive data warehouse.

[0067] Specifically, S120 includes:

[0068] S121: Monitor and obtain each operation instruction of the Hive data warehouse.

[0069] For example, you can obtain every operation instruction of the Hive data warehouse through the operation log file of the Hive data warehouse.

[0070] S123: Obtain the flow information of the data set between different nodes in the Hive data warehouse by parsing each operation instruction.

[0071] S124: Generate node information and data channel information corresponding to the data set flow according to the flow information.

[0072] The operation log of the Hive data warehouse records the various operations performed by the user in Hive, including query, insert, delete, table creation and modification, etc. The Hive data warehouse uses a query language similar to SQL, called HiveQL (HiveQuery Language). The syntax of HiveQL is very close to standard SQL, so users familiar with SQL can easily get started. HiveQL supports common SQL operations, such as creating tables, inserting data, querying data, updating and deleting data, etc. Therefore, by parsing the relevant operation instructions in the operation log file, the flow information of the data set inside the Hive data warehouse can be clearly known, and then through this information, the corresponding node information and the data channel information between the relevant nodes are generated, and then the acquisition of the flow dependency relationship of the data set inside the Hive data warehouse is completed. By fusing the dependency relationship inside the Hive data warehouse in this step with the dependency relationship outside the Hive data warehouse in the above steps, the flow dependency relationship of all data sets in the multi-source data processing system in this embodiment can be generated.

[0073] S200: Create a graph data structure corresponding to the data set dependency relationship in the multi-source data processing system based on all node information and data channel information, and store it in the HBase database.

[0074] HBase is a distributed, column-oriented open source database that runs on Hadoop's HDFS. HBase is designed for storing and processing large-scale data sets, and is particularly suitable for processing sparse data and scenarios that require real-time reading and writing. When querying related dependency chains later, you can obtain information about the related dependency chains more quickly, which can improve query speed. In addition, HBase's tight integration with the Hadoop ecosystem makes it easy to work with other big data processing tools.

[0075] In the existing lineage dependency capture system, node information collection is limited to node information within the Hive data warehouse. In order to broaden the lineage dependency path, the present invention builds a complete link of lineage dependency from data source access, data processing, data modeling to data application, and labels the data types for data nodes. It is mainly divided into Hive, Kylin, Kafka, es and MySQL. In a complex data lineage dependency network, different types of nodes have different attributes and processing methods. Clear node data types can make the construction and analysis of data lineage relationships more accurate and efficient.

[0076] S300: In response to a query request for target data set dependency information, obtain information in a dependency chain to which the target data set belongs in an HBase database.

[0077] like Figure 3 As shown, S400: write the node information of each level into the corresponding level information window in sequence according to the level order in the dependency chain. If the length of all node information in the level is greater than the length of the corresponding level information window, use the scroll bar to display different node information in the level information window. Among them, the main node is the data node currently queried.

[0078] S500: Generate flow indicator lines between each node information in the current level information window according to the data channel information in the dependency chain.

[0079] Since a multi-source data processing system usually has very complex business data, the corresponding data node dependencies are also extremely rich, so it is possible that a large amount of node information will appear in a certain level. Correspondingly, in order to display all the node information, the font ratio of the current information will be reduced, which is inconvenient for users to read and view. When the present invention displays the relevant information in the dependency chain, the size specifications of the window corresponding to each level are consistent. When the length of all node information in the level is greater than the length of the corresponding level information window, only the information in the window is displayed, and the remaining node information is displayed using a scroll bar. In this way, the overall layout of the relationship display interface can be ensured to remain stable, and the font size of the displayed information can also be ensured to remain consistent, which is convenient for viewing and use.

[0080] As another possible embodiment of the present invention, Figure 2 As shown, a method for determining the data node information modification authority is also provided, and the method includes the following steps:

[0081] S600: Generate an influence value I corresponding to the data node to be changed according to the dependency chain to which the current data node to be changed belongs; I satisfies the following conditions:

[0082]

[0083] Among them, C x is the number of node levels downstream of the data node to be changed in the dependency chain, C sum is the total number of node levels in the dependency chain, JD x JD is the total number of nodes in the dependency chain that are located in the node level below the data node to be changed. y is the preset node quantity threshold. K1 and K2 are the first weight coefficient and the second weight coefficient respectively. K1 <K2。

[0084] In this embodiment, I is mainly determined from two aspects. First, from the node level where the current data node to be modified is located, the position in the entire dependency chain. Generally, the top node level is more likely to affect more downstream node levels. Therefore, through C x / C sum This quantity can roughly reflect the degree of influence of the current data node to be changed on the downstream node level. Secondly, the number of nodes in the next level that have a direct dependency relationship with the current data node to be changed. Generally, if there are more nodes in the next level with a direct dependency relationship, the corresponding change of the current data node to be changed will directly affect more nodes, and the nodes associated with these nodes will indirectly affect a wider range of data nodes.

[0085] And the actual contribution of these two factors to I is different, so it is necessary to adjust the weights of different proportions by setting K1 and K2 to make the final calculation result more accurate. In this embodiment, usually the number of nodes in different types of services or nodes in different locations directly depends on the number of nodes at the next level, so it is necessary to set the corresponding JD according to the actual situation. y In addition, due to C x / C sum The value of is between (0,1), so in order not to x / JD y The calculation range of C is much larger than that of x / C sum The value range of C x / C sum The value of has a smaller influence on the calculation result I, thus, in this embodiment The maximum value of is 1.

[0086] Specifically, J.D. y Follow the steps below to obtain:

[0087] S601: Generate a node quantity threshold corresponding to each data node according to the average value of data nodes in a next layer directly related to each data node in a preset statistical period.

[0088] S602: Generate a corresponding mapping relationship table according to the node quantity threshold corresponding to each data node.

[0089] S603: Obtain the node quantity threshold corresponding to each data node from the mapping relationship table.

[0090] Specifically, in this embodiment, JD yIt is also possible to obtain the mode, median, etc. of the data nodes in the next level directly related to the data node, which can reflect the number of data nodes in the next level directly related under normal circumstances.

[0091] S700: Generate the cold and heat value L corresponding to the data node to be changed according to the access volume of each data node in the system during the preset historical period. L satisfies the following conditions:

[0092]

[0093] Among them, W g is the access volume of the data node to be changed during the preset historical period, and W sum is the total access volume of each data node in the system during the preset historical period. The preset historical period can be each month or every few weeks.

[0094] L can reflect the usage frequency of the data in the corresponding node within a certain period of time. Usually, due to business updates and replacements, changes in business heat, and adjustments of reference relationships, the usage frequency of some data nodes will inevitably decrease or even stop being used. Therefore, L can also reflect the importance of the data node from the perspective of actual usage.

[0095] S800: Generate the change permission Q corresponding to the current data node to be changed according to I and L. Q satisfies the following conditions:

[0096] Q = I + K3 × L;

[0097] K3 is the third weight coefficient, and K1 < K3 < K2. K1, K3, and K2 can all take values between (0, 1), such as K1 = 0.2, K3 = 0.4, and K2 = 0.5.

[0098] After generating the change permission Q corresponding to the current data node to be changed, the method further includes:

[0099] S900: Obtain the configuration permission Q of the user who requests to perform a change operation on the data node to be changed uid .

[0100] S901: If Q uid > Q, then allow the user to perform a change operation on the data in the data node to be changed.

[0101] S902: If Q uid ≤ Q, then reject the user from performing a change operation on the data in the data node to be changed.

[0102] The above change operation includes operations such as modifying, adding, or deleting the data in the data node to be changed.

[0103] In this embodiment, the importance of the data in the data node to be changed in the entire system is roughly determined based on the two factors of the level of the data node to be changed corresponding to the current time in the dependency chain to which it belongs, and the number of directly related nodes in the next level. In addition, the current popularity of the data in the data node to be changed is determined by the ratio of the number of visits to the data node to be changed in the corresponding preset historical period to the overall number of visits. Finally, the change permission corresponding to the current data node to be changed is obtained after comprehensive consideration and calculation of the importance and current popularity of the data in the data node to be changed in the entire system. In the present invention, the change permission corresponding to each data node can be updated every period of time, so that the change permission of each data node can be more closely aligned with the actual operation of the current data processing system, and thus the staff can be more accurately guided to maintain and manage the data nodes in this system.

[0104] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0105] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0106] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0107] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0108] The electronic device according to this embodiment of the present invention is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0109] The electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: the at least one processor mentioned above, the at least one storage device mentioned above, and a bus connecting different system components (including storage devices and processors).

[0110] The storage stores program codes, which can be executed by the processor, so that the processor executes the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0111] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read only memory (ROM).

[0112] The storage may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0113] The bus may represent one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0114] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device (e.g., routers, modems, etc.) that enables the electronic device to communicate with one or more other computing devices. Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0115] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0116] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0117] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0118] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0119] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0120] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0121] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0122] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0123] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for generating data flow information, characterized in that: Used to generate dependency information of data sets in a multi-source data processing system, wherein the multi-source data processing system includes a Kafka message queue, HDFS, a Hive data warehouse, and data management platforms on multiple application sides, wherein the Kafka message queue is connected to the Hive data warehouse through HDFS, and the Hive data warehouse is connected to the data management platforms on multiple application sides respectively; The method for generating data flow information comprises the following steps: By monitoring the preset operation behavior in the multi-source data processing system, the node information and data channel information of each data set flow are obtained; the node information is the storage node information during the data set flow process; the data channel information is the flow dependency relationship information between the data sending node and the data receiving node during the data set flow process; The preset operation behavior includes data reading and data transmission behavior; Based on all the node information and data channel information, a graph data structure corresponding to the data set dependency relationship in the multi-source data processing system is created and stored in the HBase database; In response to a query request for dependency information of a target data set, obtaining information in a dependency chain to which the target data set belongs in an HBase database; According to the hierarchical order in the dependency chain, the node information of each level is sequentially written into the corresponding hierarchical information window; If the length of all node information in a level is greater than the length of the corresponding level information window, a scroll bar is used to display different node information in the level information window; According to the data channel information in the dependency chain, a flow indicator line between each node information in the current level information window is generated.

2. The method according to claim 1, characterized in that By monitoring the preset operation behavior in the multi-source data processing system, node information and data channel information of each data set flow are obtained, including: By monitoring the preset operation behaviors between different types of data platforms in the multi-source data processing system, the node information and data channel information of each data set flow can be obtained; and By monitoring the preset operation behaviors in the Hive data warehouse, the node information and data channel information of each data set flow can be obtained.

3. The method according to claim 2, characterized in that When the data platform is a Kafka message queue and a Hive data warehouse, the node information and data channel information of each data set flow can be obtained by monitoring the preset operation behaviors between different types of data platforms in the multi-source data processing system, including: When a newly created Kafka connection is detected, obtain the current topic name in the Kafka message queue and the HDFS storage path corresponding to each topic; Get the table name of each newly created Hive table and the HDFS source path of the table data through the configuration information of the newly created Hive table in the Hive data warehouse; Generate corresponding node information according to the subject name and table name; The corresponding relationship between each topic name and table name is determined by matching the HDFS source path with the HDFS storage path to generate data channel information between corresponding node information.

4. The method according to claim 2, characterized in that: When the data platform is an Es or Hive data warehouse, the node information and data channel information of each data set flow are obtained by monitoring the preset operation behaviors between different types of data platforms in the multi-source data processing system, including: When a newly created data transmission task between Es and Hive data warehouse is detected, the datax-json configuration file corresponding to the data transmission task is obtained through the datax tool; Obtain the origin and destination information of the dataset through the datax-json configuration file; Generate two corresponding node information respectively according to the name of the place of origin and the destination of the data set; According to the location information of the sending place and the destination of the data set, a mapping relationship between the sending place and the destination is generated as the data channel information corresponding to the two node information.

5. The method according to claim 2, characterized in that: When the data platform is MySQL and Hive data warehouse, the node information and data channel information of each data set flow are obtained by monitoring the preset operation behaviors between different types of data platforms in the multi-source data processing system, including: When a newly created data transmission task of importing MySQL data into the Hive data warehouse is detected, the sqoop configuration file corresponding to the data transmission task is obtained through the sqoop component; Obtain the source and destination information of the data set through the sqoop configuration file; Generate two corresponding node information respectively according to the name of the place of origin and the destination of the data set; According to the location information of the sending place and the destination of the data set, a mapping relationship between the sending place and the destination is generated as the data channel information corresponding to the two node information.

6. The method according to claim 2, characterized in that When the data platform is Kylin and Hive data warehouse, the node information and data channel information of each data set flow are obtained by monitoring the preset operation behaviors between different types of data platforms in the multi-source data processing system, including: When the preset node information acquisition instruction is obtained, the cube list query interface in Kylin-API is called to obtain the basic information of all cubes in the target Kylin, including the cube name and the dependent Hive table; Generate two corresponding node information according to the cube name and the name information of the dependent Hive table; According to the Hive table that the cube depends on, a mapping relationship between the source and destination of the data set is generated as the data channel information corresponding to the two node information.

7. The method according to claim 2, characterized in that By monitoring the preset operation behavior in the Hive data warehouse, the node information and data channel information of each data set flow are obtained, including: Monitor and obtain every operation instruction of Hive data warehouse; By parsing each operation instruction, the flow information of the data set between different nodes in the Hive data warehouse is obtained; According to the flow information, node information and data channel information corresponding to the data set flow are generated.

8. The method according to claim 7, characterized in that Monitor and obtain each operation instruction of the Hive data warehouse, including: Obtain each operation instruction of the Hive data warehouse through the operation log file of the Hive data warehouse.

9. A non-transitory computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a method for generating data flow information as described in any one of claims 1 to 8 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for generating data flow information as described in any one of claims 1 to 8 is implemented.

Images