Blood relationship analysis method and device, computer device and storage medium

Abstract

The application belongs to the fields of big data and financial technology, and relates to a blood relationship analysis method, which comprises the following steps: obtaining task dependency information of a data warehouse task from a data warehouse; obtaining table field dependency information of a task data table corresponding to the data warehouse task from the data warehouse; importing the task dependency information and the table field dependency information into a target data table; importing the task dependency information and the table field dependency information in the target data table into a calling graph database; performing graph calculation processing on the task dependency information and the table field dependency information through the graph database to generate a blood relationship graph. The application also provides a blood relationship analysis device, a computer device and a storage medium. In addition, the application also relates to blockchain technology, and the blood relationship graph can be stored in the blockchain. The application can be applied to a blood analysis scene in the financial field, can automatically and quickly generate a blood relationship graph, and effectively improves the generation efficiency of the blood relationship graph.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data and the field of financial technology, and in particular to a blood relationship analysis method and device, a computer device and a storage medium. BACKGROUND

[0002] Data bloodline describes the association between data and the flow of data in the life cycle of data generation, transmission, use, storage, sharing and destruction. Data bloodline can be used for data traceability analysis, impact analysis and importance analysis in data application and governance. At present, in financial technology companies such as insurance companies and banks, there is usually a demand for analyzing the data bloodline in the internal data warehouse. The existing analysis and processing method of the data bloodline in the data warehouse in the financial technology company is usually to set up a special analysis post and manually sort and analyze each data in the data warehouse to build the corresponding blood relationship. However, such processing method has the problem of low data bloodline analysis efficiency. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a blood relationship analysis method, device, computer device and storage medium to solve the technical problem of low data bloodline analysis efficiency of the existing processing method of manually sorting and analyzing each data in the data warehouse to build the corresponding blood relationship.

[0004] To solve the above technical problem, the embodiments of the present application provide a blood relationship analysis method, which adopts the following technical solution:

[0005] Obtain task dependency information of a data warehouse task from a preset data warehouse;

[0006] Obtain table field dependency information of a task data table corresponding to the data warehouse task from the data warehouse;

[0007] Import the task dependency information and the table field dependency information into a preset target data table;

[0008] Call a preset graph database;

[0009] Import the task dependency information and the table field dependency information in the target data table into the graph database;

[0010] Perform graph calculation processing on the task dependency information and the table field dependency information through the graph database to generate a corresponding blood relationship graph.

[0011] Further, the step of obtaining task dependency information of a data warehouse task from a preset data warehouse specifically includes:

[0012] determining a target data layer from the data warehouse;

[0013] obtaining a scheduling task of the target data layer, and taking the scheduling task as the data warehouse task;

[0014] obtaining task dependency information corresponding to the data warehouse task from the data warehouse.

[0015] Further, the step of obtaining table field dependency information of a task data table corresponding to the data warehouse task from the data warehouse specifically comprises:

[0016] obtaining a task data table corresponding to the data warehouse task from the data warehouse;

[0017] performing information analysis on the task data table to obtain a dependency relationship between table fields corresponding to the task data table;

[0018] taking the dependency relationship between the table fields as the table field dependency information.

[0019] Further, the step of performing graph calculation processing on the task dependency information and the table field dependency information through the graph database to generate a corresponding blood relationship graph specifically comprises:

[0020] obtaining a task name of the data warehouse task;

[0021] obtaining a table name of the task data table;

[0022] constructing corresponding nodes in the graph database based on the task name and the table name;

[0023] generating an association relationship between the nodes based on the task dependency information and the table field dependency information;

[0024] constructing a blood relationship graph corresponding to the nodes in the graph database based on the association relationship.

[0025] Further, after the step of performing graph calculation processing on the task dependency information and the table field dependency information through the graph database to generate a corresponding blood relationship graph, the method further comprises:

[0026] obtaining a target storage type corresponding to the blood relationship graph;

[0027] determining a target storage medium corresponding to the target storage type;

[0028] storing the blood relationship graph through the target storage medium.

[0029] Further, after the step of storing the blood relationship graph in the target storage medium, specifically comprising:

[0030] determining whether the target storage medium meets a preset cleaning condition;

[0031] If yes, determining the data to be cleaned from the target storage medium;

[0032] cleaning the data to be cleaned.

[0033] Further, after the step of performing graph calculation processing on the task dependency information and the table field dependency information in the graph database to generate a corresponding blood relationship graph, further comprising:

[0034] determining whether an editing instruction for the blood relationship graph triggered by a user is received;

[0035] If yes, performing permission verification on the user based on a preset permission analysis model;

[0036] If the permission verification is passed, receiving a modification operation for the blood relationship graph input by the user;

[0037] modifying the blood relationship graph based on the modification operation to obtain a target blood relationship graph;

[0038] storing the target blood relationship graph.

[0039] To solve the above technical problems, the embodiments of the present application also provide a blood relationship analysis device, which adopts the technical scheme as follows:

[0040] A first acquisition module is configured to acquire task dependency information of a data warehouse task from a preset data warehouse;

[0041] A second acquisition module is configured to acquire table field dependency information of a task data table corresponding to the data warehouse task from the data warehouse;

[0042] A first import module is configured to import the task dependency information and the table field dependency information into a preset target data table;

[0043] A calling module is configured to call a preset graph database;

[0044] A second import module is configured to import the task dependency information and the table field dependency information in the target data table into the graph database;

[0045] The generation module is configured to perform graph calculation processing on the task dependency information and the table field dependency information by using the graph database to generate a corresponding blood relationship graph.

[0046] To solve the above technical problems, the embodiment of the application further provides a computer device, which adopts the technical scheme as follows:

[0047] Obtain task dependency information of a data warehouse task from a preset data warehouse;

[0048] Obtain table field dependency information of a task data table corresponding to the data warehouse task from the data warehouse;

[0049] Import the task dependency information and the table field dependency information into a preset target data table;

[0050] Call a preset graph database;

[0051] Import the task dependency information and the table field dependency information in the target data table into the graph database;

[0052] Perform graph calculation processing on the task dependency information and the table field dependency information by using the graph database to generate a corresponding blood relationship graph.

[0053] To solve the above technical problems, the embodiment of the application further provides a computer readable storage medium, which adopts the technical scheme as follows:

[0054] Obtain task dependency information of a data warehouse task from a preset data warehouse;

[0055] Obtain table field dependency information of a task data table corresponding to the data warehouse task from the data warehouse;

[0056] Import the task dependency information and the table field dependency information into a preset target data table;

[0057] Call a preset graph database;

[0058] Import the task dependency information and the table field dependency information in the target data table into the graph database;

[0059] Perform graph calculation processing on the task dependency information and the table field dependency information by using the graph database to generate a corresponding blood relationship graph.

[0060] Compared with the prior art, the embodiment of the application has the following beneficial effects:

[0061] This embodiment first obtains task dependency information of data warehouse tasks from a preset data warehouse; then, it obtains table field dependency information of the task data table corresponding to the data warehouse task from the data warehouse; next, it imports the task dependency information and the table field dependency information into a preset target data table; subsequently, it calls a preset graph database and imports the task dependency information and the table field dependency information from the target data table into the graph database; finally, it performs graph calculation processing on the task dependency information and the table field dependency information through the graph database to generate a corresponding lineage graph. This embodiment obtains task dependency information of data warehouse tasks and table field dependency information of the task data table corresponding to the data warehouse task from the data warehouse, and then performs graph calculation processing on the task dependency information and the table field dependency information based on the graph database to automatically and quickly generate the corresponding lineage graph, effectively improving the generation efficiency of the lineage graph. Furthermore... By using a graphical database, the lineage of tasks in a data warehouse can be visually displayed, providing a reliable basis for data developers' encryption work and greatly reducing the time cost and error probability of manual retrieval and analysis. Attached Figure Description

[0062] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0064] Figure 2 A flowchart of an embodiment of the blood relationship analysis method according to this application;

[0065] Figure 3 This is a schematic diagram of one embodiment of the blood relationship analysis device according to this application;

[0066] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application; the use herein of terms such as "comprise" and "comprising", "have" and "having", "include" and "including" and "contain" and "containing" are to be construed in an open, non-exhaustive way, i.e. in the sense of "including, but not limited to"; the use herein of terms such as "first", "second", and "other" are used to distinguish one object from another, and are not used to describe a particular sequential order.

[0068] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another.

[0069] For better understanding of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings.

[0070] As shown in Figure 1 The system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0071] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0072] The terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop portable computers, and desktop computers, etc.

[0073] The server 105 can be a server providing various services, for example, a background server providing support for a page displayed on the terminal device 101, 102, 103.

[0074] It should be noted that the blood relationship analysis method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the blood relationship analysis apparatus is generally arranged in a server / terminal device.

[0075] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0076] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0077] It should be understood that Figure 1 The number of terminal devices, networks and servers in

[0078] With reference to Figure 2 , a flowchart of one embodiment of the blood relationship analysis method according to the present application is shown. The order of the steps in the flowchart can be changed, and some steps can be omitted according to different needs. The blood relationship analysis method provided in the embodiments of the present application can be applied to any scenario requiring blood relationship analysis, and therefore the blood relationship analysis method can be applied to products in these scenarios, for example, data blood relationship analysis in a data warehouse in the financial insurance field. The blood relationship analysis method includes the following steps:

[0079] Step S201: acquiring task dependency information of a data warehouse task from a preset data warehouse.

[0080] In this embodiment, the electronic device (for example Figure 1The task dependency information can be acquired through a wired connection or a wireless connection. The execution subject of the blood relationship analysis method can be a task scheduling platform in an electronic device. It should be noted that the wireless connection can include, but is not limited to, 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other now known or future developed wireless connection methods. The data warehouse is a data warehouse using big data technology. In the financial field, the data warehouse can store business data, transaction data, payment data, and the like. In addition, the specific implementation process of acquiring the task dependency information of the data warehouse task from the preset data warehouse will be further described in detail in subsequent specific embodiments, and will not be described here.

[0081] In step S202, table field dependency information of a task data table corresponding to the data warehouse task is acquired from the data warehouse.

[0082] In this embodiment, the specific implementation process of acquiring the table field dependency information of the task data table corresponding to the data warehouse task from the data warehouse will be further described in detail in subsequent specific embodiments, and will not be described here.

[0083] In step S203, the task dependency information and the table field dependency information are imported into a preset target data table.

[0084] In this embodiment, the target data table can be a hive data table. Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation, and loading. It is a mechanism that can store, query, and analyze large-scale data stored in Hadoop. The hive data warehouse tool can map structured data files to a database table and provide SQL query functions, and can convert SQL statements into MapReduce tasks for execution. The advantage of Hive is low learning cost. It can realize fast MapReduce statistics through similar SQL statements, making MapReduce more simple without developing special MapReduce applications. Hive is very suitable for statistical analysis of data warehouses.

[0085] In step S204, a preset graph database is called.

[0086] In the embodiment, the graph database can be Neo4j. Neo4j is a high-performance NoSQL graph database, which is a network-oriented database. The network is constructed by nodes, and each node has corresponding attributes and relationships between nodes.

[0087] In step S205, the task dependency information in the target data table and the table field dependency information are imported into the graph database.

[0088] In the embodiment, the Neo4j graph database is a graph database supporting external data import.

[0089] In step S206, the graph database is used to perform graph calculation processing on the task dependency information and the table field dependency information to generate a corresponding blood relationship graph.

[0090] In the embodiment, the specific implementation process of performing graph calculation processing on the task dependency information and the table field dependency information by the graph database to generate a corresponding blood relationship graph will be further described in detail in subsequent embodiments, and will not be described in detail here.

[0091] The application first obtains task dependency information of a data warehouse task from a preset data warehouse. Then, table field dependency information of a task data table corresponding to the data warehouse task is obtained from the data warehouse. Then, the task dependency information and the table field dependency information are imported into a preset target data table. Subsequently, a preset graph database is called, and the task dependency information and the table field dependency information in the target data table are imported into the graph database. Finally, the graph database is used to perform graph calculation processing on the task dependency information and the table field dependency information to generate a corresponding blood relationship graph. By obtaining the task dependency information of the data warehouse task and the table field dependency information of the task data table corresponding to the data warehouse task from the data warehouse, and then performing graph calculation processing on the task dependency information and the table field dependency information based on the graph database, the application can automatically and quickly generate a corresponding blood relationship graph, thereby effectively improving the generation efficiency of the blood relationship graph. In addition, by using the graph database, the blood relationship of the task link in the data warehouse can be intuitively displayed in a graphical manner, thereby providing effective support for the work of data developers, and greatly reducing the time cost and error probability of manual search and analysis of developers.

[0092] In some optional implementations, step S201 includes the following steps:

[0093] A target data layer is determined from the data warehouse.

[0094] In the embodiment, the hierarchical layers of the data warehouse include ODS->DWD / DWS->DM, and there is a dependency relationship between the two data layers of the upstream and downstream, and there are tens of thousands of T+1 scheduling tasks in each data layer every day, and the scheduling tasks are scheduled through a task scheduling platform. The target data layer refers to the ODS layer and the DM layer.

[0095] Obtain the scheduling tasks of the target data layer, and take the scheduling tasks as the data warehouse tasks.

[0096] In the embodiment, the scheduling tasks of the ODS layer and the DM layer can be extracted from all the tasks contained in the data warehouse, and taken as the data warehouse tasks.

[0097] Obtain the task dependency information corresponding to the data warehouse tasks from the data warehouse.

[0098] In the embodiment, the dependency relationship between the data warehouse tasks is maintained in the task scheduling platform, so that the task dependency information corresponding to the data warehouse tasks can be obtained from the data warehouse.

[0099] The application determines the target data layer from the data warehouse, then obtains the scheduling tasks of the target data layer, and takes the scheduling tasks as the data warehouse tasks, and subsequently obtains the task dependency information corresponding to the data warehouse tasks from the data warehouse. By determining the target data layer in the data warehouse, the scheduling tasks of the target data layer can be queried, so that the task dependency information corresponding to the data warehouse tasks can be quickly and conveniently obtained from the data warehouse.

[0100] In some optional implementation manners of the embodiment, step S202 includes the following steps:

[0101] Obtain the task data table corresponding to the data warehouse tasks from the data warehouse.

[0102] In the embodiment, the bottom layer of each data warehouse task corresponds to a task data table with sql synchronization logic, and by analyzing the task data table, the dependency relationship between the table fields can be obtained.

[0103] Perform information analysis on the task data table to obtain the dependency relationship between the table fields corresponding to the task data table.

[0104] Take the dependency relationship between the table fields as the table field dependency information.

[0105] The application obtains a task data table corresponding to the data warehouse task from the data warehouse, then performs information analysis on the task data table to obtain a dependency relationship between table fields corresponding to the task data table, and subsequently takes the dependency relationship between the table fields as the table field dependency information. The application performs information analysis on the task data table corresponding to the data warehouse task obtained from the data warehouse, so that the table field dependency information of the task data table can be quickly and conveniently obtained.

[0106] In some optional implementations, step S206 includes the following steps:

[0107] Obtaining a task name of the data warehouse task.

[0108] In this embodiment, the task name of the data warehouse task can be extracted from task description information of the data warehouse task.

[0109] Obtaining a table name of the task data table.

[0110] In this embodiment, the task name of the data warehouse task can be extracted from data table description information of the task data table.

[0111] Constructing corresponding nodes in the graph database based on the task name and the table name.

[0112] In this embodiment, the task name of the data warehouse task and the table name of the task data table are taken as nodes in the graph database. In addition, the task node attribute is the task group and the task person in charge, and the table node attribute is the table encryption field.

[0113] Generating an association relationship between the nodes based on the task dependency information and the table field dependency information.

[0114] In this embodiment, the dependency relationship between the data warehouse tasks and the dependency relationship between the task data tables can be obtained based on the task dependency information and the table field dependency information. Furthermore, the association relationship between the nodes can be obtained based on the dependency relationship between the data warehouse tasks and the dependency relationship between the task data tables.

[0115] Constructing a blood relationship graph corresponding to the nodes in the graph database based on the association relationship.

[0116] In this embodiment, the nodes can be outlined in the graph database based on the association relationship, so as to construct a blood relationship graph containing the upstream and downstream relationships of the nodes.

[0117] The application obtains a task name of the data warehouse task and a table name of the task data table, then constructs corresponding nodes in the graph database based on the task name and the table name, generates an association relationship between the nodes based on the task dependency information and the table field dependency information, and constructs a blood relationship graph corresponding to the nodes in the graph database based on the association relationship. The application uses the graph database to perform graph calculation processing on the task dependency information and the table field dependency information, which can quickly generate a blood relationship graph, and improves the generation efficiency and intelligence of the blood relationship graph.

[0118] In some optional implementations, after step S206, the electronic device can further perform the following steps:

[0119] Obtain a target storage type corresponding to the blood relationship graph.

[0120] In this embodiment, the target storage type is specifically a graph type.

[0121] Determine a target storage medium corresponding to the target storage type.

[0122] In this embodiment, a plurality of storage media for storing different types of data are pre-set. The storage media can include a database, a cloud, a network disk, a blockchain, etc. A specified storage medium matching the target storage medium can be obtained from all the storage media as the target storage medium. In addition, the storage media can be used to store business data, transaction data, payment data, etc. of business systems such as insurance systems, banking systems, transaction systems, and order systems, to ensure the security of the business data, transaction data, and payment data.

[0123] Store the blood relationship graph through the target storage medium.

[0124] In this embodiment, a storage block can be added in the target storage medium, and the blood relationship graph can be stored in the storage block.

[0125] The application obtains a target storage type corresponding to the blood relationship graph, then determines a target storage medium corresponding to the target storage type, and subsequently stores the blood relationship graph through the target storage medium. The application determines the target storage medium by using the target storage type of the blood relationship graph, and then stores the blood relationship graph using the target storage medium, which can effectively improve the storage standardization and intelligence of the blood relationship graph.

[0126] In some optional implementations of the embodiment, after the step of storing the blood relationship graph in the target storage medium, the electronic device can further perform the following steps:

[0127] determining whether the target storage medium meets a preset cleaning condition.

[0128] In the embodiment, the process of determining whether the target storage medium meets the preset cleaning condition includes: obtaining a current available storage space of the target storage medium; determining whether the available storage space is less than a preset space threshold; if yes, determining that the target storage medium meets the cleaning condition, otherwise, determining that the target storage medium does not meet the cleaning condition. The value of the space threshold is not specifically limited and can be generated according to actual use test results. In addition, if the current available storage space of the storage medium is less than the space threshold, it indicates that the current available storage resource of the storage medium is insufficient and will affect the normal operation of the data in the storage medium.

[0129] if yes, determining the data to be cleaned from the target storage medium.

[0130] In the embodiment, the process of determining the data to be cleaned from the target storage medium can include: obtaining the usage frequency of each data included in the target storage medium in a preset time period; obtaining the data size of each data; generating an activity value of each data based on the usage frequency and the data size; selecting target data with an activity value less than a preset threshold from all the data, and taking the target data as the data to be cleaned. The value of the preset time period is not specifically limited and can be set according to actual business use requirements, for example, the previous two months from the current time. The value of the preset threshold is not specifically limited and can be set according to actual business use requirements. In addition, the ratio between the usage frequency of the data and the data size of the data can be calculated, and the ratio is taken as the activity value of the data.

[0131] cleaning the data to be cleaned.

[0132] In the embodiment, a cleaning time point can be set in advance, and the data to be cleaned is cleaned when the current time reaches the cleaning time point.

[0133] The application determines whether the target storage medium meets preset cleaning conditions; if yes, determines the data to be cleaned from the target storage medium; and subsequently cleans the data to be cleaned. When the target storage medium meets the cleaning conditions, the application intelligently cleans the data with a small activity value in the target storage medium, so as to ensure that sufficient available resource space can be kept in the target storage medium, thereby facilitating the stability of data operation in the target storage medium.

[0134] In some optional implementations of the embodiment, after step S206, the electronic device can further perform the following steps:

[0135] It is judged whether a user triggered editing instruction for the blood relationship graph is received.

[0136] In the embodiment, the editing instruction is an instruction issued by a user to edit the blood relationship graph.

[0137] If yes, the user is verified for permission based on a preset permission analysis model.

[0138] In the embodiment, the editing instruction can carry user information of the user, such as a user name or a user number. The user information carried in the editing instruction can be obtained, and the user information is input into the permission analysis model, so that the user information is analyzed for permission by the permission analysis model, and a permission analysis result of the user is obtained. The permission analysis model is a data model constructed according to a correspondence between user information and graph editing permission.

[0139] If the permission verification is passed, a modification operation for the blood relationship graph input by the user is received.

[0140] In the embodiment, the modification operation can include adding, deleting, replacing, and the like.

[0141] The blood relationship graph is modified based on the modification operation to obtain a target blood relationship graph.

[0142] The target blood relationship graph is stored.

[0143] In the embodiment, the storage manner of the target blood relationship graph can refer to the storage manner of the blood relationship graph, which will not be described in detail herein.

[0144] The application determines whether a user triggers an editing instruction for the blood relationship graph, if yes, performs permission verification on the user based on a preset permission analysis model, if the permission verification is passed, receives a modification operation for the blood relationship graph input by the user, then modifies the blood relationship graph based on the modification operation to obtain a target blood relationship graph, and subsequently stores the target blood relationship graph. After the blood relationship graph is generated based on the use of the graph database, the application further provides an editing function for the blood relationship graph, which can intelligently modify the blood relationship graph according to the modification operation for the blood relationship graph input by the user, so as to quickly and conveniently generate a target blood relationship graph meeting the requirements, and facilitate to improve the operation experience of the blood relationship graph and the use experience of the user.

[0145] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0146] It should be emphasized that, in order to further ensure the privacy and security of the above blood relationship graph, the above blood relationship graph can also be stored in a node of a block chain.

[0147] The block chain referred to in the application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. The block chain (block chain) is essentially a decentralized database, which is a series of data blocks associated using cryptographic methods, each data block contains information of a batch of network transactions, used to verify the validity (anti-fake) of the information and generate the next block. The block chain can include a block chain underlying platform, a platform product service layer and an application service layer.

[0148] The embodiment of the application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0149] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. Several major directions.

[0150] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through computer readable instructions, and the computer readable instructions can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiment methods. Among them, the storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0151] It should be understood that although each step in the flowchart of the accompanying drawings is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0152] Further reference is made to Figure 3 , as an implementation of the method shown in Figure 2 , the present application provides an embodiment of a blood relationship analysis device, which corresponds to the method embodiment shown in Figure 2 , and the device can be specifically applied to various electronic devices.

[0153] As shown in Figure 3 , the blood relationship analysis device 300 described in the embodiment includes a first acquisition module, a second acquisition module 302, a first import module 303, a calling module 304, a second import module 305, and a generation module 306. Among them:

[0154] The first acquisition module 301 is configured to acquire task dependency information of a data warehouse task from a preset data warehouse;

[0155] The second acquisition module 302 is configured to acquire table field dependency information of a task data table corresponding to the data warehouse task from the data warehouse;

[0156] The first import module 303 is configured to import the task dependency information and the table field dependency information into a preset target data table;

[0157] The calling module 304 is configured to call a preset graph database;

[0158] The second importing module 305 is configured to import the task dependency information in the target data table and the table field dependency information into the graph database.

[0159] The generating module 306 is configured to perform graph calculation processing on the task dependency information and the table field dependency information by using the graph database, and generate a corresponding blood relationship graph.

[0160] In the embodiment, the operations performed by the above modules or units correspond to the steps of the blood relationship analysis method of the foregoing embodiments one by one, and thus will not be described here again.

[0161] In some optional implementations of the embodiment, the first obtaining module 301 includes:

[0162] The first determining sub-module is configured to determine a target data layer from the data warehouse.

[0163] The first obtaining sub-module is configured to obtain a scheduling task of the target data layer, and take the scheduling task as the data warehouse task.

[0164] The second obtaining sub-module is configured to obtain task dependency information corresponding to the data warehouse task from the data warehouse.

[0165] In the embodiment, the operations performed by the above modules or units correspond to the steps of the blood relationship analysis method of the foregoing embodiments one by one, and thus will not be described here again.

[0166] In some optional implementations of the embodiment, the second obtaining module 302 includes:

[0167] The third obtaining sub-module is configured to obtain a task data table corresponding to the data warehouse task from the data warehouse.

[0168] The parsing sub-module is configured to perform information parsing on the task data table to obtain a dependency relationship between table fields corresponding to the task data table.

[0169] The second determining sub-module is configured to take the dependency relationship between the table fields as the table field dependency information.

[0170] In the embodiment, the operations performed by the above modules or units correspond to the steps of the blood relationship analysis method of the foregoing embodiments one by one, and thus will not be described here again.

[0171] In some optional implementations of the embodiment, the generating module 306 includes:

[0172] The fourth obtaining sub-module is configured to obtain a task name of the data warehouse task.

[0173] A fifth obtaining sub-module is configured to obtain a table name of the task data table;

[0174] A first constructing sub-module is configured to construct a corresponding node in the graph database based on the task name and the table name;

[0175] A generating sub-module is configured to generate an association relationship between the nodes based on the task dependency information and the table field dependency information;

[0176] A second constructing sub-module is configured to construct a blood relationship graph corresponding to the nodes in the graph database based on the association relationship.

[0177] In the embodiment, the modules or units are respectively configured to perform operations corresponding to the steps of the blood relationship analysis method of the foregoing embodiments, and thus no further description is provided herein.

[0178] In some optional implementations of the embodiment, the blood relationship analysis apparatus further includes:

[0179] A third obtaining module is configured to obtain a target storage type corresponding to the blood relationship graph;

[0180] A first determining module is configured to determine a target storage medium corresponding to the target storage type;

[0181] A first storage module is configured to store the blood relationship graph by using the target storage medium.

[0182] In the embodiment, the modules or units are respectively configured to perform operations corresponding to the steps of the blood relationship analysis method of the foregoing embodiments, and thus no further description is provided herein.

[0183] In some optional implementations of the embodiment, the blood relationship analysis apparatus further includes:

[0184] A first judging module is configured to judge whether the target storage medium meets a preset cleaning condition;

[0185] A second determining module is configured to determine, if yes, to-be-cleaned data from the target storage medium;

[0186] A cleaning module is configured to clean the to-be-cleaned data.

[0187] In the embodiment, the modules or units are respectively configured to perform operations corresponding to the steps of the blood relationship analysis method of the foregoing embodiments, and thus no further description is provided herein.

[0188] In some optional implementations of the embodiment, the blood relationship analysis apparatus further includes:

[0189] The second determining module is configured to determine whether an editing instruction triggered by a user for the blood relationship graph is received.

[0190] The verifying module is configured to, if yes, perform permission verification on the user based on a preset permission analysis model.

[0191] The receiving module is configured to, if the permission verification is passed, receive a modification operation input by the user for the blood relationship graph.

[0192] The modifying module is configured to modify the blood relationship graph based on the modification operation to obtain a target blood relationship graph.

[0193] The second storage module is configured to store the target blood relationship graph.

[0194] In the embodiment, the modules or units described above are respectively used to perform operations corresponding to steps of the blood relationship analysis method of the foregoing embodiments, and thus will not be described here.

[0195] To solve the above technical problems, the embodiment of the present application further provides a computer device. For details, please refer to Figure 4 , Figure 4 The basic structure block diagram of the computer device of the embodiment is shown in the figure.

[0196] The computer device 4 includes a memory 41, a processor 42 and a network interface 43 which are connected to each other through a system bus. It should be pointed out that only the computer device 4 with components 41-43 is shown in the figure, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented. Among them, the computer device herein is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0197] The computer device can be a desktop computer, a notebook computer, a palm computer and a cloud server, etc. The computer device can interact with the user through a keyboard, a mouse, a remote controller, a touchpad or a voice control device, etc.

[0198] The memory 41 includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit and the external storage device of the computer device 4. In this embodiment, the memory 41 is generally used to store an operating system and various application software installed on the computer device 4, such as computer readable instructions of the blood relationship analysis method, etc. In addition, the memory 41 can also be used to temporarily store various data that have been output or will be output.

[0199] The processor 42 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer readable instructions or process data stored in the memory 41, such as computer readable instructions of the blood relationship analysis method.

[0200] The network interface 43 can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0201] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0202] In the embodiment of the present application, firstly, task dependency information of a data warehouse task is obtained from a preset data warehouse; then, table field dependency information of a task data table corresponding to the data warehouse task is obtained from the data warehouse; then, the task dependency information and the table field dependency information are imported into a preset target data table; subsequently, a preset graph database is called, and the task dependency information and the table field dependency information in the target data table are imported into the graph database; finally, the graph database is used to perform graph calculation processing on the task dependency information and the table field dependency information, and a corresponding blood relationship graph is generated. In the embodiment of the present application, the task dependency information of the data warehouse task and the table field dependency information of the task data table corresponding to the data warehouse task are obtained from the data warehouse, and then the graph calculation processing is performed on the task dependency information and the table field dependency information based on the graph database, so that the corresponding blood relationship graph is automatically and quickly generated, and the generation efficiency of the blood relationship graph is effectively improved. In addition, the blood relationship of the task link in the data warehouse can be intuitively displayed in a graphical manner by using the graph database, so that the data developers can be effectively assisted in work, and the time cost and error probability of manual search and analysis of the developers are greatly reduced.

[0203] The present application also provides another embodiment, that is, a computer readable storage medium storing computer readable instructions, the computer readable instructions being executable by at least one processor to cause the at least one processor to perform the steps of the blood relationship analysis method as described above.

[0204] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0205] In the embodiments of the present application, firstly, task dependency information of a data warehouse task is acquired from a preset data warehouse; then, table field dependency information of a task data table corresponding to the data warehouse task is acquired from the data warehouse; then, the task dependency information and the table field dependency information are imported into a preset target data table; subsequently, a preset graph database is called, and the task dependency information and the table field dependency information in the target data table are imported into the graph database; finally, the task dependency information and the table field dependency information are subjected to graph calculation processing by the graph database, and a corresponding blood relationship graph is generated. In the embodiments of the present application, the task dependency information of a data warehouse task and the table field dependency information of a task data table corresponding to the data warehouse task are acquired from a data warehouse, and then the task dependency information and the table field dependency information are subjected to graph calculation processing based on a graph database, so as to automatically and quickly generate a corresponding blood relationship graph, thereby effectively improving the generation efficiency of the blood relationship graph. In addition, by using the graph database, the blood relationship of a task link in the data warehouse can be intuitively displayed in a graphical manner, thereby providing effective support for the work of data developers, and greatly reducing the time cost and error probability of manual search and analysis of the developers.

[0206] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and a general hardware platform as required, and of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk), and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the methods described in the various embodiments of the present application.

[0207] Obviously, the above-described embodiments are only some of the embodiments of the present application, rather than all the embodiments, and the preferred embodiments of the present application are given in the drawings, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and contrary to the above-described embodiments, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can modify the technical solutions recorded in the foregoing specific embodiments, or equivalently replace some of the technical features. Any equivalent structure made by using the contents of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.

Claims

1. A method of analyzing blood relationships, characterized by, The method comprises the following steps: obtaining task dependency information of a data warehouse task from a preset data warehouse; obtaining table field dependency information of a task data table corresponding to the data warehouse task from the data warehouse; importing the task dependency information and the table field dependency information into a preset target data table; calling a preset graph database; importing the task dependency information and the table field dependency information in the target data table into the graph database; performing graph calculation processing on the task dependency information and the table field dependency information through the graph database to generate a corresponding blood relationship graph; The step of obtaining task dependency information of a data warehouse task from a preset data warehouse comprises: determining a target data layer from the data warehouse; obtaining a scheduling task of the target data layer and taking the scheduling task as the data warehouse task; obtaining task dependency information corresponding to the data warehouse task from the data warehouse; The step of performing graph calculation processing on the task dependency information and the table field dependency information through the graph database to generate a corresponding blood relationship graph comprises: obtaining a task name of the data warehouse task; obtaining a table name of the task data table; constructing corresponding nodes in the graph database based on the task name and the table name; generating an association relationship between the nodes based on the task dependency information and the table field dependency information; constructing a blood relationship graph corresponding to the nodes in the graph database based on the association relationship; The dependency relationship between each data warehouse task and the dependency relationship between each task data table are obtained based on the task dependency information and the table field dependency information, and the association relationship between the nodes is obtained based on the dependency relationship between each data warehouse task and the dependency relationship between each task data table.

2. The blood relationship analysis method according to claim 1, characterized by, The step of obtaining table field dependency information of a task data table corresponding to the data warehouse task from the data warehouse comprises: obtaining a task data table corresponding to the data warehouse task from the data warehouse; performing information analysis on the task data table to obtain a dependency relationship between table fields corresponding to the task data table; taking the dependency relationship between the table fields as the table field dependency information.

3. The blood relationship analysis method according to claim 1, characterized by, After the step of performing graph calculation processing on the task dependency information and the table field dependency information through the graph database to generate a corresponding blood relationship graph, the method further comprises: obtaining a target storage type corresponding to the blood relationship graph; determining a target storage medium corresponding to the target storage type; storing the blood relationship graph through the target storage medium.

4. The blood relationship analysis method according to claim 3, characterized by, After the step of storing the blood relationship graph through the target storage medium, the method specifically comprises: determining whether the target storage medium meets a preset cleaning condition; if yes, determining to-be-cleaned data from the target storage medium; cleaning the to-be-cleaned data.

5. The blood relationship analysis method according to claim 1, characterized by, After the step of performing graph calculation processing on the task dependency information and the table field dependency information by the graph database to generate a corresponding blood relationship graph, the method further includes: determining whether an editing instruction triggered by a user for the blood relationship graph is received; if yes, performing permission verification on the user based on a preset permission analysis model; if the permission verification is passed, receiving a modification operation input by the user for the blood relationship graph; modifying the blood relationship graph based on the modification operation to obtain a target blood relationship graph; storing the target blood relationship graph.

6. A blood relationship analysis device characterized by comprising: The method comprises: a first obtaining module configured to obtain task dependency information of a data warehouse task from a preset data warehouse; a second obtaining module configured to obtain table field dependency information of a task data table corresponding to the data warehouse task from the data warehouse; a first importing module configured to import the task dependency information and the table field dependency information into a preset target data table; a calling module configured to call a preset graph database; a second importing module configured to import the task dependency information and the table field dependency information in the target data table into the graph database; a generating module configured to perform graph calculation processing on the task dependency information and the table field dependency information by the graph database to generate a corresponding blood relationship graph; the generating module comprises: a fourth obtaining submodule configured to obtain a task name of the data warehouse task; a fifth obtaining submodule configured to obtain a table name of the task data table; a first constructing submodule configured to construct corresponding nodes in the graph database based on the task name and the table name; a generating submodule configured to generate an association relationship between the nodes based on the task dependency information and the table field dependency information; a second constructing submodule configured to construct a blood relationship graph corresponding to the nodes in the graph database based on the association relationship; wherein the dependency relationship between each data warehouse task and the dependency relationship between each task data table are obtained based on the task dependency information and the table field dependency information, and the association relationship between each node is further obtained based on the dependency relationship between each data warehouse task and the dependency relationship between each task data table.

7. A computer device comprising a memory and a processor, wherein the memory stores computer readable instructions, and the processor executes the computer readable instructions to implement the steps of the blood relationship analysis method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the steps of the blood relationship analysis method according to any one of claims 1 to 5.

Images