Entity table information processing method, device, equipment and storage medium

Abstract

Embodiments of the present invention disclose a method, apparatus, device, and storage medium for processing entity table information. The method comprises: capturing workflow information in the workflow task via a first hook function during workflow task execution, generating and storing entity table association relationship data based on the workflow information; capturing entity table change information in the workflow task via a second hook function, generating and storing corresponding entity table data based on the entity table change information; and upon receiving a target entity table query instruction, querying and generating feedback results based on the entity table association relationship data and the entity table data, the feedback results including the entity table data and the entity table association relationship data. This solution achieves efficient capture of entity table association relationship data, can provide feedback on entity table association relationships based on queries, and improves the efficiency of querying entity table association data.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for processing entity table information. Background Art

[0002] Data is the cornerstone of the information age. With technological advancements, the scale of data collected is growing. Skilled in processing and mining data for value can help us understand current business development, gain a deeper understanding of user behavior, and identify potential business opportunities. However, as data mining deepens, the data processing chain becomes increasingly complex. When data issues arise, the cost of locating and repairing them increases exponentially. When data formats need to be changed, assessing the scope of impact and locating and notifying relevant downstream dependencies are also pressing challenges in enterprise data management. To reduce data management costs and risks, it is crucial to clearly and accurately organize data relationships.

[0003] Existing techniques primarily use structured query languages ​​(SQLs) to extract and process data when determining data relationships. This SQL is used to parse upstream and downstream entity tables to determine entity table relationships. However, this approach suffers from poor real-time performance and fails to capture upstream and downstream information from execution plans. Another approach involves using a data processing engine to parse workflow execution plans and extract upstream and downstream entity table information. However, this approach severely limits the amount of information collected. Summary of the Invention

[0004] The embodiments of the present invention provide a method, apparatus, device and storage medium for processing entity table information, which solve the problems of poor timeliness and limited information collection in the prior art when processing entity table information, and improve the query efficiency of entity table associated data.

[0005] In a first aspect, an embodiment of the present invention provides a method for processing entity table information, the method comprising:

[0006] When a workflow task is executed, the workflow information in the workflow task is captured by a set first hook function, and entity table association relationship data is generated based on the workflow information and stored;

[0007] The entity table change information in the workflow task is captured by the set second hook function, and corresponding entity table data is generated and stored based on the entity table change information;

[0008] When a target entity table query instruction is received, a query is performed based on the entity table association relationship data and the entity table data to generate a feedback result, where the feedback result includes the entity table data and the entity table association relationship data.

[0009] In a second aspect, an embodiment of the present invention further provides an entity table information processing device, the device comprising:

[0010] A first data capture module is configured to capture workflow information in a workflow task through a set first hook function when the workflow task is executed, generate entity table association relationship data based on the workflow information, and store the data;

[0011] A second data capture module is configured to capture entity table change information in the workflow task through a set second hook function, generate corresponding entity table data based on the entity table change information, and store the data;

[0012] The query processing module is used to generate a feedback result based on the entity table association relationship data and the entity table data when receiving a target entity table query instruction. The feedback result includes the entity table data and the entity table association relationship data.

[0013] In a third aspect, an embodiment of the present invention further provides an entity table information processing device, the device comprising:

[0014] one or more processors;

[0015] a storage device for storing one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the entity table information processing method described in the embodiment of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention further provides a storage medium storing computer-executable instructions, which, when executed by a computer processor, are used to execute the entity table information processing method described in the embodiment of the present invention.

[0018] In an embodiment of the present invention, when a workflow task is executed, a first hook function is set to capture workflow information in the workflow task, and entity table association relationship data is generated and stored based on the workflow information. A second hook function is set to capture entity table change information in the workflow task, and corresponding entity table data is generated and stored based on the entity table change information. When a target entity table query instruction is received, a query is performed based on the entity table association relationship data and the entity table data to generate feedback results, wherein the feedback results include the entity table data and the entity table association relationship data. This solution solves the problems of poor timeliness and limited information collection when determining entity table association relationships in the prior art, and improves the efficiency of querying entity table association data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a method for processing entity table information provided by an embodiment of the present invention;

[0020] Figure 2 A flowchart of a method for querying entity table data and entity table association relationship data provided by an embodiment of the present invention;

[0021] Figure 3 A flowchart of another entity table information processing method provided by an embodiment of the present invention;

[0022] Figure 4 A flowchart of another entity table information processing method provided by an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of a module of an entity table information processing device provided by an embodiment of the present invention;

[0024] Figure 6 A schematic structural diagram of an entity table information processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0026] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0027] The entity table information processing method provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0028] Figure 1 This is a flowchart of a method for processing entity table information provided by an embodiment of the present invention. This embodiment can capture and query entity table data and entity table relationship data. The method can be executed by a device with computing capabilities, such as a server, laptop, desktop computer, etc., and specifically includes the following steps:

[0029] Step S101: When a workflow task is executed, the workflow information in the workflow task is captured by a set first hook function, and entity table association relationship data is generated based on the workflow information and stored.

[0030] Among them, the workflow task can be a task created and scheduled in real time by the workflow scheduling engine. In order to execute a certain function, the developer configures the corresponding workflow task to execute the workflow task through the workflow task execution engine. For example, the system platform internally stores an entity table containing a large amount of web page information. Now it is necessary to obtain certain information from the entity table, such as click information, page view information, or registered user information in a certain period of time, to generate another entity table. At this time, the executable workflow task is to read the original entity table and obtain data that meets the conditions to generate a new entity table. In one embodiment, the open source Oozie workflow scheduling engine can be used to create and schedule workflow tasks. The workflow scheduling engine supports task scheduling such as Spark SQL, Hive scripts, and Sqoop, and can be used to schedule data processing tasks.

[0031] In one embodiment, when a workflow task is executed, the workflow information in the workflow task is captured through a first hook function. The hook function is a setting procedure that preferentially captures the data message before the data message calls the function it is supposed to call, and then obtains control through the hook function to process the data message. After the workflow information in the workflow task is captured by the first hook function, entity table association relationship data is generated and stored based on the workflow information. The workflow information is relevant information during the execution of the workflow task, such as the workflow task's execution plan data, task execution parameters, etc.

[0032] In one embodiment, capturing workflow information in a workflow task via a configured first hook function includes: when a processing engine based on in-memory computing executes the workflow task, capturing workflow information in the workflow task via a configured first hook function corresponding to the processing engine. Exemplarily, the in-memory computing processing engine may be a Spark engine, which is a fast and versatile computing engine for large-scale data processing. Exemplarily, the first hook function may be recorded as a Spark Hook.

[0033] Entity table association data is used to represent the association between entity tables. For example, if entity table 1 is read and entity table 2 is created based on entity table 1, then entity table 1 and entity table 2 are in an upstream-downstream association relationship. In one embodiment, after workflow information is captured by the first hook function, it is parsed to obtain basic information about the upstream and downstream entity tables during the execution of the workflow task. Based on this information, entity table association data is determined and generated for storage.

[0034] In one embodiment, the entity table association relationship data is stored in the form of a graph structure, specifically in the form of graph vertices and directed edges within the graph structure. Graph vertices represent entity tables (e.g., recording entity table identifiers), and directed edges between graph vertices represent the association relationships between graph vertices. The directions of the directed edges can be determined by recorded input and output labels. For example, if a graph vertex corresponding to entity table 1 points to a graph vertex corresponding to entity table 2 as an outgoing edge, and corresponding graph vertex 2 is connected to graph vertex 1 as an incoming edge, to represent the entity table association relationship data between entity tables 1 and 2, when recording the direction of the directed edge connecting entity tables 1 and 2, the graph vertex label corresponding to entity table 1 can be recorded as input, and the graph vertex label corresponding to entity table 2 can be recorded as output.

[0035] Step S102: capturing entity table change information in the workflow task through a set second hook function, generating corresponding entity table data based on the entity table change information, and storing the data.

[0036] The second hook function is used to capture entity table change information in workflow tasks. As previously described, the first hook function captures workflow information and generates and stores entity table association data based on this workflow information. This entity table association data, taking the entity table corresponding to a graph vertex as an example, optionally only records relevant attribute information of the graph vertex, such as the project name and instance ID, but does not record the specific entity table metadata. In this step, the second hook function captures entity table change information in workflow tasks, generates and stores corresponding entity table data based on the entity table change information, and thus records the complete entity table data and the entity table association data related to the entity table data for the current workflow task.

[0037] In one embodiment, the entity table change information may be data associated with Hive. Hive is a Hadoop-based data warehouse tool that maps structured data files to database tables and provides SQL query capabilities. Data stored in Hive is displayed as Hive tables. The second hook function can be exemplarily recorded as a Hive Hook. When an entity table is created, deleted, or modified, the Hive Hook receives the corresponding change operation notification and, based on the specific entity table change information, retrieves the latest metadata, the corresponding change type, basic entity table information, entity table fields, and so on.

[0038] In one embodiment, based on the captured entity table change information, metadata is modified accordingly to obtain and store entity table data. The entity table data is stored as graph vertices in a graph structure. The stored data at the graph vertices represents the entity table data and includes metadata captured by a Hive hook and related descriptive information describing the entity table captured by a corresponding Spark hook.

[0039] Step S103: When a target entity table query instruction is received, query is performed based on the entity table association relationship data and the entity table data to generate a feedback result.

[0040] In one embodiment, when a workflow task is executed, the corresponding entity table data and entity table associated data are stored to provide feedback on the target entity table query instruction. Optionally, the target entity table query instruction can be generated by a web front-end. Specifically, the web front-end displays an interface and provides a search interface for entity table associated information. Users can search and query entity tables, related metadata, and entity table associated data through the display interface of the web front-end. For example, a search box is displayed on the web page, and the user can enter the information they want to query in the search box, such as the table name, attribute information, field name, etc. of the entity table. The web front-end generates a target entity table associated information query instruction based on the received user search operation. Exemplarily, the user search operation can be a click operation of a search button, and the target entity table associated information query instruction can be an associated information query instruction corresponding to the search content obtained in the search box.

[0041] In one embodiment, when a target entity table query instruction is received, a search is performed in a database storing entity table data and entity table association relationships. Optionally, the entity table data and entity table association relationship data are stored in a graph database, which are stored in the form of vertices and edges in the graph database respectively. The specific query process is as follows: Figure 2 As shown, Figure 2A flowchart of a method for querying entity table data and entity table association relationship data provided by an embodiment of the present invention specifically includes:

[0042] Step S1031: Determine the target entity table in the target entity table query instruction.

[0043] The target entity table may be represented by an entity table name, an entity table identifier, etc.

[0044] Step S1032: query the graph vertex corresponding to the target entity table in the graph database, query the connected graph directed edges based on the graph vertex, determine the graph vertex at the other end based on the graph directed edges obtained by the query, and so on.

[0045] In one embodiment, a search is performed in a graph database for an entity table record corresponding to a target entity table. If no entity table record is found, a response indicating no search data was found is returned. If entity table data corresponding to the target entity table is found, a search is performed starting with the graph vertex corresponding to the entity table data, such as by traversing the graph using a breadth-first algorithm, querying for connected directed graph edges. The vertex at the other end of the graph edge is then determined based on the query, and so on until the last connected graph vertex is found.

[0046] Step S1033: Generate a relationship graph based on the graph vertices and directed edges obtained from the query.

[0047] In one embodiment, a relationship graph is generated based on the graph vertices and directed edges obtained from the query and displayed on a web front-end. The graph vertices are the target entity table obtained from the query and the entity tables associated with it, and the directed edges are the relationships between the target entity table obtained from the query and the associated entity tables.

[0048] As can be seen from the above, when a workflow task is executed, the workflow information in the workflow task is captured through the first hook function, and entity table association relationship data is generated and stored based on the workflow information. The entity table change information in the workflow task is captured through the second hook function, and corresponding entity table data is generated and stored based on the entity table change information. When a target entity table query instruction is received, a query is performed based on the entity table association relationship data and the entity table data to generate feedback results, where the feedback results include the entity table data and the entity table association relationship data. This solution solves the problems of poor timeliness and limited information collection when determining entity table association relationships in the prior art, and improves the efficiency of querying entity table association data.

[0049] Based on the above technical solution, before executing a workflow task, the system further includes receiving user-configured scheduling task information and creating a workflow task based on the scheduling task information. Specifically, the user configures the scheduling task information, and the system automatically creates a workflow task based on the scheduling task information. During the execution of the workflow task, the hook function provided by this solution captures and stores entity table data and entity table relationship data for subsequent query feedback. This solves the problems of poor timeliness and limited information collection when determining entity table relationships in the prior art, and improves the efficiency of querying entity table relationship data.

[0050] On the basis of the above technical solution, after querying and generating feedback results based on the entity table association relationship data and the entity table data, it also includes: determining the status of the entity table in the feedback results based on the execution status of the workflow task, and displaying it. In one embodiment, after obtaining the feedback results of the target entity table, taking the feedback result as a relationship graph as an example, the relationship graph records the entity tables that have mutual association relationships, as well as the workflows between the entity tables. By further querying the current scheduling status of the workflow in the relationship graph, as well as the scheduling time and scheduling results of the latest workflow task instance and other information, and judging the processing status of the entity table based on the input workflow status of the entity table in the relationship graph, and displaying it. This further enriches the display function of the entity table and the associated data, making it easier for users to intuitively understand the actual situation of the entity table data and the entity table association relationship data in a visual form.

[0051] Figure 3 A flowchart of another entity table information processing method provided by an embodiment of the present invention provides a specific method for capturing and storing workflow information in the workflow task by setting a first hook function, specifically including:

[0052] Step S201: Receive scheduling task information configured by a user, and create a workflow task according to the scheduling task information.

[0053] Step S202: When the processing engine based on memory computing executes the workflow task, the execution plan data and task execution parameters in the workflow task are captured through the set first hook function.

[0054] In one embodiment, the workflow task includes execution plan data and corresponding task execution parameters, which are obtained through the first hook function.

[0055] Step S203: parse the execution plan data to obtain upstream and downstream entity table information, and obtain the workflow instance identifier based on the task execution parameters, format the upstream and downstream entity table information and the workflow instance identifier, and generate entity table association relationship data based on the formatted upstream and downstream entity table information and workflow instance identifier and store it.

[0056] In one embodiment, after the execution plan data is obtained through the first hook function, it is parsed to obtain information about upstream and downstream entity tables and the corresponding executing users. For example, the upstream entity table of entity table 2 obtained by parsing is entity table 1, and the downstream entity table is entity table 3. The parsed upstream and downstream entity table information includes basic information about entity tables 1 and 3, such as type, name identifier, and project.

[0057] In one embodiment, after obtaining task execution parameters through a first hook function, a determination is made as to whether the currently executing task is a workflow scheduling task, and the instance identifier of the workflow is obtained. Optionally, after obtaining information such as upstream and downstream entity table information, the executing user, and the instance identifier, the information is formatted and sent to a Kafka message queue for processing by the data management platform system. This formatting includes formatting the obtained information such as upstream and downstream entity table information, the executing user, and the instance identifier into JSON format. The data management platform system generates and stores entity table association data based on the formatted upstream and downstream entity table information and the workflow instance identifier. In one embodiment, after receiving the JSON-formatted workflow information, the data management platform system uses the instance identifier to query the task scheduling service in the system to obtain the corresponding task identifier and parent task identifier, and then performs association. Since the task scheduling service's resident workflow instances are generated on demand, if association is not performed, the data management platform system will generate different instances, resulting in a surge in the number of instances stored in the system, increasing the system burden and hindering users from viewing the associations between corresponding processing tasks. After association, the data management platform system will associate each workflow instance generated by the task scheduling service with the same workflow instance, that is, the same vertex in the graph database. Finally, the system will save the scheduled workflow task in the form of vertices and edges in the graph database. The vertex contains the basic information of the scheduling task, and the edge contains the relationship between different entities.

[0058] Step S204: Capture entity table change information in the workflow task through the set second hook function, generate corresponding entity table data based on the entity table change information, and store the data.

[0059] Step S205: When a target entity table query instruction is received, query is performed based on the entity table association relationship data and the entity table data to generate a feedback result.

[0060] From the above, it can be seen that in the process of processing entity table information, the upstream and downstream entity table information is obtained by parsing the execution plan data, and the workflow instance identifier is obtained according to the task execution parameters. The upstream and downstream entity table information and the workflow instance identifier are formatted, and the entity table association relationship data is generated and stored based on the formatted upstream and downstream entity table information and workflow instance identifier. This realizes the efficient capture and storage of the entity table association relationship in the workflow information, optimizes the existing entity table information processing mechanism, and can realize efficient query of entity table association data.

[0061] Figure 4 A flowchart of another entity table information processing method provided by an embodiment of the present invention provides a specific method for capturing and storing workflow information in the workflow task by setting a second hook function, specifically including:

[0062] Step S301: When a workflow task is executed, the workflow information in the workflow task is captured through a set first hook function, and entity table association relationship data is generated based on the workflow information and stored.

[0063] Step S302: Acquire entity table change events in the workflow task through the set second hook function, where the entity table change events include create events, modify events, and delete events.

[0064] When an entity table change event occurs, it can be captured by the second hook function. Entity table change events include create events, modify events, and delete events.

[0065] Step S303: Extract the metadata information from the entity table change event and format it, and send the formatted metadata information and the corresponding change type to the data management platform through the message queue component. The data management platform processes the metadata based on the received metadata information and the corresponding change type and stores it in the graph database in the form of vertices.

[0066] The message queue may be a Kafka message queue. The metadata information obtained includes basic information of the entity table, entity table fields, change operation type, etc.

[0067] In one embodiment, when the data management platform receives metadata information of an entity table, it creates, updates, deletes, and performs other processing on the corresponding entity table according to the type of change in the metadata information. The entity table metadata is saved in the graph database in the form of vertices. Different entity tables correspond to different vertices in the graph database, and the vertex is saved after adding relevant attribute information.

[0068] Step S304: When a target entity table query instruction is received, query is performed based on the entity table association relationship data and the entity table data to generate a feedback result.

[0069] From the above, it can be seen that in the process of processing the entity table information, the entity table change event in the workflow task is obtained through the set second hook function, the metadata information in the entity table change event is extracted and formatted, and the formatted metadata information and the corresponding change type are sent to the data management platform through the message queue component. The data management platform performs metadata processing based on the received metadata information and the corresponding change type and stores it in the form of vertices in the graph database, which optimizes the existing entity table information processing mechanism and can realize efficient query of entity table related data.

[0070] Figure 5 This is a schematic diagram of a module of an entity table information processing device provided by an embodiment of the present invention. The device is used to execute the entity table information processing method described above and has the corresponding functional modules and beneficial effects of the execution method. Figure 5 As shown, the device specifically includes: a first data capture module 101, a second data capture module 102 and a query processing module 103, wherein,

[0071] A first data capture module 101 is configured to capture workflow information in a workflow task through a set first hook function when the workflow task is executed, generate entity table association relationship data based on the workflow information, and store the data;

[0072] A second data capture module 102 is configured to capture entity table change information in the workflow task through a set second hook function, generate corresponding entity table data based on the entity table change information, and store the data;

[0073] The query processing module 103 is configured to, upon receiving a target entity table query instruction, perform a query based on the entity table association data and the entity table data to generate a feedback result, wherein the feedback result includes the entity table data and the entity table association data.

[0074] As can be seen from the above scheme, when a workflow task is executed, the workflow information in the workflow task is captured through the first hook function, and entity table association relationship data is generated and stored based on the workflow information. The entity table change information in the workflow task is captured through the second hook function, and corresponding entity table data is generated and stored based on the entity table change information. When a target entity table query instruction is received, a query is performed based on the entity table association relationship data and the entity table data to generate feedback results, where the feedback results include the entity table data and the entity table association relationship data. This scheme solves the problems of poor timeliness and limited information collection when determining entity table association relationships in the prior art, and improves the efficiency of querying entity table association data.

[0075] In a possible embodiment, the device further includes a workflow processing module, which is configured to receive user-configured scheduling task information before the workflow task is executed, and create a workflow task according to the scheduling task information;

[0076] The first data capture module 101 is specifically configured to:

[0077] When the processing engine based on memory calculation executes the workflow task, the workflow information in the workflow task is captured through the first hook function that is set and corresponds to the processing engine.

[0078] In a possible embodiment, capturing the workflow information in the workflow task by setting a first hook function includes:

[0079] The execution plan data and task execution parameters in the workflow task are captured through the set first hook function.

[0080] In a possible embodiment, the first data capturing module 101 is specifically configured to:

[0081] Parsing the execution plan data to obtain upstream and downstream entity table information, and obtaining a workflow instance identifier based on the task execution parameters;

[0082] The upstream and downstream entity table information and the workflow instance identifier are formatted, and entity table association relationship data is generated based on the formatted upstream and downstream entity table information and the workflow instance identifier and stored.

[0083] In a possible embodiment, the first data capturing module 101 is specifically configured to:

[0084] Formatting the upstream and downstream entity table information and the workflow instance identifier, and sending the formatted data to the data management platform through a message queue component;

[0085] The data management platform generates association relationship data in the form of graph vertices according to the formatted workflow instance identifier and in the form of graph structure according to the formatted upstream and downstream entity table information, and stores the generated association relationship data in the graph database.

[0086] In a possible embodiment, the second data capturing module 102 is specifically configured to:

[0087] The entity table change event in the workflow task is obtained through the set second hook function, and the entity table change event includes a create event, a modify event, and a delete event.

[0088] In a possible embodiment, the second data capturing module 102 is specifically configured to:

[0089] Extracting metadata information from the entity table change event and formatting it, and sending the formatted metadata information and the corresponding change type to the data management platform through the message queue component;

[0090] The data management platform performs metadata processing based on the received metadata information and the corresponding change type and stores the metadata in the form of vertices in the graph database.

[0091] In a possible embodiment, the entity table association relationship data and the entity table data are stored in a graph database in the form of vertices and edges, and the query processing module 103 is specifically configured to:

[0092] Determine the target entity table in the target entity table query instruction;

[0093] Querying the graph vertex corresponding to the target entity table in the graph database, querying the connected graph directed edges based on the graph vertex, determining the graph vertex at the other end based on the queried directed edges, and so on;

[0094] Generate a relationship graph based on the graph vertices and directed edges obtained from the query.

[0095] In a possible embodiment, the query processing module 103 is further configured to:

[0096] After generating feedback results by querying the entity table association relationship data and the entity table data, the state of the entity table in the feedback results is determined according to the execution status of the workflow task and displayed.

[0097] Figure 6 A schematic diagram of the structure of an entity table information processing device provided by an embodiment of the present invention, such as Figure 6 As shown, the device includes a processor 201 and a memory 202; the number of processors 201 in the device can be one or more. Figure 6In the embodiment, a processor 201 is used as an example; the processor 201 and the memory 202 in the device can be connected via a bus or other means. Figure 6 In this example, a bus connection is used. Memory 202, as a computer-readable storage medium, can be used to store software programs, computer executable programs, and modules, such as the program instructions / modules corresponding to the entity table information processing method in the embodiments of the present invention. Processor 201 executes the software programs, instructions, and modules stored in memory 202 to execute various functional applications and data processing of the device, thereby implementing the aforementioned entity table information processing method.

[0098] An embodiment of the present invention further provides a storage medium containing computer-executable instructions, which may be stored in the form of a server application. When the computer-executable instructions are executed by a computer processor, they are used to perform an entity table information processing method, the method comprising:

[0099] When a workflow task is executed, the workflow information in the workflow task is captured by a set first hook function, and entity table association relationship data is generated based on the workflow information and stored;

[0100] The entity table change information in the workflow task is captured by the set second hook function, and corresponding entity table data is generated and stored based on the entity table change information;

[0101] When a target entity table query instruction is received, a query is performed based on the entity table association relationship data and the entity table data to generate a feedback result, where the feedback result includes the entity table data and the entity table association relationship data.

[0102] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0103] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be an unmanned device, a mobile phone, a computer, a server or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0104] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An entity table information processing method, characterized in that: include: When a workflow task is executed, the execution plan data and task execution parameters in the workflow task are captured through a set first hook function, the execution plan data is parsed to obtain upstream and downstream entity table information, and a workflow instance identifier is obtained according to the task execution parameters, the upstream and downstream entity table information and the workflow instance identifier are formatted, and entity table association relationship data is generated based on the formatted upstream and downstream entity table information and the workflow instance identifier and stored; The entity table change information in the workflow task is captured by the set second hook function, and corresponding entity table data is generated and stored based on the entity table change information; When a target entity table query instruction is received, a query is performed based on the entity table association relationship data and the entity table data to generate a feedback result, where the feedback result includes the entity table data and the entity table association relationship data.

2. The entity table information processing method according to claim 1, characterized in that: Before the workflow task is executed, it also includes: Receive user-configured scheduling task information, and create a workflow task based on the scheduling task information; When the workflow task is executed, the workflow information in the workflow task is captured by setting a first hook function, including: When the processing engine based on memory calculation executes the workflow task, the workflow information in the workflow task is captured through the first hook function that is set and corresponds to the processing engine.

3. The entity table information processing method according to claim 1, characterized in that: The formatting of the upstream and downstream entity table information and the workflow instance identifier, and generating and storing entity table association relationship data based on the formatted upstream and downstream entity table information and the workflow instance identifier, includes: Formatting the upstream and downstream entity table information and the workflow instance identifier, and sending the formatted data to the data management platform through a message queue component; The data management platform generates association relationship data in the form of graph vertices according to the formatted workflow instance identifier and in the form of graph structure according to the formatted upstream and downstream entity table information, and stores the generated association relationship data in the graph database.

4. The entity table information processing method according to claim 1, characterized in that: The capturing of the entity table change information in the workflow task by setting the second hook function includes: The entity table change event in the workflow task is obtained through the set second hook function, and the entity table change event includes a create event, a modify event, and a delete event.

5. The entity table information processing method according to claim 4, characterized in that: The generating and storing corresponding entity table data based on the entity table change information includes: Extracting metadata information from the entity table change event and formatting it, and sending the formatted metadata information and the corresponding change type to the data management platform through the message queue component; The data management platform performs metadata processing based on the received metadata information and the corresponding change type and stores the metadata in the form of vertices in the graph database.

6. The entity table information processing method according to any one of claims 1 to 5, characterized in that: The entity table association relationship data and the entity table data are stored in the form of vertices and edges in a graph database, and querying and generating feedback results based on the entity table association relationship data and the entity table data includes: Determine the target entity table in the target entity table query instruction; Querying the graph vertex corresponding to the target entity table in the graph database, querying the connected graph directed edges based on the graph vertex, determining the graph vertex at the other end based on the queried directed edges, and so on; Generate a relationship graph based on the graph vertices and directed edges obtained from the query.

7. The entity table information processing method according to any one of claims 1 to 5, characterized in that: After querying and generating feedback results based on the entity table association relationship data and the entity table data, the method further includes: The status of the entity table in the feedback result is determined according to the execution status of the workflow task and displayed.

8. An entity table information processing device, characterized in that: include: a first data capture module configured to capture, when a workflow task is executed, execution plan data and task execution parameters in the workflow task through a set first hook function, parse the execution plan data to obtain upstream and downstream entity table information, obtain a workflow instance identifier based on the task execution parameters, format the upstream and downstream entity table information and the workflow instance identifier, and generate and store entity table association relationship data based on the formatted upstream and downstream entity table information and the workflow instance identifier; A second data capture module is configured to capture entity table change information in the workflow task through a set second hook function, generate corresponding entity table data based on the entity table change information, and store the data; The query processing module is used to generate a feedback result based on the entity table association relationship data and the entity table data when receiving a target entity table query instruction. The feedback result includes the entity table data and the entity table association relationship data.

9. An entity table information processing device, comprising: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the entity table information processing method according to any one of claims 1 to 7.

10. A storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the entity table information processing method according to any one of claims 1 to 7 when executed by a computer processor.

Images