Operation and maintenance management method and device for multi-platform data processing task, equipment and medium

Abstract

The invention discloses an operation and maintenance management method and device for a multi-platform data processing task, equipment and a medium. The method comprises the following steps: acquiring task information of each data processing task on a plurality of platforms and data source information of processing data; constructing an entity relation graph and a virtual relation graph according to the task information and the data source information; wherein the entity relation graph comprises circulation paths of different entity data nodes among different task nodes, and the virtual relation graph comprises processing logics of different task nodes to different virtual data nodes; the entity data nodes correspond to the virtual data nodes; and performing operation and maintenance management on the data processing task and the processing data according to the entity relation graph and the virtual relation graph. According to the method, the entity relation graph and the virtual relation graph are constructed, unified management of the multi-platform data tasks is achieved, the relation between the data processing tasks and the data is displayed from different angles through the two graphs, and the operation and maintenance management efficiency of the data processing tasks is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to an operation and maintenance management method, device, equipment and medium for multi-platform data processing tasks. Background Art

[0002] In today's digital age, the business development of enterprises increasingly depends on data processing on multiple platforms. For example, in related enterprises such as securities, aviation, and banking, it is necessary to obtain relevant enterprise data on multiple platforms, and respectively perform corresponding task processing on the data on different platforms, and there is data flow and interaction between different tasks on different platforms. Therefore, in order to improve the operation and maintenance efficiency of all tasks processed on multiple platforms, it is necessary to uniformly manage all tasks on multiple platforms.

[0003] However, the operation and maintenance management of multi-platform data processing tasks faces many technical problems, seriously affecting the operation efficiency and data processing quality of enterprises. The existing technologies have the following deficiencies in the operation and maintenance management of multi-platform data processing tasks: it is difficult to troubleshoot faults. When a processing task has problems, it is difficult to quickly locate the fault point and determine the cause of the fault; there is a lack of effective full-link monitoring means, and it is impossible to grasp the processing status and flow situation of data on each platform in real time, and it is difficult to discover potential problems in advance, etc. Summary of the Invention

[0004] The present invention provides an operation and maintenance management method, device, equipment and medium for multi-platform data processing tasks to solve the problem of low operation and maintenance management efficiency of multiple data processing tasks on multiple platforms.

[0005] According to one aspect of the present invention, there is provided an operation and maintenance management method for multi-platform data processing tasks, including:

[0006] Obtaining task information of each data processing task on multiple platforms and data source information of the processed data;

[0007] Constructing an entity relationship graph and a virtual relationship graph according to the task information and the data source information; wherein, the entity relationship graph includes the flow paths of different entity data nodes between different task nodes, and the virtual relationship graph includes the processing logics of different task nodes for different virtual data nodes; the entity data nodes correspond to the virtual data nodes;

[0008] Performing operation and maintenance management on the data processing tasks and the processed data according to the entity relationship graph and the virtual relationship graph.

[0009] According to another aspect of the present invention, there is provided an operation and maintenance management device for multi-platform data processing tasks, including:

[0010] A platform operation and maintenance information acquisition module, which is used to acquire task information of each data processing task and data source information of processed data on multiple platforms;

[0011] A relationship graph construction module, which is used to construct an entity relationship graph and a virtual relationship graph according to the task information and the data source information; wherein, the entity relationship graph includes the transfer paths of different entity data nodes between different task nodes, and the virtual relationship graph includes the processing logics of different task nodes for different virtual data nodes; the entity data nodes correspond to the virtual data nodes;

[0012] An operation and maintenance management module, which is used to perform operation and maintenance management on the data processing tasks and the processed data according to the entity relationship graph and the virtual relationship graph.

[0013] According to another aspect of the present invention, there is provided an electronic device, which includes:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the operation and maintenance management method of the multi-platform data processing task according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, there is provided a computer-readable storage medium, which stores computer instructions for causing a processor to implement the operation and maintenance management method of the multi-platform data processing task according to any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention represents nodes and edges for data processing tasks and processed data on multiple platforms, constructs an entity relationship graph and a virtual relationship graph, realizes unified operation and maintenance management of multi-platform data tasks, and shows the relationship between data processing tasks and data from different perspectives through the two graphs, improving the efficiency of operation and maintenance management of data processing tasks according to the entity relationship graph and the virtual relationship graph.

[0019] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. Description of the Drawings

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0021] Figure 1 It is a flowchart of an operation and maintenance management method for multi-platform data processing tasks provided according to an embodiment of the present invention;

[0022] Figure 2 It is a flowchart of another operation and maintenance management method for multi-platform data processing tasks provided according to an embodiment of the present invention

[0023] Figure 3 It is a flowchart of yet another operation and maintenance management method for multi-platform data processing tasks provided according to an embodiment of the present invention

[0024] Figure 4 It is a schematic structural diagram of an operation and maintenance management device for multi-platform data processing tasks provided according to an embodiment of the present invention;

[0025] Figure 5 It is a schematic structural diagram of an electronic device for implementing the operation and maintenance management method of multi-platform data processing tasks in the embodiments of the present invention. Detailed implementation manners

[0026] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0027] It should be noted that the terms "candidate", "target", etc. in the specification and claims of the present invention are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0028] Figure 1 FIG. 1 is a flowchart of an operation and maintenance management method for multi-platform data processing tasks. This embodiment is applicable to the situation of unified operation and maintenance and management of all tasks and data sources on multiple platforms. This method can be executed by an operation and maintenance management device for multi-platform data processing tasks. The operation and maintenance management device for multi-platform data processing tasks can be implemented in the form of hardware and / or software, and can be configured in a server with communication and computing capabilities. As Figure 1 shown, the method includes:

[0029] S110. Obtain the task information of each data processing task on multiple platforms and the data source information of the processed data.

[0030] Among them, the multiple platforms refer to data providers with task processing requirements. For example, for a certain type of data sampling task in Platform A, the result output by the sampling task is provided to Platform B for corresponding task processing, and then the data after the corresponding task processing is sent to Platform C for storage and display tasks. Similarly, the tasks between multiple platforms not only include the logic of sequential execution, but also include cross-logic or parallel logic, etc., that is, the data processed by Platform B is sent to Platform A, or the data displayed by Platform C is continued to be sent to Platform A for processing, or the data sampled by Platform A is simultaneously sent to Platform B and Platform C for corresponding processing, etc.

[0031] The data processing task refers to the task of processing the data on the corresponding platform set on different platforms. For example, the data processing task on Platform A is the target type data sampling task, the data processing task on Platform B is the data preprocessing process, and the data processing task on Platform C is the data storage and display task, etc. The data processing tasks on different platforms are determined according to the platform functions and platform services. The task information is used to describe the input data, output data, and task processing logic of the data processing task. Exemplarily, the task information at least includes the data source information and storage location of the input data, the storage location of the output data, and the processing priority information of the task for the input data, etc.

[0032] The processed data refers to the data that needs to be processed on different platforms. This data can be obtained by its own platform or obtained from other platforms, etc. The data source information refers to the source information corresponding to the processed data, such as including information such as the data storage location, data storage table name, data storage index, and data storage data structure.

[0033] Specifically, in the information acquisition stage, an automatic collection combined with manual supplementation method is adopted to accurately obtain relevant information such as data processing tasks, data sources, and data structures of each platform from data sources of multiple platforms. The businesses of each platform are different, and the generated data processing tasks are also different. The logical relationships between them are determined according to the relationships between platform businesses. Exemplarily, relevant data is obtained from each platform at regular intervals through pre-set automatic data collection scripts and interfaces to ensure the timeliness and comprehensiveness of the data; for some special cases or data not covered by automatic collection, it is supplemented manually to ensure the integrity of the data.

[0034] Optionally, before S120, the method further includes: performing a data integration operation on the data source information.

[0035] Specifically, after obtaining the data source information, a data integration operation is performed. The data integration operation includes steps of data filtering, data conversion, data error correction, and data standardization. First, data filtering is performed to remove obviously incorrect and duplicate data records; then data conversion is performed to uniformly convert data in different formats into a standard format, such as date format, numerical unit, etc.; then data error correction is performed to correct the error values in the data by comparing with a standard data source or applying pre-set verification rules; data standardization is to unify data encoding, field naming, etc., eliminate redundancy and inconsistency, and lay a solid foundation for subsequent data processing and analysis.

[0036] Since the data sources of different platforms are complex and diverse, and there are significant differences in data formats, storage methods, and processing logics among different platforms, it is difficult to integrate the data and comprehensively understand the context of the data; through the solution of the embodiments of the present invention, the data of different platforms is standardized, providing an accurate data basis for unified operation and maintenance management of data processing tasks of different platforms.

[0037] S120: Construct an entity relationship graph and a virtual relationship graph according to the task information and the data source information.

[0038] Among them, the entity relationship graph includes the transfer paths of different entity data nodes between different task nodes, and the virtual relationship graph includes the processing logics of different task nodes for different virtual data nodes; the entity data nodes correspond to the virtual data nodes.

[0039] Determine the task nodes according to the task information, and determine the entity data nodes and virtual data nodes according to the data source information. Among them, the entity data nodes are used to represent the actual data sources, for example, the entity data nodes are used to represent the table names or indexes of the data sources; the virtual data nodes are used to represent the specific data processed by the tasks in the entity data nodes, for example, the virtual data nodes are used to represent a certain type of relevant data under the table name of the data source.

[0040] Determine the edge relationships between different nodes according to the data processing logic included in the task information of the data processing task, and then construct an entity relationship graph based on the edge relationships between the entity data nodes and the task nodes, and construct a virtual relationship graph based on the edge relationships between the virtual data nodes and the task nodes. Optionally, determine the attribute information of each node according to the task information and data source information of the data processing task. For example, take the task processing priority as the attribute information of the task node according to the task information, and take the dependency relationship between different data as the attribute information of the data node according to the data source information. For example, there is an influence relationship between data node A and data node B, that is, when the data in data node A changes, the data in data node B will also be affected accordingly, which is a dependency relationship. The data node can be an entity data node or a virtual data node.

[0041] Specifically, construct a relationship graph based on the integrated data source information and task information, and store it in the graph database. The basis for constructing the relationship graph lies in clarifying the definitions of data nodes and edges. The nodes represent entities such as data processing tasks and data sources, and the edges represent the relationships between entities, such as task dependency relationships and data flow relationships. When constructing the graph from a logical perspective, it focuses on showing business relationships, sorting out the roles and interconnections of each task in the business process, presenting the sequence and business logic of task execution. The constructed relationship graph is a virtual relationship graph; when constructing from a physical perspective, it focuses on the data transfer path, depicting the specific trajectory of data from generation, transmission to processing, and identifying the flow of data between different platforms and systems. The constructed relationship graph is an entity relationship graph.

[0042] In a feasible embodiment, S120 includes:

[0043] Determine task nodes according to the task information of each data processing task;

[0044] Determine data source nodes according to the data source information of the processed data as entity data nodes;

[0045] Determine the data processing logic according to the task information of the task nodes, and determine the virtual data nodes corresponding to the entity data nodes according to the data processing logic and the entity data nodes;

[0046] Construct an entity relationship graph based on the transfer trajectory of the entity data nodes from generation, transmission to processing between different task nodes;

[0047] Determine the virtual relationship graph according to the execution order and processing relationship of the task nodes for the virtual data nodes corresponding to the entity data nodes.

[0048] Specifically, task nodes are determined according to the task information of the data processing task, and each data processing task is decomposed into at least one task node. For example, the data processing task is divided according to the data generation results in the task information to obtain each subtask, each subtask corresponds to a task node, and each subtask corresponds to a type of data generation result.

[0049] Data source nodes are determined according to the data source information of the processed data, and each data source node is used as an entity data node. The data source corresponding to each data source node includes at least one data type. The data processing logic of the task is determined according to the task information corresponding to different task nodes, that is, the input data information and output data information of the task. According to the relationship between the input data information and output data information and the data source information corresponding to the entity data node, the corresponding data information is determined from the data source information as a virtual data node. Then, the virtual data node has an associated correspondence with the entity data node corresponding to the corresponding data source information. In summary, an entity data node has a corresponding relationship with at least one virtual data node.

[0050] According to the process of the data in the entity data node from generation to transmission and from transmission to processing between different task nodes, the blood relationship between the entity data node and the task node is determined, and the entity relationship graph is determined according to the blood relationship.

[0051] According to the processing order and processing relationship of each virtual data node by different task nodes, the blood relationship between the virtual data node and the task node is determined, and then the virtual relationship graph is determined according to the blood relationship.

[0052] At the same time, according to the relationship between the virtual data node and the entity data node, the relationship between the entity relationship graph and the virtual relationship graph is constructed.

[0053] The entity relationship graph and the virtual relationship graph are constructed through different logics to represent the relationship between data and tasks from different perspectives, so as to improve the clarity and traceability of the associated impact between data and tasks, and further improve the operation and maintenance management efficiency of tasks.

[0054] S130. Perform operation and maintenance management on the data processing task and the processed data according to the entity relationship graph and the virtual relationship graph.

[0055] Perform operation and maintenance management on the data processing task through the blood relationship between different data processing tasks and different processed data in the entity relationship graph and the virtual relationship graph.

[0056] Specifically, the blood relationship query of data processing tasks and processed data is performed based on the entity relationship graph and the virtual relationship graph, and the source and evolution process of the data are determined based on the queried blood relationship. Exemplarily, the relationship graph is visualized, and the blood relationship in the entity relationship graph and the virtual relationship graph is quickly and conveniently queried based on the graph database nebula to trace the source and evolution process of the data. The entity relationship graph and the virtual relationship graph are presented in a visual way on the front end, and the target to be queried determined by the user through operations such as mouse clicks and drags are received, and the detailed information of the target task to be queried is determined, such as the input and output data of the task, the execution time, the person in charge, etc., and the data processing process is determined according to the data flow path in the relationship graph, so as to realize convenient interaction with the graph. The visualization implementation adopts a professional graphic visualization library to convert the graph data into an intuitive graphical interface, and optimizes the rendering algorithm to ensure the fluency when displaying large-scale graphs.

[0057] According to the entity relationship map and virtual relationship map, data processing tasks and processed data can also be applied in other scenarios, such as troubleshooting, real-time determination of data status, multi-platform task coordination, and decision-making assistance. Specifically, multi-scenario applications cover multiple aspects, and the intuitive process display presents the entire process of the data processing task in the relationship map in the form of a flowchart through a graphical interface. Each node represents a task or data source (topic, table name, index, etc.), and the connection between the nodes represents the execution order of the task and the data flow direction. The user can understand the overall picture of the task at a glance; when the data of a certain node is disconnected, the fault node of the disconnection can be quickly located according to the input and output indicators of the data flow task. At the same time, when a fault occurs, the relationship map and related data are automatically analyzed, combined with the fault characteristics and historical fault records, and the task nodes or data links that may have problems are quickly locked, providing accurate troubleshooting directions for operation and maintenance personnel. Real-time grasp of data status, through real-time monitoring and data collection, the latest status of each platform data is displayed in the form of a visual relationship map, such as data flow, data quality indicators, etc., so that users can understand the dynamic changes of data at any time. Coordinate multi-platform tasks, intelligently schedule and coordinate task execution on different platforms based on the relationship map and the resource conditions of each platform, optimize resource allocation, and improve task processing efficiency. Assist in decision-making, based on the analysis of historical task data and real-time data, provide data statistical reports and analysis models to provide users with decision-making basis in task planning, resource allocation, business optimization, etc.

[0058] Optionally, ensure the accuracy and timeliness of the relationship graph through real-time updates and regular maintenance. When changes in the processed data or data processing tasks are detected, obtain the changed data and synchronize it to the relationship graph to ensure the real-time nature of the graph; optimize the structure of the relationship graph at a preset cycle, and optimize, clean, and supplement the processed data. According to changes in business requirements, adjust the display method and content of the relationship graph, continuously optimize the performance, and make it better adapt to business development.

[0059] In a feasible embodiment, S130 includes:

[0060] Determine the entity node parameters of each entity data node and each task node in the entity relationship graph, and determine the virtual node parameters of each virtual data node and each task node in the virtual relationship graph;

[0061] Determine the task execution parameters of the data processing task and the data quality parameters of the processed data according to the entity node parameters and the virtual node parameters;

[0062] Optimize and schedule the data processing tasks on multiple platforms according to the task execution parameters and the data quality parameters.

[0063] Specifically, determine the entity data node parameters according to the changed data of the entity data nodes in the entity relationship graph, and determine the entity task node parameters according to the changed data of the task nodes. Determine the entity node parameters according to the entity data node parameters and the entity task node parameters. Determine the virtual data node parameters according to the changed data of the virtual data nodes in the virtual relationship graph, and determine the virtual task node parameters according to the changed data of the task nodes. Determine the virtual node parameters according to the virtual data node parameters and the virtual task node parameters. Exemplarily, the entity data node parameters include the changed data volume of the data source corresponding to the entity data node, and the entity task node parameters include the processed data volume of the data processing task corresponding to the task node; the virtual data node parameters include the changed data volume of the data type corresponding to the virtual data node, and the virtual task node parameters include the processed data volume of the data processing task corresponding to the task node.

[0064] Determine the data quality parameters of the corresponding data source according to the entity data node parameters in the entity node parameters, and this data quality parameter is used to characterize the processing efficiency of this data source; determine the data quality parameters of the corresponding data type according to the virtual data node parameters in the virtual node parameters, and this data quality parameter is used to characterize the processing efficiency of this data type; jointly determine the final data quality parameters of this data source according to the data quality parameters of the data source and the data quality parameters of the corresponding data type. For example, sort the data sources according to the final data quality parameters of different data sources.

[0065] Determine the task execution parameters for the corresponding data processing task according to the entity task node parameters in the entity node parameters and the virtual task node parameters in the virtual node parameters, and the task execution parameters are used to characterize the processing efficiency of the data processing task.

[0066] Determine the processing priorities of different data processing tasks according to the task execution parameters and the data quality parameters, and optimize the execution order of the data processing tasks on different platforms according to the processing priorities. Exemplarily, determine at least two data processing tasks in the parallel processing logic, and sort the at least two data processing tasks according to the task execution parameters of the at least two data processing tasks and the data quality parameters of the corresponding associated data nodes. Exemplarily, the priority of the data processing task with high task execution parameters and data quality parameters is greater than that of the data processing task with low task execution parameters and data quality parameters.

[0067] The embodiment of the present invention optimizes the scheduling of data processing tasks on multiple platforms according to the task execution parameters and the data quality parameters, improves the efficiency of unified execution of data processing tasks, preferentially processes the data processing tasks with high processing efficiency, and avoids the impact on computing power caused by the accumulation of tasks in parallel processing.

[0068] The technical solution of the embodiment of the present invention represents the data processing tasks and the processed data on multiple platforms in terms of nodes and edges, constructs an entity relationship graph and a virtual relationship graph, realizes the unified operation and maintenance management of multi-platform data tasks, and through the two graphs, shows the relationship between the data processing tasks and the data from different perspectives, improving the efficiency of operation and maintenance management of data processing tasks according to the entity relationship graph and the virtual relationship graph.

[0069] Figure 2 It is a flowchart of another method for operation and maintenance management of multi-platform data processing tasks provided by the embodiment of the present invention. In this embodiment, the fault troubleshooting process in the operation and maintenance management in the above embodiment is further refined. As Figure 2 shown, the method includes:

[0070] S210. Obtain the task information of each data processing task on multiple platforms and the data source information of the processed data.

[0071] S220. Construct an entity relationship graph and a virtual relationship graph according to the task information and the data source information.

[0072] Among them, the entity relationship graph includes the transfer paths of different entity data nodes between different task nodes, and the virtual relationship graph includes the processing logics of different task nodes for different virtual data nodes; the entity data nodes correspond to the virtual data nodes.

[0073] S230. Determine the faulty node according to the node parameters of each virtual data node and each task node in the virtual relationship graph.

[0074] Since the virtual relationship graph includes the execution business logic of data processing tasks, and the entity relationship graph is used to represent the data flow path, when determining the faulty node, first select the node parameters of each virtual data node and each task node in the virtual relationship graph to improve the efficiency and accuracy of determining the faulty node.

[0075] Specifically, monitor the node parameters of each virtual data node and each task node in the virtual relationship graph. When a problem occurs in the node parameters of any node, determine that node as the faulty node. Exemplarily, the node parameters are the data input volume and data output volume of each node. When the data output volume of a task node is less than the preset threshold, determine that task node as the faulty node; when the data input volume of a task node is less than the preset threshold, determine the upstream virtual data node of that task node as the faulty node; when the data input volume of a virtual data node is less than the preset threshold, determine the upstream task node of that virtual data node as the faulty node; when the data output volume of a virtual data node is less than the preset threshold, determine that virtual data node as the faulty node.

[0076] In a feasible embodiment, S230 includes:

[0077] Determine the data input volume and data output volume of each virtual data node and each task node, and use the data input volume and data output volume as node parameters;

[0078] Determine the faulty node according to the comparison between the node parameters and the historical node parameters.

[0079] Statistically analyze the historical node parameters of each node in the virtual relationship graph, and determine the preset threshold corresponding to each node according to the historical node parameters. Exemplarily, obtain the historical node parameters at multiple historical time points, and determine the preset threshold corresponding to that node according to the average value of the multiple historical node parameters.

[0080] Determine whether the faulty node judgment condition is satisfied according to the comparison result between the node parameters at the current time point and the preset threshold determined according to the historical node parameters.

[0081] Alternatively, pre-train a faulty node prediction model according to the historical node parameters, input the currently obtained node parameters of each node into the faulty node prediction model, and the model outputs the prediction result of the faulty node.

[0082] This embodiment judges the current node through the historical node parameters, improves the accuracy of the judgment basis of the faulty node, and further improves the accuracy of determining the faulty node.

[0083] S240. Determine the faulty virtual graph associated with the faulty node according to the positional relationship of the faulty node in the virtual relationship graph.

[0084] Taking the faulty node as the starting point, along all the edges connected by the faulty node, determine a preset number of associated nodes on each edge, and determine the faulty virtual graph according to the edge relationship between the preset number of associated nodes on each edge and the faulty node. The specific value of the preset number can be determined according to the actual situation and scenario requirements, and is not limited here.

[0085] S250. Determine the corresponding faulty entity graph in the entity relationship graph according to the faulty virtual graph.

[0086] According to the corresponding relationship between the virtual data nodes in the virtual relationship graph and the entity data nodes in the entity relationship graph, respectively determine the associated entity data nodes corresponding to each virtual data node in the faulty virtual graph, and determine the corresponding faulty entity graph according to the edges and task nodes including all the associated entity data nodes.

[0087] S260. Locate the faulty source node according to the node parameters of the entity data nodes and task nodes in the faulty entity graph.

[0088] Since the faulty node is the node that reflects the faulty problem, and the appearance of the faulty node is caused by problems in other nodes, it is necessary to determine the faulty source node. That is, the faulty source node is the source of the faulty problem. There is one faulty source node for each type of fault, and there are multiple resulting faulty nodes. The business logic of the faulty node itself may not be problematic. When the fault of the faulty source node is detected and resolved, the faulty node will also become a normal node.

[0089] Specifically, determine the node parameters of each entity data node and task node in the faulty entity graph, trace back each node according to the node parameters, and determine the node that first appears faulty as the faulty source node. Exemplarily, check each entity data node and task node in the faulty entity graph according to the node parameters, determine the nodes with abnormal node parameters, and according to the trajectory information of the data generation, transmission and processing in the faulty entity graph, determine the node that first appears abnormal on the trajectory, and determine the specific node state of this node. If the node state is normal, determine that this abnormal node is not the faulty source node, and continue to check the abnormal node along the trajectory; if it is determined that the node state is abnormal according to the specific node state of this node, determine that this abnormal node is the faulty source node, and determine the cause of the fault according to the abnormal state information.

[0090] In a feasible embodiment, S260 includes:

[0091] Determine the node parameters of each entity data node and each task node in the faulty entity graph;

[0092] Determine candidate fault source nodes from each entity data node and each task node according to a pre-trained fault prediction model and node parameters;

[0093] Send a node status query instruction to the candidate fault source nodes and receive the node status query results returned by the candidate fault source nodes;

[0094] Determine the final fault source node from the candidate fault source nodes according to the node status query results, and determine the fault cause according to the node status query results of the fault source node.

[0095] Pre-obtain the historical fault entity graph of historical fault nodes, obtain the historical node parameters of each node in the historical fault entity graph, and the historical fault source nodes in the historical fault entity graph, and perform model training according to the historical node parameters of each node in the historical fault entity graph and the historical fault source nodes to obtain a fault prediction model. Among them, the fault prediction model learns the association features between the fault source nodes and the node parameters.

[0096] Input the fault entity graph and the node parameters of each entity data node and each task node in the fault entity graph into the fault prediction model. The output of the fault prediction model is at least two candidate fault source nodes. Send a node status query instruction to each candidate fault source node. After receiving the node status query instruction, the candidate fault source node returns the corresponding node status query result according to the instruction content. Judge the candidate fault source nodes according to the node status query results of each candidate fault source node to obtain the final fault source node, and determine the fault cause of the fault source node according to the abnormal status in the node status query results. So that the operation and maintenance personnel can repair according to the determined fault source node and fault cause, improving the efficiency of fault troubleshooting.

[0097] Exemplarily, the node status query instruction is determined according to the node type of the candidate fault source node. For example, the node status query instructions corresponding to the data node and the task node are different, and the node status query instruction corresponding to each type of node can be determined in advance according to the node type and historical fault information. For example, the node status query instruction includes a memory query instruction, an online status query instruction, etc.

[0098] Exemplarily, pre-train a fault source determination model, input the node status query results of each candidate fault source node and the position information of the candidate fault source node in the relationship graph into the fault source determination model, and the output of the model is the final fault source node.

[0099] The solution of the embodiment of the present invention determines a faulty node by monitoring the node parameters of each node in the virtual relationship graph, determines a faulty virtual graph and a faulty entity graph according to the faulty node, and determines a faulty source node according to the node parameters of each node in the faulty entity graph, making full use of the data service processing logic of the virtual relationship graph and the data flow track information in the entity relationship graph, improving the accuracy and efficiency of determining the faulty source node, and realizing fast and accurate positioning of the fault cause.

[0100] Figure 3 FIG. is a flowchart of another operation and maintenance management method for multi-platform data processing tasks provided by the embodiment of the present invention. This embodiment further refines the full-link monitoring process in the operation and maintenance management in the above embodiment. As Figure 2 shown, the method includes:

[0101] S310. Obtain the task information of each data processing task on multiple platforms and the data source information of the processed data.

[0102] S320. Construct an entity relationship graph and a virtual relationship graph according to the task information and the data source information.

[0103] Among them, the entity relationship graph includes the transfer paths between different entity data nodes among different task nodes, and the virtual relationship graph includes the processing logics of different task nodes for different virtual data nodes; the entity data nodes correspond to the virtual data nodes.

[0104] S330. Determine the target query information according to the data processing task and the processed data.

[0105] Among them, the target query information is determined according to at least one of the following information: platform, task, and metadata.

[0106] The target query information is the information for which the user has a query requirement. The user selects the target query information from the data processing task and the processed data according to the query requirement. Exemplarily, if the user needs to query the data lineage of the data processing task on the target platform, the target query information is the platform information of the target platform, such as the platform name or code, etc.; if the user needs to query the data lineage of the target data processing task, the target query information is the task information of the target data processing task, such as the task name or code, etc.; if the user needs to query the task processing lineage of the target metadata, the target query information is the data information of the target metadata, and the target metadata can be the data source information or a certain data type in the data source, etc.

[0107] S340. Determine the target query node according to the target query information.

[0108] Determine the target query node in the relationship graph corresponding to the target query information. Exemplarily, if the target query information is determined based on the platform information of the target platform, the target query node is the task node corresponding to all data processing tasks on the target platform; if the target query information is determined based on the target data processing task, the target query node is the task node corresponding to the target data processing task; if the target query information is determined based on the metadata, the target query node is all data nodes corresponding to the metadata. Among them, the target query node includes the associated nodes in the entity relationship graph and the associated nodes in the virtual relationship graph.

[0109] S350. Determine the full-link lineage relationship from the virtual relationship graph based on the target query node, and determine the target query virtual graph according to the full-link lineage relationship.

[0110] Taking the target query node as the starting point, along all the edges connected by the target query node, determine all the associated nodes on each edge, determine the full-link lineage relationship according to the edge relationship between all the associated nodes on each edge and the target query node, and determine the target query virtual graph according to the full-link lineage relationships corresponding to all the edges.

[0111] S360. Determine the corresponding target query entity graph in the entity relationship graph according to the target query virtual graph.

[0112] According to the corresponding relationship between the virtual data nodes in the virtual relationship graph and the entity data nodes in the entity relationship graph, respectively determine the associated entity data nodes corresponding to each virtual data node in the target query virtual graph, and determine the corresponding target query entity graph according to the edges and task nodes including all the associated entity data nodes.

[0113] S370. Visualize the target query virtual graph and the target query entity graph, and perform associated visualization on the entity data nodes and virtual data nodes with corresponding relationships in the target query virtual graph and the target query entity graph.

[0114] Visualize the target query virtual graph and the target query entity graph so that users can view the full-link lineage relationship of the target query information, and in the visualization, associate the entity data nodes and virtual data nodes with corresponding relationships so that users can view the target query virtual graph and the target query entity graph in an associated manner, improving the data link query efficiency of the target query information.

[0115] The technical solution of the embodiment of the present invention determines the corresponding full-link blood relationship through the target query information, determines the target query virtual graph and the target query entity graph according to the full-link blood relationship, and performs associated visualization display on the target query virtual graph and the target query entity graph, improving the full-link monitoring efficiency of the target query information.

[0116] Figure 4 It is a schematic structural diagram of an operation and maintenance management device for multi-platform data processing tasks provided by an embodiment of the present invention. As Figure 4 shown, the device includes:

[0117] A platform operation and maintenance information acquisition module 410, configured to acquire task information of each data processing task on multiple platforms and data source information of the processed data;

[0118] A relationship graph construction module 420, configured to construct an entity relationship graph and a virtual relationship graph according to the task information and the data source information; wherein, the entity relationship graph includes the transfer paths of different entity data nodes between different task nodes, and the virtual relationship graph includes the processing logics of different task nodes for different virtual data nodes; the entity data nodes correspond to the virtual data nodes;

[0119] An operation and maintenance management module 430, configured to perform operation and maintenance management on the data processing tasks and the processed data according to the entity relationship graph and the virtual relationship graph.

[0120] The technical solution of the embodiment of the present invention represents nodes and edges for data processing tasks and processed data on multiple platforms, constructs an entity relationship graph and a virtual relationship graph, realizes unified operation and maintenance management of multi-platform data tasks, and improves the efficiency of operation and maintenance management of data processing tasks according to the entity relationship graph and the virtual relationship graph by showing the relationships between data processing tasks and data from different perspectives through the two graphs.

[0121] Optionally, the relationship graph construction module is specifically configured to:

[0122] Determine task nodes according to the task information of each data processing task;

[0123] Determine data source nodes according to the data source information of the processed data as entity data nodes;

[0124] Determine data processing logics according to the task information of the task nodes, and determine virtual data nodes corresponding to the entity data nodes according to the data processing logics and the entity data nodes;

[0125] Construct an entity relationship graph according to the transfer trajectories of entity data nodes from generation, transmission to processing between different task nodes;

[0126] Determine the virtual relationship graph according to the execution order and processing relationship of the virtual data nodes corresponding to the entity data nodes for each task node.

[0127] Optionally, the operation and maintenance management module includes a first operation and maintenance management sub-module, including:

[0128] A fault node determination unit, configured to determine a fault node according to the node parameters of each virtual data node and each task node in the virtual relationship graph;

[0129] A fault virtual graph determination unit, configured to determine a fault virtual graph associated with the fault node according to the positional relationship of the fault node in the virtual relationship graph;

[0130] A fault entity graph determination unit, configured to determine a corresponding fault entity graph in the entity relationship graph according to the fault virtual graph;

[0131] A fault source node determination unit, configured to locate a fault source node according to the node parameters of the entity data node and the task node in the fault entity graph.

[0132] Optionally, the fault node determination unit is specifically configured to:

[0133] Determine the data input volume and data output volume of each virtual data node and each task node, and use the data input volume and data output volume as node parameters;

[0134] Determine a fault node according to the node parameters and historical node parameters.

[0135] Optionally, the fault source node determination unit is specifically configured to:

[0136] Determine the node parameters of each entity data node and each task node in the fault entity graph;

[0137] Determine candidate fault source nodes from each entity data node and each task node according to a pre-trained fault prediction model and node parameters;

[0138] Send a node status query instruction to the candidate fault source nodes, and receive the node status query results returned by the candidate fault source nodes;

[0139] Determine the final fault source node from the candidate fault source nodes according to the node status query results, and determine the fault cause according to the node status query results of the fault source node.

[0140] Optionally, the operation and maintenance management module includes a second operation and maintenance management sub-module, specifically configured to:

[0141] Determine target query information according to the data processing task and the processed data; wherein, the target query information is determined according to at least one of the following information: platform, task, and metadata;

[0142] Determine the target query node according to the target query information;

[0143] Determine the full-link blood relationship from the virtual relationship graph according to the target query node, and determine the target query virtual graph according to the full-link blood relationship;

[0144] Determine the corresponding target query entity graph in the entity relationship graph according to the target query virtual graph;

[0145] Visually display the target query virtual graph and the target query entity graph, and visually associate the entity data nodes and virtual data nodes with corresponding relationships in the target query virtual graph and the target query entity graph.

[0146] Optionally, the operation and maintenance management module includes a third operation and maintenance management sub-module, which is specifically used for:

[0147] Determine the entity node parameters of each entity data node and each task node in the entity relationship graph, and determine the virtual node parameters of each virtual data node and each task node in the virtual relationship graph;

[0148] Determine the task execution parameters of the data processing task and the data quality parameters of the processed data according to the entity node parameters and the virtual node parameters;

[0149] Optimize and schedule the data processing tasks on multiple platforms according to the task execution parameters and the data quality parameters.

[0150] The operation and maintenance management device for multi-platform data processing tasks provided by the embodiments of the present invention can execute the operation and maintenance management method for multi-platform data processing tasks provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0151] In the technical solution of this application, the acquisition, storage, use, processing, etc. of data all comply with the relevant regulations of national laws and regulations and do not violate public order and good customs.

[0152] According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0153] Figure 5FIG. 0 shows a schematic structural diagram of an electronic device 10 that can be used to implement embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as, for example, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0154] As Figure 5 shown, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. The memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0155] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0156] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the operation and maintenance management method for multi-platform data processing tasks.

[0157] In some embodiments, the operation and maintenance management method for multi-platform data processing tasks can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the operation and maintenance management method for multi-platform data processing tasks described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the operation and maintenance management method for multi-platform data processing tasks by any other suitable means (e.g., by means of firmware).

[0158] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0159] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a dedicated computer, or other programmable data processing apparatus, such that when the computer programs are executed by the processor, the functions / operations specified in the flowchart and / or block diagram are implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0160] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0161] To provide for interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0162] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0163] A computing system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0164] It should be understood that various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0165] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for operation and maintenance management of multi-platform data processing tasks, characterized in that: The method includes: Obtain task information of each data processing task on multiple platforms and data source information of processed data; An entity relationship graph and a virtual relationship graph are constructed according to the task information and the data source information; wherein the entity relationship graph includes the flow paths of different entity data nodes between different task nodes, and the virtual relationship graph includes the processing logic of different task nodes on different virtual data nodes; the entity data nodes correspond to the virtual data nodes; The data processing tasks and the processed data are operated and managed according to the entity relationship graph and the virtual relationship graph.

2. The method according to claim 1, characterized in that Constructing an entity relationship graph and a virtual relationship graph according to the task information and the data source information, including: Determining a task node according to the task information of each of the data processing tasks; Determine a data source node as an entity data node according to the data source information of the processed data; Determine a data processing logic according to the task information of the task node, and determine a virtual data node corresponding to the physical data node according to the data processing logic and the physical data node; Constructing an entity relationship graph according to the flow track of the entity data nodes from generation, transmission to processing between different task nodes; A virtual relationship graph is determined according to the execution order and processing relationship of the task nodes on the virtual data nodes corresponding to the physical data nodes.

3. The method according to claim 1, characterized in that Performing operation and maintenance management on the data processing task and the processed data according to the entity relationship graph and the virtual relationship graph includes: Determine the faulty node according to the node parameters of each virtual data node and each task node in the virtual relationship graph; Determine a fault virtual graph associated with the fault node according to the position relationship of the fault node in the virtual relationship graph; Determine the corresponding fault entity graph in the entity relationship graph according to the fault virtual graph; The fault source node is located according to the node parameters of the entity data node and the task node in the fault entity graph.

4. The method according to claim 3, characterized in that Determining the faulty node according to the node parameters of each virtual data node and each task node in the virtual relationship graph includes: Determine the data input amount and data output amount of each virtual data node and each task node, and use the data input amount and the data output amount as node parameters; The faulty node is determined according to the node parameters and historical node parameters.

5. The method according to claim 3, characterized in that: Locating the fault source node according to the node parameters of the entity data node and the task node in the fault entity graph includes: Determine node parameters of each entity data node and each task node in the fault entity graph; Determine a candidate fault source node from each of the entity data nodes and each of the task nodes according to a pre-trained fault prediction model and the node parameters; Sending a node status query instruction to the candidate fault source node, and receiving a node status query result returned by the candidate fault source node; A final fault source node is determined from the candidate fault source nodes according to the node status query result, and a fault cause is determined according to the node status query result of the fault source node.

6. The method according to claim 1, characterized in that Performing operation and maintenance management on the data processing task and the processed data according to the entity relationship graph and the virtual relationship graph includes: Determine target query information according to the data processing task and the processed data; wherein the target query information is determined according to at least one of the following information: platform, task and metadata; Determine a target query node according to the target query information; Determine a full-link blood relationship from the virtual relationship graph according to the target query node, and determine a target query virtual graph according to the full-link blood relationship; Determine the target query entity graph corresponding to the entity relationship graph according to the target query virtual graph; The target query virtual graph and the target query entity graph are visualized, and the entity data nodes and the virtual data nodes having a corresponding relationship in the target query virtual graph and the target query entity graph are associated and visualized.

7. The method according to claim 1, characterized in that Performing operation and maintenance management on the data processing task and the processed data according to the entity relationship graph and the virtual relationship graph includes: Determine the entity node parameters of each of the entity data nodes and each of the task nodes in the entity relationship graph, and determine the virtual node parameters of each of the virtual data nodes and each of the task nodes in the virtual relationship graph; Determine the task execution parameters of the data processing task and the data quality parameters of the processed data according to the physical node parameters and the virtual node parameters; The data processing tasks on the multiple platforms are optimized and scheduled according to the task execution parameters and the data quality parameters.

8. An operation and maintenance management device for multi-platform data processing tasks, characterized in that: The device includes: The platform operation and maintenance information acquisition module is used to obtain the task information of each data processing task on multiple platforms and the data source information of the processed data; A relationship graph construction module, used to construct an entity relationship graph and a virtual relationship graph according to the task information and the data source information; wherein the entity relationship graph includes the flow paths between different entity data nodes and different task nodes, and the virtual relationship graph includes the processing logic of different task nodes on different virtual data nodes; the entity data nodes correspond to the virtual data nodes; An operation and maintenance management module is used to perform operation and maintenance management on the data processing tasks and the processed data according to the entity relationship graph and the virtual relationship graph.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the operation and maintenance management method of multi-platform data processing tasks described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the operation and maintenance management method for multi-platform data processing tasks according to any one of claims 1 to 7 when executed.

Images