Intelligent operation and maintenance method and device and medium

Abstract

The invention provides an intelligent operation and maintenance method and device and a medium, and relates to the technical field of operation and maintaining.The method comprises the steps that in response to a data process execution error, a blood relationship table and an execution record table corresponding to a data process are obtained; constructing a data flow fault graph model according to the blood relationship table and the execution record table, wherein the data flow fault graph model comprises a front-back relationship of data flow execution of each data node and the number of pieces of real data; and based on the data flow fault graph model, obtaining the number of fitting data of each data node according to the number of real data of the preposed data node, and judging the fault root cause data node of the data flow according to the number of real data and the number of fitting data. According to the method and the device, through the blood relationship table and the execution record table, the front-back relationship and the number of the real data of each data node for executing the data process are respectively obtained, then the number of the fitting data is obtained according to the front-back relationship, and finally, the number of the real data is compared with the number of the fitting data to judge whether the data node is the fault node or not.

Description

Technical Field

[0001] The present disclosure at least relates to the field of operation and maintenance technology, and in particular, to an intelligent operation and maintenance method, an intelligent operation and maintenance device, and a computer-readable storage medium. Background Art

[0002] In order to help operation and maintenance personnel quickly process massive, multi-source, and heterogeneous operation and maintenance data, the concept of "intelligent operation and maintenance" has been proposed in the field of operation and maintenance. Intelligent operation and maintenance is based on artificial intelligence technology, and uses big data, modern machine learning, and more advanced analysis technologies to enhance the analysis and decision-making capabilities required to achieve operation and maintenance goals. However, existing technologies still have many shortcomings in accurately locating the actual location of faults. Summary of the invention

[0003] The technical problem to be solved by the present disclosure is to provide an intelligent operation and maintenance method, an intelligent operation and maintenance device, and a computer-readable storage medium in order to solve the problem of how to realize fault location through intelligent operation and maintenance.

[0004] In a first aspect, the present disclosure provides an intelligent operation and maintenance method, the method comprising:

[0005] In response to a data flow execution error, obtaining a blood relationship table and an execution record table corresponding to the data flow;

[0006] A data flow fault graph model is constructed based on the blood relationship table and the execution record table. The data flow fault graph model includes the before and after relationship of each data node executing the data flow and the number of real data items.

[0007] Based on the data flow fault graph model, the number of fitted data items of each data node is obtained according to the real data items of the preceding data node, and the data node of the root cause of the fault in the data flow is determined according to the real data items and the fitted data items.

[0008] Further, in response to a data flow execution error, obtaining a blood relationship table and an execution record table corresponding to the data flow specifically includes:

[0009] Automatically inspect data processes based on robotic process automation (RPA);

[0010] The data center collects and obtains the blood relationship table and execution record table of the data process in real time;

[0011] In response to the automatic inspection discovering a data flow error, the blood relationship table and execution record table corresponding to the data flow are read from the data center based on the hook sensor HookSensor.

[0012] Furthermore, the automatic inspection found data flow errors, including:

[0013] Based on RPA receiving the data flow, the data flow includes a specified time when the specified data should arrive at a certain data node. In response to the specified data not arriving at the certain data node at the specified time, based on RPA, the data flow error information is displayed on the monitoring alarm page of the data middle station.

[0014] Furthermore, the method further comprises:

[0015] Based on RPA, check the database connection or audit the data table. In response to discovering abnormal database connection or data table audit error, read the database connection status or data table field from the data center based on HookSensor to locate the abnormal database connection or the wrong data table field.

[0016] Automatically reconnect the database based on RPA, or send data table field error information to the operation and maintenance personnel based on RPA and receive the data table fields corrected by the operation and maintenance personnel, or send the root cause data node information of the fault to the operation and maintenance personnel based on RPA.

[0017] Furthermore, a data flow fault graph model is constructed based on the blood relationship table and the execution record table. The data flow fault graph model includes the before and after relationship of each data node executing the data flow and the number of real data items, specifically including:

[0018] Obtain each data node of the execution data flow and its execution before and after relationship from the blood relationship table;

[0019] Obtain the actual number of data items inserted into each data node for executing the data flow from the execution record table;

[0020] A directed acyclic graph (DAG) is established for each data node that executes the data flow. Directed arrows are used in the DAG to connect each data node according to the execution before and after relationship, and the actual number of data inserted into each data node is marked.

[0021] Furthermore, the number of fitted data items of each data node is obtained according to the number of real data items of the preceding data node, specifically including:

[0022] Obtain all preceding data nodes [1, 2, ..., j] for each data node i according to the connection relationship in DAG;

[0023] The number of fitted data for each data node i is obtained according to the following formula: i =a0+a1*x1+a2*x2+…+a j *x j , where [a0, a1, ..., a j ] is the fitting coefficient obtained in advance based on the number of past real data, [x1, x2, ..., x j ] is the actual number of data items in each front-end data node.

[0024] Furthermore, the root cause data node of the data flow is determined according to the number of real data and the number of fitted data, including:

[0025] Response to M*Y i ≤Z i ≤N*Y i , judge that data node i is normal, otherwise judge that data node i is a faulty data node in the data flow, where M and N are pre-acquired judgment coefficients, and Z i is the actual number of data items in data node i;

[0026] All faulty data nodes are displayed in the DAG, and the faulty data node located closest to the front is the data node that is the root cause of the fault in the data flow.

[0027] Furthermore, the method further comprises:

[0028] Based on RPA receiving the scheduling request input through the dialogue robot, in response to the fault root cause data node requiring manual processing, the scheduling list including the fault root cause data node information and the operation and maintenance personnel responsible for processing the fault root cause data node is output according to the scheduling request.

[0029] In a second aspect, the present disclosure provides an intelligent operation and maintenance device, the device comprising:

[0030] An operation and maintenance data collection unit, used to obtain a blood relationship table and an execution record table corresponding to the data flow in response to a data flow execution error;

[0031] The fault graph model building unit is connected to the operation and maintenance data collection unit and is used to build a data flow fault graph model according to the blood relationship table and the execution record table. The data flow fault graph model includes the before and after relationship of each data node executing the data flow and the number of real data items.

[0032] The fault root cause analysis unit is connected to the fault graph model construction unit, and is used to obtain the number of fitted data items of each data node according to the real data items of the preceding data node based on the data flow fault graph model, and to determine the fault root cause data node of the data flow according to the real data items and the fitted data items.

[0033] In a third aspect, the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the intelligent operation and maintenance method as described above is implemented.

[0034] The present disclosure provides an intelligent operation and maintenance method, an intelligent operation and maintenance device, and a computer-readable storage medium. Through a blood relationship table and an execution record table, the front and back relationships and the real data counts of each data node executing a data flow are respectively obtained, and a data flow fault graph model is constructed accordingly. Then, on the basis of the data flow fault graph model, the fitted data count of each data node is obtained according to the preceding data node in the front and back relationships. Finally, the real data count and the fitted data count are compared to determine whether the data node is a fault node that causes a data flow execution error, thereby locating the fault and improving the operation and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of an intelligent operation and maintenance method according to an embodiment of the present disclosure;

[0036] Figure 2 is a structural schematic diagram of an intelligent operation and maintenance device according to an embodiment of the present disclosure;

[0037] Figure 3 is a schematic diagram of the architecture of an intelligent operation and maintenance system according to an embodiment of the present disclosure;

[0038] Figure 4 It is a schematic diagram of the architecture of a HookSensor troubleshooting technology according to an embodiment of the present disclosure;

[0039] Figure 5 is an example diagram of a data flow fault graph model according to an embodiment of the present disclosure;

[0040] Figure 6 This is an example diagram of a data center display interface according to an embodiment of the present disclosure;

[0041] Figure 7 is an example diagram of a method for implementing a shift scheduling according to an embodiment of the present disclosure;

[0042] Figure 8 It is a schematic diagram of interactive functions of an intelligent operation and maintenance system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0044] It should be understood that the specific embodiments and drawings described herein are only used to explain the present disclosure rather than to limit the present disclosure.

[0045] It can be understood that, in the absence of conflict, the various embodiments of the present disclosure and the various features in the embodiments can be combined with each other.

[0046] It can be understood that, for the convenience of description, the drawings of the present disclosure only show parts related to the present disclosure, while parts irrelevant to the present disclosure are not shown in the drawings.

[0047] It can be understood that each module or unit involved in the embodiments of the present disclosure may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple modules or units may be integrated into one entity structure.

[0048] It will be understood that, in the absence of conflict, the functions and steps marked in the flowcharts and block diagrams of the present disclosure may occur in an order different from that marked in the drawings.

[0049] It is understood that the flowcharts and block diagrams of the present disclosure illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present disclosure. Each box in the flowchart or block diagram may represent a module, unit, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based device that implements the specified functions, or may be implemented by a combination of hardware and computer instructions.

[0050] It can be understood that the modules and units involved in the embodiments of the present disclosure can be implemented by software or hardware, for example, the modules and units can be located in a processor.

[0051] In order to facilitate the understanding of the present disclosure, the main inventive concept of the present disclosure is first introduced in combination with specific application methods.

[0052] The present disclosure can specifically provide a data middle platform intelligent operation and maintenance method based on RPA (Robotic Process Automation). In the research in the field of intelligent operation and maintenance, in recent years, because RPA can realize the process linkage between different systems within the enterprise in a non-invasive way, automate repetitive tasks to reduce human errors, improve operational efficiency, release human resources, and improve the quality of operation and maintenance, the use of RPA to participate in intelligent operation and maintenance is particularly favored by researchers. Telecom operators build a data middle platform to manage data processes, and use the data middle platform to package processing tasks into "processes" to simplify the complexity of database tasks. With the advancement of the construction of the digital middle platform, users have more and more urgent demands for the timeliness and correctness of data operations. Therefore, building a data middle platform based on RPA to achieve intelligent operation and maintenance is a development direction.

[0053] However, the existing RPA technology for operation and maintenance is often just a fixed program. When RPA development is separated from the operation and maintenance of the data center, it is difficult to adjust dynamically based on the feedback of the operation and maintenance personnel. At the same time, the existing technology is often difficult to meet the root cause analysis of processing failures in the data center. A more intelligent and adaptive operation and maintenance robot process automation solution is needed. In summary, the existing RPA development and operation and maintenance technology has the following problems:

[0054] 1) Scripts are often run at fixed times and locations based on rules. This approach is relatively mechanical and difficult to dynamically adjust based on feedback from operation and maintenance personnel and changes in demand. At the same time, the lack of effective interaction with operation and maintenance personnel makes it difficult for operation and maintenance personnel to quickly obtain RPA support when facing problems.

[0055] 2) In actual applications, the data center often faces the problem of insufficient adaptability to complex business scenarios. Since the data center involves a large number of data processing, conversion and analysis tasks, and traditional RPA operation and maintenance technologies usually rely on fixed scripts and process templates, it is difficult to flexibly respond to these changes. Therefore, when a fault occurs in the data processing process, traditional RPA operation and maintenance technology often cannot accurately locate the actual location of the fault, which not only increases the difficulty of troubleshooting and repairing, but also seriously affects the stability and efficiency of business operations.

[0056] The new generation of operation and maintenance RPA not only needs to be able to execute preset tasks, but also needs to be able to learn and adapt to changes in the operation and maintenance environment to achieve close integration and real-time interaction with the data center operation and maintenance. The innovative features of this disclosure include:

[0057] 1) Apply RPA technology to the data warehouse (data center) operation and maintenance field. Through the RPA solution, 7×24 hours of automated data warehouse regular inspections are realized, ensuring the high availability and stability of the data warehouse system. Whether it is daily data quality inspection, performance monitoring, or timely discovery and handling of abnormal situations, RPA can respond quickly, providing strong technical support and decision-making basis for the operation and maintenance team.

[0058] 2) In order to further improve the fault location capability of RPA in data warehouse operation and maintenance, HookSensor technology is proposed to sense data processing failures and data fluctuations in the data warehouse system. By combining HookSensor technology with optical character recognition (OCR) technology, it can accurately capture and analyze various logs, error messages, and operation interfaces in the data warehouse system, thereby greatly shortening the time for troubleshooting and repairing, and improving overall operation and maintenance efficiency.

[0059] 3) In order to give RPA more intelligent interactive capabilities and higher convenience, RPA is also integrated with Rasa conversational robot technology. With its powerful natural language processing capabilities and flexible conversation management mechanism, Rasa conversational robots allow users to control RPA to perform complex operation and maintenance tasks through simple conversational instructions. This combination not only greatly reduces the threshold for using RPA, but also significantly improves RPA's response speed and user experience. Whether it is querying the status of the data warehouse, initiating inspection tasks, or requesting troubleshooting suggestions, users only need to communicate with the Rasa conversational robot through natural language (either through RPA automatic inspection or through the RASA conversational robot to control RPA) to obtain timely, accurate, and personalized services, truly realizing the intelligence and automation of data warehouse operation and maintenance.

[0060] The specific embodiments of the present disclosure will be described in more detail below in conjunction with the above applications.

[0061] Embodiment 1:

[0062] like Figure 1 As shown, the present disclosure provides an intelligent operation and maintenance method, the method comprising:

[0063] S1. In response to a data flow execution error, obtain a blood relationship table and an execution record table corresponding to the data flow;

[0064] S2. Construct a data flow fault graph model based on the blood relationship table and the execution record table. The data flow fault graph model includes the before and after relationship of each data node executing the data flow and the number of real data items.

[0065] S3. Based on the data flow fault graph model, the number of fitted data items of each data node is obtained according to the number of real data items of the preceding data node, and the data node of the root cause of the fault in the data flow is determined according to the number of real data items and the number of fitted data items.

[0066] In this embodiment, the contextual relationship table and the execution record table are used to obtain the contextual relationship and the real number of data items of each data node executing the data flow, and then the number of fitted data items is obtained based on the preceding data nodes of each data node in the contextual relationship. Finally, the real number of data items and the number of fitted data items are compared to determine whether the data node is a faulty node that causes an execution error in the data flow.

[0067] Specifically, this embodiment can provide Figure 2 The intelligent operation and maintenance device shown corresponds to the method, and at least includes an operation and maintenance data acquisition unit 1 that executes step S1, a fault graph model construction unit 2 that executes step S2, and a fault root cause analysis unit 3 that executes step S3.

[0068] More specifically, the device provided in this embodiment may also be, for example Figure 3 The intelligent operation and maintenance system shown in the figure or the HookSensor troubleshooting part of the system. The system is based on the RPA platform, and combines the HookSensor troubleshooting technology with the OCR technology to form an information acquisition module. Through OCR operation and maintenance detection, the database operation and maintenance information is queried and supplemented, and handed over to HookSensor for troubleshooting, which improves the timeliness of RPA operation in the data center; by combining RPA with the RASA dialogue robot, an intelligent dialogue module is formed, including obtaining HookSensor troubleshooting results and / or operation and maintenance personnel commands as information input, and through NLU intent recognition and entity recognition, Tracker dialogue state tracking, Policy dialogue strategy learning and ACTION dialogue action generation, the RPA support dialogue function is realized, which improves the timeliness and flexibility of operation and maintenance; through RPA empowering data operation and maintenance and data governance work, a capability expansion module is formed, and by receiving the results generated by the intelligent dialogue module, intelligent operation and maintenance, intelligent notification, daily report generation and other functions are realized; the operation and maintenance efficiency of the data center is greatly improved, and cost reduction and efficiency improvement are achieved. Among them, the HookSensor troubleshooting technology is as follows Figure 4 As shown in the figure, it includes three functions: ① operation and maintenance data collection, ② fault graph model construction and ③ fault root cause analysis. The data flow fault graph constructed is as follows Figure 5 shown.

[0069] In one implementation, in response to a data flow execution error, obtaining a blood relationship table and an execution record table corresponding to the data flow specifically includes:

[0070] Automatically inspect data processes based on robotic process automation (RPA);

[0071] The data center collects and obtains the blood relationship table and execution record table of the data process in real time;

[0072] In response to the automatic inspection discovering a data flow error, the blood relationship table and execution record table corresponding to the data flow are read from the data center based on the hook sensor HookSensor.

[0073] In this embodiment, in the operation and development practice of the data middle platform, many error situations are often encountered. For common errors, such as database connection exceptions, due to the use of the re-entry mechanism, by writing specific RPA scripts, it is usually possible to effectively implement re-entry processing (re-execute the data warehouse processing process), or send notifications to the development team in time to take action. However, in the face of other problems, such as delayed startup or empty data, since the root causes of these problems may be deeply hidden in the previous data source, if different faults occur simultaneously, the actual situation will be even more complicated. Therefore, it is necessary to introduce more advanced mechanisms for in-depth verification. These mechanisms need to have stronger traceability and diagnosis capabilities in order to accurately identify and solve the fundamental problems hidden in complex data flows. In view of this, this embodiment proposes HookSensor troubleshooting technology, which is specifically a data middle platform fault perception technology based on a linear model, which is used to troubleshoot missing and abnormal data nodes. The architecture of HookSensor Figure 4 The diagram includes: setting an operation and maintenance aggregation layer and an indicator collection layer in the operation and maintenance data collection unit. The indicator collection layer collects, formats, processes and stores operation and maintenance indicators in real time. The operation and maintenance aggregation layer uses the OCR operation and maintenance page to identify and retrieve the operation and maintenance indicators stored in the indicator collection layer, and submits them to RPA for data statistics, and outputs indicator data related to the fault; setting a process processing layer and a service call layer in the fault graph model construction unit. The process processing layer processes the data indicators related to the fault into Figure 5 The data flow fault graph shown in the figure shows the lineage relationship and execution record of each node in the data flow. The service call layer displays the relevant data nodes in the data flow fault graph in combination with their actual topological structure in the network; the fault root cause analysis unit combines the data flow fault graph with the HookSensor linear analysis model to locate the fault data node and the root cause fault data node, such as Figure 4 In the figure, F2 and F3 are both faulty data nodes, and F3 is the root cause faulty data node.

[0074] In one implementation, the automatic inspection finds that the data flow reports an error, specifically including:

[0075] Based on RPA receiving the data flow, the data flow includes a specified time when the specified data should arrive at a certain data node. In response to the specified data not arriving at the certain data node at the specified time, based on RPA, the data flow error information is displayed on the monitoring alarm page of the data middle station.

[0076] In this embodiment, the data process can be obtained through RPA and formulated by the operation and maintenance personnel. For example, a data process is to store broadband traffic data, and the subsequent process is to label the active broadband users of the day based on the above data. Data storage and labeling form the relationship between the front node and the back node. The actual data process may trigger a large amount of subsequent data processing. After RPA obtains the information of each execution link of the data process, for example, the process shows that a node hopes to see a certain table at 10 o'clock, but if this table does not arrive on time one day, it may be that a certain table is delayed. The reason may be that the data suddenly surges on that day, etc. It may be caused by the front table or even the front table of the front table. At this time, it is necessary to use the HookSensor linear model for prediction. The data center will provide an operation and maintenance page (such as Figure 6 The current error is displayed, or the data processing flow page of the fault is provided. RPA can read the information on these pages to determine whether there is an error message. The error message can be exported to Excel and handed over to RPA for running. RPA can see the detailed error information and preliminarily determine the type of error based on the error message.

[0077] In one embodiment, the method further comprises:

[0078] Based on RPA, check the database connection or audit the data table. In response to discovering abnormal database connection or data table audit error, read the database connection status or data table field from the data center based on HookSensor to locate the abnormal database connection or the wrong data table field.

[0079] Automatically reconnect the database based on RPA, or send data table field error information to the operation and maintenance personnel based on RPA and receive the data table fields corrected by the operation and maintenance personnel, or send the root cause data node information of the fault to the operation and maintenance personnel based on RPA.

[0080] In this embodiment, for other errors, such as database connection exceptions or audit errors, HookSensor also supports obtaining metadata information from the data center, such as database connection status, table field length, etc., and comprehensively locating and judging the final fault through metadata information and processing flow information from the data center, thereby realizing intelligent judgment of the fault location. By accurately judging the fault location, RPA can intelligently realize error notification and error processing. The following Table 1 shows the common error types and their processing measures of the data center.

[0081] Table 1 Common error types and treatment measures of the data center

[0082]

[0083] In one implementation, a data flow fault graph model is constructed based on the blood relationship table and the execution record table. The data flow fault graph model includes the before and after relationship of each data node executing the data flow and the number of real data items, specifically including:

[0084] Obtain each data node of the execution data flow and its execution before and after relationship from the blood relationship table;

[0085] Obtain the actual number of data items inserted into each data node for executing the data flow from the execution record table;

[0086] A directed acyclic graph (DAG) is established for each data node that executes the data flow. Directed arrows are used in the DAG to connect each data node according to the execution before and after relationship, and the actual number of data inserted into each data node is marked.

[0087] In this embodiment, HookSensor mainly includes three parts: ① operation and maintenance data collection, ② fault graph model construction, and ③ fault root cause analysis. HookSensor uses OCR to sort and count the data on the operation and maintenance page, and reads operation and maintenance related data indicators from the database of the data center (exports data from the database). Since the data center provides a lineage table and an execution record table, the lineage table shows the front and back relationships of the data processing flow, that is, each piece of data in the lineage table corresponds to the possible predecessor node information and successor node information of a process, and each piece of data in the execution record table corresponds to the number of data inserted each time the process is executed, which can be used to judge the quality of the data processing flow. The data flow of the data center can be modeled and represented by a DAG (Directed Acyclic Graph), such as Figure 5 As shown, F_PROCESS indicates the sequence of nodes, and DF_COUNT indicates the number of inserted data.

[0088] The table structure of the blood relationship table is as follows:

[0089] CREATE TABLE SH_031_SJJH.META_RELATION_ALL COMMENT 'blood relationship table' (

[0091] MONTH_ID VARCHAR(10),

[0092] DAY_ID VARCHAR(5),

[0093] SOURCE_OWNER VARCHAR(200) comment'Source table user',

[0094] SOURCE_TABLE_CODE VARCHAR(200) comment'Source table name',

[0095] SOURCE_TABLE_NAME VARCHAR(500)comment'Source table name',

[0096] SOURCE_RESOURCE_NAME VARCHAR(100) comment'Source table belongs to the resource',

[0097] DES_OWNER VARCHAR(200) comment'target table user',

[0098] DES_TABLE_CODE VARCHAR(500) comment'target table name',

[0099] DES_TABLE_NAME VARCHAR(200) comment'target table name',

[0100] DES_RESOURCE_NAME VARCHAR(200) comment'The resource to which the target table belongs',

[0101] WORKFLOW_NAME VARCHAR(500)comment'Process name',

[0102] WORKFLOW_DIR VARCHAR(500) comment'Process directory',

[0103] MAPPING_LAST_EXECUTE_BATCH VARCHAR(500) comment'Mapping the last scheduled batch referenced by the process',

[0104] MAPPING_LAST_EXECUTE_TIME VARCHAR(200) comment'Last execution time' )

[0106] PARTITIONED BY(MONTH_ID,DAY_ID)

[0107] The execution record table records how many data items a certain table has run. You can view this from a system table, such as:

[0108] CREATE TABLE SH_031_SJJH.ETL_FIELD_INFORMATION_SHANGHAI COMMENT 'Full process log table' (

[0110] MONTH_ID CHAR(10)comment'month',

[0111] DAY_ID CHAR(8)comment'day',

[0112] BATCHVALUE VARCHAR(200)comment'batch',

[0113] WORKFLOWNAME VARCHAR(200)comment'Process name',

[0114] TABLENAME VARCHAR(200)comment'table name',

[0115] IS_PARTITIONED_TABLE CHAR(3) comment'Is it a partitioned table?',

[0116] PARTITIONFIELDNAME VARCHAR(200)comment'partition name',

[0117] DSTRESOURCECODE VARCHAR(200)comment'table belongs to resources',

[0118] DSTRESOURCECODE_DESC VARCHAR(12) comment'Description of the resource to which the table belongs',

[0119] TASKSTATUS VARCHAR(12)comment'task status',

[0120] TARGSUCCESSROWS VARCHAR(200)comment'data volume',

[0121] MAPPINGTYPE VARCHAR(12) comment'mapping type',

[0122] MAPPINGTYPE_DESC VARCHAR(12) comment'Mapping type description',

[0123] STARTTIME VARCHAR(200)comment'start time',

[0124] ENDTIME VARCHAR(200)comment'end time',

[0125] DURATION LONG comment'Processing time',

[0126] PROJECTNAME VARCHAR(21) comment'process belongs to the project',

[0127] EXCEPTIONINFO VARCHAR(32767)comment'error information',

[0128] WORKFLOWRUNKEY VARCHAR(200)comment'process instance id' )

[0130] PARTITIONED BY(MONTH_ID,DAY_ID)

[0131] This table records which batches and how many data records were run for each process every day.

[0132] If you know how many data entries are in a table's T-2, T-3, T-4... predecessor table and the current table, then you will also know how many data entries are in its T-1 predecessor table and can deduce the theoretical value of the table.

[0133] In one implementation, obtaining the number of fitted data items of each data node according to the number of real data items of the preceding data node specifically includes:

[0134] Obtain all preceding data nodes [1, 2, ..., j] for each data node i according to the connection relationship in DAG;

[0135] The number of fitted data for each data node i is obtained according to the following formula: r =a0+a1*x1+a2*x2+…+a j *x j , where [a0,a1,…,a j ] is the fitting coefficient obtained in advance based on the number of past real data, [x1,x2,…,x j ] is the actual number of data items in each front-end data node.

[0136] In this embodiment, due to the particularity of the big data platform, most big data tasks present a linear property in space, that is, the number of data items generated by the subsequent tasks is approximately linearly determined by the previous tasks, so a multivariate linear model can be used to fit the results:

[0137] Y i =a0+a1*x1+a2*x2+…+a j *x j

[0138] Among them, x represents the number of data items in each predecessor node, and a represents the coefficient obtained by least square fitting calculation based on the amount of data inserted in the past. According to this model, when the number of data items in the predecessor node is known, the number of data items in its successor node can be approximately estimated.

[0139] In one implementation, determining the fault root cause data node of the data flow according to the number of real data items and the number of fitted data items specifically includes:

[0140] Response to M*Y i ≤Z i ≤N*Y i , judge that data node i is normal, otherwise judge that data node i is a faulty data node in the data flow, where M and N are pre-acquired judgment coefficients, and Z i is the actual number of data items in data node i;

[0141] All faulty data nodes are displayed in the DAG, and the faulty data node located closest to the front is the data node that is the root cause of the fault in the data flow.

[0142] In this embodiment, when the number of successful entries of the data node corresponding to the error reporting process is not within a specific interval, that is, M*Y i ≤Z i ≤N*Y i , where M and N are coefficients used for judgment, which can be set manually or obtained based on the number of normal real data in the past, Z i It is the actual value of the number of data inserted into the node to be judged. It can be determined that the node is not the fault process that causes data problems. Otherwise, it is determined that the node is the fault process that causes data problems (if it is too small, it means that the data may be missing, and if it is too large, it means that the data may be inserted repeatedly).

[0143] In one embodiment, the method further comprises:

[0144] Based on RPA receiving the scheduling request input through the dialogue robot, in response to the fault root cause data node requiring manual processing, the scheduling list including the fault root cause data node information and the operation and maintenance personnel responsible for processing the fault root cause data node is output according to the scheduling request.

[0145] In this embodiment, after completing the precise fault location based on OCR and HookSensor technology, the HookRasa system will perform semantic-level orchestration processing based on the detected fault type and its detailed information, convert it into a form that is easy to parse by Rasa Natural Language Understanding (NLU) and input it into it. Rasa's NLU will perform intent recognition and entity extraction tasks on this problem.

[0146] On this basis, developers can expand the functional scope of the conversational robot by defining personalized actions. For example, RPA technology can be called to perform error processing, error information notification tasks, and dynamically adjust the specific configuration of RPA. Similarly, operation and maintenance personnel can also flexibly modify certain parameters of RPA or implement other functional requirements through interaction with the conversational robot. Through the conversational robot, RPA can be controlled to handle errors, such as updating the information of the on-duty personnel on that day, generating and outputting the duty schedule, etc.

[0147] For example, the process of using Rasa framework for shift scheduling is as follows Figure 7 As shown in the figure, in this process, the problem of missing data needs to be transferred to manual processing, that is, the operation and maintenance personnel are arranged to handle the root cause node located after the data flow error. In the operation and maintenance scenario, when the operation and maintenance personnel make a request "RPA, help me arrange the duty schedule for 202410", the natural language understanding (NLU) component immediately performs an in-depth analysis of the sentence and accurately identifies the user's intention as "duty scheduling request". After identifying the intention, the NLU passes the information to the dialogue state tracker (Tracker). After receiving this intention, the Tracker further classifies it into the predefined "duty scheduling" story line (story). After the Policy confirms the user's request, the Action module is immediately activated. The module performs the corresponding scheduling operation according to the user's request content, arranges the corresponding maintenance person in charge according to the location, type and other information of the root cause node, and generates an Excel file containing the scheduling information. After the scheduling is completed, RPA sends the generated Excel file to the relevant personnel or departments according to the scheduling information provided by the Action module.

[0148] like Figure 8 As shown, Figure 2 The system shown in the figure is dispatched by Rasa and based on the RPA platform, including HookSensor perception technology, to achieve real-time process inspection, real-time error notification, key report audit, common issuance notification, process modification notification and other operation and maintenance functions. Real-time process inspection means that RPA can perceive the completion of the current process in real time and perform corresponding fault repair. Real-time error notification means that RPA can dynamically pay attention to the current process of error or audit error, and automatically notify the on-duty person to handle it. Key report audit means that RPA can focus on the generation and audit of important reports. Intelligent daily report generation means that RPA can summarize the completion and response of the day's duty and send duty emails. Process modification notification means that RPA can read the process modification of the day in the ETL data warehouse (data warehouse), and verify and mark the process modification that has not been reported.

[0149] This embodiment uses RPA to empower data operation and maintenance and data governance. It uses RPA to implement extended notification sending without modifying the underlying code of the ETL (Extract, Transform, Load, which stands for extraction, transformation and loading, a data processing technology) data warehouse. The improved data middle platform can not only send regular text messages and outbound calls through RPA, but also use WeChat and email to report operation and maintenance status and send alarms. RPA can also handle common error types. It is estimated that more than 375 hours of work will be saved each year, which greatly saves the amount of manual operations.

[0150] Embodiment 2:

[0151] like Figure 2 As shown, the present disclosure provides an intelligent operation and maintenance device, the device comprising:

[0152] The operation and maintenance data collection unit 1 is used to obtain the blood relationship table and execution record table corresponding to the data flow in response to the data flow execution error;

[0153] The fault graph model building unit 2 is connected to the operation and maintenance data collection unit 1 and is used to build a data flow fault graph model according to the blood relationship table and the execution record table. The data flow fault graph model includes the before and after relationship of each data node executing the data flow and the number of real data items;

[0154] The fault root cause analysis unit 3 is connected to the fault graph model construction unit 2, and is used to obtain the number of fitted data items of each data node according to the real data items of the preceding data node based on the data flow fault graph model, and to determine the fault root cause data node of the data flow according to the real data items and the fitted data items.

[0155] It should be noted that the device may specifically be a data center based on RPA (Robotic process automation).

[0156] In one embodiment, the operation and maintenance data collection unit 1 specifically includes:

[0157] Automatic inspection subunit, used to automatically inspect data processes based on Robotic Process Automation (RPA);

[0158] The real-time recording subunit is used for the data center to collect and obtain the blood relationship table and execution record table of the data process in real time;

[0159] The data reading sub-unit is connected to the automatic inspection sub-unit and the real-time recording sub-unit, and is used to respond to data flow errors found by automatic inspection, and read the blood relationship table and execution record table corresponding to the data flow from the data middle platform based on the hook sensor HookSensor.

[0160] In one embodiment, the automatic inspection subunit specifically includes a data flow error reporting subunit, which is used for automatic inspection to find data flow errors, specifically for:

[0161] Based on RPA receiving the data flow, the data flow includes a specified time when the specified data should arrive at a certain data node. In response to the specified data not arriving at the certain data node at the specified time, based on RPA, the data flow error information is displayed on the monitoring alarm page of the data middle station.

[0162] In one embodiment, in the device:

[0163] The automatic inspection subunit is also used to check database connections or audit data tables based on RPA;

[0164] The data reading subunit is also used to respond to the discovery of database connection abnormality or data table audit error, and read the database connection status or data table field from the data middle platform based on HookSensor to locate the database with abnormal connection or the wrong data table field;

[0165] The device also includes a fault processing unit, a data reading subunit and a fault root cause analysis unit 3, which is used to automatically perform database reconnection based on RPA, or send data table field error information to the operation and maintenance personnel based on RPA and receive the data table fields corrected by the operation and maintenance personnel, or send fault root cause data node information to the operation and maintenance personnel based on RPA.

[0166] In one implementation, the fault graph model building unit 2 specifically includes:

[0167] The blood relationship subunit is used to obtain the data nodes of the execution data flow and their execution before and after relationships from the blood relationship table;

[0168] An execution record subunit is used to obtain the actual number of data items inserted into each data node for executing the data flow from the execution record table;

[0169] The fault graph subunit is connected to the blood relationship subunit and the execution record subunit, and is used to establish a directed acyclic graph DAG including each data node that executes the data flow. In the DAG, directed arrows are used to connect each data node according to the execution before and after relationship, and the actual number of data inserted into each data node is marked.

[0170] In one embodiment, the fault root cause analysis unit 3 includes a fitting calculation subunit, which is used to obtain the number of fitting data items of each data node according to the number of real data items of the front data node, specifically for:

[0171] Obtain all preceding data nodes [1, 2, ..., j] for each data node i according to the connection relationship in DAG;

[0172] The number of fitted data for each data node i is obtained according to the following formula: i =a0+a1*x1+a2*x2+…+a j *x j , where [a0, a1, ..., a j ] is the fitting coefficient obtained in advance based on the number of past real data, [x1, x2, ..., x j ] is the actual number of data items in each front-end data node.

[0173] In one embodiment, the fault root cause analysis unit 3 includes a comparison judgment subunit, which is used to judge the fault root cause data node of the data flow according to the number of real data items and the number of fitted data items, and is specifically used to:

[0174] Response to M*Y i ≤Z i ≤N*Y i , judge that data node i is normal, otherwise judge that data node i is a faulty data node in the data flow, where M and N are pre-acquired judgment coefficients, and Z i is the actual number of data items in data node i;

[0175] All faulty data nodes are displayed in the DAG, and the faulty data node located closest to the front is the data node that is the root cause of the fault in the data flow.

[0176] In one embodiment, the device further includes a fault handling unit, specifically including a shift scheduling subunit, for:

[0177] Based on RPA receiving the scheduling request input through the dialogue robot, in response to the fault root cause data node requiring manual processing, the scheduling list including the fault root cause data node information and the operation and maintenance personnel responsible for processing the fault root cause data node is output according to the scheduling request.

[0178] Embodiment 3:

[0179] Embodiment 3 of the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the intelligent operation and maintenance method as described in Embodiment 1 or the intelligent operation and maintenance device as described in Embodiment 2 is implemented.

[0180] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program elements or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0181] In addition, the present disclosure may also provide a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes the intelligent operation and maintenance method as described in Example 1. The computer device may be the intelligent operation and maintenance device as described in Example 2.

[0182] The memory is connected to the processor, the memory may be a flash memory or a read-only memory or other memory, and the processor may be a central processing unit or a single-chip microcomputer.

[0183] Embodiments 1-3 of the present disclosure provide an intelligent operation and maintenance method, an intelligent operation and maintenance device, and a computer-readable storage medium. Through a blood relationship table and an execution record table, the before and after relationships and the actual number of data items of each data node executing a data flow are respectively obtained, and a data flow fault graph model is constructed accordingly. Then, on the basis of the data flow fault graph model, the number of fitted data items of each data node is obtained according to the preceding data nodes in the before and after relationships. Finally, the number of actual data items and the number of fitted data items are compared to determine whether the data node is a fault node causing an error in the execution of the data flow, thereby locating the fault and improving the operation and maintenance efficiency.

[0184] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. An intelligent operation and maintenance method, characterized in that: The method comprises: In response to a data flow execution error, obtaining a blood relationship table and an execution record table corresponding to the data flow; A data flow fault graph model is constructed based on the blood relationship table and the execution record table. The data flow fault graph model includes the before and after relationship of each data node executing the data flow and the number of real data items. Based on the data flow fault graph model, the number of fitted data items of each data node is obtained according to the real data items of the preceding data node, and the data node of the root cause of the fault in the data flow is determined according to the real data items and the fitted data items.

2. The method according to claim 1, characterized in that In response to a data flow execution error, a blood relationship table and an execution record table corresponding to the data flow are obtained, specifically including: Automatically inspect data processes based on robotic process automation (RPA); The data center collects and obtains the blood relationship table and execution record table of the data process in real time; In response to the automatic inspection discovering a data flow error, the blood relationship table and execution record table corresponding to the data flow are read from the data center based on the hook sensor HookSensor.

3. The method according to claim 2, characterized in that Automatic inspection finds data flow errors, including: Based on RPA receiving the data flow, the data flow includes a specified time when the specified data should arrive at a certain data node. In response to the specified data not arriving at the certain data node at the specified time, based on RPA, the data flow error information is displayed on the monitoring alarm page of the data middle station.

4. The method according to claim 2, characterized in that: The method further comprises: Based on RPA, check the database connection or audit the data table. In response to discovering abnormal database connection or data table audit error, read the database connection status or data table field from the data center based on HookSensor to locate the abnormal database connection or the wrong data table field. Automatically reconnect the database based on RPA, or send data table field error information to the operation and maintenance personnel based on RPA and receive the data table fields corrected by the operation and maintenance personnel, or send the root cause data node information of the fault to the operation and maintenance personnel based on RPA.

5. The method according to any one of claims 1 to 4, characterized in that: The data flow fault graph model is constructed based on the blood relationship table and the execution record table. The data flow fault graph model includes the before and after relationship of each data node executing the data flow and the number of real data items, including: Obtain each data node of the execution data flow and its execution before and after relationship from the blood relationship table; Obtain the actual number of data items inserted into each data node for executing the data flow from the execution record table; A directed acyclic graph (DAG) is established for each data node that executes the data flow. Directed arrows are used in the DAG to connect each data node according to the execution before and after relationship, and the actual number of data inserted into each data node is marked.

6. The method according to claim 5, characterized in that The number of fitted data items of each data node is obtained according to the number of real data items of the preceding data node, including: Obtain all preceding data nodes [1, 2, ..., j] for each data node i according to the connection relationship in DAG; The number of fitted data for each data node i is obtained according to the following formula: i =a0+a1*x1+a2*x2+…+a j *x j , where [a0,a1,…,a j ] is the fitting coefficient obtained in advance based on the number of past real data, [x1,x2,…,x j ] is the actual number of data items in each front-end data node.

7. The method according to claim 6, characterized in that The root cause data node of the data flow is determined based on the number of real data and the number of fitted data, including: Response to M*Y i ≤Z i ≤N*Y i , judge that data node i is normal, otherwise judge that data node i is a faulty data node in the data flow, where M and N are pre-acquired judgment coefficients, and Z i is the actual number of data items in data node i; All faulty data nodes are displayed in the DAG, and the faulty data node located closest to the front is the data node that is the root cause of the fault in the data flow.

8. The method according to claim 7, characterized in that The method further comprises: Based on RPA receiving the scheduling request input through the dialogue robot, in response to the fault root cause data node requiring manual processing, the scheduling list including the fault root cause data node information and the operation and maintenance personnel responsible for processing the fault root cause data node is output according to the scheduling request.

9. An intelligent operation and maintenance device, characterized in that: The device comprises: An operation and maintenance data collection unit, used to obtain a blood relationship table and an execution record table corresponding to the data flow in response to a data flow execution error; The fault graph model building unit is connected to the operation and maintenance data collection unit and is used to build a data flow fault graph model according to the blood relationship table and the execution record table. The data flow fault graph model includes the before and after relationship of each data node executing the data flow and the number of real data items. The fault root cause analysis unit is connected to the fault graph model construction unit, and is used to obtain the number of fitted data items of each data node according to the real data items of the preceding data node based on the data flow fault graph model, and to determine the fault root cause data node of the data flow according to the real data items and the fitted data items.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the intelligent operation and maintenance method according to any one of claims 1 to 8 is implemented.

Images