Troubleshooting methods, apparatus and computer-readable storage media

Abstract

This application relates to the field of big data processing and discloses a fault handling method, apparatus, and computer-readable storage medium. The method includes: acquiring tool logs and task execution logs when a data warehouse task failure occurs; the tool logs are the work logs of big data tools and devices in a big data processing system executing the data warehouse task; the task execution logs are the work logs of a big data platform in the big data processing system executing the data warehouse task; processing the tool logs and task execution logs using a target fault classification model to obtain a fault category; the fault category is a tool-side fault, a data warehouse-side fault, or a platform-side fault; determining the fault responsible party based on the fault category and sending alarm information to the fault responsible terminal; the fault responsible terminal is the fault responsible terminal corresponding to the fault responsible party; the alarm information is used to instruct the fault responsible party to handle the fault occurring in the data warehouse task. Applying the technical solution of this invention can improve fault handling efficiency.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of big data technology, specifically to a fault handling method, apparatus, and computer-readable storage medium. Background Technology

[0002] With the development of the big data industry, every company has a large number of ETL (Extract-Transform-Load) tasks to maintain, especially when data failures occur, requiring alerts to the responsible parties for rapid remediation. However, because the execution chain of ETL tasks is relatively long, involving data warehouses, big data platforms, and big data tools, ETL task failure alerts are often sent to people who are not actually responsible for the failure or capable of resolving it. Furthermore, the entire process requires manual step-by-step judgment and contacting potential responsible parties, resulting in significant waste of manpower and communication costs, and low efficiency in resolving the failure. Summary of the Invention

[0003] In view of the above problems, embodiments of the present invention provide a fault handling method, apparatus and computer-readable storage medium to solve the problems of difficulty in determining fault responsibility and low fault resolution efficiency in the prior art.

[0004] Firstly, this application provides a fault handling method, which includes: when a data warehouse task fault occurs, obtaining tool logs and task execution logs; the tool logs are the work logs of big data tools and devices in the big data processing system executing the data warehouse task; the task execution logs are the work logs of the big data platform in the big data processing system executing the data warehouse task; inputting the tool logs and task execution logs into a target fault classification model to obtain a fault category; the fault category is any one of the following: tool-side fault, data warehouse-side fault, platform-side fault; based on the fault category, determining the fault responsible party and sending alarm information to the fault responsible terminal; the fault responsible terminal is the fault responsible terminal corresponding to the fault responsible party; the alarm information is used to instruct the fault responsible party to handle the fault that occurred in the data warehouse task.

[0005] In one possible design approach of the first aspect, the responsible party for the fault is determined according to the fault category, and an alarm message is sent to the terminal responsible for the fault. This includes: if the fault category is a tool-side fault, then the responsible party for the big data tool and the communication information of the responsible party are determined according to the responsibility table of the big data tool and equipment; the responsibility table includes the communication information of multiple tool managers, and the correspondence between the tool managers and the time periods they are responsible for the big data tool and equipment; the responsible party for the fault is determined, and an alarm message is sent to the terminal responsible for the fault using the communication information of the responsible party.

[0006] In one possible design approach of the first aspect, the responsible party for the fault is determined and an alarm message is sent to the terminal responsible for the fault based on the fault category, including: if the fault category is a data warehouse side fault, obtaining the first change record of the first subtask of the data warehouse task; the first change record is either a version change record of the first subtask or a table change record of the first subtask; if it is determined from the first change record that the first subtask has a task version change or a task table change, then the latest first changer corresponding to the first subtask is determined as the responsible party for the fault and an alarm message is sent to the first change terminal corresponding to the first changer.

[0007] In one possible design of the first aspect, after obtaining the first change record of the first subtask of the data warehouse task, the method further includes: if it is determined from the first change record that the first subtask does not have a task version change or a task table change, then obtaining the second change record of the second subtask; the second subtask is the upstream task of the first subtask in the data warehouse task; the second change record is the version change record of the second subtask or the table change record of the second subtask; if it is determined from the second change record that the second subtask has a task version change or a task table change, then identifying the second person responsible for the new version change of the second subtask as the person responsible for the fault and sending an alarm message to the second terminal corresponding to the second person responsible for the fault; if it is determined from the second change record that the second subtask does not have a task version change or a task table change, then identifying the latest first person responsible for the fault corresponding to the first subtask as the person responsible for the fault and sending an alarm message to the first terminal corresponding to the first person responsible for the fault.

[0008] In one possible design approach of the first aspect, the responsible party for the fault is determined and an alarm message is sent to the terminal responsible for the fault based on the fault category. This includes: if the fault category is a platform fault, obtaining the cluster operation status parameters of the big data platform; if the cluster operation status parameters indicate that the target component of the big data platform is abnormal, determining the component responsible party corresponding to the target component as the responsible party for the fault, and sending an alarm message to the terminal responsible for the component responsible party; if the cluster operation status parameters indicate that the configuration parameters of the big data platform have changed, determining the platform responsible party corresponding to the big data platform as the responsible party for the fault, and sending an alarm message to the terminal responsible for the platform responsible party.

[0009] In one possible design of the first aspect, after obtaining the cluster operating status parameters of the big data platform, the method further includes: if the cluster operating status parameters indicate that there are no abnormalities in all components of the big data platform and the configuration parameters of the big data platform have not been changed, then the latest first changer corresponding to the first subtask of the data warehouse task is identified as the fault responsible person and an alarm message is sent to the first change terminal corresponding to the first changer.

[0010] In one possible design approach of the first aspect, before calling the target fault classification model to process the tool logs and task execution logs, the method further includes: obtaining multiple sets of sample data and the sample categories corresponding to the multiple sets of sample data; the sample data are the tool logs and task execution logs when a data warehouse task failure occurs in the big data processing system; the sample category corresponding to the sample data is the fault category of the sample data; using the sample data as training data and the sample category as supervision information, the initial fault classification model is iteratively trained to obtain the target fault classification model.

[0011] Based on the technical solution provided in this application, the working logs (i.e., tool logs) of big data tools and devices and the working logs (i.e., task execution logs) of the big data platform during the execution of a data warehouse task (i.e., an ETL task) can be obtained first. Since the tool logs and task execution logs can characterize which part of the big data processing system caused the failure in the data warehouse task, the tool logs and task execution logs can be input into the target fault classification model to obtain the fault category of the data warehouse task. The fault category can be a tool-side fault, a data warehouse-side fault, or a platform-side fault. That is, at this point, it can be roughly determined which part of the big data processing system caused the failure in the data warehouse task, preventing the task from executing normally. Because it is now known which part of the big data processing system caused the fault, the responsible party can be accurately identified based on the determined fault category, and an alarm message can be sent to the corresponding fault-responsible terminal so that the responsible party can handle the fault in the data warehouse task. It can be seen that, based on the technical solution provided in this application, the fault category when a data warehouse task fails can be automatically determined, thereby accurately identifying the responsible party. Compared to existing technologies, this method reduces the complexity of determining fault liability, lowers communication costs during the fault liability determination process, and improves the efficiency of fault resolution.

[0012] Secondly, this application provides a fault handling device, which includes an acquisition module and a processing module.

[0013] The module includes an acquisition module, which acquires tool logs and task execution logs when a data warehouse task fails. Tool logs are the work logs of the big data tools and devices in the big data processing system executing the data warehouse task; task execution logs are the work logs of the big data platform in the big data processing system executing the data warehouse task. The processing module inputs the tool logs and task execution logs acquired by the acquisition module into a target fault classification model to obtain the fault category. The fault category can be any of the following: tool-side fault, data warehouse-side fault, or platform-side fault. The processing module also determines the person responsible for the fault based on the fault category and sends alarm information to the responsible terminal. The responsible terminal is the terminal corresponding to the person responsible for the fault; the alarm information instructs the person responsible for the fault to handle the fault occurring in the data warehouse task.

[0014] Thirdly, an electronic device is provided, including a processor, a memory, a communication interface, and a communication bus. The processor, memory, and communication interface communicate with each other via the communication bus. The memory stores computer instructions. When the computer instructions are executed on the processor, the processor performs the fault handling method as described in any one of the first aspects above.

[0015] Fourthly, a computer-readable storage medium is provided that stores computer instructions, which, when executed on an electronic device, cause the electronic device to perform the fault handling method as described in any one of the first aspects above.

[0016] Fifthly, a computer program product containing computer instructions is provided, which, when executed on an electronic device, causes the electronic device to perform the fault handling method as described in any one of the first aspects above.

[0017] Understandably, the solutions provided in aspects two through ten above are all used to implement the corresponding methods provided in aspect one above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0018] It should be understood that, in this application, the names of the aforementioned fault handling devices and electronic devices do not constitute a limitation on the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the function of each device or functional module is similar to that of the present invention, it falls within the scope of the claims of this disclosure and their equivalents. Furthermore, it should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this disclosure.

[0019] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0020] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0021] Figure 1 A flowchart illustrating a data warehouse task fault handling method provided by existing technology;

[0022] Figure 2 This is a schematic diagram of the structure of a big data processing system provided in an embodiment of this application;

[0023] Figure 3 A flowchart illustrating a fault handling method provided in an embodiment of this application;

[0024] Figure 4 A flowchart illustrating another fault handling method provided in an embodiment of this application;

[0025] Figure 5 A flowchart illustrating another fault handling method provided in an embodiment of this application;

[0026] Figure 6 A flowchart illustrating a training method for a target fault classification model provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of the structure of a fault handling device provided in an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more. "A and / or B" includes the following three combinations: only A, only B, and a combination of A and B.

[0031] With the development of the big data industry, every company has a large number of ETL (Extract-Transform-Load) tasks (i.e., data warehouse tasks in this application) that need to be maintained, especially when data failures occur, requiring alerts to relevant personnel for rapid repair. In practice, the execution chain of data ETL tasks is relatively long, and the big data processing systems used in them involve data warehouses (referred to as data warehouses), big data platforms (referred to as platforms), and big data tools (referred to as tools).

[0032] The data warehouse is primarily responsible for connecting with various business systems, and then extracting business data from the corresponding business systems to perform ETL tasks when a business request is received. The data warehouse is also responsible for the development of all programs and functions related to ETL tasks.

[0033] When implementing ETL tasks and developing related programs or functions, data warehouses need to utilize big data tools. After the data warehouse has developed the ETL-related programs or functions using the development environment provided by the big data tools, it sends these programs or functions, along with their execution sequence logic within the ETL task, to the big data tools so that the big data tools can schedule the various programs or functions.

[0034] A big data platform is a server cluster composed of multiple servers, where different servers can serve as different components, performing different functions. These include storage components, computing components, management components, and access control components. When a data warehouse performs ETL tasks using big data tools, it relies on the computing and storage resources provided by the big data platform to successfully complete the ETL tasks.

[0035] Reference Figure 1 As shown, when an existing ETL task fails, since the ETL task is mainly completed by the data warehouse using tools with the help of the platform's computing and storage resources, an alarm message will first be generated on the data warehouse side when an ETL task fails.

[0036] Upon receiving an alert, the data warehouse manager needs to first determine if there are any new version changes (or deployments) of programs or functions related to the ETL task. If it is confirmed that there are new version changes of programs or functions related to the ETL task, an alert is sent to the terminal of the person who made the change, instructing them to handle it.

[0037] If it is determined that there are no new version changes to the programs or functions related to the ETL task, then it is determined whether the data warehouse side can handle the fault. If it is determined that the data warehouse side can handle the fault, then an alarm message is sent to the terminal corresponding to the person responsible for the current subtask of the ETL task to instruct them to handle the fault.

[0038] If it is determined that the data warehouse cannot handle the fault, a first instruction message is sent to the person in charge of the platform (this first instruction message may be sent to the platform) so that the person in charge of the platform can identify the specific person responsible for the fault.

[0039] Upon receiving the initial instruction, the platform administrator will determine whether the anomaly lies on the tool side (i.e., whether it's a tool malfunction) based on the platform logs (i.e., the platform's operational logs). If it is a tool malfunction, an alarm message will be sent to the terminal corresponding to the administrator, instructing them to handle the issue. Upon receiving the alarm message, the administrator will review the tool logs (or application logs) and address the fault accordingly.

[0040] If the issue isn't with the tool itself, the next step is to determine if there's an anomaly in the ETL task's configuration. If so, the platform provides a solution based on the specific details of the anomaly and sends an alert to the terminal corresponding to the task manager of the current subtask, instructing them to handle the fault according to the solution. If the issue isn't with the task configuration, the problem is likely on the platform side. In this case, the platform manager can obtain the cluster status parameters and component logs of the platform's server cluster to address the fault.

[0041] In summary, it can be seen that existing ETL task fault alarms are often sent to people who are not actually responsible for the fault or capable of resolving it. Furthermore, the entire process requires manual intervention at each level to identify and contact potential fault-holders, resulting in significant waste of manpower and communication costs, and low fault resolution efficiency.

[0042] To address the aforementioned issues, this application provides a fault handling method. In this method, when a data warehouse task (i.e., an ETL task) encounters a fault, tool logs (obtained from the tool side) and task execution logs (obtained from the platform side) capable of representing all possible fault categories during the data warehouse task's execution can be acquired. These tool logs and task execution logs can then be input into a pre-trained target fault classification model to obtain the fault category. Since the fault category can determine which part of the big data processing system the fault occurred in—whether it occurred on the tool side, the data warehouse side, or the platform side—the responsible party can be automatically and accurately identified based on the fault category, and alarm information can be sent to the corresponding fault-responsible terminal. This reduces the complexity of the fault-responsibility identification process, lowers the communication costs incurred during fault-responsibility identification, and improves the efficiency of fault resolution.

[0043] In the embodiments of this application, the data warehouse task may specifically refer to the ETL task. The embodiments disclosed in this application use the data warehouse task as an ETL task as an example for explanation, and will not be explained again thereafter.

[0044] Figure 2 This is a schematic diagram of the structure of a big data processing system for applying a fault handling method according to an exemplary embodiment. (Refer to...) Figure 2 As shown, the big data processing system may include a tool server 01 and a platform server 02. The tool server 01 and the platform server 02 can communicate with each other via wired or wireless communication.

[0045] For example, the tool server 01 in this disclosure can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center; this disclosure does not limit this. The tool server 01 primarily hosts a data warehouse and big data tools for executing ETL tasks. The tool server 01 can also establish communication connections with multiple business servers 03 that host different business systems, facilitating the acquisition of business data from these systems for implementing ETL tasks.

[0046] For example, the platform server 02 in this disclosure can be a server cluster consisting of multiple servers, or a cloud computing service center. The platform server 02 is mainly used to provide the tool server 01 with the storage and computing resources required to perform ETL tasks. Its server cluster can be divided into multiple components, such as storage components, computing components, management components, and access control components.

[0047] Furthermore, tool server 01 or platform server 02 can also be used to train a target fault classification model. This model can then be used to determine the fault category, enabling the fault handling method provided in this application to be implemented smoothly.

[0048] The fault handling method provided in the embodiments of this application will be described in detail below with reference to the service.

[0049] This embodiment provides a fault handling method applied to a fault handling device. The fault handling device may be an electronic device or a part of an electronic device, which may be part of a platform server or a tool server. (See also...) Figure 3 As shown, the method may include S301-S303:

[0050] S301. When an ETL task failure occurs, obtain the tool log and task execution log.

[0051] When an ETL task fails, since the data warehouse is the primary execution entity, it will automatically generate a prominent fault message (e.g., the client used by the operator performing the ETL task) to indicate the failure. Because the electronic device executing this fault handling method can be part of the platform server or the tool server, the fault message can be detected promptly through real-time monitoring or short-term periodic monitoring. Any feasible detection method can be used.

[0052] The tool logs are the work logs of the big data tools and devices in the big data processing system that execute ETL tasks; the task execution logs are the work logs of the big data platform in the big data processing system that executes ETL tasks.

[0053] In actual ETL task execution, an ETL task mainly includes several core stages or sub-tasks such as task scheduling, task distribution, task execution plan generation, and task submission and execution. Anomalies in the task scheduling and task distribution stages are considered tool-side failures. Anomalies in generating the execution plan are usually due to code parsing / compilation errors, which are data warehouse-side failures. Anomalies in the task submission and execution stage are most likely platform-side failures. Since both the data warehouse and the tool side reside on the tool server hosting the big data tool, task scheduling, task distribution, and task execution plan generation are all implemented on the tool server. Therefore, tool-side and data warehouse-side failures can be characterized by the contents of the tool server's corresponding tool logs; that is, the tool logs can be used to determine whether it is one of these two types of failures.

[0054] Task submission and execution are mainly implemented on the platform side, so platform-side failures can be characterized by task execution logs generated by the platform server. In other words, it can be determined whether a platform-side failure is caused by task execution logs.

[0055] Therefore, in order to determine what kind of failure the ETL task caused, it is necessary to first obtain the tool logs and task execution logs.

[0056] S302. Call the target fault classification model to process the tool logs and task execution logs to obtain fault categories.

[0057] Based on the foregoing description, in the implementation of this application, the fault category can be any of the following: tool-side fault, data warehouse-side fault, or platform-side fault.

[0058] In some embodiments, after the electronic device processes the tool logs and task execution logs using the target fault classification model, three probabilities corresponding to tool-side faults, data warehouse-side faults, and platform-side faults can be obtained. For example, v1 corresponds to a tool-side fault, v2 to a data warehouse-side fault, and v3 to a platform-side fault. The final fault category can be the fault category corresponding to the highest of these three probabilities. For example, if the three probability values ​​corresponding to v1-v3 are 0.4, 0.5, and 0.1 respectively, then the final fault category can be the tool-side fault corresponding to v1.

[0059] S303. Based on the fault category, determine the person responsible for the fault and send alarm information to the terminal responsible for the fault.

[0060] Among them, the fault responsibility terminal is the fault responsibility terminal corresponding to the fault responsibility person; the alarm information is used to instruct the fault responsibility person to handle the fault that occurred in the ETL task.

[0061] Once the person responsible for the fault has been identified, their communication information, such as phone number and terminal account, can be obtained based on the correlation between them and their communication records. Furthermore, the electronic device can then send the alarm information to the terminal responsible for the fault based on this communication information.

[0062] For example, the fault-handling terminal in this application embodiment can be a terminal capable of wirelessly communicating with the electronic device executing the fault handling method, such as a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, personal computer (PC), etc. This application embodiment does not impose any special limitations on the specific form of the terminal.

[0063] In some embodiments, to facilitate faster handling of faults by the responsible party, alarm information may also include fault categories and corresponding log information, such as tool logs or task execution logs.

[0064] Based on the technical solution provided in this application, the working logs (i.e., tool logs) of big data tools and devices and the working logs (i.e., task execution logs) of the big data platform during the execution of the ETL task (i.e., data warehouse task) can be obtained first. Since the tool logs and task execution logs can characterize which part of the big data processing system caused the failure in the ETL task, the tool logs and task execution logs can be input into the target fault classification model to obtain the fault category of the ETL task. The fault category can be a tool-side fault, a data warehouse-side fault, or a platform-side fault. That is, at this point, it can be roughly determined which part of the big data processing system caused the failure in the ETL task, preventing the ETL task from executing normally. Because it is now known which part of the big data processing system caused the failure, the responsible party can be accurately identified based on the determined fault category, and an alarm message can be sent to the corresponding fault-responsible terminal so that the responsible party can handle the fault in the ETL task. It can be seen that, based on the technical solution provided in this application, the fault category when an ETL task fails can be automatically determined, thereby accurately identifying the responsible party. Compared to existing technologies, this method reduces the complexity of determining fault liability, lowers communication costs during the fault liability determination process, and improves the efficiency of fault resolution.

[0065] For tool-side failures (or task scheduling / deployment anomalies), they are mainly caused by errors in the tool itself, so the person responsible for the failure can be directly identified as the manager currently in charge of managing the tool.

[0066] Data warehouse-side failures (or code parsing / compile-time exceptions) are primarily caused by code errors corresponding to the data warehouse. These code errors mainly arise because subtasks being executed by the ETL task have undergone logical or table structure modifications (or task version or task table modifications) without rigorous testing and verification, leading to failures when the new version of the subtask is deployed. Similarly, if an upstream subtask has similar changes, the current subtask may also fail. Therefore, for data warehouse-side failures, the specific responsible party can be identified by reviewing the change logs of the current subtask or its upstream subtasks.

[0067] The data warehouse, through its tools, ultimately submits each subtask in the ETL task to the server cluster within the platform for execution. Therefore, in the event of a platform-side failure (or task execution anomaly), if a component, resource, or configuration of the platform's server cluster malfunctions, the entire task will fail even if the code corresponding to the ETL task is correct. Thus, for this type of failure, it is necessary to obtain cluster operational status parameters that reflect the aforementioned anomalies to determine the specific person responsible for the failure.

[0068] Based on the foregoing description, this application also provides another embodiment of a fault handling method. This method can be applied to a fault handling device, which may be an electronic device or a part of an electronic device. The electronic device may be part of a platform server or a tool server. (Refer to...) Figure 4 As shown, the fault handling method in this embodiment may include S401-S413:

[0069] S401. When an ETL task failure occurs, obtain the tool log and task execution log.

[0070] The specific implementation of S401 can be referred to the specific description of S301 in the aforementioned embodiments, and will not be repeated here.

[0071] S402. Call the target fault classification model to process the tool logs and task execution logs to obtain fault categories.

[0072] The specific implementation of S401 can be referred to the specific description of S302 in the aforementioned embodiments, and will not be repeated here.

[0073] After S402, if the fault category is a tool-side fault, the electronic device can determine the fault responsible person based on the shift schedule (which can be called the responsibility table) and send an alarm message to the corresponding tool responsibility terminal. That is, execute the subsequent S403 and S404.

[0074] After S402, if the fault category is a data warehouse-side fault, the electronic device can determine from the change record of the currently executing subtask of the ETL task that there is a task version change or task table change. If so, the person responsible for the fault is identified by the person making the change, and an alarm message is sent to the corresponding terminal. That is, the subsequent S405-S406 are executed.

[0075] Alternatively, if the electronic device determines that the currently executing subtask has not undergone task version changes or task table changes, it can determine the specific fault responsible party based on the change records of its upstream subtasks and send alarm information to the corresponding changed terminal. This triggers subsequent steps S407-S409.

[0076] After S402, if the fault category is a platform-side fault, the specific person responsible for the fault can be determined based on the cluster operation status parameters of the big data platform. If the cluster operation status parameters determine that the anomaly is in certain parts of the platform side, the corresponding person in charge can be identified as the fault responsible party, and alarm information can be sent to the corresponding terminal. That is, execute S410-S413.

[0077] If the cluster's operational status parameters determine that the anomaly is not on the platform side, then the problem can be attributed to an anomaly in the task configuration parameters of the current subtask, leading to the operational error. In this case, the person in charge of the current subtask can be identified as the fault handler, and an alarm message should be sent to the corresponding terminal. This involves executing S414.

[0078] S403. If the fault category is a tool-side fault, then determine the person in charge of the big data tool and equipment and their communication information according to the responsibility table of the big data tool and equipment.

[0079] The responsibility table includes contact information for multiple tool administrators, as well as the correspondence between tool administrators and the time periods for which they are responsible for the big data tools and equipment. For example, this responsibility table can be shown in Table 1 below.

[0080] Table 1 - Responsibility Table

[0081]

[0082]

[0083] S404. Identify the person responsible for the tool as the person responsible for the fault, and use the communication information of the person responsible for the tool to send alarm information to the terminal responsible for the tool.

[0084] Based on the technical solutions corresponding to S403 and S404, when the fault category is a tool-side fault, the current tool responsible person can be automatically identified as the fault responsible person, and an alarm message can be sent to the corresponding tool responsible terminal. This allows ETL task faults to be handled as quickly as possible, improving fault handling efficiency.

[0085] S405. If the fault type is a data warehouse side fault, obtain the first change record of the first subtask of the ETL task.

[0086] The first change record is either the version change record of the first subtask or the table change record of the first subtask. The first subtask can be the subtask currently being executed by the ETL task.

[0087] In practice, every change to the code of each subtask has a corresponding person responsible for the change—the individual who specifically updated the code for the subtask. Since code development is based on the development environment provided by the tool, this person leaves a corresponding change record (the first change record) on the tool side (i.e., the tool server) when making changes. The electronic device can then use this first change record to determine if the first subtask has undergone a version change or a task table change, and if so, identify the responsible person, thus executing S406.

[0088] S406. If it is determined from the first change record that the first subtask has a task version change or a task table change, then the latest first changer corresponding to the first subtask is identified as the person responsible for the fault and an alarm message is sent to the first changer terminal corresponding to the first changer.

[0089] Specifically, if the first change record contains change records within a certain time range (e.g., within 5 seconds) at the current moment, it indicates that the first subtask has undergone a task version change or task table change. Furthermore, to facilitate identification of the responsible party, the first change record will contain the contact information of the person responsible for each change record. Based on this, the electronic device can, upon determining that the first subtask has undergone a task version change or task table change, identify the first person responsible for the fault corresponding to the latest change record in the first change record and send an alarm message to the first change terminal using the first person's contact information.

[0090] Based on the technical solutions corresponding to S405 and S406, if the fault category is a data warehouse-side fault and it is determined that the first subtask being executed has undergone a task version change or task table change, the first person making the change can be identified as the person responsible for the fault, and an alarm message can be sent to the corresponding first change terminal. This allows ETL task faults to be handled as quickly as possible, improving fault handling efficiency.

[0091] S407. If it is determined from the first change record that there is no task version change or task table change for the first subtask, then obtain the second change record for the second subtask.

[0092] In this context, the second subtask is the upstream task of the first subtask in the ETL task. For example, taking an ETL task that includes extraction, transformation, and loading subtasks, the extraction subtask should be executed first, followed by the transformation subtask, and finally the loading subtask. Therefore, it can be determined that the extraction subtask is the upstream task of the transformation subtask, and the transformation subtask is the upstream task of the loading subtask. Furthermore, if the extraction subtask also contains multiple subtasks, then similar upstream and downstream relationships exist between these subtasks. This application does not impose specific limitations in this regard.

[0093] The second change record is either the version change record of the second subtask or the change record of the second subtask table.

[0094] S408. If it is determined from the second change record that the second subtask has a task version change or a task table change, then the second person who made the new version change of the second subtask is identified as the person responsible for the fault and an alarm message is sent to the second change terminal corresponding to the second person who made the change.

[0095] Specifically, if the second change record contains change records within a certain time range (e.g., within 5 seconds) at the current moment, it indicates that the second subtask has undergone a task version change or task table change. Furthermore, to facilitate identification of the responsible party, the second change record will contain the contact information of the person responsible for each change record. Based on this, the electronic device can, upon determining that the second subtask has undergone a task version change or task table change, identify the person responsible for the fault corresponding to the latest change record in the second change record and send an alarm message to the second change terminal using the contact information of the person responsible for the change.

[0096] S409. If it is determined from the second change record that there is no change in the task version or the task table for the second subtask, then the latest first changer corresponding to the first subtask is identified as the person responsible for the fault, and an alarm message is sent to the first changer terminal corresponding to the first changer.

[0097] If the first change record indicates that there are currently no changes to the task version or task table for the first subtask, and the second change record also indicates that there are currently no changes to the task version or task table for the second subtask, then it can be assumed that the anomaly in the currently executing first subtask is likely caused by the previous change. In this case, the person who made the last change in the first change record (i.e., the latest first changer) can be identified as the person responsible for the fault, and an alarm message should be sent to their corresponding first change terminal.

[0098] Based on the technical solutions corresponding to S407-S409, when the fault category is a data warehouse-side fault, if it is determined that the currently executing first subtask has not undergone task version or task table changes, the true fault-holder can be accurately identified based on the second change record of its upstream second subtask, and an alarm message can be sent to the corresponding change terminal. This allows ETL task faults to be handled as quickly as possible, improving fault handling efficiency.

[0099] S410. If the fault category is platform fault, then obtain the cluster operation status parameters of the big data platform.

[0100] Specifically, electronic devices can obtain the cluster operating status parameters of their server cluster from the big data platform in any feasible way. These cluster operating status parameters may include at least: information on whether each component in the server cluster is abnormal, the parameter configuration of the server cluster, and hardware configuration information.

[0101] S411. If the cluster operation status parameters indicate that the target component of the big data platform is abnormal, the component responsible person corresponding to the target component shall be identified as the fault responsible person, and an alarm message shall be sent to the component responsible terminal corresponding to the component responsible person.

[0102] Each component of the big data platform has a corresponding responsible person, whose information (including communication information) can be stored in the big data platform. When an electronic device determines that a target component of the big data platform is abnormal, it can identify the responsible person for the target component from the big data platform and send an alarm message to the corresponding responsible terminal based on their communication information.

[0103] S412. If the cluster running status parameters indicate that the configuration parameters of the big data platform have changed, then the person in charge of the big data platform will be identified as the person responsible for the fault, and an alarm message will be sent to the platform responsibility terminal corresponding to the person in charge of the platform.

[0104] The configuration parameters of a big data platform can specifically include software and hardware configuration parameters. These parameters primarily determine the amount of resources allocated to each component within the big data platform. If the configuration parameters are changed without corresponding changes to the content of the ETL task, the ETL task may fail or malfunction due to insufficient resources being available when it needs to utilize a component of the big data platform.

[0105] The configuration parameters of the big data platform are adjusted and modified by the person responsible for the platform, and this person's information (including communication information) can be stored in the big data platform. When an electronic device determines that the configuration parameters of the big data platform are abnormal, it can identify the person responsible for the platform from the big data platform and send alarm information to the corresponding responsible terminal based on their communication information.

[0106] Based on the technical solutions corresponding to S410-S412, when the fault category is a platform-side fault, the faulty part of the big data platform can be determined according to the cluster's operating status parameters. The responsible party can then be identified, and alarm information can be sent to the corresponding responsible terminal. This ensures that faults in ETL tasks are handled as quickly as possible, improving fault handling efficiency.

[0107] S413. If the cluster running status parameters indicate that there are no abnormalities in all components of the big data platform and the configuration parameters of the big data platform have not been changed, then the latest first changer corresponding to the first subtask of the ETL task is identified as the fault responsible person and an alarm message is sent to the first changer terminal corresponding to the first changer.

[0108] If the cluster's operational status parameters determine that the components and control parameters of the big data platform are free of anomalies or changes, then it's highly likely that the task configuration parameters of the currently executing first subtask are abnormal. For example, the task configuration parameters of this first subtask might indicate that it's executed by server A in the big data platform. However, if server A itself is executing other subtasks, leading to resource shortages, then the first subtask will malfunction. Since the task configuration parameters of the first subtask were modified by the last person to make changes to it, in this case, the latest person who made the change can be identified as the person responsible for the fault, and an alarm message can be sent to the first terminal corresponding to that person.

[0109] For details on how to determine the first person to make the change and how to send alarm information to the first terminal to make the change, please refer to the relevant descriptions in the foregoing embodiments, which will not be repeated here.

[0110] Based on the technical solution corresponding to S413, if the fault category is a platform-side fault, and the cluster operating status parameters determine that the fault is not caused by an anomaly generated by the big data platform, then it can be assumed that the task configuration parameters configured by the last person to make changes to the currently executing first task (i.e., the latest first person to make changes) are abnormal. Therefore, this first person to make changes can be identified as the person responsible for the fault, and an alarm message can be sent to the corresponding first change terminal. This allows faults in ETL tasks to be handled as quickly as possible, improving fault handling efficiency.

[0111] It should be noted that after the responsible party for the fault has been identified in the electronic device and an alarm message has been sent to the corresponding responsible terminal, the responsible party will rerun the subtask that caused the error after handling the corresponding anomaly, so that the entire ETL task can be implemented smoothly. If, after receiving the alarm message, the responsible party finds that the actual fault is not their responsibility, then the responsible party will assign the handling of the fault to the actual responsible party based on the actual situation.

[0112] Based on the technical solutions corresponding to S401-S413 described above, the specific responsible party can be automatically and accurately identified according to the type of fault, and alarm information can be sent to the corresponding fault-responsible terminal. This ensures that faults occurring in ETL tasks are handled as quickly as possible, improving fault handling efficiency.

[0113] To more clearly illustrate the technical solutions provided in the embodiments of this application, another embodiment of the fault handling method is also provided. In this embodiment, taking an electronic device as part of a platform server as an example, refer to... Figure 5 As shown, the method may specifically include S501-S511:

[0114] S501. In the event of an ETL task error (i.e. a failure), obtain the tool log and task execution log.

[0115] The specific implementation of S501 can be referred to S301 in the aforementioned embodiments, and will not be repeated here.

[0116] S502. Determine the fault category based on the tool log and task execution log.

[0117] The specific implementation of S502 can be referred to S302 in the aforementioned embodiment, and will not be repeated here.

[0118] If the fault category is a tool-side fault, execute S503; if the fault category is a data warehouse-side fault, execute S504; if the fault category is a platform-side fault, execute S509.

[0119] S503, handling of alarm tools by the responsible person.

[0120] Specifically, S503 involves sending alarm information to the tool responsibility terminal of the person in charge of the tool. The specific implementation can be found in the description of S404 in the aforementioned embodiment, and will not be repeated here.

[0121] S504. Determine if there are any new version changes in the first subtask.

[0122] Here, the current subtask is the first subtask. Specifically, S504 determines whether the first subtask has undergone a task version change or a task table change based on the first change record of the first subtask. For a detailed implementation, please refer to the relevant description in S406 of the aforementioned embodiment.

[0123] If it is determined that the current subtask has a new version change, then execute S505; if it is determined that the current subtask does not have a new version change, then execute S506.

[0124] S505, alarm first change of person handling.

[0125] The first person to make the change is the latest person to make the change in the change record of the first subtask. The specific implementation of S505 can be found in the relevant description in S406, and will not be repeated here.

[0126] S506. Determine the upstream task of the first subtask.

[0127] Specifically, the electronic device can determine the upstream task of the first subtask based on the settings of each subtask in the ETL task on the data warehouse side. The explanation of the upstream and downstream relationships between subtasks can be found in the relevant descriptions in the foregoing embodiments, and will not be repeated here.

[0128] S507. Determine if there are any new version changes in the second subtask.

[0129] The second subtask is the upstream task of the first subtask.

[0130] Specifically, S507 determines whether the second subtask has undergone a task version change or a task table change based on the second change record of the second subtask. The specific implementation can be found in the relevant descriptions of S407 and S408 in the aforementioned embodiments, and will not be repeated here.

[0131] If it is determined that there is a new version change in the second subtask, then execute S508.

[0132] If it is determined that there are no new version changes in the second subtask, it can be assumed that the anomaly in the currently executing first subtask is likely caused by the previous change. In this case, the person who made the last change in the first change record (i.e., the latest first changer) can be identified as the person responsible for the fault, and an alarm message should be sent to their corresponding first change terminal. This is equivalent to executing S505.

[0133] S508, Alarm Second Change of Person Handling.

[0134] The second person to make the change is the latest person to make the change in the change record of the second subtask. The specific implementation of S508 can be found in the relevant description in S408, and will not be repeated here.

[0135] S509. Obtain cluster running status parameters.

[0136] The specific implementation of S509 can be referred to the relevant description of S410 in the aforementioned embodiments, and will not be repeated here.

[0137] S510. Determine if there are any abnormalities in the target components or changes in configuration parameters in the big data platform.

[0138] If there is an anomaly in the target component or a change in configuration parameters in the big data platform, then execute S511.

[0139] If there are no abnormalities in the target components or changes in configuration parameters in the big data platform, it is highly likely that there is an abnormality in the task configuration parameters of the currently executing first subtask. The person who made the latest change to the first subtask can be identified as the person in charge of the fault, and an alarm can be issued to them, i.e., S505 can be executed.

[0140] S511. If the target component is abnormal, an alarm will be issued to the person in charge of the component; if the configuration parameters are changed, an alarm will be issued to the person in charge of the platform.

[0141] The specific implementation of S511 can be referred to the relevant descriptions of S411 and S412 in the aforementioned embodiments, and will not be repeated here.

[0142] The technical effects of the aforementioned S501-S511 can be referred to the technical effects of the technical solutions disclosed in the foregoing embodiments, and will not be repeated here.

[0143] In this embodiment, to enable accurate fault classification by the electronic device, a target fault classification model needs to be trained in advance (at least before the electronic device inputs tool logs and task execution logs into the target fault classification model). Therefore, the fault handling method provided in this application should also include a method for training the target fault classification model, which can be implemented before S302 and S402 of the aforementioned embodiment. (Refer to...) Figure 6 As shown, the training methods for the target fault classification model can include S601 and S602:

[0144] S601. Obtain multiple sets of sample data and the corresponding sample categories for each set of sample data.

[0145] Each set of sample data consists of tool logs and task execution logs from when an ETL task fails in the big data processing system; the sample category corresponding to the sample data is the failure category of the sample data.

[0146] The sample data obtained in S601 can be acquired from the big data processing system before the current moment. Specifically, it could be obtained manually when a failure occurs in the big data processing system while processing an ETL task, by manually determining the fault category and acquiring the corresponding tool logs and task execution logs. In this acquired set of data, the tool logs and task execution logs constitute one sample data set, and the corresponding fault category is the sample category corresponding to that sample data set.

[0147] S602. Using sample data as training data and sample categories as supervision information, iteratively train the initial fault classification model to obtain the target fault classification model.

[0148] For example, S802 may specifically include: initializing a fault classification model; inputting sample data into the fault classification model to obtain a predicted category; determining a loss value based on the predicted category and the sample category; iteratively updating the fault classification model based on the loss value; and repeatedly executing the step of inputting sample data into the fault classification model to obtain a predicted category until a target fault classification model that meets preset conditions is obtained.

[0149] Specifically, initializing the fault classification model can refer to setting the corresponding hyperparameters and the weight parameters that will be optimized during the initialization training process, based on the selected model framework.

[0150] Specifically, the loss value can be calculated using any feasible loss function. For example, the mean squared error (MSE) is used to calculate the Euclidean distance between the predicted value (predictive control decision) and the true value (sample control decision). This application does not impose any specific restrictions on this.

[0151] Subsequently, based on this loss value, the parameters in the fault classification model can be adjusted using any feasible parameter tuning method. Examples include stochastic gradient descent (SGD). This application does not impose specific limitations in this regard.

[0152] Furthermore, a target fault classification model that meets the preset conditions may include: the number of iterations of the fault classification model meets the preset number, or the loss value is less than a predetermined threshold. That is, during the training process of the fault classification model, if the number of iterations of the fault classification model after a certain iteration is greater than or equal to the preset number, or the loss value corresponding to the fault classification model after a certain iteration is less than the predetermined threshold, then the fault classification model is the target fault classification model.

[0153] The preset number of attempts and the predetermined threshold can be determined based on requirements and model training experience. Specific values ​​can be set according to actual needs, and this application does not impose specific restrictions. In this way, the training of the target fault classification model can be terminated promptly once the training objective is achieved, based on user requirements.

[0154] Furthermore, the S602 can employ any feasible decision tree algorithm for training the target fault classification model. During training, all sample data and their corresponding categories can be divided into a training set and a validation set. After training the training set to obtain a potential fault classification model, the accuracy, recall, and precision of this model can be evaluated using the validation set. Then, based on the required accuracy, recall, and precision, the parameters of the potential fault classification model can be adjusted to obtain the target fault classification model.

[0155] It should be noted that the electronic device for training the target fault classification model can be a device implementing the corresponding technical solution in the foregoing embodiments, or it can be other devices capable of communicating with the aforementioned device. When the electronic device is another device, after training the target fault classification model, it can send the model to the device implementing the corresponding technical solution in the foregoing embodiments. This application does not impose specific limitations on how this is implemented.

[0156] Based on the technical solutions corresponding to S801 and S802 above, a target fault classification model can be trained using machine learning. This model has the ability to predict the fault category of an ETL task using tool logs and task execution logs. This allows the fault handling method provided in this application to conveniently and quickly determine the fault category using this model.

[0157] It is understood that, in order to achieve the aforementioned functions, the electronic device includes corresponding hardware structures and / or software modules for performing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0158] In the case of dividing each function into corresponding modules, embodiments of this application also provide a fault handling device. For example... Figure 7 The diagram shown is a structural schematic of a fault handling device provided in an embodiment of this application. The device may include an acquisition module 701 and a processing module 702.

[0159] The acquisition module 701 is used to acquire tool logs and task execution logs when an ETL task failure occurs. The tool logs are the work logs of the big data tools and devices in the big data processing system that execute the ETL task. The task execution logs are the work logs of the big data platform in the big data processing system that executes the ETL task. The processing module 702 is used to call the target fault classification model to process the tool logs and task execution logs acquired by the acquisition module 701 to obtain the fault category. The fault category is any one of the following: tool-side fault, data warehouse-side fault, or platform-side fault. The processing module 702 is also used to determine the person responsible for the fault based on the fault category and send alarm information to the fault-responsible terminal. The fault-responsible terminal is the fault-responsible terminal corresponding to the person responsible for the fault. The alarm information is used to instruct the person responsible for the fault to handle the fault that occurred in the ETL task.

[0160] In one possible implementation, the processing module 702 is specifically used to include: if the fault type is a tool-side fault, then determine the tool responsible person and the tool responsible person's communication information according to the responsibility table of the big data tool equipment; the responsibility table includes the communication information of multiple tool managers, and the correspondence between the tool managers and the time periods they are responsible for the big data tool equipment; determine the tool responsible person as the fault responsible person, and use the communication information of the tool responsible person to send alarm information to the tool responsible terminal.

[0161] In one possible implementation, the processing module 702 is specifically used to include: if the fault type is a data warehouse side fault, obtaining the first change record of the first subtask of the ETL task; the first change record is the first subtask version change record or the first subtask table change record; if it is determined from the first change record that the first subtask has a task version change or a task table change, then the latest first changer corresponding to the first subtask is identified as the fault responsible person and an alarm message is sent to the first change terminal corresponding to the first changer.

[0162] In one possible implementation, after obtaining the first change record of the first subtask of the ETL task, the processing module 702 is further configured to: if it is determined from the first change record that the first subtask does not have a task version change or a task table change, then obtain the second change record of the second subtask; the second subtask is the upstream task of the first subtask in the ETL task; the second change record is the version change record of the second subtask or the table change record of the second subtask; if it is determined from the second change record that the second subtask has a task version change or a task table change, then determine the second person responsible for the new version change of the second subtask as the person responsible for the fault and send an alarm message to the second terminal corresponding to the second person responsible for the fault; if it is determined from the second change record that the second subtask does not have a task version change or a task table change, then determine the latest first person responsible for the fault corresponding to the first subtask as the person responsible for the fault and send an alarm message to the first terminal corresponding to the first person responsible for the fault.

[0163] In one possible implementation, the processing module 702 is specifically used to: if the fault type is a platform fault, obtain the cluster operation status parameters of the big data platform; if the cluster operation status parameters indicate that the target component of the big data platform is abnormal, determine the component responsible person corresponding to the target component as the fault responsible person, and send alarm information to the component responsible terminal corresponding to the component responsible person; if the cluster operation status parameters indicate that the configuration parameters of the big data platform have changed, determine the platform responsible person corresponding to the big data platform as the fault responsible person, and send alarm information to the platform responsible terminal corresponding to the platform responsible person.

[0164] In one possible implementation, after obtaining the cluster operating status parameters of the big data platform, the processing module 702 is further configured to: if the cluster operating status parameters indicate that there are no abnormalities in all components of the big data platform and the configuration parameters of the big data platform have not been changed, then determine the latest first changer corresponding to the first subtask of the ETL task as the fault responsible person and send alarm information to the first changer corresponding to the first changer terminal.

[0165] In one possible implementation, the device further includes a training module 703. The training module 703 is specifically used to: acquire multiple sets of sample data and corresponding sample categories; the sample data consists of tool logs and task execution logs from when an ETL task failure occurs in the big data processing system; the sample categories corresponding to the sample data are the fault categories of the sample data; using the sample data as training data and the sample categories as supervisory information, iteratively train the initial fault classification model to obtain the target fault classification model.

[0166] Regarding the fault handling device in the above embodiments, the specific methods of operation of each module and the corresponding beneficial effects have been described in detail in the embodiments of the fault handling method described above, and will not be repeated here.

[0167] Figure 8 This is a schematic diagram illustrating a possible structure of an electronic device according to an exemplary embodiment. The electronic device may be the aforementioned fault handling device, or it may be a terminal or server including the fault handling device. Figure 8 As shown, the electronic device includes a processor 81 and a memory 82. The memory 82 stores instructions executable by the processor 81, which in turn implements the functions of the various modules in the fault handling device described in the above embodiments. The memory 82 stores at least one instruction, which is loaded and executed by the processor 81 to implement the methods provided in the above method embodiments.

[0168] In a specific implementation, as one example, processor 81 (81-1 and 81-2) may include one or more CPUs, for example... Figure 8 CPU0 and CPU1 are shown in the diagram. As one embodiment, the electronic device may include multiple processors 81, such as... Figure 8 The processors 81-1 and 81-2 are shown. Each CPU in these processors 81 can be a single-core processor or a multi-core processor. Here, processor 81 can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0169] The memory 82 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), disk computer storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 82 may exist independently and be connected to the processor 81 via a communication bus 83. The memory 82 may also be integrated with the processor 81.

[0170] The communication bus 83 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. This communication bus 83 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0171] In addition, to facilitate information interaction between the electronic device and other devices (e.g., information interaction between the electronic device as a terminal and a server, or information interaction between the electronic device as a server and a terminal), the electronic device includes a communication interface 84. The communication interface 84, using any transceiver-like device, is used to communicate with other devices or communication networks, such as control systems, radio access networks (RAN), wireless local area networks (WLAN), etc. The communication interface 84 may include a receiving unit to implement receiving functions and a transmitting unit to implement transmitting functions. The communication interface 84, processor 81, and memory 82 are connected via a communication bus 83 to complete mutual communication.

[0172] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the fault handling method described in the above method embodiments.

[0173] For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0174] This application also provides a computer program product containing computer instructions, which, when executed on an electronic device, cause the electronic device to perform the fault handling method described in the above method embodiments.

[0175] In this application, the electronic device, computer-readable storage medium, or computer program product provided in the embodiments are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0176] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, the division of the above functional modules is only used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device (e.g., electronic device) can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device (e.g., electronic device) and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0177] In the embodiments provided in this application, it should be understood that the disclosed systems, devices (e.g., electronic devices) and methods can be implemented in other ways. For example, the device (e.g., electronic device) embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, and may be electrical, mechanical, or other forms.

[0178] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0180] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0181] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A fault handling method, characterized in that, The method includes: When a data warehouse task failure occurs, the tool log and task execution log are obtained; the tool log is the working log of the big data tool device in the big data processing system that executes the data warehouse task; the task execution log is the working log of the big data platform in the big data processing system that executes the data warehouse task. The target fault classification model is invoked to process the tool logs and the task execution logs to obtain fault categories; the fault category is any one of the following: tool-side fault, data warehouse-side fault, or platform-side fault. Based on the fault category, the person responsible for the fault is determined and an alarm message is sent to the fault-responsible terminal; the fault-responsible terminal is the fault-responsible terminal corresponding to the person responsible for the fault; the alarm message is used to instruct the person responsible for the fault to handle the fault that occurred in the data warehouse task; The step of determining the person responsible for the fault and sending alarm information to the terminal responsible for the fault based on the fault category includes: If the fault category is a platform fault, then obtain the cluster operation status parameters of the big data platform; If the cluster operation status parameters indicate that the target component of the big data platform is abnormal, the component responsible person corresponding to the target component is identified as the fault responsible person, and the alarm information is sent to the component responsible terminal corresponding to the component responsible person. If the cluster running status parameters indicate that the configuration parameters of the big data platform have changed, then the platform person in charge of the big data platform will be identified as the person responsible for the fault, and the alarm information will be sent to the platform responsibility terminal corresponding to the platform person in charge. If the cluster running status parameters indicate that there are no abnormalities in all components of the big data platform and the configuration parameters of the big data platform have not been changed, then the latest first changer corresponding to the first subtask of the data warehouse task is identified as the fault responsible person and the alarm information is sent to the first change terminal corresponding to the first changer.

2. The method according to claim 1, characterized in that, The step of determining the person responsible for the fault based on the fault category and sending alarm information to the terminal responsible for the fault includes: If the fault category is a tool-side fault, then the person in charge of the big data tool and the communication information of the person in charge are determined according to the responsibility table of the big data tool and equipment; the responsibility table includes the communication information of multiple tool managers, and the correspondence between the tool managers and the time periods in charge of the big data tool and equipment. The person responsible for the tool is identified as the person responsible for the fault, and the alarm information is sent to the terminal responsible for the tool using the communication information of the person responsible for the tool.

3. The method according to claim 1, characterized in that, The step of determining the person responsible for the fault based on the fault category and sending alarm information to the terminal responsible for the fault includes: If the fault category is a data warehouse side fault, obtain the first change record of the first subtask of the data warehouse task; the first change record is the first subtask version change record or the first subtask table change record. If it is determined from the first change record that the first subtask has undergone a task version change or a task table change, then the latest first changer corresponding to the first subtask is identified as the person responsible for the fault, and the alarm information is sent to the first changer terminal corresponding to the first changer.

4. The method according to claim 3, characterized in that, After obtaining the first change record of the first subtask of the data warehouse task, the method further includes: If it is determined from the first change record that the first subtask does not have a task version change or a task table change, then the second change record of the second subtask is obtained; the second subtask is the upstream task of the first subtask in the data warehouse task; the second change record is the version change record of the second subtask or the table change record of the second subtask. If it is determined from the second change record that the second subtask has a task version change or a task table change, then the second person who made the new version change of the second subtask is identified as the person responsible for the fault and the alarm information is sent to the second change terminal corresponding to the second person who made the change. If it is determined from the second change record that there is no change in the task version or task table for the second subtask, then the latest first changer corresponding to the first subtask is identified as the person responsible for the fault, and the alarm information is sent to the first change terminal corresponding to the first changer.

5. The method according to claim 1, characterized in that, Before the method calls the target fault classification model to process the tool logs and the task execution logs, it further includes: Obtain multiple sets of sample data and their corresponding sample categories; the sample data consists of tool logs and task execution logs from when the big data processing system experiences a data warehouse task failure; the sample category corresponding to the sample data is the failure category of the sample data. Using the sample data as training data and the sample categories as supervision information, the initial fault classification model is iteratively trained to obtain the target fault classification model.

6. A fault handling device, characterized in that, The device includes: The acquisition module is used to acquire tool logs and task execution logs when a data warehouse task failure occurs; the tool logs are the work logs of the big data tools and devices in the big data processing system that execute the data warehouse task; the task execution logs are the work logs of the big data platform in the big data processing system that executes the data warehouse task. The processing module is used to input the tool logs and task execution logs obtained by the acquisition module into the target fault classification model to obtain the fault category; the fault category is any one of the following: tool-side fault, data warehouse-side fault, platform-side fault; The processing module is further configured to determine the person responsible for the fault based on the fault category and send alarm information to the fault-responsible terminal; the fault-responsible terminal is the fault-responsible terminal corresponding to the person responsible for the fault; the alarm information is used to instruct the person responsible for the fault to handle the fault that occurred in the data warehouse task; The processing module is also used to obtain the cluster operation status parameters of the big data platform if the fault category is a platform fault. If the cluster operation status parameters indicate that the target component of the big data platform is abnormal, the component responsible person corresponding to the target component is identified as the fault responsible person, and the alarm information is sent to the component responsible terminal corresponding to the component responsible person. If the cluster running status parameters indicate that the configuration parameters of the big data platform have changed, then the platform person in charge of the big data platform will be identified as the person responsible for the fault, and the alarm information will be sent to the platform responsibility terminal corresponding to the platform person in charge. If the cluster running status parameters indicate that there are no abnormalities in all components of the big data platform and the configuration parameters of the big data platform have not been changed, then the latest first changer corresponding to the first subtask of the data warehouse task is identified as the fault responsible person and the alarm information is sent to the first change terminal corresponding to the first changer.

7. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the fault handling method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on the electronic device, causes the electronic device to perform the operation of the fault handling method as described in any one of claims 1-5.

Images