ETL system maintenance method and device based on neural network model

Abstract

The invention provides an ETL system maintenance method and device based on a neural network model, and relates to the technical field of system maintaining.The method comprises the steps that log data of ETL systems from multiple different sources are obtained, and the log data comprise multiple pieces of log information; according to the log data, feature information corresponding to each piece of log information is determined, and the feature information comprises time, task identification and fault categories; in response to a fault reason analysis instruction which is input by a user and is used for indicating to determine the ETL system in the target time period, determining the corresponding quantity of the log information of each fault category in the target time period; and inputting the number corresponding to the log information of each fault category in the target time period into the trained first neural network model to obtain the fault reason of the ETL system in the target time period. According to the method provided by the invention, the log analysis time can be shortened, and the fault reason of the ETL system can be quickly determined, so that the system maintenance efficiency is effectively improved, and the maintenance cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of system maintenance technology, and in particular to an ETL system maintenance method and apparatus based on a neural network model. Background Art

[0002] As enterprises become increasingly digital, ETL (Extract, Transform, Load) systems have become core components of data warehouses (EDW) and business intelligence (BI), used to extract, transform, and load data from heterogeneous sources into target databases to support subsequent data analysis and decision-making. However, with the surge in data volume and increasing business complexity, maintaining current ETL systems faces numerous challenges:

[0003] ETL systems generate massive amounts of log data during operation, including execution status, error messages, and resource usage. These logs are typically stored in unstructured or semi-structured formats. When an ETL system malfunctions, manually troubleshooting is not only time-consuming but also prone to missing critical information. This is especially true in distributed environments, where cross-service log correlation analysis is difficult, leading to high maintenance costs.

[0004] Therefore, there is an urgent need for an ETL system maintenance method and device based on a neural network model, which can quickly and accurately determine the cause of the fault based on the log information of the ETL system, reduce processing time, and lower maintenance costs. Summary of the Invention

[0005] This invention provides an ETL system maintenance method and apparatus based on a neural network model, which can quickly and accurately determine the cause of the fault based on the log information of the ETL system, reduce processing time, and lower maintenance costs.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0007] Firstly, a method for maintaining an ETL system based on a neural network model is provided. This method includes: acquiring log data from multiple different sources of ETL systems, the log data including multiple log entries; determining feature information corresponding to each log entry based on the log data, the feature information including time, task identifier, and fault category; responding to a user-inputted instruction to determine the cause of faults in the ETL system within a target time period, determining the quantity of log entries corresponding to each fault category within the target time period; and inputting the quantity of log entries corresponding to each fault category within the target time period into a trained first neural network model to obtain the cause of faults in the ETL system within the target time period.

[0008] In one possible implementation of the first aspect, after determining the number of log messages corresponding to each fault category within the target time period in response to a user-input instruction for indicating fault cause analysis of the ETL system within a target time period, the method further includes: generating a first alarm message when the number of log messages corresponding to any fault category is greater than a first threshold, the first alarm message being used to indicate an ETL system malfunction.

[0009] In one possible implementation of the first aspect, the method further includes: determining repair suggestions for the ETL system from a database based on the causes of failures in the ETL system within a target time period, wherein the database stores repair suggestions corresponding to each of multiple causes of failure; and displaying the causes of failures and repair suggestions for the ETL system within the target time period on a preset interface.

[0010] In one possible implementation of the first aspect, before inputting the number of log information corresponding to each fault category within the target time period into the trained first neural network model to obtain the fault cause of the ETL system within the target time period, the method further includes: obtaining a first training dataset, the first training dataset including multiple first training samples, each first training sample including the number of log information corresponding to each fault category in multiple fault categories, and the fault cause; iteratively training the first neural network model according to the first training dataset to obtain the trained first neural network model.

[0011] In one possible implementation of the first aspect, the method further includes: acquiring time-series data of the ETL system, the time-series data including the resource utilization rate corresponding to each time step from the i-th time step to every n-th time step, where i and n are positive integers, and n is greater than i; inputting the time-series data of the ETL system into a trained second neural network model, and outputting the resource utilization rate of the ETL system at the (n+1)-th time step; when the resource utilization rate of the ETL system at the (n+1)-th time step is greater than a second threshold, generating a second alarm message, the second alarm message being used to indicate that the resource utilization rate of the ETL system is abnormal.

[0012] In one possible implementation of the first aspect, the ETL system is configured with multiple interfaces, and the method further includes: obtaining the response time and service type of each interface; performing a weighted summation of the response time and service type of each interface to obtain the importance score corresponding to each interface, wherein different response times correspond to different weight coefficients, and different service types correspond to different weight coefficients; sending a service processing instruction to the ETL system, the service processing instruction carrying the importance score corresponding to each interface, the service processing instruction being used to instruct the ETL system to process the pending services corresponding to each interface in descending order of importance score.

[0013] The beneficial effects of this invention are as follows: The method provided by this invention acquires log data from multiple ETL systems from different sources, including multiple log messages; based on the log data, it determines the feature information corresponding to each log message, including time, task identifier, and fault category; responding to a user-inputted instruction to determine the fault cause analysis of the ETL system within a target time period, it determines the quantity of log messages corresponding to each fault category within the target time period; and inputs the quantity of log messages corresponding to each fault category within the target time period into a trained first neural network model to obtain the fault cause of the ETL system within the target time period. The method provided by this invention can reduce log analysis time, quickly determine the fault cause of the ETL system, thereby effectively improving system maintenance efficiency and reducing maintenance costs. On the other hand, the method provided by this invention can accurately predict resource utilization at future time nodes, facilitating timely adjustments by technicians to the ETL system and maintaining its stable operation. Finally, the method provided by this invention also determines the importance score of each interface based on its response time and business type, thereby instructing the TEL system to process the pending business of each interface sequentially according to its importance, effectively improving system processing efficiency and meeting the needs of different usage scenarios.

[0014] Secondly, the present invention provides an ETL system maintenance device based on a neural network model. The device includes: a data acquisition unit for acquiring log data from multiple different sources of ETL systems, the log data including multiple log information; a feature determination unit for determining feature information corresponding to each log information based on the log data, the feature information including time, task identifier, and fault category; a quantity determination unit for determining the quantity of log information corresponding to each fault category within the target time period in response to a user-input instruction for determining the fault cause analysis of the ETL system within a target time period; and a cause determination unit for inputting the quantity of log information corresponding to each fault category within the target time period into a trained first neural network model to obtain the fault cause of the ETL system within the target time period.

[0015] In one possible implementation of the second aspect, after the quantity determination unit determines the quantity of log information corresponding to each fault category within the target time period in response to a user-input instruction for analyzing the cause of failure in the ETL system within the target time period, the device further includes an alarm unit for generating a first alarm message when the quantity of log information corresponding to any fault category is greater than a first threshold. The first alarm message is used to indicate an ETL system malfunction.

[0016] In one possible implementation of the second aspect, the apparatus further includes: a suggestion determination unit, configured to determine repair suggestions for the ETL system from a database based on the causes of failures in the ETL system within a target time period, wherein the database stores repair suggestions corresponding to each of a plurality of causes of failure; and a unit configured to display the causes of failures and repair suggestions for the ETL system within a preset interface within the target time period.

[0017] Thirdly, an electronic device is provided, the electronic device including a memory and one or more processors; the memory is coupled to the processors; wherein the memory stores computer program code, the computer program code including computer instructions, which, when executed by the processor, cause the electronic device to perform the method as described in any implementation of the first aspect.

[0018] Fourthly, a computer-readable storage medium is provided, including computer instructions that, when executed on an electronic device, cause the electronic device to perform a method as described in any implementation of the first aspect.

[0019] Fifthly, a computer program product is provided that, when run on a computer, causes the computer to perform the method in any implementation of the first aspect.

[0020] Understandably, the beneficial effects achieved by the system of the second aspect, the electronic device of the third aspect, the computer-readable storage medium of the fourth aspect, and the computer program product of the fifth aspect provided above can be referred to with reference to the beneficial effects of the first aspect and any of its possible design embodiments, which will not be repeated here. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention;

[0022] Figure 2 A flowchart illustrating an ETL system maintenance method based on a neural network model, provided as an embodiment of the present invention;

[0023] Figure 3 A flowchart of another ETL system maintenance method based on a neural network model provided in an embodiment of the present invention;

[0024] Figure 4 A flowchart illustrating another ETL system maintenance method based on a neural network model provided in this embodiment of the invention;

[0025] Figure 5 A flowchart illustrating another ETL system maintenance method based on a neural network model provided in this embodiment of the invention;

[0026] Figure 6This is a schematic diagram of a maintenance device provided in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings. In the description of the present invention, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. The "or" in the present invention is merely a description of the relationship between the related objects, indicating that three relationships can exist. For example, A or B can represent: A alone, A and B simultaneously, and B alone. A and B can be singular or plural. Furthermore, in the description of the present invention, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items.

[0028] Furthermore, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0029] In this embodiment of the invention, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this embodiment of the invention should not be construed as superior or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0030] As enterprises become increasingly digital, ETL (Extract, Transform, Load) systems have become core components of data warehouses (EDW) and business intelligence (BI), used to extract, transform, and load data from heterogeneous sources into target databases to support subsequent data analysis and decision-making. However, with the surge in data volume and increasing business complexity, maintaining current ETL systems faces numerous challenges:

[0031] ETL systems generate massive amounts of log data during operation, including execution status, error messages, and resource usage. These logs are typically stored in unstructured or semi-structured formats. When an ETL system malfunctions, manually troubleshooting is not only time-consuming but also prone to missing critical information. This is especially true in distributed environments, where cross-service log correlation analysis is difficult, leading to high maintenance costs.

[0032] Therefore, there is an urgent need for an ETL system maintenance method and device based on a neural network model, which can quickly and accurately determine the cause of the fault based on the log information of the ETL system, reduce processing time, and lower maintenance costs.

[0033] In view of this, embodiments of the present invention provide an ETL system maintenance method based on a neural network model. The method includes: acquiring log data from multiple ETL systems from different sources, the log data including multiple log information; determining feature information corresponding to each log information based on the log data, the feature information including time, task identifier, and fault category; responding to a user input instruction for determining the fault cause analysis of the ETL system within a target time period, determining the quantity of log information corresponding to each fault category within the target time period; inputting the quantity of log information corresponding to each fault category within the target time period into a trained first neural network model to obtain the fault cause of the ETL system within the target time period.

[0034] The method provided by this invention acquires log data from multiple ETL systems from different sources. The log data includes multiple log entries. Based on the log data, it determines the characteristic information corresponding to each log entry, including time, task identifier, and fault category. Responding to a user-inputted instruction to determine the cause of ETL system faults within a target time period, it determines the quantity of log entries for each fault category within the target time period. The quantity of log entries for each fault category within the target time period is input into a trained first neural network model to obtain the cause of ETL system faults within the target time period. This method reduces log analysis time, quickly identifies the cause of ETL system faults, and thus effectively improves system maintenance efficiency and reduces maintenance costs. Furthermore, this method can accurately predict resource utilization at future time points, facilitating timely adjustments by technicians to the ETL system and maintain its stable operation. Finally, this method also determines the importance score of each interface based on its response time and service type, instructing the TEL system to process the pending services of each interface sequentially according to its importance, effectively improving system processing efficiency and meeting the needs of different usage scenarios.

[0035] In some embodiments, the ETL system maintenance method based on a neural network model provided in this invention can be executed by an ETL system maintenance device 100 based on a neural network model (hereinafter referred to as maintenance device 100).

[0036] As an example, the maintenance device 100 can be any electronic device 200 with data processing capabilities, such as a general-purpose computer, personal computer, laptop computer, switch, or tablet computer. The specific implementation of the maintenance device 100 is not limited here.

[0037] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention is shown. The electronic device 200 includes a processor 210, a memory 220, and a communication interface 230.

[0038] Processor 210 may include one or more processing cores. Processor 210 connects to various parts within electronic device 200 using various interfaces and lines, and performs various functions and processes data of electronic device 200 by running or executing instructions, programs, code sets, or instruction sets stored in memory 220, and by calling data stored in memory 220. Optionally, processor 210 may be implemented using at least one of the following hardware forms: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA).

[0039] The memory 220 may include random access memory (RAI) or read-only memory (ROI). Optionally, the memory 220 may include non-transitory computer-readable storage ledger. The memory 220 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 220 may include a program storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as video acquisition, feature extraction, and process detection), and instructions for implementing the various method embodiments described above.

[0040] The communication interface 230 is used to communicate with other devices, equipment, or communication networks, such as data storage devices, image processing devices, or Ethernet, wireless access networks (RAN), wireless local area networks (WLAN), etc.

[0041] In terms of physical implementation, the aforementioned devices (such as processor 210, memory 220, and communication interface 230) can each be devices within the same device (such as a laptop computer). Alternatively, at least two of these devices can be located within the same device, i.e., as different devices within the same device, similar to the deployment of devices or components in a distributed system.

[0042] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 200. In other embodiments of the present invention, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0043] The following description, in conjunction with the accompanying drawings, illustrates an ETL system maintenance method based on a neural network model provided by an embodiment of the present invention.

[0044] Figure 2 This is a flowchart illustrating an ETL system maintenance method based on a neural network model, provided as an embodiment of the present invention. Optionally, this method can be... Figure 1 The illustrated electronic device 200 performs this operation. The method may include the following steps:

[0045] S1. Obtain log data from multiple different ETL systems. The log data includes multiple log messages.

[0046] Specifically, the log data sources include the ETL system itself, the database communicating with the ETL system, the target system, the infrastructure system, and custom monitoring scripts. Database logs include database logs, file system logs, and API call logs. Database logs record data changes, file system logs record file creation and deletion, and API call logs record requests or responses during data extraction. ETL system logs include job scheduling logs, data processing logs, and connector logs. Target system logs include load logs and target database audit logs. Infrastructure system logs include server logs, container logs, and network logs. Custom monitoring script logs include verification script logs.

[0047] Furthermore, log information includes different types, such as runtime status logs, error logs, performance logs, data quality logs, operation audit logs, and security logs.

[0048] It should be understood that the sources of the log data and the types of log information mentioned above are merely illustrative examples, and the embodiments of the present invention do not impose any special restrictions on the sources of log data and the types of log information.

[0049] S2. Based on the log data, determine the characteristic information corresponding to each log message. The characteristic information includes time, task identifier, and fault category.

[0050] Specifically, the time refers to the time the log information was generated, the task identifier is the task (business) corresponding to the log information, and the fault category is the fault category corresponding to the log information. Fault categories include connection failure, data extraction error, data transformation error, data loading error, insufficient resources, missing data, data inconsistency, data quality violation, data timeliness failure, business logic error, scheduling system failure, dependent service failure, security permission issues, and version compatibility issues.

[0051] Specifically, the log information includes the following fields: Connection failure, Timeout, Authentication failed; Extract error, Table not found, Invalid SQL query, Permission denied; Transform error, Type mismatch, Null value violation, Aggregation overflow; Load error, Connection refused, Timeout, Authentication failed; Primary key conflict, Constraint violation, Disk full; Insufficient resources, Out of memory, Thread pool exhausted, CPU throttling; Missing data, Source file not found, Zero records extracted; Inconsistent data, Row count mismatch, Checksum validation failed; and Data quality violation, Null value rate > 5%, Duplicate records. The log information for fault categories of data timeliness failure carries fields such as SLQ missed and Job delayed by 2 hours; the log information for fault categories of business logic errors carries fields such as Incorrect aggregation result and Unexpected data truncation; the log information for fault categories of scheduling system failure carries fields such as DAG failed to trigger and Schedulemissed; the log information for fault categories of dependent service failure carries fields such as Kafka broker unavailable and HDFS NameNode down; the log information for fault categories of security permission issues carries fields such as Access denied and SSL handshake failed.Log messages indicating connection failures carry the fields "Unsupported JDBCdriver" and "Schema migration failed".

[0052] In one possible implementation, the maintenance device 100 processes each log message based on a natural language processing algorithm, identifies the semantic information carried by the fields in each log message, and then determines the fault type corresponding to each log message.

[0053] It should be noted that there are fault-free log messages in the ETL system, such as log messages recording data changes. The maintenance device 100 determines the fault category of such fault-free log messages as fault-free.

[0054] S3. In response to the user input command for determining the cause of failure in the ETL system within a target time period, determine the number of log messages corresponding to each failure category within the target time period.

[0055] In some embodiments, after S3 described above, the method provided by the present invention further includes:

[0056] If the number of log messages for any fault category exceeds the first threshold, a first alarm message is generated. The first alarm message is used to indicate that the ETL system is malfunctioning.

[0057] S4. Input the number of log information corresponding to each fault category within the target time period into the trained first neural network model to obtain the fault causes of the ETL system within the target time period.

[0058] In one example, if the number of connection failures is 200 and the number of data transformation errors is 100 within the target time period (with other failure categories all less than 20), the first neural network model outputs that the cause of the ETL system failure within the target time period is database service instability and changes in source system field types. In another example, if the number of scheduling system failures is 150 and the number of resource shortages is 100 within the target time period (with other failure categories all less than 30), the first neural network model outputs that the cause of the ETL system failure within the target time period is scheduling server resource exhaustion.

[0059] In some embodiments, see Figure 3 Prior to S4 above, the method provided in this embodiment of the invention further includes:

[0060] S31. Obtain the first training dataset. The first training dataset includes multiple first training samples. Each first training sample includes the number of log information corresponding to each fault category in multiple fault categories, as well as the cause of the fault.

[0061] S32. Iteratively train the first neural network model based on the first training dataset to obtain the trained first neural network model.

[0062] As described in S1-S4 above, the method provided by this invention acquires log data from multiple ETL systems from different sources. The log data includes multiple log entries. Based on the log data, it determines the feature information corresponding to each log entry, including time, task identifier, and fault category. Responding to a user-inputted instruction to determine the cause of faults in the ETL system within a target time period, it determines the quantity of log entries corresponding to each fault category within the target time period. The quantity of log entries corresponding to each fault category within the target time period is input into a trained first neural network model to obtain the cause of faults in the ETL system within the target time period. The method provided by this invention can reduce log analysis time, quickly determine the cause of faults in the ETL system, thereby effectively improving system maintenance efficiency and reducing maintenance costs.

[0063] In one possible implementation, the method provided by the embodiments of the present invention further includes: determining repair suggestions for the ETL system from a database based on the causes of failure of the ETL system within a target time period, wherein the database stores repair suggestions corresponding to each of a plurality of causes of failure; and displaying the causes of failure and repair suggestions for the ETL system within the target time period on a preset interface.

[0064] Specifically, the database pre-stores remedial suggestions for each of the multiple fault causes. For example, if the fault is due to database service instability and changes in the field types of the source system, the remedial suggestion is to check the database cluster monitoring and whether the field definitions of the source tables and ETL processes are updated synchronously. If the fault is due to the exhaustion of scheduling server resources, the remedial suggestion is to scale up or optimize the task scheduling strategy.

[0065] The method provided in this invention stores repair suggestions corresponding to different fault causes in a database in advance, enabling technicians to quickly obtain repair suggestions for the ETL system without relying on their own professional experience, thereby achieving rapid repair of the ETL system, improving the maintenance efficiency of the ETL system, and reducing maintenance costs.

[0066] In some embodiments, see Figure 4 The method provided in this embodiment of the invention further includes:

[0067] S41. Obtain the time series data of the ETL system. The time series data includes the resource utilization rate corresponding to each time from the i-th time to every n-th time, where i and n are positive integers, and n is greater than i.

[0068] S42. Input the time series data of the ETL system into the trained second neural network model, and output the resource utilization rate of the ETL system at time n+1.

[0069] S43. When the resource utilization rate of the ETL system at time n+1 is greater than the second threshold, a second alarm message is generated. The second alarm message is used to indicate that the resource utilization rate of the ETL system is abnormal.

[0070] The method provided in this invention uses a second neural network model to accurately predict the resource utilization rate at future time nodes (the resource utilization rate at time n+1) based on historical resource utilization rates (the resource utilization rate corresponding to each time from time i to every n time). This can effectively achieve early warning of resource utilization, enabling technicians to quickly adjust resource utilization in a timely manner, avoiding impact on the normal operation of the ETL system, improving operation and maintenance efficiency, and meeting the user's needs in different scenarios.

[0071] In one possible implementation, see Figure 5 The ETL system is configured with multiple interfaces, and the method provided in this embodiment of the invention further includes:

[0072] S51. Obtain the response time and business type of each interface.

[0073] S52. Calculate the weighted sum of the response time and business type for each interface to obtain the importance score for each interface.

[0074] Different response times correspond to different weighting coefficients, and different business types correspond to different weighting coefficients.

[0075] Specifically, the maintenance device 100 pre-stores multiple response time intervals and the weight coefficient corresponding to each response time interval. The maintenance device 100 also pre-stores the weight coefficients corresponding to different business types. Then, the maintenance device determines the importance score of each interface based on the response time and business type of each interface.

[0076] In one example, the importance score L is calculated using the following formula:

[0077] L = A1 + A2;

[0078] Where A1 is the weighting coefficient corresponding to response time, and A2 is the weighting coefficient corresponding to business type.

[0079] S53. Send a service processing instruction to the ETL system. The service processing instruction carries the importance score corresponding to each interface. The service processing instruction is used to instruct the ETL system to process the pending services corresponding to each interface in descending order of importance score.

[0080] As can be seen from the above, the method provided by the embodiments of the present invention also determines the importance score of each interface based on the response time and service type of each interface, and then instructs the TEL system to process the pending services of each interface in sequence according to the importance of each interface, which can effectively improve the processing efficiency of the system and meet the usage needs in different usage scenarios.

[0081] The foregoing mainly describes the solutions of the embodiments of the present invention from a methodological perspective. It is understood that, in order to achieve the above functions, the maintenance device 100 includes at least one of the hardware structures and software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present invention.

[0082] In this embodiment of the invention, the maintenance device 100 can be divided into functional units according to the above method example. For example, the maintenance device 100 can be divided into functional units corresponding to various functions, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this embodiment of the invention is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0083] For example, Figure 6A schematic diagram of the hardware structure of a maintenance device provided in an embodiment of the present invention is shown. The maintenance device 100 includes: a data acquisition unit 110, used to acquire log data from multiple ETL systems from different sources, the log data including multiple log information; a feature determination unit 120, used to determine feature information corresponding to each log information based on the log data, the feature information including time, task identifier, and fault category; a quantity determination unit 130, used to determine the quantity of log information corresponding to each fault category within the target time period in response to a user input instruction for determining the fault cause analysis of the ETL system within a target time period; and a cause determination unit 140, used to input the quantity of log information corresponding to each fault category within the target time period into a trained first neural network model to obtain the fault cause of the ETL system within the target time period.

[0084] Optionally, after the quantity determination unit 130 determines the quantity of log information corresponding to each fault category within the target time period in response to a user-input instruction for analyzing the cause of failure in the ETL system within the target time period, the device further includes an alarm unit 140, which generates a first alarm message when the quantity of log information corresponding to any fault category is greater than a first threshold. The first alarm message is used to indicate an ETL system malfunction.

[0085] Optionally, the above-mentioned device further includes: a suggestion determination unit 150, used to determine the repair suggestions of the ETL system from the database based on the fault causes of the ETL system within the target time period, wherein the database stores the repair suggestions corresponding to each of the multiple fault causes; and used to display the fault causes and repair suggestions of the ETL system within the target time period on a preset interface.

[0086] It should be understood that specific descriptions of the above-mentioned optional methods can be found in the foregoing method embodiments, and will not be repeated here. Furthermore, explanations of any of the maintenance devices 100 provided above, as well as descriptions of their beneficial effects, can be found in the corresponding method embodiments described above, and will not be repeated here.

[0087] This invention also provides a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to implement the methods of the various embodiments described above. Explanations of the relevant content and descriptions of the beneficial effects of any of the computer-readable storage media provided above can be found in the corresponding embodiments described above, and will not be repeated here.

[0088] This invention also provides a chip. This chip integrates a control circuit for implementing the functions of the maintenance device 100 described above, and one or more ports. Optionally, the functions supported by this chip are as described above, and will not be repeated here.

[0089] Those skilled in the art will understand that the program for implementing all or part of the steps of the above embodiments, which can be executed by a program instructing related hardware, can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a random access memory, etc. The processing unit or processor mentioned above can be a central processing unit, a general-purpose processor, an application-specific integrated circuit (ASIC), a microprocessor (DSP), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0090] This invention also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform any of the methods described in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this invention is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), etc.

[0091] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of the present invention, such as, but not limited to, the aforementioned memory, computer-readable storage medium, and communication chip, are all non-transitory. Those skilled in the art should recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0092] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for maintaining an ETL system based on a neural network model, characterized in that, The method includes: Obtain log data from multiple different ETL systems, the log data including multiple log information; Based on the log data, determine the characteristic information corresponding to each log message, including time, task identifier, and fault category; In response to a user-input command to determine the cause of failure in the ETL system within a target time period, the number of log messages corresponding to each failure category within the target time period is determined. Input the number of log information corresponding to each fault category within the target time period into the trained first neural network model to obtain the fault causes of the ETL system within the target time period.

2. The method according to claim 1, characterized in that, After determining the number of log entries for each fault category within the target time period in response to a user-input command instructing the analysis of fault causes in the ETL system within that time period, the method further includes: If the number of log messages corresponding to any fault category exceeds a first threshold, a first alarm message is generated. The first alarm message is used to indicate that the ETL system is malfunctioning.

3. The method according to claim 2, characterized in that, The method further includes: Based on the causes of failures in the ETL system within the target time period, repair suggestions for the ETL system are determined from the database, where the database stores repair suggestions corresponding to each of multiple causes of failure. The default interface displays the causes of ETL system failures and repair suggestions for the target time period.

4. The method according to claim 3, characterized in that, Before inputting the number of log information corresponding to each fault category within the target time period into the trained first neural network model to obtain the fault causes of the ETL system within the target time period, the method further includes: Obtain a first training dataset, which includes multiple first training samples. Each first training sample includes the number of log information corresponding to each fault category in multiple fault categories, as well as the cause of the fault. The first neural network model is iteratively trained based on the first training dataset to obtain the trained first neural network model.

5. The method according to claim 4, characterized in that, The method further includes: Obtain time series data of the ETL system, the time series data including the resource utilization rate corresponding to each time from the i-th time to every n-th time, where i and n are positive integers, and n is greater than i; The time series data of the ETL system is input into the trained second neural network model, and the resource utilization rate of the ETL system at time n+1 is output. When the resource utilization rate of the ETL system at time n+1 is greater than the second threshold, a second alarm message is generated. The second alarm message is used to indicate that the resource utilization rate of the ETL system is abnormal.

6. The method according to claim 5, characterized in that, The ETL system is configured with multiple interfaces, and the method further includes: Get the response time and business type of each interface; The response time and business type of each interface are weighted and summed to obtain the importance score for each interface. Different response times and different business types correspond to different weight coefficients. A service processing instruction is sent to the ETL system. The service processing instruction carries the importance score corresponding to each interface. The service processing instruction is used to instruct the ETL system to process the pending services corresponding to each interface in descending order of importance score.

7. An ETL system maintenance device based on a neural network model, characterized in that, The device includes: The data acquisition unit is used to acquire log data from multiple different ETL systems, the log data including multiple log information; The feature determination unit is used to determine the feature information corresponding to each log information based on the log data. The feature information includes time, task identifier, and fault category. The quantity determination unit is used to determine the quantity of log information corresponding to each fault category within the target time period in response to a user input instruction for indicating the analysis of fault causes in the ETL system within a target time period. The cause determination unit is used to input the number of log information corresponding to each fault category within the target time period into the trained first neural network model to obtain the fault cause of the ETL system within the target time period.

8. The apparatus according to claim 7, characterized in that, After the quantity determination unit determines the quantity of log information corresponding to each fault category within the target time period in response to a user-input instruction for analyzing the cause of failure in the ETL system within the target time period, the device further includes an alarm unit, which generates a first alarm message when the quantity of log information corresponding to any fault category is greater than a first threshold. The first alarm message is used to indicate that the ETL system is experiencing an operational failure.

9. The apparatus according to claim 8, characterized in that, The device further includes: The suggestion determination unit is used to determine the repair suggestions for the ETL system from the database based on the causes of failure of the ETL system within the target time period. The database stores the repair suggestions corresponding to each of the multiple causes of failure. The unit is also used to display the causes of failure and repair suggestions of the ETL system within the target time period on a preset interface.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the performance parameter determination method for engine regenerative cooling simulation as described in any one of claims 1-6.

Images