Method, device, equipment, medium and product for network training and determining fault

Abstract

This embodiment discloses a method, apparatus, device, medium, and product for network training and fault determination. The network training method includes: acquiring system logs and container metrics corresponding to cloud-native applications, wherein the system logs are at least a portion of the logs generated by each microservice in the cloud-native application; constructing a log feature set based on the system logs and the container metrics, wherein the log feature set is training data for a neural network, and the neural network is used to determine fault information of the cloud-native application; and training the neural network based on the log feature set.

Description

Technical Field

[0001] This application belongs to the field of cloud-native technology, and in particular relates to methods, apparatus, equipment, media and products for network training and fault diagnosis. Background Technology

[0002] In related technologies, cloud-native operation and maintenance management systems mostly rely on alarm policies and monitoring scripts. To determine alarm policies and monitoring scripts, operation and maintenance personnel need to summarize based on product architecture, performance indicators, and past failures. This may result in inaccurate alarm policies and monitoring scripts, which reduces the fault identification accuracy of the operation and maintenance management system. Summary of the Invention

[0003] To address the issue of low fault identification accuracy in cloud-native operation and maintenance management systems, embodiments of this application propose a method, apparatus, device, medium, and product for network training and fault determination.

[0004] This application provides a method for training a neural network, the method comprising:

[0005] Obtain system logs and container metrics corresponding to cloud-native applications, wherein the system logs are at least a portion of the logs generated by each microservice in the cloud-native application;

[0006] Based on the system logs and container metrics, a log feature set is constructed. The log feature set serves as training data for a neural network, which is used to determine fault information of the cloud-native application.

[0007] The neural network is trained based on the log feature set.

[0008] In some embodiments, constructing a log feature set based on the system logs and the container metrics includes: searching for a first interval with an accuracy metric higher than a preset accuracy metric within the log chain length interval of the system logs using a binary search method; the log chain length interval represents the interval from the minimum log chain length to the maximum log chain length; the log chain length represents the number of logs generated by a single request in the system logs; and the accuracy metric represents the accuracy of the logs corresponding to the interval; if the first interval is successfully found, constructing a first sliding window based on the first interval, and selecting logs through the first sliding window; and constructing a log feature set based on the logs selected in the first sliding window and the container metrics.

[0009] In some embodiments, the step of finding the first interval with an accuracy index higher than a preset accuracy index in the log chain length interval of the system log by means of binary search includes: dividing the log chain length interval of the system log into N intervals by means of binary search, and for each interval obtained by each interval division, finding the first interval with an accuracy index higher than the preset accuracy index, where N is an integer greater than 1.

[0010] In some embodiments, the preset accuracy rate represents the accuracy rate corresponding to logs with a preset log chain length; the method further includes: after performing the N interval divisions, if a first interval with an accuracy rate higher than the preset accuracy rate is not found, constructing a second sliding window based on the preset log chain length, and selecting logs through the second sliding window; constructing a log feature set based on the logs selected in the second sliding window and the container index.

[0011] In some embodiments, the fault information of the cloud-native application includes at least one of the following: existing fault information and fault warning information.

[0012] This application embodiment also provides a method for determining a fault, the method comprising:

[0013] Obtain log features to be processed; the log features to be processed are constructed based on real-time collected system logs and real-time collected container metrics of cloud-native applications;

[0014] The log features to be processed are processed using a pre-trained neural network to obtain the fault information of the cloud-native application. The neural network is trained based on any of the above training methods.

[0015] In some embodiments, after obtaining the fault information of the cloud-native application, the method further includes: determining the contribution of each log feature among the log features to be processed for the fault information; and determining the root cause of the fault information by fault tracing for the K log features with the highest contribution, wherein the root cause includes at least one of log and container metrics, and K is an integer greater than or equal to 1.

[0016] In some embodiments, the log feature set used to train the neural network includes system logs for a specific time period, wherein the specific time period represents the time period during which a first type of alarm log is generated, and the fault represented by the first type of alarm log has a risk level less than a preset threshold; after obtaining the fault information of the cloud-native application, the method further includes: filtering faults with a risk level less than the preset threshold from the fault information of the cloud-native application.

[0017] This application also provides a neural network training device, the device comprising:

[0018] The first acquisition module is used to acquire system logs and container metrics corresponding to cloud-native applications. The system logs are at least a portion of the logs generated by each microservice in the cloud-native application.

[0019] The first processing module is used to construct a log feature set based on the system logs and the container metrics. The log feature set is the training data of a neural network, and the neural network is used to determine the fault information of the cloud-native application.

[0020] The training module is used to train the neural network based on the log feature set.

[0021] This application also provides an apparatus for determining a fault, the apparatus comprising:

[0022] The second acquisition module is used to acquire log features to be processed; the log features to be processed are constructed based on real-time collected system logs and real-time collected container metrics of cloud-native applications.

[0023] The second processing module is used to process the log features to be processed using a pre-trained neural network to obtain the fault information of the cloud-native application. The neural network is trained based on any of the above-mentioned neural network training methods.

[0024] This application also provides an electronic device, which includes a processor and a memory for storing a computer program that can run on the processor; wherein the processor is used to run the computer program to execute any of the above-described neural network training methods or any of the above-described fault determination methods.

[0025] This application also provides a computer storage medium storing a computer program that, when executed by a processor, implements any of the above-described neural network training methods or any of the above-described fault determination methods.

[0026] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described neural network training methods or any of the above-described fault determination methods.

[0027] As can be seen, the embodiments of this application can combine system logs and container metrics corresponding to cloud-native applications to construct a log feature set containing more information. Therefore, the neural network trained based on the log feature set can more accurately identify faults. Attached Figure Description

[0028] Figure 1 This is a flowchart of a neural network training method according to an embodiment of this application;

[0029] Figure 2 This application provides an architecture diagram of a cloud-native intelligent operation and maintenance system.

[0030] Figure 3 This is a schematic diagram illustrating the construction of a log feature set in an embodiment of this application;

[0031] Figure 4 This is a schematic diagram illustrating the use of a gating mechanism to control feature introduction in a long short-term memory network, as proposed in an embodiment of this application.

[0032] Figure 5 This is a flowchart of a method for determining a fault according to an embodiment of this application;

[0033] Figure 6 This is another flowchart of a method for determining a fault according to an embodiment of this application;

[0034] Figure 7 This is a schematic diagram of the structure of a neural network training device according to an embodiment of this application;

[0035] Figure 8 This is a schematic diagram of the structure of a fault-determining device according to an embodiment of this application;

[0036] Figure 9 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] In related technologies, cloud-native operation and maintenance management systems mostly rely on alarm policies and monitoring scripts. To determine alarm policies and monitoring scripts, operation and maintenance personnel need to summarize based on product architecture, performance indicators, and past failures. This may result in inaccurate alarm policies and monitoring scripts, which reduces the fault identification accuracy of the operation and maintenance management system.

[0038] In view of the technical problems existing in related technologies, this application proposes a technical solution that can be applied to scenarios such as operation and maintenance management, natural language processing, data mining, and cloud-native technologies.

[0039] In this embodiment of the application, the influence of building factors within the city can be considered to establish a network signal evaluation model for a large-scale three-dimensional airspace in the city, providing guidance for UAV flights within the city.

[0040] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the embodiments provided herein are merely illustrative of the embodiments of this application and are not intended to limit the embodiments of this application. Furthermore, the embodiments provided below are some embodiments for implementing this application, and not all embodiments for implementing this application. Unless otherwise specified, the technical solutions described in the embodiments of this application can be implemented in any combination.

[0041] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus, such as portions of circuitry, processors, programs, or software, etc.) in the method or apparatus that includes that element.

[0042] The neural network training method and fault determination method provided in this application embodiment include a series of steps. However, the neural network training method and fault determination method provided in this application embodiment are not limited to the steps described. Similarly, the neural network training device and fault determination device provided in this application embodiment include a series of modules. However, the device provided in this application embodiment is not limited to the modules explicitly described, but may also include modules that need to be set up for obtaining relevant information or processing based on information.

[0043] Figure 1 This is a flowchart of a neural network training method according to an embodiment of this application, such as... Figure 1 As shown, the process includes:

[0044] Step 101: Obtain system logs and container metrics corresponding to cloud-native applications. System logs are at least a portion of the logs generated by each microservice in the cloud-native application.

[0045] This application provides a cloud-native intelligent operation and maintenance system based on natural language processing technology. In the cloud-native intelligent operation and maintenance system, various monitoring methods such as Promethus and Zabbix can be used to collect data. An adaptive data feature construction scheme is proposed, which is suitable for monitoring multi-product, big data, and high-performance cloud environments.

[0046] Reference Figure 2In cloud-native intelligent operations and maintenance systems, Promethus can be used to monitor services within Kubernetes. Data collection is achieved by gathering various container metrics and local mount logs. Here, the collected container metrics are the container metrics corresponding to the aforementioned cloud-native applications, and the collected local mount logs are the logs generated by each microservice within the cloud-native application. In some embodiments, after obtaining the logs generated by each microservice within the cloud-native application, all logs generated by each microservice can be used as system logs, or only a portion of the logs generated by each microservice can be used as system logs.

[0047] In some embodiments, refer to Figure 2 Alternatively, the collected data can be categorized using existing alarm policies, or it can be categorized based on faults that occur during operation and maintenance.

[0048] Step 102: Based on system logs and container metrics, construct a log feature set. The log feature set serves as training data for a neural network, which is used to determine fault information in cloud-native applications.

[0049] In some embodiments, refer to Figure 2 In the algorithm analysis section, natural language processing techniques can be applied to perform feature mining on the collected data, thereby constructing a log feature set based on the feature mining results. In some embodiments, log mining algorithms can be used to perform feature mining on the collected data, thereby constructing a fault log chain based on the feature mining results, which can then be used for source tracing.

[0050] In related technologies, monitoring processes and ports in specific scenarios is often achieved by writing scripts or configuring monitoring policies in the monitoring system. This monitoring method relies too heavily on past experience, often resulting in a massive number of invalid alarms. To effectively train neural network models, system logs used to construct log feature sets can be divided into three categories: no-alarm period logs, low-risk alarm period logs, and fault period logs. No-alarm period logs represent logs that generate alarm information; low-risk alarm period logs represent system logs from specific time periods, where the first type of alarm logs are generated and the fault severity represented by these logs is less than a preset threshold; fault period logs represent the second type of alarm logs, where the fault severity represented by these logs is greater than or equal to a preset threshold.

[0051] To effectively build system logs, it's possible to agree with developers to add parameters such as the user's identity document (ID) and request ID to each application log entry. Log sets filtered by these IDs can effectively represent information such as user behavior and system status.

[0052] This application embodiment combines system logs and container metrics to characterize the current system operation status. System logs are generated by each microservice, and their format and fields are typically specified by developers in the code. During product development, developers are required to record key information such as request IDs and user IDs for subsequent log chain construction. First, a log parsing algorithm is applied to convert the system logs into structured data, removing log items such as request IDs, user IDs, and timestamps that do not reflect the system's operational status. Then, natural application processing techniques are applied to convert the logs into feature vectors, referring to... Figure 3 The word vector (word2vec) model can be used to analyze n log files [log1, ..., log2]. n The value is converted into a corresponding log feature vector, where n is an integer greater than 1. Container monitoring systems can collect container metrics, such as container group (pod) running status, Central Processing Unit (CPU) utilization, memory utilization, and network utilization. These container metrics are acquired every second and appended to all log feature vectors within that time period. In some embodiments, refer to... Figure 3 When obtaining m container metrics [a1,...,a], m After that, we can process the indices of m containers [a1,...,a] m The normalization process is performed to obtain the normalized container index [a1′,...,a′]. m Then, a log feature set can be constructed based on the normalized container metrics and log feature vectors, where m is an integer greater than 1.

[0053] In some embodiments of this application, the fault information of cloud-native applications includes at least one of the following: existing fault information and fault warning information.

[0054] Step 103: Train the neural network based on the log feature set.

[0055] Reference Figure 2 In the algorithm analysis section, a neural network can be trained based on the log feature set to obtain the trained neural network.

[0056] In some embodiments, the neural network model described above can be a Long Short-Term Memory (LSTM) model. For example, in this embodiment, a word vector model is used to construct semantic vectors from system logs, which are then combined with container metrics to obtain a log feature set. This log feature set is then used to train the LSTM model. The LSTM model employs a gating mechanism to control the introduction of features. By adding input, forget, and output thresholds, the weights of the self-loop continuously change. With fixed model parameters, the integral scale can be dynamically changed at different times, thereby avoiding the problems of gradient vanishing or gradient inflation. (Refer to...) Figure 4 The gate inputs are all the current input features X. i Compared with the previous feature hidden state H i-1 The output is calculated by the fully connected layer of the activation function. Where F... i The forgetting threshold is used to compute the hidden state H of the previous feature. i-1 Weights retained for the current feature; I i The input threshold is used to calculate the current feature input X. i Preserve the weights of the current state; O i The output threshold is used to calculate the output value of the current state. The calculation formulas for the forget threshold, input threshold, and output threshold are formulas (1) to (3), respectively.

[0057] F i =σ(X) i W xF +H i-1 W hF +b F (1)

[0058] I i =σ(X) i W xI +H i-1 W hI +b I (2)

[0059] O i =σ(X) i W xO +H i-1 W hO +b O (3)

[0060] Where σ represents the fully connected layer and the activation function, W xF W hF W xI W hI W xO and W hO This represents 6 weight parameters, bF b I and b O This represents three deviation parameters.

[0061] In practical applications, steps 101 to 103 can be implemented based on a processor, which can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor.

[0062] As can be seen, the embodiments of this application can combine system logs and container metrics corresponding to cloud-native applications to construct a log feature set containing more information. Therefore, the neural network trained based on the log feature set can more accurately identify faults.

[0063] In some embodiments of this application, the process of constructing a log feature set based on system logs and container metrics includes:

[0064] Within the log chain length range of the system logs, a binary search method is used to find the first range where the accuracy index is higher than the preset accuracy. The log chain length range represents the range from the minimum log chain length to the maximum log chain length. The log chain length represents the number of logs generated by a single request in the system logs, and the accuracy index represents the accuracy of the logs within the range.

[0065] If the first interval is successfully found, a first sliding window is constructed based on the first interval, and logs are selected through the first sliding window;

[0066] A log feature set is constructed based on the log and container metrics selected in the first sliding window.

[0067] In some embodiments, the process of finding a first interval with an accuracy index higher than a preset accuracy index within the log chain length range of the system log using a binary search method may include:

[0068] The log chain length range of the system log is divided into N intervals using a binary search method. For each interval obtained from each interval division, the first interval with an accuracy index higher than the preset accuracy is found, where N is an integer greater than 1.

[0069] Here, N can be preset according to actual needs. For example, the value of N can be 2, 3 or 4. In some embodiments, the preset accuracy rate can also be set according to actual needs. For example, when the log chain length range is [n, m], the preset accuracy rate is the accuracy rate corresponding to the log selected with (m+n) / 2 as the sliding window size.

[0070] Because the log chain lengths for requests from different ports vary significantly, determining the sliding window size based on the request with the most logs would result in a large amount of invalid data in the log chains of requests with relatively simple logic. For an application's logs, the number of log entries generated by a single request can range from a few to dozens. This embodiment applies the binary search algorithm to model training, specifically using binary search to find the optimal sliding window size. This reduces the occurrence of invalid data in system logs, thereby improving the training efficiency and accuracy of the neural network while saving computing power costs. Therefore, the technical solution of this embodiment is applicable to large-scale cloud scenarios.

[0071] In some embodiments, the preset accuracy rate represents the accuracy rate corresponding to logs with a preset log chain length; the method further includes: after performing N interval divisions, if a first interval with an accuracy rate higher than the preset accuracy rate is not found, a second sliding window is constructed based on the preset log chain length, and logs are selected through the second sliding window; a log feature set is constructed based on the logs and container metrics selected in the second sliding window.

[0072] For example, 90% of the system logs generated by the request ID can be used as the training set, and the remaining 10% as the test set. When the first interval is [n, m], logs are selected with a sliding window size of (m+n) / 2. A log feature set is constructed based on the selected logs. According to the accuracy metric ACC1 corresponding to this log feature set, a binary search method is applied to find the most suitable log chain length interval between [n, (m+n) / 2] and [(m+n) / 2, m]. Within the above two intervals, the binary search method is used multiple times to find the first interval with an accuracy metric higher than ACC1. If no first interval with an accuracy metric higher than ACC1 is found after three interval divisions, then (m+n) / 2 is selected as the sliding window size.

[0073] It can be seen that if, after N interval divisions, a first interval with an accuracy metric higher than the preset accuracy is not found, the next interval division will not be performed. Instead, a second sliding window will be constructed directly based on the preset log chain length, thereby constructing a log feature set. This can improve the efficiency of constructing the log feature set, and thus improve the training efficiency of the neural network.

[0074] Figure 5This is a flowchart of a method for determining a fault according to an embodiment of this application, such as... Figure 5 As shown, the process includes:

[0075] Step 501: Obtain log features to be processed; the log features to be processed are constructed based on real-time collected system logs and real-time collected container metrics of cloud-native applications.

[0076] Step 502: Use a pre-trained neural network to process the log features to obtain fault information of cloud-native applications.

[0077] Here, the neural network is trained using any of the training methods mentioned above.

[0078] In practical applications, steps 501 to 502 can be implemented based on a processor, which can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0079] As can be seen, the embodiments of this application can combine system logs and container metrics corresponding to cloud-native applications to construct a log feature set containing more information. Therefore, the neural network trained based on the log feature set can more accurately identify faults.

[0080] In some embodiments of this application, after obtaining the fault information of the cloud-native application, the contribution of each log feature in the log features to be processed can be determined based on the fault information; for the K log features with the highest contribution, the root cause of the fault information can be determined through fault tracing, and the root cause includes at least one of log and container metrics, where K is an integer greater than or equal to 1.

[0081] This application utilizes data mining technology to trace faults by calculating feature contribution. It analyzes log entries for features with high contribution. The solution in this application possesses fault tracing capabilities, accurately identifying indicators and keywords strongly correlated with faults based on feature contribution, thus assisting operations and maintenance personnel in optimizing monitoring strategies. By employing the technical solution of this application, high-value data can be extracted from massive logs, assisting operations and maintenance personnel in formulating monitoring strategies.

[0082] This application proposes a data processing scheme for extracting features from system logs and container metrics. It applies a feature contribution algorithm to extract strongly correlated features of different product faults, so that operation and maintenance and R&D personnel can better design monitoring strategies.

[0083] In some embodiments of this application, the log feature set used to train the neural network includes system logs for a specific time period, where the specific time period represents the time period during which a first type of alarm log is generated, and the risk level of the fault represented by the first type of alarm log is less than a preset threshold; after obtaining the fault information of the cloud-native application, the method further includes: filtering faults with a risk level less than a preset threshold from the fault information of the cloud-native application.

[0084] For example, real-time monitoring and alarms can be achieved by training a three-class classification model, with the three classes of data selected corresponding to no faults, low-risk alarms, and product faults, respectively. Alarm grading is implemented for different scenarios, and the model has the ability to predict faults for new businesses and scenarios where they have not yet occurred. To better assist maintenance personnel in optimizing monitoring strategies, source analysis is performed on features with high contribution. This application embodiment can use gradient boosting decision trees to calculate feature contribution, train the model using LSTM training data, and then mine the data to perform reverse source tracing of high-contribution features to assist in determining the root cause of faults. The feature contribution of each feature set judged as a high-risk fault is calculated, the top 5 features are selected and their coordinate values ​​are recorded, and reverse source tracing is performed according to the method described above for constructing log feature sets. Specific fields or container indicators in the logs are found. If multiple feature sets show that the logs describe product faults for this field, this field is used as a keyword to set an alarm; if a container indicator has high contribution and high frequency, it is considered to set the high-contribution indicator as an alarm threshold.

[0085] In real-world applications, the large number of cloud products and resource pools generates a massive amount of alarms daily. To address this, this application embodiment can train a neural network based on real logs and construct a training set based on logs from low-risk alarm periods. This enables the trained neural network to filter out invalid alarms from massive logs, thereby improving operational efficiency.

[0086] In some embodiments, a pre-trained neural network can be used to parse log features to obtain fault information of cloud-native applications. The fault information of cloud-native applications may include existing fault information and fault warning information. The embodiments of this application use natural language processing and artificial intelligence algorithms to realize automated operation and maintenance, analyze system logs and container metrics in real time, and have the ability to provide past fault alarms and predict fault occurrence.

[0087] As can be seen from the foregoing description, fault information for cloud-native applications can also be filtered for faults. For example, refer to... Figure 2 In the monitoring dashboard section, it is possible to filter low-frequency alarms, update alarm testing strategies, and provide fault warnings.

[0088] This application embodiment can analyze the causes of failures in cloud-native scenarios and assist the operation and maintenance team in scheduling personnel. In this application embodiment, natural language processing and deep learning algorithms are applied to quickly report product failures, filter invalid alarms, and collect container logs and status data through Prometheus to build a fault knowledge base.

[0089] Reference Figure 6 After performing operations such as log collection, log parsing, natural language processing, and container metric collection, a log feature set can be constructed. Then, a neural network can be trained based on this log feature set. The trained neural network can then identify fault information in cloud-native applications, and further, alarm filtering and fault warnings can be implemented based on this fault information. Additionally, after assessing the contribution of each log feature, fault detection strategies can be optimized based on the contribution of each feature.

[0090] In this embodiment, existing data can be used for training, and past faults can be used to label the data to improve the accuracy of fault identification. A sliding window algorithm is used to segment log sequences from different products, and the log set with the highest accuracy is selected as the window size for real-time construction. Container metrics collected by Promethus are used as row feature values, and the log template set and container features are combined into a feature set. Finally, low-risk alarms are filtered and fault predictions are made during real-time detection to assist maintenance personnel in maintaining stable product operation. To obtain more accurate correlation results, alarm mining results and correlation results can be jointly judged to avoid the limitations of a single method.

[0091] The embodiments of this application are applicable to the monitoring and management of complex cloud scenarios involving large scale and multiple products. They possess functions such as fault tracing, alarm filtering, fault early warning, and monitoring optimization, improving operational efficiency and reducing manpower investment, thus demonstrating practical value.

[0092] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0093] Figure 7 This is a schematic diagram of the structure of a neural network training device according to an embodiment of this application, as shown below. Figure 7 As shown, the device includes:

[0094] The first acquisition module 701 is used to acquire system logs and container metrics corresponding to cloud-native applications, wherein the system logs are at least a portion of the logs generated by each microservice in the cloud-native application.

[0095] The first processing module 702 is used to construct a log feature set based on the system log and the container metrics. The log feature set is training data for a neural network, and the neural network is used to determine the fault information of the cloud-native application.

[0096] Training module 703 is used to train the neural network based on the log feature set.

[0097] In some embodiments, the first processing module 702 is configured to construct a log feature set based on the system logs and the container metrics, including:

[0098] Within the log chain length range of the system logs, a first interval with an accuracy index higher than a preset accuracy index is found using a binary search method. The log chain length range represents the interval from the minimum log chain length to the maximum log chain length. The log chain length represents the number of logs generated by a single request in the system logs. The accuracy index represents the accuracy of the logs within the interval.

[0099] If the first interval is successfully found, a first sliding window is constructed based on the first interval, and logs are selected through the first sliding window;

[0100] A log feature set is constructed based on the logs selected in the first sliding window and the container metrics.

[0101] In some embodiments, the first processing module 702 is configured to, within the log chain length range of the system log, use a binary search method to find a first interval where the accuracy index is higher than a preset accuracy, including:

[0102] The log chain length range of the system log is divided into N intervals using a binary search method. For each interval obtained from each interval division, the first interval with an accuracy index higher than the preset accuracy is found, where N is an integer greater than 1.

[0103] In some embodiments, the preset accuracy rate represents the accuracy rate corresponding to logs with a preset log chain length; the first processing module 702 is further configured to, after performing the N interval divisions, if a first interval with an accuracy rate higher than the preset accuracy rate is not found, construct a second sliding window based on the preset log chain length, select logs through the second sliding window; and construct a log feature set based on the logs selected in the second sliding window and the container index.

[0104] In some embodiments, the fault information of the cloud-native application includes at least one of the following: existing fault information and fault warning information.

[0105] In practical applications, the first acquisition module 701, the first processing module 702, and the training module 703 can be implemented based on a processor.

[0106] Figure 8 This is a schematic diagram of the structure of the device for determining a fault according to an embodiment of this application, as shown below. Figure 8 As shown, the device includes:

[0107] The second acquisition module 801 is used to acquire log features to be processed; the log features to be processed are constructed based on real-time collected system logs and real-time collected container metrics of cloud-native applications.

[0108] The second processing module 802 is used to process the log features to be processed using a pre-trained neural network to obtain the fault information of the cloud-native application. The neural network is trained based on any of the above training methods.

[0109] In some embodiments, the second processing module 802 is further configured to, after obtaining the fault information of the cloud-native application, determine the contribution of each log feature among the log features to be processed for the fault information; and determine the root cause of the fault information by fault tracing for the K log features with the highest contribution, wherein the root cause includes at least one of log and container metrics, and K is an integer greater than or equal to 1.

[0110] In some embodiments, the log feature set used to train the neural network includes system logs for a specific time period, wherein the specific time period represents the time period during which a first type of alarm log is generated, and the severity of the fault represented by the first type of alarm log is less than a preset threshold.

[0111] The second processing module 802 is further configured to, after obtaining the fault information of the cloud-native application, filter faults with a risk level less than the preset threshold from the fault information of the cloud-native application.

[0112] In practical applications, the second acquisition module 801 and the second processing module 802 can be implemented based on a processor.

[0113] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0114] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a terminal, server, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0115] Correspondingly, this application embodiment further provides a computer program product, the computer program product including computer executable instructions, which are used to implement any of the neural network training methods provided in this application embodiment.

[0116] Accordingly, this application embodiment further provides a computer storage medium storing computer-executable instructions, which are used to implement any of the neural network training methods provided in the above embodiments.

[0117] This application also provides an electronic device. Figure 9 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application, as shown below. Figure 9 As shown, the electronic device 90 may include:

[0118] Memory 901 is used to store executable instructions;

[0119] The processor 902 is used to implement any of the above-described neural network training methods or fault determination methods when executing executable instructions stored in the memory 901.

[0120] The processor 902 mentioned above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0121] The aforementioned computer-readable storage medium and memory 902 may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; or it may be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0122] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0123] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0124] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict.

[0125] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0126] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0128] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A method for training a neural network, characterized in that, The method includes: Obtain system logs and container metrics corresponding to cloud-native applications, wherein the system logs are at least a portion of the logs generated by each microservice in the cloud-native application; Based on the system logs and container metrics, a log feature set is constructed. The log feature set serves as training data for a neural network, which is used to determine fault information of the cloud-native application. The neural network is trained based on the log feature set; The step of constructing a log feature set based on the system logs and the container metrics includes: Within the log chain length range of the system logs, a first interval with an accuracy index higher than a preset accuracy index is found using a binary search method. The log chain length range represents the interval from the minimum log chain length to the maximum log chain length. The log chain length represents the number of logs generated by a single request in the system logs. The accuracy index represents the accuracy of the logs within the interval. If the first interval is successfully found, a first sliding window is constructed based on the first interval, and logs are selected through the first sliding window; A log feature set is constructed based on the logs selected in the first sliding window and the container metrics.

2. The method according to claim 1, characterized in that, The step of finding a first interval with an accuracy index higher than a preset accuracy index within the log chain length range of the system log using a binary search method includes: The log chain length range of the system log is divided into N intervals using a binary search method. For each interval obtained from each interval division, the first interval with an accuracy index higher than the preset accuracy is searched, where N is an integer greater than 1.

3. The method according to claim 2, characterized in that, The preset accuracy rate represents the accuracy rate corresponding to logs with a preset log chain length. The method further includes: After performing the N interval divisions, if a first interval with an accuracy index higher than the preset accuracy is not found, a second sliding window is constructed based on the preset log chain length, and logs are selected through the second sliding window. A log feature set is constructed based on the logs selected in the second sliding window and the container metrics.

4. The method according to claim 1, characterized in that, The fault information of the cloud-native application includes at least one of the following: existing fault information and fault warning information.

5. A method for determining a fault, characterized in that, The method includes: Obtain log features to be processed; the log features to be processed are constructed based on real-time collected system logs and real-time collected container metrics of cloud-native applications; The log features to be processed are processed using a pre-trained neural network to obtain fault information of the cloud-native application, wherein the neural network is trained based on the method described in any one of claims 1 to 4.

6. The method according to claim 5, characterized in that, After obtaining the fault information of the cloud-native application, the method further includes: For the fault information, determine the contribution of each log feature among the log features to be processed; For the K log features with the highest contribution, the root cause of the fault information is determined by fault tracing. The root cause includes at least one of log and container metrics, where K is an integer greater than or equal to 1.

7. The method according to claim 5, characterized in that, The log feature set used to train the neural network includes system logs for a specific time period, wherein the specific time period represents the time period during which the first type of alarm log is generated, and the risk of the fault represented by the first type of alarm log is less than a preset threshold. After obtaining the fault information of the cloud-native application, the method further includes: In the fault information of the cloud-native application, faults with a danger level less than the preset threshold are filtered out.

8. A training device for a neural network, characterized in that, The device includes: The first acquisition module is used to acquire system logs and container metrics corresponding to cloud-native applications. The system logs are at least a portion of the logs generated by each microservice in the cloud-native application. The first processing module is used to construct a log feature set based on the system logs and the container metrics. The log feature set is the training data of a neural network, and the neural network is used to determine the fault information of the cloud-native application. The training module is used to train the neural network based on the log feature set; The first processing module is configured to construct a log feature set based on the system logs and the container metrics, including: Within the log chain length range of the system logs, a first interval with an accuracy index higher than a preset accuracy index is found using a binary search method. The log chain length range represents the interval from the minimum log chain length to the maximum log chain length. The log chain length represents the number of logs generated by a single request in the system logs. The accuracy index represents the accuracy of the logs within the interval. If the first interval is successfully found, a first sliding window is constructed based on the first interval, and logs are selected through the first sliding window; A log feature set is constructed based on the logs selected in the first sliding window and the container metrics.

9. A device for determining a fault, characterized in that, The device includes: The second acquisition module is used to acquire log features to be processed; the log features to be processed are constructed based on real-time collected system logs and real-time collected container metrics of cloud-native applications. The second processing module is used to process the log features to be processed using a pre-trained neural network to obtain the fault information of the cloud-native application, wherein the neural network is trained based on the method described in any one of claims 1 to 4.

10. An electronic device, characterized in that, The electronic device includes a processor and a memory for storing computer programs that can run on the processor; wherein, The processor is used to run the computer program to perform the method according to any one of claims 1 to 4 or the method according to any one of claims 5 to 7.

11. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method of any one of claims 1 to 4 or the method of any one of claims 5 to 7.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4 or the method of any one of claims 5 to 7.

Images