Network training and fault determination method, device, equipment, medium and product

Abstract

The embodiment of the invention discloses a network training and fault determination method and device, equipment, a medium and a product, and the network training method comprises the steps: obtaining a system log and a container index corresponding to a cloud native application, the system log being at least a part of logs generated by each micro-service in the cloud native application; according to the system log and the container index, a log feature set is constructed, the log feature set is training data of a neural network, and the neural network is used for determining fault information of the cloud native application; and training the neural network according to the log feature set.

Description

Technical Field

[0001] The present application belongs to the technical field of cloud native, and in particular relates to methods, devices, equipment, media and products for network training and fault determination. Background Art

[0002] In related technologies, in cloud-native operation and maintenance management systems, most of them are implemented through alarm strategies and monitoring scripts. In order to determine the alarm strategies and monitoring scripts, operation and maintenance personnel are required to summarize according to the product architecture, performance indicators and past failures. In this way, inaccurate alarm strategies and monitoring scripts may appear, resulting in a reduction in the fault identification accuracy of the operation and maintenance management system. Summary of the invention

[0003] In order to solve the problem of low fault identification accuracy in cloud-native operation and maintenance management systems existing in related technologies, the embodiments of the present application propose a method, device, equipment, medium and product for network training and fault determination.

[0004] The present application provides a neural network training method, the method comprising:

[0005] Obtaining system logs and container metrics corresponding to cloud native applications, where the system logs are at least part of the logs generated by each microservice in the cloud native application;

[0006] Constructing a log feature set according to the system log and the container indicator, wherein the log feature set is training data of a neural network, and the neural network is used to determine fault information of the cloud native application;

[0007] The neural network is trained according to the log feature set.

[0008] In some embodiments, constructing a log feature set based on the system log and the container indicator includes: in the log chain length interval of the system log, searching for a first interval whose accuracy index is higher than a preset accuracy by a binary search method, the log chain length interval represents an interval from a minimum log chain length to a maximum log chain length, the log chain length represents the number of logs generated by a request in the system log, and the accuracy index represents the accuracy corresponding to the logs in the interval; when the first interval is successfully found, constructing a first sliding window based on the first interval, and selecting logs through the first sliding window; constructing a log feature set according to the logs selected in the first sliding window and the container indicator.

[0009] In some embodiments, in the log chain length interval of the system log, a first interval with an accuracy index higher than a preset accuracy is found by a binary search method, including: dividing the log chain length interval of the system log N times by a binary search method, and for each interval obtained by each interval division, finding the first interval with an accuracy index higher than the preset accuracy, where N is an integer greater than 1.

[0010] In some embodiments, the preset accuracy rate represents the accuracy rate corresponding to the log of the preset log chain length; the method further includes: after performing the N interval divisions, if the first interval whose accuracy index is higher than the preset accuracy rate cannot be found, constructing a second sliding window according to the preset log chain length, and selecting the log through the second sliding window; constructing a log feature set according to the log selected in the second sliding window and the container index.

[0011] In some embodiments, the fault information of the cloud native application includes at least one of the following: existing fault information and fault warning information.

[0012] The present application also provides a method for determining a fault, the method comprising:

[0013] Obtaining log features to be processed; the log features to be processed are constructed based on real-time collected system logs and real-time collected container indicators of cloud native applications;

[0014] The log features to be processed are processed using a pre-trained neural network to obtain fault information of the cloud native application, wherein the neural network is trained based on any one of the above training methods.

[0015] In some embodiments, after obtaining the fault information of the cloud native application, the method further includes: determining, for the fault information, the contribution of each log feature in the log features to be processed; for the K log features with the highest contribution, determining the root cause of the fault information through fault tracing, wherein the root cause includes at least one of the log and container indicators, and K is an integer greater than or equal to 1.

[0016] In some embodiments, the log feature set used to train the neural network includes system logs of a specific time period, and the specific time period represents a time period in which a first type of alarm log is generated, and the criticality of the fault represented by the first type of alarm log is less than a preset threshold; after obtaining the fault information of the cloud native application, the method also includes: in the fault information of the cloud native application, filtering faults whose criticality is less than the preset threshold.

[0017] The present application also provides a neural network training device, the device comprising:

[0018] A first acquisition module is used to acquire system logs and container indicators corresponding to cloud native applications, where the system logs are at least part of the logs generated by each microservice in the cloud native application;

[0019] A first processing module, configured to construct a log feature set according to the system log and the container indicator, wherein the log feature set is training data of a neural network, and the neural network is used to determine fault information of the cloud native application;

[0020] A training module is used to train the neural network according to the log feature set.

[0021] The embodiment of the present application also provides a device for determining a fault, the device comprising:

[0022] A second acquisition module is used to acquire log features to be processed; the log features to be processed are constructed based on real-time collected system logs and real-time collected container indicators of cloud native applications;

[0023] The second processing module is used to process the log features to be processed by using a pre-trained neural network to obtain the fault information of the cloud native application, and the neural network is trained based on any one of the above-mentioned neural network training methods.

[0024] An embodiment of the present application also provides an electronic device, comprising a processor and a memory for storing a computer program that can be run on the processor; wherein the processor is used to run the computer program to execute any one of the above-mentioned neural network training methods or any one of the above-mentioned fault determination methods.

[0025] An embodiment of the present application also provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned neural network training methods or any of the above-mentioned fault determination methods.

[0026] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned neural network training methods or any of the above-mentioned fault determination methods.

[0027] It can be seen that the embodiment of the present application can combine the system log and the container indicators corresponding to the cloud-native application to construct a log feature set containing more information. Therefore, the neural network trained based on the log feature set can more accurately realize fault identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flow chart of a neural network training method according to an embodiment of the present application;

[0029] Figure 2 An architectural diagram of the cloud-native intelligent operation and maintenance system provided in an embodiment of the present application;

[0030] Figure 3 A schematic diagram of constructing a log feature set in an embodiment of the present application;

[0031] Figure 4 A schematic diagram of using a gate mechanism to control feature introduction in a long short-term memory network proposed in an embodiment of the present application;

[0032] Figure 5 A flow chart of a method for determining a fault according to an embodiment of the present application;

[0033] Figure 6 Another flow chart of the method for determining a fault according to an embodiment of the present application;

[0034] Figure 7 A schematic diagram of the structure of a neural network training device according to an embodiment of the present application;

[0035] Figure 8 A schematic diagram of the structure of a device for determining a fault according to an embodiment of the present application;

[0036] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In related technologies, in cloud-native operation and maintenance management systems, most of them are implemented through alarm strategies and monitoring scripts. In order to determine the alarm strategies and monitoring scripts, operation and maintenance personnel are required to summarize according to the product architecture, performance indicators and past failures. In this way, inaccurate alarm strategies and monitoring scripts may appear, resulting in a reduction in the fault identification accuracy of the operation and maintenance management system.

[0038] In response to the technical problems existing in the related technologies, a technical solution of the embodiments of the present application is proposed. The embodiments of the present application can be applied to scenarios such as operation and maintenance management, natural language processing, data mining, and cloud native.

[0039] In an embodiment of the present application, the influence of building factors within the city can be considered, and a network signal evaluation model for a large-scale three-dimensional airspace in the city can be established to provide guidance for drone flights within the city.

[0040] The following is a further detailed description of the embodiments of the present application in conjunction with the accompanying drawings and examples. It should be understood that the embodiments provided herein are only used to explain the embodiments of the present application and are not intended to limit the embodiments of the present application. In addition, the embodiments provided below are partial embodiments for implementing the present application, rather than providing all embodiments for implementing the present application. In the absence of conflict, the technical solutions recorded in the embodiments of the present application can be implemented in any combination.

[0041] It should be noted that, in the embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or device including a series of elements includes not only the elements explicitly recorded, but also includes other elements not explicitly listed, or also includes elements inherent to the implementation of the method or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other related elements (such as steps in a method or units in a device, for example, a unit may be a part of a circuit, a part of a processor, a part of a program or software, etc.) in the method or device including the element.

[0042] The neural network training method and the fault determination method provided in the embodiments of the present application include a series of steps, but the neural network training method and the fault determination method provided in the embodiments of the present application are not limited to the recorded steps. Similarly, the neural network training device and the fault determination device provided in the embodiments of the present application include a series of modules, but the device provided in the embodiments of the present application is not limited to including the modules explicitly recorded, and may also include modules required to obtain relevant information or perform processing based on information.

[0043] Figure 1 A flow chart of a neural network training method according to an embodiment of the present application is shown in FIG. Figure 1 As shown, the process includes:

[0044] Step 101: Obtain system logs and container indicators corresponding to cloud native applications. The system logs are at least part of the logs generated by each microservice in the cloud native application.

[0045] The embodiment of the present application provides a cloud-native intelligent operation and maintenance system based on natural language processing technology. In the cloud-native intelligent operation and maintenance system, various monitoring methods such as Promethus and Zabbix can be used to collect data, and an adaptive data feature construction solution is proposed, which is suitable for monitoring multi-product, big data, and high-performance cloud environments.

[0046] Reference Figure 2In the cloud-native intelligent operation and maintenance system, Promethus can be used to monitor the services in K8S, and data is collected by collecting various container indicators and local mount logs. Here, the collected container indicators are the container indicators corresponding to the above-mentioned cloud-native applications, and the collected local mount logs are the logs generated by each microservice in the cloud-native application. In some embodiments, after obtaining the logs generated by each microservice in the cloud-native application, all logs generated by each microservice in the cloud-native application can be used as system logs, or part of the logs generated by each microservice in the cloud-native application can be used as system logs.

[0047] In some embodiments, reference Figure 2 , the collected data can also be classified according to the existing alarm strategy, or the collected data can be classified based on the faults that occur during the operation and maintenance process.

[0048] Step 102: Build a log feature set based on the system log and container indicators. The log feature set is training data for the neural network. The neural network is used to determine the fault information of the cloud native application.

[0049] In some embodiments, reference Figure 2 In the algorithm analysis part, natural language processing technology can be applied to perform feature mining on the above-collected data, so as to construct a log feature set based on the result of feature mining. In some embodiments, a log mining algorithm can be used to perform feature mining on the above-collected data, so as to construct a fault log chain based on the result of feature mining, and traceability can be performed based on the fault log chain.

[0050] In the related art, scripts are often written to monitor processes and ports in specific scenarios, or monitoring strategies are configured in the monitoring system. This monitoring method relies too much on past experience, and often results in a large number of invalid alarms. In order to effectively train the neural network model, the system logs used to construct the log feature set can be divided into the following three types of logs: no alarm period logs, low-risk alarm period logs, and fault period logs, where the no alarm period logs represent logs that generate alarm information, and the low-risk alarm period logs represent system logs for specific periods of time. The specific period represents the period when the first type of alarm log is generated, and the danger level of the fault represented by the first type of alarm log is less than the preset threshold; the fault period log represents the second type of alarm log, and the danger level of the fault represented by the second type of alarm log is greater than or equal to the preset threshold.

[0051] In order to effectively construct system logs, you can agree with R&D personnel to add parameters such as user identity document (ID) and request ID to each application log. The log set filtered by the above ID can effectively represent user behavior, system status and other information.

[0052] The embodiments of the present application can combine system logs and container indicators to characterize the current system operation status. System logs are logs generated by each microservice. The format and fields are usually specified by developers in the code. During the product development stage, developers are required to record key messages such as request IDs and user IDs for subsequent log chain construction. First, the log parsing algorithm is used to convert the system log into structured data, and log items such as request IDs, user IDs, and time that do not reflect the system operation status are removed. Then, the natural application processing technology is used to convert the log into a feature vector. Figure 3 , we can use the word2vec model to transform n logs [log1,…,log n ] is converted into a corresponding log feature vector, where n is an integer greater than 1. The container monitoring system can be used to collect container indicators, such as: container group (pod) operation status, central processing unit (CPU) usage, memory usage, network utilization, etc. The above container indicators are obtained once every second and appended to all log feature vectors within the time. In some embodiments, refer to Figure 3 , after obtaining m container indices [a1,...,a m ], we can then calculate the m container indices [a1,...,a m ] is normalized to obtain the normalized container index [a1′,...,a′ m ] Then, the log feature set can be constructed according to the normalized container index and log feature vector, where m is an integer greater than 1.

[0053] In some embodiments of the present application, the fault information of the cloud-native application includes at least one of the following: existing fault information and fault warning information.

[0054] Step 103: Train a neural network based on the log feature set.

[0055] Reference Figure 2 ,In the algorithm analysis part, the neural network can be trained according to the ,log feature set to obtain the trained neural network.

[0056] In some embodiments, the model of the neural network can be a long short-term memory network (Long Short-Term Memory, LSTM) model. Exemplarily, in an embodiment of the present application, a word vector model is used to construct the system log into a semantic vector, and a log feature set is obtained after combining it with the container index, and the log feature set is used to train the LSTM model. The LSTM model uses a gate mechanism to control the introduction of features. By adding input thresholds, forgetting thresholds, and output thresholds, the weights of the self-loop are constantly changing. When the model parameters are fixed, the integral scale at different times can be changed dynamically, thereby avoiding the problem of gradient disappearance or gradient expansion. Reference Figure 4 , the gate input is the current input feature X i With the previous feature hidden state H i-1 , the output is calculated by the fully connected layer of the activation function. i is the forgetting threshold, which is used to calculate the hidden state H of the previous feature i-1 The weight retained to the current feature; I i is the input threshold, used to calculate the current feature input X i Keep the weight of the current state; O i is the output threshold, which is used to calculate the output value of the current state. The calculation formulas for the forgetting threshold, input threshold, and output threshold are respectively as follows:

[0057] F i =σ(X i W xF +H i-1 W hF +b F ) (1)

[0058] I i =σ(X i W xI +H i-1 W hI +b I ) (2)

[0059] O i =σ(X i W xO +H i-1 W hO +b O ) (3)

[0060] Among them, σ represents the fully connected layer and activation function, W xF , W hF , W xI , W hI , W xO and W hO Represents 6 weight parameters, bF 、b I and b O Represents 3 deviation parameters.

[0061] In practical applications, steps 101 to 103 can be implemented based on a processor, and the processor can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor.

[0062] It can be seen that the embodiment of the present application can combine the system log and the container indicators corresponding to the cloud-native application to construct a log feature set containing more information. Therefore, the neural network trained based on the log feature set can more accurately realize fault identification.

[0063] In some embodiments of the present application, the process of constructing a log feature set according to system logs and container indicators includes:

[0064] In the log chain length interval of the system log, the first interval whose accuracy index is higher than the preset accuracy is found by binary search method, the log chain length interval represents the interval from the minimum log chain length to the maximum log chain length, the log chain length represents the number of logs generated by one request in the system log, and the accuracy index represents the accuracy corresponding to the logs in the interval;

[0065] When the first interval is successfully found, a first sliding window is constructed based on the first interval, and logs are selected through the first sliding window;

[0066] A log feature set is constructed based on the log and container indicators selected in the first sliding window.

[0067] In some embodiments, in the log chain length interval of the system log, the process of finding the first interval whose accuracy index is higher than the preset accuracy by using the binary search method may include:

[0068] The log chain length interval of the system log is divided into N intervals by binary division. For each interval obtained by each interval division, the first interval whose accuracy index is higher than the preset accuracy is found, where N is an integer greater than 1.

[0069] Here, N can be pre-set according to actual needs. For example, the value of N is 2, 3 or 4. In some embodiments, the preset accuracy rate can also be set according to actual needs. For example, when the log chain length interval is [n, m], the preset accuracy rate is the accuracy rate corresponding to the log selected with (m+n) / 2 as the sliding window size.

[0070] Since the length of the log chain for different port requests varies greatly, if the sliding window size is determined based on the request with the most logs, a large amount of invalid data will be present in the request log chain with relatively simple logic. For the application log of a product, the number of logs generated by a request ranges from a few to dozens. In the embodiment of the present application, the binary search algorithm is applied to the model training, that is, the optimal sliding window size is found through the binary search method, which can reduce the occurrence of invalid data in the system log, thereby helping to improve the training efficiency and accuracy of the neural network and save computing power costs. Therefore, the technical solution of the embodiment of the present application can be applied to large-scale cloud scenarios.

[0071] In some embodiments, the preset accuracy rate represents the accuracy rate corresponding to the log of the preset log chain length; the method further includes: after performing N interval divisions, if the first interval whose accuracy index is higher than the preset accuracy rate cannot be found, then constructing a second sliding window according to the preset log chain length, and selecting the log through the second sliding window; constructing a log feature set according to the log and container index selected in the second sliding window.

[0072] Exemplarily, 90% of the system logs generated by the request id can be used as a training set, and the other 10% can be used as a test set. When the first interval is [n, m], select logs with (m+n) / 2 as the sliding window size, build a log feature set based on the selected logs, and use the binary search method to find the most suitable log chain length interval in [n, (m+n) / 2] and [(m+n) / 2, m] according to the accuracy index ACC1 corresponding to the log feature set. Use the binary search method to find the first interval with an accuracy index higher than ACC1 in the above two intervals multiple times. If the first interval with an accuracy index higher than ACC1 is not found after three interval divisions, then select (m+n) / 2 as the sliding window size.

[0073] It can be seen that if after N interval divisions, the first interval whose accuracy index is higher than the preset accuracy cannot be found, the next interval division will not be continued. Instead, the second sliding window will be directly constructed according to the preset log chain length, and then the log feature set will be constructed. In this way, the construction efficiency of the log feature set can be improved, and therefore, the training efficiency of the neural network can be improved.

[0074] Figure 5A flow chart of a method for determining a fault according to an embodiment of the present application is shown in FIG. Figure 5 As shown, the process includes:

[0075] Step 501: Obtain the log features to be processed; the log features to be processed are constructed based on the real-time collected system logs and the real-time collected container indicators of the cloud native application.

[0076] Step 502: Use a pre-trained neural network to process the log features to be processed to obtain fault information of the cloud native application.

[0077] Here, the neural network is trained according to any of the above training methods.

[0078] In practical applications, steps 501 to 502 may be implemented based on a processor, and the processor may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0079] It can be seen that the embodiment of the present application can combine the system log and the container indicators corresponding to the cloud-native application to construct a log feature set containing more information. Therefore, the neural network trained based on the log feature set can more accurately realize fault identification.

[0080] In some embodiments of the present application, after obtaining the fault information of the cloud native application, the contribution of each log feature in the log features to be processed can be determined for the fault information; for the K log features with the highest contribution, the root cause of the fault information is determined by fault tracing, and the root cause includes at least one of the log and container indicators, and K is an integer greater than or equal to 1.

[0081] The embodiment of the present application uses data mining technology to trace the fault by calculating the contribution of features, and analyzes the logs one by one for the features with higher contribution. The solution of the embodiment of the present application has the ability to trace the source, and can accurately locate the indicators and keywords that are strongly associated with the fault according to the contribution of the features, and can assist the operation and maintenance personnel in optimizing the monitoring strategy. The technical solution of the embodiment of the present application can extract high-value data from massive logs to assist the operation and maintenance personnel in formulating monitoring strategies.

[0082] The embodiment of the present application proposes a data processing solution for extracting features from system logs and container indicators, and applies a feature contribution algorithm to extract strongly correlated features of different product failures, so that operation and maintenance and R&D personnel can better design monitoring strategies.

[0083] In some embodiments of the present application, the log feature set used to train the neural network includes system logs of a specific time period, the specific time period represents the time period for generating a first type of alarm log, and the criticality of the fault represented by the first type of alarm log is less than a preset threshold; after obtaining the fault information of the cloud native application, the method also includes: in the fault information of the cloud native application, filtering faults whose criticality is less than a preset threshold.

[0084] Exemplarily, real-time monitoring alarms can be achieved by training a three-classification model, and the three types of data selected for data selection correspond to no fault, low-risk alarms, and product failures. For the early warning classification processing of unavailable scenarios, it has the ability to predict faults for new services and scenarios that have not occurred. In order to better assist operation and maintenance personnel in optimizing monitoring strategies, traceability analysis is performed on features with higher contribution. In the embodiment of the present application, a gradient boosting decision tree can be used to calculate feature contribution, and a model can be trained using LSTM training data to mine the data, and features with high contribution can be reversely traced to assist in determining the root cause of the fault. Calculate the feature contribution of each feature set judged as a high-risk fault, filter out the top 5 features and record the coordinate values, and perform reverse tracing according to the above method of constructing a log feature set. Find a specific field or container indicator in the log. If the log of the field is found in multiple feature sets to express product failures, use the field as a keyword to set an alarm; if the container indicator has a high contribution and occurs frequently, consider setting the high contribution indicator as the alarm threshold.

[0085] In actual application scenarios, a large number of cloud products and resource pools generate massive alarms every day. In response to this situation, the embodiment of the present application can train a neural network based on real logs and construct a training set based on low-risk alarm period logs, so that the trained neural network has the ability to filter invalid alarms from massive logs, thereby improving operation and maintenance efficiency.

[0086] In some embodiments, a pre-trained neural network can be used to parse log features to obtain fault information of cloud-native applications. The fault information of cloud-native applications may include existing fault information and fault warning information. The embodiments of the present application use natural language processing and artificial intelligence algorithms to achieve automated operation and maintenance, analyze system logs and container indicators in real time, and have the ability to warn of past faults and predict the occurrence of faults.

[0087] Combined with the above-mentioned contents, it can be seen that fault filtering can also be performed for fault information of cloud native applications. Figure 2 In the monitoring dashboard, low-frequency alarms can be filtered, alarm test strategies can be updated, and fault warnings can be issued.

[0088] The embodiments of the present application can analyze the causes of failures in cloud-native scenarios and assist in the scheduling of operation and maintenance team personnel; in the embodiments of the present application, natural language processing and deep learning algorithms are used to quickly feedback product failures, filter invalid alarms, collect container logs and status data through Prometheus, and build a fault knowledge base.

[0089] Reference Figure 6 After performing operations such as log collection, log parsing, natural language processing, and container indicator collection, a log feature set can be constructed; then, a neural network can be trained based on the log feature set, and the trained neural network can determine the fault information of cloud native applications, and then alarm filtering and fault warning can be performed based on the fault information of cloud native applications. In addition, after the contribution of each log feature is determined, the fault detection strategy can be optimized based on the contribution of the log feature.

[0090] In an embodiment of the present application, stock data can be used for training, and data can be marked with past faults to improve the accuracy of fault identification. The log sequences of different products are segmented using a sliding window algorithm, and the log set with the highest accuracy is selected as the window size for real-time construction. The container indicators collected by Promethus are used as a row of feature values, and the log template set and container features are combined into a feature set. Finally, low-risk alarms and fault predictions are filtered during real-time detection to assist operation and maintenance personnel in maintaining stable product operation. In order to obtain more accurate correlation results, the alarm mining results can be jointly judged with the correlation results to avoid the limitations of a single method.

[0091] The embodiment of the present application can be applied to the monitoring and management of large-scale, multi-product complex cloud scenarios. It has the functions of fault tracing, filtering alarms, fault warning and monitoring optimization, improves operation and maintenance efficiency, reduces operation and maintenance manpower investment, and has application value.

[0092] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0093] Figure 7 Schematic diagram of the structure of the neural network training device of the embodiment of the present application, such as Figure 7 As shown, the device comprises:

[0094] The first acquisition module 701 is used to obtain system logs and container indicators corresponding to cloud native applications, where the system logs are at least part of the logs generated by each microservice in the cloud native application;

[0095] A first processing module 702 is used to construct a log feature set according to the system log and the container indicator, wherein the log feature set is training data of a neural network, and the neural network is used to determine the fault information of the cloud native application;

[0096] The training module 703 is used to train the neural network according to the log feature set.

[0097] In some embodiments, the first processing module 702 is used to construct a log feature set according to the system log and the container indicator, including:

[0098] In the log chain length interval of the system log, a first interval whose accuracy index is higher than a preset accuracy is searched by a binary search method, wherein the log chain length interval represents an interval from a minimum log chain length to a maximum log chain length, the log chain length represents the number of logs generated by one request in the system log, and the accuracy index represents the accuracy corresponding to the logs in the interval;

[0099] In the case where the first interval is successfully found, a first sliding window is constructed based on the first interval, and a log is selected through the first sliding window;

[0100] A log feature set is constructed according to the logs selected in the first sliding window and the container indicators.

[0101] In some embodiments, the first processing module 702 is used to find a first interval whose accuracy index is higher than a preset accuracy rate in the log chain length interval of the system log by a binary search method, including:

[0102] The log chain length interval of the system log is divided into N intervals by binary division, and for each interval obtained by each interval division, the first interval whose accuracy index is higher than the preset accuracy is found, where N is an integer greater than 1.

[0103] In some embodiments, the preset accuracy rate represents the accuracy rate corresponding to the logs of the preset log chain length; the first processing module 702 is also used to construct a second sliding window according to the preset log chain length if the first interval whose accuracy index is higher than the preset accuracy rate cannot be found after performing the N interval divisions, and select the log through the second sliding window; construct a log feature set according to the logs selected in the second sliding window and the container index.

[0104] In some embodiments, the fault information of the cloud native application includes at least one of the following: existing fault information and fault warning information.

[0105] In practical applications, the first acquisition module 701, the first processing module 702 and the training module 703 can be implemented based on a processor.

[0106] Figure 8 Schematic diagram of the structure of the device for determining a fault according to an embodiment of the present application. Figure 8 As shown, the device comprises:

[0107] The second acquisition module 801 is used to acquire the log features to be processed; the log features to be processed are constructed based on the system logs collected in real time and the container indicators of the cloud native application collected in real time;

[0108] The second processing module 802 is used to process the log features to be processed by using a pre-trained neural network to obtain the fault information of the cloud native application, and the neural network is trained based on any one of the above training methods.

[0109] In some embodiments, the second processing module 802 is further used to determine, after obtaining the fault information of the cloud-native application, the contribution of each log feature in the log features to be processed for the fault information; for the K log features with the highest contribution, determine the root cause of the fault information by fault tracing, where the root cause includes at least one of the log and container indicators, and K is an integer greater than or equal to 1.

[0110] In some embodiments, the log feature set used to train the neural network includes system logs of a specific period, the specific period represents a period in which a first type of alarm log is generated, and the danger level of the fault represented by the first type of alarm log is less than a preset threshold;

[0111] The second processing module 802 is further configured to filter faults having a risk level less than a preset threshold in the fault information of the cloud native application after obtaining the fault information of the cloud native application.

[0112] In practical applications, the second acquisition module 801 and the second processing module 802 can be implemented based on a processor.

[0113] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0114] It should be noted that in the embodiments of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a terminal, a server, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0115] Correspondingly, an embodiment of the present application further provides a computer program product, which includes computer executable instructions, and the computer executable instructions are used to implement any neural network training method provided in the embodiment of the present application.

[0116] Accordingly, an embodiment of the present application further provides a computer storage medium, on which computer executable instructions are stored, and the computer executable instructions are used to implement any one of the neural network training methods provided in the above embodiments.

[0117] An embodiment of the present application also provides an electronic device. Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the electronic device 90 may include:

[0118] Memory 901, used to store executable instructions;

[0119] The processor 902 is used to implement any of the above-mentioned neural network training methods or fault determination methods when executing the executable instructions stored in the memory 901.

[0120] The processor 902 may be at least one of an ASIC, a DSP, a DSPD, a PLD, a FPGA, a CPU, a controller, a microcontroller, and a microprocessor.

[0121] The above-mentioned computer-readable storage medium and memory 902 can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and other memories; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0122] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0123] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0124] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0125] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0126] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0128] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A neural network training method, characterized in that: The method comprises: Obtaining system logs and container metrics corresponding to cloud native applications, where the system logs are at least part of the logs generated by each microservice in the cloud native application; Constructing a log feature set according to the system log and the container indicator, wherein the log feature set is training data of a neural network, and the neural network is used to determine fault information of the cloud native application; The neural network is trained according to the log feature set.

2. The method according to claim 1, characterized in that The step of constructing a log feature set according to the system log and the container indicator includes: In the log chain length interval of the system log, a first interval whose accuracy index is higher than a preset accuracy is searched by a binary search method, wherein the log chain length interval represents an interval from a minimum log chain length to a maximum log chain length, the log chain length represents the number of logs generated by one request in the system log, and the accuracy index represents the accuracy corresponding to the logs in the interval; In the case where the first interval is successfully found, a first sliding window is constructed based on the first interval, and a log is selected through the first sliding window; A log feature set is constructed according to the logs selected in the first sliding window and the container indicators.

3. The method according to claim 2, characterized in that The step of searching, in the log chain length interval of the system log, for a first interval having an accuracy index higher than a preset accuracy by a binary search method comprises: The log chain length interval of the system log is divided into N intervals by a binary search method, and for each interval obtained by each interval division, the first interval whose accuracy index is higher than a preset accuracy is searched, where N is an integer greater than 1.

4. The method according to claim 3, characterized in that The preset accuracy rate represents the accuracy rate corresponding to logs of a preset log chain length; The method further comprises: After performing the N interval divisions, if the first interval whose accuracy index is higher than the preset accuracy cannot be found, constructing a second sliding window according to the preset log chain length, and selecting logs through the second sliding window; A log feature set is constructed according to the logs selected in the second sliding window and the container indicators.

5. The method according to claim 1, characterized in that The fault information of the cloud native application includes at least one of the following: existing fault information and fault warning information.

6. A method for determining a fault, characterized in that: The method comprises: Obtaining log features to be processed; the log features to be processed are constructed based on real-time collected system logs and real-time collected container indicators of cloud native applications; The log features to be processed are processed using a pre-trained neural network to obtain fault information of the cloud native application, wherein the neural network is trained based on the method described in any one of claims 1 to 5.

7. The method according to claim 6, characterized in that After obtaining the fault information of the cloud native application, the method further includes: Determine, for the fault information, the contribution of each log feature in the log features to be processed; For the K log features with the highest contribution, the root cause of the fault information is determined by fault tracing, where the root cause includes at least one of the log and the container indicator, and K is an integer greater than or equal to 1.

8. The method according to claim 6, characterized in that The log feature set used to train the neural network includes system logs of a specific period, wherein the specific period represents a period in which a first type of alarm log is generated, and the risk level of the fault represented by the first type of alarm log is less than a preset threshold; After obtaining the fault information of the cloud native application, the method further includes: In the fault information of the cloud native application, faults with a risk level less than the preset threshold are filtered.

9. A neural network training device, characterized in that: The device comprises: A first acquisition module is used to acquire system logs and container indicators corresponding to cloud native applications, where the system logs are at least part of the logs generated by each microservice in the cloud native application; A first processing module, configured to construct a log feature set according to the system log and the container indicator, wherein the log feature set is training data of a neural network, and the neural network is used to determine fault information of the cloud native application; A training module is used to train the neural network according to the log feature set.

10. A device for determining a fault, characterized in that: The device comprises: A second acquisition module is used to acquire the log features to be processed; the log features to be processed are constructed based on the system logs collected in real time and the container indicators of the cloud native application collected in real time; The second processing module is used to process the log features to be processed by using a pre-trained neural network to obtain the fault information of the cloud native application, wherein the neural network is trained based on the method described in any one of claims 1 to 5.

11. An electronic device, characterized in that: The electronic device comprises a processor and a memory for storing a computer program that can be run on the processor; wherein, The processor is configured to run the computer program to perform the method according to any one of claims 1 to 5 or the method according to any one of claims 6 to 8.

12. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method described in any one of claims 1 to 5 or the method described in any one of claims 6 to 8 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 5 or the method according to any one of claims 6 to 8.

Images