Flink log management method, system and device based on large model and medium

Abstract

The invention discloses a Flink log management method, system and device based on a large model and a medium, and the method comprises the steps: obtaining a historical log sample through multiple data sources, and carrying out the preprocessing of the historical log sample, so as to obtain the preprocessed log data; according to the preprocessed log data, training a log translation model, and training an anomaly detection model; and receiving a to-be-processed Flink log file to output a corresponding anomaly detection report through the trained anomaly detection model, and outputting corresponding log translation information through the trained log translation model to realize Flink log management. According to the method, the complexity of Flink log management is greatly simplified, the requirement of manual intervention is reduced, and the efficiency and effect of log management are improved.

Description

Technical Field

[0001] The present application relates to the field of log processing technology, and in particular to a Flink log management method, system, device and medium based on a large model. Background Art

[0002] Currently, in the field of big data processing, Apache Flink, as a high-performance, highly available stream processing framework, is widely used in real-time data processing scenarios. However, despite Flink's outstanding performance in data processing capabilities, it has certain limitations in log recording and processing. The traditional Flink log recording method is mainly for technical personnel, and the content, format, and interpretation of the records are highly dependent on the technical personnel's professional knowledge and experience.

[0003] This limitation is particularly prominent in actual business applications. When business requirements are complex and changeable and problems need to be quickly located and solved, the communication barrier between business personnel and technical personnel becomes an issue that cannot be ignored. Business personnel often lack sufficient technical background and find it difficult to directly obtain useful information from logs; while technical personnel need to spend extra time and energy to interpret the log content, which undoubtedly increases the time and manpower costs of the project.

[0004] Furthermore, Flink programs may encounter various complex situations during operation, such as abnormal data format, insufficient system resources, external dependency failure, etc., which may cause program failure or system blockage. However, the current logging mechanism can only simply capture and print out error information, but cannot deeply identify and analyze these errors, let alone convert them into text output that is easy for business personnel to understand. Summary of the invention

[0005] The embodiments of the present application provide a Flink log management method, system, device and medium based on a large model to solve the above technical problems.

[0006] On the one hand, the embodiment of the present application provides a Flink log management method based on a large model, including:

[0007] Acquire historical log samples through multiple data sources, and preprocess the historical log samples to obtain preprocessed log data;

[0008] According to the preprocessed log data, a log translation model is trained, and an anomaly detection model is trained;

[0009] Receive the Flink log files to be processed, output the corresponding anomaly detection report through the trained anomaly detection model, and output the corresponding log translation information through the trained log translation model to implement Flink log management.

[0010] In one implementation of the present application, historical log samples are obtained through multiple data sources, and the historical log samples are preprocessed to obtain preprocessed log data, specifically including:

[0011] Obtain historical log samples through the Flink log library or open source community, and perform data cleaning on the historical log samples to remove abnormal log entries in the historical log samples; the abnormal log entries are repeated, invalid or erroneous log entries;

[0012] Key features are extracted from the cleaned historical log samples to obtain corresponding log data; the key features include at least keywords, error codes and system components.

[0013] In one implementation of the present application, the log translation model is trained according to the preprocessed log data, specifically including:

[0014] Input the preprocessed log data into the fastText model to extract word vectors describing log features, and input the word vectors describing log features into the log translation model;

[0015] The word vector describing the log feature is trained through the log translation model, and the association between the word vector and the natural language is learned to obtain the trained log translation model.

[0016] In one implementation of the present application, training the anomaly detection model specifically includes:

[0017] Determine the time series corresponding to the preprocessed log data, and model the time dimension based on the log data with time series;

[0018] Build the covariance matrix of the Flink run configuration and obtain the linear transformation corresponding to the memory usage of the Flink task to build an anomaly detection model for linear transformation.

[0019] In one implementation of the present application, a Flink log file to be processed is received to output a corresponding anomaly detection report through a trained anomaly detection model, specifically including:

[0020] Receive the Flink log files to be processed generated over time, and input the Flink logs to be processed into the trained anomaly detection model;

[0021] The trained anomaly detection model is used to output an anomaly detection report corresponding to the Flink log file to be processed.

[0022] In one implementation of the present application, the trained log translation model is used to output corresponding log translation information to implement Flink log management, which specifically includes:

[0023] Inputting the to-be-processed Flink log into the trained log translation model, so as to decode the to-be-processed Flink log through the trained log translation model;

[0024] The log type and problem category corresponding to the Flink log to be processed are identified according to the decoding result, and the log translation information corresponding to the Flink log file to be processed is output.

[0025] In one implementation of the present application, after outputting corresponding log translation information through a trained log translation model to implement Flink log management, the method further includes:

[0026] Based on Flink's business rules, standard operations are performed on the decoding results corresponding to the Flink logs to be processed, and analysis reports corresponding to the anomaly detection report and the log translation information are generated to implement Flink log management;

[0027] The standard operations include filtering operations, sorting operations and searching operations.

[0028] On the other hand, the embodiment of the present application further provides a Flink log management system based on a large model, the system comprising:

[0029] The front-end display layer is used to build a viewing interface through the front-end framework, display log analysis results, and provide a user interaction interface;

[0030] The backend service layer uses the Spring Boot framework to receive frontend requests, call data processing and model inference services, and return corresponding results.

[0031] The data processing layer is used to collect, clean, and format historical log samples;

[0032] The model training and inference layer is used to train the preprocessed log data and provide an API interface for backend services to call for real-time or batch inference;

[0033] The data storage layer is used to use the database to store structured data corresponding to configuration parameters, and use HDFS to store log files and model files.

[0034] On the other hand, the embodiment of the present application further provides a Flink log management device based on a large model, the device comprising:

[0035] at least one processor;

[0036] and, a memory communicatively coupled to the at least one processor;

[0037] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the Flink log management method based on a large model as described above.

[0038] On the other hand, an embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, which, when executed, implements a Flink log management method based on a large model as described above.

[0039] The embodiments of the present application provide a Flink log management method, system, device, and medium based on a large model, which at least have the following beneficial effects:

[0040] Obtaining historical log samples from multiple data sources can cover a wider range of usage scenarios and abnormal situations, thereby improving the diversity and richness of log data and providing a more comprehensive data foundation for subsequent model training; preprocessing multi-source historical log samples can remove noise, unify the format, and improve data quality, so that the log translation model and anomaly detection model based on these data have stronger generalization capabilities after training, and better adapt to different log environments and abnormal situations; using preprocessed high-quality log data to train the anomaly detection model can enable the model to learn more accurate abnormal patterns, improve the accuracy of identifying abnormal behaviors in Flink logs and the speed of response, and effectively reduce missed reports and false positives; through the trained log translation model, complex Flink log information can be converted into an easy-to-understand form, helping operation and maintenance personnel or developers to quickly locate problems and understand the meaning of logs, thereby improving the efficiency and quality of log analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 A flowchart of a Flink log management method based on a large model provided in an embodiment of the present application;

[0043] Figure 2A schematic diagram of the organizational structure of a Flink log management system based on a large model provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the internal structure of a Flink log management device based on a large model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0046] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0047] Figure 1 A flowchart of a Flink log management method based on a large model is provided in an embodiment of the present application.

[0048] The analysis method involved in the embodiments of the present application can be implemented by a terminal device or a server, and the present application does not impose any special restrictions on this. For the convenience of understanding and description, the following embodiments are described in detail by taking a server as an example.

[0049] It should be noted that the server may be a single device or a system consisting of multiple devices, that is, a distributed server, and this application does not make any specific limitation on this.

[0050] like Figure 1 As shown, a Flink log management method based on a large model provided in an embodiment of the present application includes:

[0051] 101. Obtain historical log samples through multiple data sources, and preprocess the historical log samples to obtain preprocessed log data.

[0052] Specifically, in one embodiment of the present application, historical log samples are obtained through multiple data sources, and the historical log samples are preprocessed to obtain preprocessed log data, specifically including:

[0053] Obtain historical log samples through the Flink log library or open source community, and perform data cleaning on the historical log samples to remove abnormal log entries in the historical log samples; abnormal log entries are repeated, invalid, or malformed log entries;

[0054] Extract key features from the cleaned historical log samples to obtain corresponding log data; the key features include at least keywords, error codes, and system components.

[0055] In one embodiment, the historical log storage location is accessed through the Flink log library, and the Flink task running logs in the past month are collected as historical log samples using log collection tools (such as Flume, Logstash, etc.). At the same time, Flink log files shared by users can also be downloaded from open source communities such as GitHub and the Apache Flink official forum as supplementary samples.

[0056] Preprocess the collected historical log samples. First, remove duplicate log entries. Use hashing algorithms or direct string comparison to identify and delete duplicates. Next, check the validity of log entries and remove those that do not contain valid information, such as entries that only contain timestamps but no actual log content. Finally, verify the log format and remove entries that do not conform to the Flink log standard format, such as logs that are missing key fields or have the wrong field order.

[0057] In the cleaned log data, use regular expressions or log parsing libraries (such as Logstash's grok plugin) to extract keywords, such as operation types (such as "START", "END", "ERROR"), task names, etc. Identify and record error codes, which are usually numeric or string codes representing specific error types and are critical for troubleshooting. Extract system component information, including the node that executed the task, the resources used (such as CPU, memory), and any related external system or service names.

[0058] 102. Based on the preprocessed log data, the log translation model is trained, and the anomaly detection model is trained.

[0059] Specifically, in one embodiment of the present application, the log translation model is trained according to the preprocessed log data, specifically including:

[0060] Input the preprocessed log data into the fastText model to extract word vectors that describe log features, and then input the word vectors that describe log features into the log translation model;

[0061] The word vectors describing log features are trained through the log translation model, and the association between word vectors and natural language is learned to obtain a trained log translation model.

[0062] In one embodiment, first, the collected Flink log data is preprocessed, including removing duplicate logs, invalid logs, and malformed log entries to ensure the accuracy and consistency of the log data. The cleaned log data is segmented to split each log into separate words or phrases for subsequent word vector extraction.

[0063] Input the preprocessed log data into the fastText model. fastText is an efficient word vector extraction tool that can map words to points in a fixed-dimensional vector space. Through the fastText model, word vectors with descriptive log features are extracted. These word vectors can capture the semantic relationship between words and provide rich feature representation for subsequent log translation.

[0064] The word vectors describing the log features are input into the log translation model. The log translation model is a machine learning model that aims to learn the relationship between word vectors and natural language. A large number of log samples and corresponding natural language descriptions are used as training data to train the log translation model. During the training process, the model will gradually learn how to map word vectors to natural language descriptions. Through continuous iteration and optimization, a trained log translation model is finally obtained. The model can translate the input log data into easy-to-understand natural language descriptions, helping operation and maintenance personnel or developers to quickly locate problems and understand the meaning of logs.

[0065] Deploy the trained log translation model to the actual Flink log management system to translate the log data generated in real time. Perform a quality assessment on the translated log data to compare whether the log data before translation and the natural language description after translation are consistent and accurate. Further optimize and adjust the log translation model based on the evaluation results to improve the accuracy and efficiency of the translation. At the same time, you can also collect user feedback and continuously optimize the model to meet actual needs.

[0066] In one embodiment of the present application, training the anomaly detection model specifically includes:

[0067] Determine the time series corresponding to the preprocessed log data, and model the time dimension based on the log data with time series;

[0068] Build the covariance matrix of the Flink run configuration and obtain the linear transformation corresponding to the memory usage of the Flink task to build an anomaly detection model for linear transformation.

[0069] In one embodiment, Flink log data is collected and cleaned to remove duplicate, invalid or malformed log entries to ensure data purity. The cleaned log data is standardized, such as unifying the time format, normalizing the log level, etc., for subsequent processing. The preprocessed log data is input into the fastText model, and the key words in the log are extracted using fastText's deep learning algorithm and converted into word vectors. These word vectors can capture the semantic relationship between words and provide rich feature representation for log translation.

[0070] Design a log translation model that uses the word vectors extracted by fastText as input and outputs the corresponding natural language description. Select a suitable machine learning algorithm (such as neural network, LSTM, etc.) as the basis of the log translation model, and define the input and output structure of the model. Use a large number of log samples and corresponding natural language descriptions as training data to train the log translation model. During the training process, adjust the model parameters through the back propagation algorithm so that the model gradually learns the relationship between word vectors and natural language. Optimize model performance through cross-validation, adjusting hyperparameters, and other means to ensure the generalization ability and accuracy of the model.

[0071] Evaluate the trained log translation model, use the test data set to test the translation effect of the model, and calculate indicators such as accuracy and recall. Deploy the trained log translation model to the actual Flink log management system to translate the log data generated in real time. Monitor the running effect of the model and update the model regularly to adapt to changes in log data and new translation requirements. At the same time, collect user feedback and continuously optimize the model to improve user satisfaction.

[0072] 103. Receive the Flink log file to be processed, output the corresponding anomaly detection report through the trained anomaly detection model, and output the corresponding log translation information through the trained log translation model to implement Flink log management.

[0073] Specifically, in one embodiment of the present application, a Flink log file to be processed is received, and a corresponding anomaly detection report is outputted through a trained anomaly detection model, specifically including:

[0074] Receive the pending Flink log files generated over time and input the pending Flink logs into the trained anomaly detection model;

[0075] Through the trained anomaly detection model, the anomaly detection report corresponding to the Flink log file to be processed is output.

[0076] In one embodiment, the Flink tasks in the system are monitored in real time to capture log files generated over time. The captured log files are stored in a specified storage location for subsequent processing. The received Flink log files to be processed are preprocessed, including removing duplicate logs, invalid logs, and malformed log entries. The cleaned log data is standardized to ensure that the data format is consistent with the data format used when training the anomaly detection model.

[0077] The preprocessed Flink log data to be processed is input into the trained anomaly detection model. The anomaly detection model is trained based on historical log data and can identify abnormal patterns in logs. The input Flink log data to be processed is analyzed in real time by the anomaly detection model to detect whether there is abnormal behavior. If abnormal behavior is detected, a corresponding anomaly detection report is generated, which contains a detailed description of the anomaly, the time of occurrence, possible causes, and other information. If no abnormal behavior is detected, a normal log processing report is generated, indicating that no abnormality is found in the log file.

[0078] Output the generated anomaly detection report or normal log processing report to the specified location, such as the monitoring system interface, log file, or email notification. According to the severity of the anomaly detection report, trigger the corresponding response mechanism, such as sending an alarm to the operation and maintenance personnel, automatically adjusting the Flink task configuration, etc. Regularly evaluate the performance of the anomaly detection model, and update and optimize it according to the actual situation to improve the accuracy and efficiency of anomaly detection.

[0079] In one embodiment of the present application, the corresponding log translation information is output through the trained log translation model to implement Flink log management, which specifically includes:

[0080] Input the Flink logs to be processed into the trained log translation model, so that the Flink logs to be processed can be decoded by the trained log translation model.

[0081] According to the decoding result, the log type and problem category corresponding to the Flink log to be processed are identified, and the log translation information corresponding to the Flink log file to be processed is output.

[0082] In one embodiment, the unprocessed Flink log files generated in the system are captured in real time. These log files are preprocessed, including cleaning, standardization, and formatting, to ensure that they are consistent with the data format used when training the log translation model. The preprocessed unprocessed Flink log data is input into the trained log translation model. The log translation model is trained based on historical log data and can decode the log data into natural language descriptions.

[0083] The input Flink log data to be processed is decoded through the log translation model and converted into an easy-to-understand natural language description. Based on the decoding results, the log type (such as error log, warning log, etc.) and problem category (such as insufficient resources, configuration error, etc.) corresponding to the Flink log to be processed are identified. The decoded log data, the identified log type and the problem category are integrated into log translation information. The log translation information is output to a specified location, such as the monitoring system interface, log file or email notification, so that operation and maintenance personnel or developers can quickly locate and understand the log content.

[0084] According to the log type and problem category in the log translation information, the corresponding response mechanism is triggered. For example, for serious error logs, an alert can be sent to the operation and maintenance team; for configuration warning logs, the configuration can be automatically adjusted and relevant personnel can be notified. The performance of the log translation model is regularly evaluated, and it is updated and optimized according to actual conditions to improve the accuracy and efficiency of log decoding. At the same time, user feedback can also be collected to continuously adjust the model to meet actual needs.

[0085] In one embodiment of the present application, after outputting corresponding log translation information through the trained log translation model to implement Flink log management, the method further includes:

[0086] Based on Flink's business rules, standard operations are performed on the decoding results of the Flink logs to be processed, and analysis reports corresponding to anomaly detection reports and log translation information are generated to implement Flink log management.

[0087] Among them, standard operations include filtering operations, sorting operations, and searching operations.

[0088] In one embodiment, the unprocessed Flink log files generated in the system are captured in real time. These log files are preprocessed, including cleaning, standardization, and formatting, to ensure that they are consistent with the data format of the subsequent processing flow. The preprocessed unprocessed Flink log data is input into the trained log translation model. The log translation model decodes the log data into natural language descriptions for subsequent analysis and processing.

[0089] Based on Flink's business rules, standard operations are performed on the decoded log data. First, according to the business rules, key log entries such as error logs and warning logs are filtered out. Second, the logs are sorted according to log levels, occurrence times, and other dimensions to quickly locate problems. Then, a keyword search function is provided to allow users to quickly find specific log entries.

[0090] Based on the results of standard operations, an anomaly detection report and an analysis report corresponding to the log translation information are generated. The anomaly detection report contains a detailed description of the anomaly, the time of occurrence, possible causes, and recommended solutions. The analysis report corresponding to the log translation information provides an overview of the decoded log data, log type, problem category, and business rule execution results.

[0091] Output the analysis report to a specified location, such as the monitoring system interface, log file, or email notification, so that the operation and maintenance personnel or developers can view it. According to the information in the analysis report, trigger the corresponding response mechanism, such as sending an alarm, automatically adjusting the configuration, etc. Regularly evaluate and optimize the log translation model and business rules to improve the accuracy and efficiency of log management. At the same time, collect user feedback and continuously adjust and improve the log management system to meet actual needs.

[0092] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, this application embodiment also provides a Flink log management system based on a large model, whose organizational structure is as follows Figure 2 shown.

[0093] Figure 2 The following is a schematic diagram of the organizational structure of a Flink log management system based on a large model provided in an embodiment of the present application. Figure 2 As shown, the system includes:

[0094] The front-end display layer 201 is used to build a viewing interface through the front-end framework, display log analysis results, and provide a user interaction interface;

[0095] The backend service layer 202 is used to use the Spring Boot framework to receive frontend requests, call data processing and model inference services, and return corresponding results;

[0096] The data processing layer 203 is used to collect, clean and format historical log samples;

[0097] The model training and reasoning layer 204 is used to train the preprocessed log data and provide an API interface for backend services to call for real-time or batch reasoning;

[0098] The data storage layer 205 is used to use a database to store structured data corresponding to configuration parameters, and use HDFS to store log files and model files.

[0099] Figure 3 The internal structure diagram of a Flink log management device based on a large model provided in an embodiment of the present application. Figure 3 As shown, the device includes:

[0100] at least one processor;

[0101] and, a memory communicatively coupled to the at least one processor;

[0102] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to:

[0103] Obtain historical log samples through multiple data sources, and preprocess the historical log samples to obtain preprocessed log data;

[0104] Based on the preprocessed log data, the log translation model is trained and the anomaly detection model is trained;

[0105] Receive the Flink log files to be processed, output the corresponding anomaly detection report through the trained anomaly detection model, and output the corresponding log translation information through the trained log translation model to implement Flink log management.

[0106] The present application also provides a non-volatile computer storage medium storing computer executable instructions. When the computer executable instructions are executed, they can:

[0107] Obtain historical log samples through multiple data sources, and preprocess the historical log samples to obtain preprocessed log data;

[0108] Based on the preprocessed log data, the log translation model is trained and the anomaly detection model is trained;

[0109] Receive the Flink log files to be processed, output the corresponding anomaly detection report through the trained anomaly detection model, and output the corresponding log translation information through the trained log translation model to implement Flink log management.

[0110] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0111] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0113] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0114] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0117] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0118] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0119] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0120] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0121] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A Flink log management method based on a large model, characterized in that: The method comprises: Acquire historical log samples through multiple data sources, and preprocess the historical log samples to obtain preprocessed log data; According to the preprocessed log data, a log translation model is trained, and an anomaly detection model is trained; Receive the Flink log files to be processed, output the corresponding anomaly detection report through the trained anomaly detection model, and output the corresponding log translation information through the trained log translation model to implement Flink log management.

2. According to the Flink log management method based on a large model according to claim 1, it is characterized in that: Obtaining historical log samples from multiple data sources and preprocessing the historical log samples to obtain preprocessed log data, specifically including: Obtain historical log samples through the Flink log library or open source community, and perform data cleaning on the historical log samples to remove abnormal log entries in the historical log samples; the abnormal log entries are repeated, invalid or erroneous log entries; Extract key features from the cleaned historical log samples to obtain corresponding log data; the key features include at least keywords, error codes and system components.

3. According to a Flink log management method based on a large model according to claim 1, it is characterized in that: According to the preprocessed log data, the log translation model is trained, specifically including: Input the preprocessed log data into the fastText model to extract word vectors describing log features, and input the word vectors describing log features into the log translation model; The word vectors describing log features are trained through the log translation model, and the association between the word vectors and natural language is learned to obtain the trained log translation model.

4. According to a Flink log management method based on a large model according to claim 1, it is characterized in that: Training the anomaly detection model includes: Determine the time series corresponding to the preprocessed log data, and model the time dimension based on the log data with time series; Build the covariance matrix of the Flink run configuration and obtain the linear transformation corresponding to the memory usage of the Flink task to build an anomaly detection model for linear transformation.

5. According to a Flink log management method based on a large model according to claim 1, it is characterized in that: Receive the Flink log files to be processed and output the corresponding anomaly detection report through the trained anomaly detection model, including: Receive the Flink log files to be processed generated over time, and input the Flink logs to be processed into the trained anomaly detection model; The trained anomaly detection model is used to output an anomaly detection report corresponding to the Flink log file to be processed.

6. According to a Flink log management method based on a large model according to claim 1, it is characterized in that: Through the trained log translation model, the corresponding log translation information is output to implement Flink log management, including: Inputting the to-be-processed Flink log into the trained log translation model, so as to decode the to-be-processed Flink log through the trained log translation model; The log type and problem category corresponding to the Flink log to be processed are identified according to the decoding result, and the log translation information corresponding to the Flink log file to be processed is output.

7. A Flink log management method based on a large model according to claim 6, characterized in that: After outputting corresponding log translation information through the trained log translation model and implementing Flink log management, the method further includes: Based on Flink's business rules, standard operations are performed on the decoding results corresponding to the Flink logs to be processed, and analysis reports corresponding to the anomaly detection report and the log translation information are generated to implement Flink log management; The standard operations include filtering operations, sorting operations and searching operations.

8. A Flink log management system based on a large model, characterized in that: The system comprises: The front-end display layer is used to build a viewing interface through the front-end framework, display log analysis results, and provide a user interaction interface; The backend service layer uses the Spring Boot framework to receive frontend requests, call data processing and model inference services, and return corresponding results. The data processing layer is used to collect, clean, and format historical log samples; The model training and inference layer is used to train the preprocessed log data and provide an API interface for backend services to call for real-time or batch inference; The data storage layer is used to use the database to store structured data corresponding to configuration parameters, and use HDFS to store log files and model files.

9. A Flink log management device based on a large model, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a Flink log management method based on a large model as described in any one of claims 1 to 7.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed, a Flink log management method based on a large model as described in any one of claims 1 to 7 is implemented.

Images