Log processing method, collector agent device and computer equipment
By loading user-defined preprocessing logic units in the log collection system, encoding and detection and conversion of log files is solved, and the existing system's low efficiency is solved when processing logs in different encoding formats is achieved, and efficient and flexible log collection and preprocessing processes are realized.
Patent Information
- Application Number
- CN202411778844.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-05
- Publication Date
- 2025-03-21
AI Technical Summary
Existing log acquisition systems are inefficient when processing log files in different encoding formats, and it is difficult to quickly integrate custom functions, resulting in complex log acquisition and preprocessing processes, affecting the accuracy and efficiency of analysis.
It provides a log processing method, which uses the built-in character encoding model library to determine the original encoding type, and preprocess the preprocessing logic unit to be loaded, and finally outputs the appropriate log file.
It significantly improves the acquisition efficiency and data quality of the log collector, enhances the scalability, stability and maintainability of the system, and can flexibly deal with a variety of complex log scenarios.
Smart Images
Figure CN119668991A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a log processing method, a collector agent device, a computer device, a computer-readable storage medium, and a computer program product. Background Art
[0002] In modern information systems, log files play a vital role as an important means of recording system operation status, tracking events and troubleshooting. With the rapid development of information technology, the IT infrastructure of enterprises and organizations is becoming increasingly complex, and the log data generated is of various types, originating from different systems, applications and devices. These log files often use different character encoding formats, including but not limited to ASCII, UTF-8, UTF-16, etc. Due to the inconsistency of encoding formats, problems such as garbled characters and data loss are often encountered during log collection, which seriously affects the accuracy and effectiveness of log analysis. At the same time, log data processing also has low processing efficiency and high processing complexity due to the wide variety of text types.
[0003] Traditional log collection systems usually rely on manual configuration file encoding or simple encoding conversion tools to process log files in different encoding formats, which is inefficient. In addition, traditional systems also face the problem of difficulty in quickly integrating custom functions. The log collection and preprocessing process often needs to be customized for different application scenarios. The low degree of automation and poor adaptability of traditional systems make it difficult for log collection systems to meet the needs of actual applications. Summary of the invention
[0004] Based on this, it is necessary to provide a log processing method, a collector agent device, a computer device, a computer-readable storage medium and a computer program product that can improve the efficiency of file encoding conversion in response to the above technical problems.
[0005] In a first aspect, the present application provides a log processing method, comprising:
[0006] After receiving a startup instruction for the collector agent device, loading at least one pre-processing logic unit defined by the user;
[0007] Performing encoding detection on the log file to be processed to obtain the original encoding type of the log file to be processed; performing encoding conversion on the log file to be processed based on the original encoding type to obtain a target log file of a target encoding type;
[0008] The target log files are preprocessed in sequence by at least one preprocessing logic unit to obtain log files to be output.
[0009] In one embodiment, encoding detection is performed on the log file to be processed to obtain the original encoding type of the log file to be processed, including:
[0010] Read the byte data of the target part of the log file to be processed, where the target part is the part from the beginning of the file content to the Nth byte in the log file;
[0011] Detect the byte order mark BOM in the log file to be processed and obtain the possible candidate encoding type of the log file to be processed;
[0012] Perform byte statistics on the byte data to obtain byte distribution graph data; read the model byte distribution data corresponding to the candidate encoding type from the built-in trained character encoding model library;
[0013] Based on the byte distribution graph data and the model byte distribution data, the original encoding type of the log file to be processed is determined.
[0014] In one embodiment, there are multiple candidate encoding types; determining the original encoding type of the log file to be processed based on the byte distribution map data and the model byte distribution data includes:
[0015] For each candidate encoding type, calculate the vector cosine between the byte distribution graph data and the model byte distribution data corresponding to the candidate encoding type, obtain the similarity corresponding to the candidate encoding type, and convert the similarity into a probability score;
[0016] The probability scores corresponding to each candidate encoding type are normalized to obtain the confidence of each candidate encoding type, and the candidate encoding type with the highest confidence is used as the original encoding type of the log file to be processed.
[0017] In one embodiment, before performing encoding detection on the log file to be processed, the method further includes:
[0018] Searching for at least one target file to be processed according to the file index path configured by the collector; the file index path includes at least one of an absolute file path or a regular file path;
[0019] For each target file, the validity of the target file is confirmed. If the confirmation result is valid, the target file is used as a file to be processed, and the handle of the file to be processed is obtained by interacting with the operating system;
[0020] Perform encoding detection on the log files to be processed, including:
[0021] Perform encoding detection on the log files to be processed based on the handle.
[0022] In one embodiment, loading at least one pre-processing logic unit defined by a user includes:
[0023] After initializing at least one preprocessing logic unit defined by the user, read configuration metadata from the configuration file; perform dependency analysis on the configuration metadata to obtain the running order of at least one preprocessing logic unit;
[0024] Preprocessing the target log file in sequence by at least one preprocessing logic unit includes:
[0025] Based on the target log file, the preprocessing logic units are started in sequence according to the running order to perform preprocessing.
[0026] In one embodiment, the method further comprises:
[0027] When a change in the configuration metadata is detected, the dependency relationship of the configuration metadata is re-parsed to obtain the latest running order; and the corresponding preprocessing logic unit is loaded and started based on the latest running order.
[0028] In a second aspect, the present application also provides a collector agent device, including:
[0029] An input module, used for receiving a startup instruction for a collector agent device;
[0030] A preprocessing logic dynamic loading module, used to load at least one preprocessing logic unit defined by a user;
[0031] The file encoding recognition module is used to perform encoding detection on the log file to be processed to obtain the original encoding type of the log file to be processed; based on the original encoding type, the log file to be processed is encoded and converted to obtain a target log file of a target encoding type;
[0032] The preprocessing logic dynamic loading module is also used to preprocess the target log file in sequence through at least one preprocessing logic unit to obtain a log file to be output;
[0033] The output module is used to output the log files to be output to the log collector.
[0034] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0035] An input module, used for receiving a startup instruction for a collector agent device;
[0036] A preprocessing logic dynamic loading module, used to load at least one preprocessing logic unit defined by a user;
[0037] The file encoding recognition module is used to perform encoding detection on the log file to be processed to obtain the original encoding type of the log file to be processed; based on the original encoding type, the log file to be processed is encoded and converted to obtain a target log file of a target encoding type;
[0038] The preprocessing logic dynamic loading module is also used to preprocess the target log file in sequence through at least one preprocessing logic unit to obtain a log file to be output;
[0039] The output module is used to output the log files to be output to the log collector.
[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:
[0041] An input module, used for receiving a startup instruction for a collector agent device;
[0042] A preprocessing logic dynamic loading module, used to load at least one preprocessing logic unit defined by a user;
[0043] The file encoding recognition module is used to perform encoding detection on the log file to be processed to obtain the original encoding type of the log file to be processed; based on the original encoding type, the log file to be processed is encoded and converted to obtain a target log file of a target encoding type;
[0044] The preprocessing logic dynamic loading module is also used to preprocess the target log file in sequence through at least one preprocessing logic unit to obtain a log file to be output;
[0045] The output module is used to output the log files to be output to the log collector.
[0046] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:
[0047] An input module, used for receiving a startup instruction for a collector agent device;
[0048] A preprocessing logic dynamic loading module, used to load at least one preprocessing logic unit defined by a user;
[0049] The file encoding recognition module is used to perform encoding detection on the log file to be processed to obtain the original encoding type of the log file to be processed; based on the original encoding type, the log file to be processed is encoded and converted to obtain a target log file of a target encoding type;
[0050] The preprocessing logic dynamic loading module is also used to preprocess the target log file in sequence through at least one preprocessing logic unit to obtain a log file to be output;
[0051] The output module is used to output the log files to be output to the log collector.
[0052] The above-mentioned log processing method, collector agent device, computer equipment, computer-readable storage medium and computer program product, after receiving the startup instruction for the collector agent device, load at least one preprocessing logic unit customized by the user; perform encoding detection on the log file to be processed to obtain the original encoding type of the log file to be processed; perform encoding conversion on the log file to be processed based on the original encoding type to obtain a target log file of the target encoding type; preprocess the target log file in turn through at least one preprocessing logic unit to obtain a log file to be output. The method can automatically process a variety of log file encodings and flexibly integrate a variety of preprocessing logic units, thereby significantly improving the collection efficiency and data quality of the collector, and can flexibly cope with a variety of complex log scenarios. The extensibility, stability and maintainability of the log file collector can be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0054] Figure 1 A flow chart of a log processing method in one embodiment;
[0055] Figure 2 A flow chart of a log processing method according to another embodiment;
[0056] Figure 3 A flow chart of a log processing method in another embodiment;
[0057] Figure 4 A schematic diagram of a flow chart of a method for loading and executing a preprocessing logic unit in one embodiment;
[0058] Figure 5 is a structural block diagram of a collector agent device in an embodiment;
[0059] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0061] In an exemplary embodiment, Figure 1 As shown, a log processing method is provided, which is applied to a collector agent device, and includes the following steps 102 to 106. Among them:
[0062] Step 102: After receiving a startup instruction for the collector agent device, load at least one pre-processing logic unit defined by the user.
[0063] The collector agent device may be a log file collector agent device. The collector may be a log file collector.
[0064] Optionally, the start instruction for the collector proxy device may be triggered by a user, or by an event, and the specific content of the event may be flexibly set according to actual conditions.
[0065] The user can customize the pre-processing logic unit in the collector agent device, and the collector agent device supports the customization of multiple pre-processing logic units.
[0066] After receiving the startup instruction for the collector agent device, the collector agent device may load at least one pre-processing logic unit defined by the user.
[0067] Step 104: Perform coding detection on the log file to be processed to obtain the original coding type of the log file to be processed; perform coding conversion on the log file to be processed based on the original coding type to obtain a target log file of a target coding type.
[0068] Optionally, the collector is usually configured with a file index path. The collector agent device can search for the target file based on the file index path, perform a touch action on the target file to ensure the validity of the target file, and when the target file is confirmed to be valid, treat the target file as a file to be processed.
[0069] Optionally, the collector agent device can obtain the handle of the file to be processed, read the byte data of the target part in the log file to be processed based on the handle, obtain the byte distribution map data of the log file to be processed based on the byte data, match the byte distribution map data with the model byte distribution data in the character encoding model library, and use the encoding type corresponding to the model byte distribution data with the highest matching degree as the original encoding type of the log file to be processed.
[0070] After determining the original encoding type of the log file to be processed, a code conversion algorithm can be determined according to the original encoding type and the target encoding type, and the code conversion algorithm is used to perform code conversion on the log file to be processed to obtain a target log file of the target encoding type.
[0071] Optionally, the target encoding type may be determined based on compatibility with the log file collector.
[0072] Optionally, the target log file of the target encoding type can be tagged, using a digest or hash value to generate a unique data identifier for quick comparison and verification. In addition, the above steps are all equipped with an exception capture and fallback mechanism to ensure the stability and reliability of the system, provide useful information when errors occur, and avoid resource consumption exceptions and process freezes.
[0073] Optionally, the collector agent device also supports user-defined encoding conversion methods to enhance the personalization of encoding conversion.
[0074] Step 106: Preprocess the target log files in sequence through at least one preprocessing logic unit to obtain log files to be output.
[0075] Among them, the user can customize the preprocessing logic unit in the collector agent device according to actual needs. The collector agent device supports customizing multiple preprocessing logic units. The collector agent device can generate configuration metadata based on the user's customized content and store the configuration metadata in the configuration file.
[0076] Optionally, after receiving a startup instruction for the collector agent device, the collector agent device first initializes at least one preprocessing logic unit defined by the user, then reads the configuration metadata from the configuration file, determines the running order of at least one preprocessing logic unit based on the configuration metadata, and after obtaining the target log file of the target encoding type, starts the preprocessing logic units in sequence according to the running order for preprocessing. After obtaining the log file to be output, the log file to be output is output to the log file collector.
[0077] In the above embodiment, after receiving the startup instruction for the collector agent device, at least one preprocessing logic unit customized by the user is loaded; the log file to be processed is subjected to encoding detection to obtain the original encoding type of the log file to be processed; the log file to be processed is subjected to encoding conversion based on the original encoding type to obtain the target log file of the target encoding type; the target log file is preprocessed in turn by at least one preprocessing logic unit to obtain the log file to be output. This method can automatically process a variety of log file encodings and flexibly integrate a variety of preprocessing logic units, thereby significantly improving the collection efficiency and data quality of the collector, and can flexibly cope with a variety of complex log scenarios. The scalability, stability, and maintainability of the log file collector can be greatly improved.
[0078] In some embodiments, encoding detection is performed on the log file to be processed to obtain the original encoding type of the log file to be processed, including: reading byte data of a target part of the log file to be processed, the target part being the part consisting of the beginning of the file content in the log file to the Nth byte; detecting the byte order mark BOM in the log file to be processed to obtain possible candidate encoding types of the log file to be processed; performing byte statistics on the byte data to obtain byte distribution graph data; reading model byte distribution data corresponding to the candidate encoding type from a built-in trained character encoding model library; and determining the original encoding type of the log file to be processed based on the byte distribution graph data and the model byte distribution data.
[0079] Among them, the byte data of the target part of the log file to be processed can be read based on the handle of the file to be processed, and the target part is the part from the beginning of the file content in the log file to the Nth byte; N can be flexibly set according to actual conditions, and for example, N can be 2000. This method of reading only part of the content for encoding detection does not occupy too many system resources, and improves resource utilization.
[0080] Among them, the byte order mark BOM in the log file to be processed can be detected to quickly determine the possible candidate encoding types of the log file to be processed.
[0081] The read byte data can be preprocessed, such as removing invisible characters or control characters to ensure the purity of the sample data. For each byte in the preprocessed data, the number of occurrences of the byte is counted, and the number of occurrences of each byte can be used as byte distribution graph data. The model byte distribution data corresponding to each candidate encoding type can be read from the built-in trained character encoding model library; based on the byte distribution graph data and the model byte distribution data, the original encoding type of the log file to be processed is determined.
[0082] In the above embodiment, the byte data of the target part of the log file to be processed is read, and the target part is the part consisting of the beginning of the file content in the log file to the Nth byte; the byte order mark BOM in the log file to be processed is detected to obtain the possible candidate encoding type of the log file to be processed; the byte data is counted by bytes to obtain byte distribution graph data; the model byte distribution data corresponding to the candidate encoding type is read from the built-in trained character encoding model library; based on the byte distribution graph data and the model byte distribution data, the original encoding type of the log file to be processed is determined. This method of only reading part of the content for encoding detection does not occupy too many system resources and improves resource utilization. Subsequently, the log file to be processed is converted based on the original encoding type to obtain a target log file of the target encoding type, which realizes automatic identification and conversion of log files of different encoding types and prevents garbled code and data loss.
[0083] In some embodiments, there are multiple candidate encoding types; based on the byte distribution map data and the model byte distribution data, the original encoding type of the log file to be processed is determined, including: for each candidate encoding type, calculating the vector cosine between the byte distribution map data and the model byte distribution data corresponding to the candidate encoding type, obtaining the similarity corresponding to the candidate encoding type, and converting the similarity into a probability score; normalizing the probability scores corresponding to each candidate encoding type to obtain the confidence of each candidate encoding type, and taking the candidate encoding type with the highest confidence as the original encoding type of the log file to be processed.
[0084] Among them, based on the BOM in the log file to be processed, there are multiple candidate encoding types that are quickly determined. The original encoding type of the log file to be processed can be determined in the following way: for each candidate encoding type, calculate the vector cosine between the byte distribution map data and the model byte distribution data corresponding to the candidate encoding type to obtain the similarity corresponding to the candidate encoding type, and convert the similarity into a probability score; normalize the probability scores corresponding to each candidate encoding type to ensure that the sum of the scores is 1. The probability scores corresponding to each candidate encoding type after normalization represent the confidence of each encoding type. The candidate encoding type with the highest confidence is used as the original encoding type of the log file to be processed.
[0085] In some embodiments, see Figure 2As shown, a log processing method is provided, firstly obtaining a log file to be processed, then reading the byte data of the target part of the log file to be processed, and preprocessing the read byte data, such as removing invisible characters or control characters, to ensure the purity of the sample data. Then load at least one preprocessing logic unit defined by the user; for each byte in the preprocessed data, count the number of occurrences of the byte, and the number of occurrences corresponding to each byte can be used as byte distribution graph data. Read the model byte distribution data corresponding to each candidate encoding type from the built-in trained character encoding model library; match the encoding type based on the byte distribution graph data and the model byte distribution data to obtain the original encoding type of the log file to be processed. After determining the original encoding type of the log file to be processed, the encoding conversion algorithm can be determined according to the original encoding type and the target encoding type, and the encoding conversion algorithm is used to convert the log file to be processed to obtain the target log file of the target encoding type. The target log file is preprocessed in turn by at least one preprocessing logic unit to obtain the log file to be output. The log file to be output can be marked, and a data unique identifier is generated using a summary or hash value for rapid comparison and verification. In addition, the above steps are all equipped with exception capture and fallback mechanisms to ensure the stability and reliability of the system, provide useful information when errors occur, and avoid resource consumption exceptions and process freezes. Output the log files to be output to the log file collector.
[0086] In the above embodiment, when there are multiple candidate encoding types, for each candidate encoding type, the vector cosine between the byte distribution map data and the model byte distribution data corresponding to the candidate encoding type is calculated to obtain the similarity corresponding to the candidate encoding type, and the similarity is converted into a probability score; the probability scores corresponding to each candidate encoding type are normalized to obtain the confidence of each candidate encoding type, and the candidate encoding type with the highest confidence is used as the original encoding type of the log file to be processed. The original encoding type determined in this way has a higher accuracy.
[0087] In some embodiments, before performing encoding detection on the log file to be processed, the method also includes: searching for at least one target file to be processed according to the file index path configured by the collector; the file index path includes at least one of the absolute file path or the regular file path; for each target file, confirming the validity of the target file, and when the confirmation result is valid, using the target file as the file to be processed, and obtaining the handle of the file to be processed by interacting with the operating system; performing encoding detection on the log file to be processed, including: performing encoding detection on the log file to be processed based on the handle.
[0088] Among them, it is necessary to perform file accessibility analysis on the log file before reading the log file. Usually, the file index path will be configured in the collector. The file index path may be an absolute file path or a regular file path. File accessibility analysis includes the following steps: according to the file index path, at least one target file to be processed is determined. For each target file, a touch action can be performed on the target file to ensure that the target file is authentic and valid. When the target file is confirmed to be valid, the target file is used as the file to be processed, and the handle of the file to be processed is obtained by interacting with the operating system to prepare for obtaining the file content. Subsequently, the file content can be obtained based on the handle, and encoding detection can be performed on the log file to be processed based on the obtained file content.
[0089] In some embodiments, see Figure 3 As shown, a log processing method is provided, which determines at least one target file to be processed according to the file index path, performs file accessibility analysis for each target file, and then preloads the file, reads the byte data of the target part in the log file to be processed, and then performs encoding detection in the manner provided by the above embodiment to obtain the original encoding type of the log file to be processed; performs encoding conversion on the log file to be processed based on the original encoding type to obtain a target log file of the target encoding type; and preprocesses the target log file in turn through at least one preprocessing logic unit to obtain a log file to be output. The above steps are all provided with an exception capture and a bottom-up mechanism to ensure the stability and reliability of the system, provide useful information when an error occurs, and avoid the occurrence of resource consumption exceptions and process stuck problems. The log file to be output is output to the log file collector.
[0090] In the above embodiment, before encoding detection is performed on the log file to be processed, at least one target file to be processed is searched according to the file index path configured by the collector; for each target file, the validity of the target file is confirmed, and when the confirmation result is valid, the target file is used as the file to be processed, and the handle of the file to be processed is obtained by interacting with the operating system; encoding detection is performed on the log file to be processed based on the handle; and subsequent operations are performed after confirming that the target file is authentic and valid, thereby avoiding waste of resources caused by encoding detection on invalid files.
[0091] In some embodiments, loading at least one preprocessing logic unit customized by the user includes: after initializing at least one preprocessing logic unit customized by the user, reading configuration metadata from a configuration file; performing dependency parsing on the configuration metadata to obtain an operation order of at least one preprocessing logic unit; and preprocessing a target log file in sequence through at least one preprocessing logic unit, including: based on the target log file, starting the preprocessing logic units in sequence in the operation order for preprocessing.
[0092] In the process of loading at least one user-defined preprocessing logic unit, first initialize at least one user-defined preprocessing logic unit, and then read configuration metadata from the configuration file; the configuration metadata includes the preprocessing logic unit directory, the metadata of the preprocessing logic unit, etc.; then, perform dependency analysis on the configuration metadata to obtain the running order of at least one preprocessing logic unit; the running order can ensure the orderly loading and initialization of the preprocessing logic. After obtaining the target log file of the target encoding type, use the preprocessing logic unit ranked first in the running order to preprocess the target log file to obtain the corresponding processing result; then use the preprocessing logic unit ranked second in the running order to continue preprocessing the processing result to obtain the corresponding processing result; and so on, until all preprocessing logic units are executed.
[0093] Optionally, the collector agent device may subscribe to relevant events via the event bus and execute corresponding pre-processing logic units as required.
[0094] In some embodiments, the log processing method provided by the embodiments of the present application also includes: when a change in the configuration metadata is detected, re-parsing the configuration metadata for dependencies to obtain the latest running order; and loading and starting the corresponding preprocessing logic unit based on the latest running order.
[0095] Among them, users can add or delete custom preprocessing logic units in the configuration metadata at any time. The collector agent device can detect the configuration metadata at a certain frequency. When it detects that the configuration metadata has changed, it re-parses the dependency relationship of the configuration metadata to obtain the latest running order; loads and starts the corresponding preprocessing logic unit according to the latest running order, so that the collector agent device can always perform preprocessing operations according to the user's most recent configuration content.
[0096] Optionally, the preprocessing logic unit can be executed by a process or thread, and the started process or thread can be marked, using hashing to generate a unique identifier for fast search and comparison verification.
[0097] Optionally, regarding the loading and execution process of the preprocessing logic unit, exception capture and backup can be performed to ensure stable operation of the system and leave traces when errors occur to facilitate subsequent investigation and tracing.
[0098] Optionally, the collector agent device can perform resource management and control on the pre-processing logic unit, and terminate the operation in time when the resource consumption of a pre-processing logic unit is abnormal, so as to ensure the stability of the dynamic loading mechanism.
[0099] Optionally, the preprocessing logic unit can be in the form of a plug-in. For example, a preprocessing logic plug-in is: automatically filter DEBUG level logs and desensitize the content (mask sensitive information such as IP passwords, etc.). The identification of the preprocessing logic plug-in is F1, and the F1 entry file is assumed to be "logproc_main.py". The steps of loading and running the plug-in include: a. Configuring and loading the plug-in; b. Starting the plug-in, subscribing to events through the event bus, and executing the plug-in; c. Exception management; After the above steps are completed, the collector agent device completes the loading and running of a custom preprocessing logic unit. Repeating the above steps can expand more custom preprocessing logic units.
[0100] See also Figure 4 As shown, a method for loading and executing a preprocessing logic unit is provided, comprising the following steps: after initializing at least one preprocessing logic unit customized by the user, reading configuration metadata from a configuration file; performing dependency analysis on the configuration metadata to obtain the running order of at least one preprocessing logic unit; and starting the preprocessing logic units in sequence according to the running order for preprocessing based on the target log file. When a change in the configuration metadata is detected during runtime, the configuration metadata is re-parsed for dependency to obtain the latest running order; and the corresponding preprocessing logic unit is loaded and started based on the latest running order. The preprocessing logic unit can be executed by a process or a thread, and the started process or thread can be marked, and a hash is used to generate a unique identifier for quick search and comparison verification. Regarding the loading and execution process of the preprocessing logic unit, exception capture and a safety net can be performed to ensure stable operation of the system and leave traces when errors occur, which is convenient for subsequent investigation and tracing.
[0101] In the above embodiment, when a change in the configuration metadata is detected, the configuration metadata is reparsed for dependency to obtain the latest running order, and the corresponding preprocessing logic unit is loaded and started based on the latest running order, thereby achieving dynamic expansion of the preprocessing logic unit.
[0102] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0103] Based on the same inventive concept, the embodiment of the present application also provides a collector agent device for implementing the log processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more collector agent device embodiments provided below can refer to the limitations on the log processing method above, and will not be repeated here.
[0104] In an exemplary embodiment, a collector agent device is provided, comprising: an input module, a file encoding identification module, a preprocessing logic dynamic loading module and an output module.
[0105] Wherein, the input module is used to receive a startup instruction for the collector agent device;
[0106] The preprocessing logic dynamic loading module is used to load at least one preprocessing logic unit defined by the user;
[0107] The file encoding recognition module is used to perform encoding detection on the log file to be processed to obtain the original encoding type of the log file to be processed; based on the original encoding type, the log file to be processed is encoded and converted to obtain a target log file of a target encoding type;
[0108] The preprocessing logic dynamic loading module is also used to preprocess the target log file in sequence through at least one preprocessing logic unit to obtain the log file to be output. The output module is used to output the log file to be output to the log collector.
[0109] In some embodiments, Figure 5 As shown, the collector agent device also includes an initialization module, the file encoding recognition module includes: a preprocessing module, a pattern matching module, a byte statistics module and a confidence calculation module, and the preprocessing logic dynamic loading module includes: a configuration management module, an event bus module, a dynamic loader module and a logic unit management module.
[0110] In some embodiments, the file encoding identification module is specifically used to read the byte data of the target part of the log file to be processed, where the target part is the part consisting of the beginning of the file content in the log file to the Nth byte; detect the byte order mark BOM in the log file to be processed to obtain possible candidate encoding types of the log file to be processed; perform byte statistics on the byte data to obtain byte distribution map data; read the model byte distribution data corresponding to the candidate encoding type from the built-in trained character encoding model library; and determine the original encoding type of the log file to be processed based on the byte distribution map data and the model byte distribution data.
[0111] In some embodiments, the file encoding identification module is specifically used to calculate the vector cosine between the byte distribution map data and the model byte distribution data corresponding to the candidate encoding type for each candidate encoding type, obtain the similarity corresponding to the candidate encoding type, and convert the similarity into a probability score; normalize the probability scores corresponding to each candidate encoding type to obtain the confidence of each candidate encoding type, and use the candidate encoding type with the highest confidence as the original encoding type of the log file to be processed.
[0112] In some embodiments, the file encoding identification module is specifically used to search for at least one target file to be processed according to the file index path configured by the collector; the file index path includes at least one of the absolute file path or the regular file path; for each target file, the validity of the target file is confirmed, and if the confirmation result is valid, the target file is used as the file to be processed, and the handle of the file to be processed is obtained by interacting with the operating system; encoding detection is performed on the log file to be processed based on the handle.
[0113] In some embodiments, the preprocessing logic dynamic loading module is specifically used to initialize at least one preprocessing logic unit customized by the user, and then read the configuration metadata from the configuration file; perform dependency analysis on the configuration metadata to obtain the running order of at least one preprocessing logic unit; and start the preprocessing logic units in sequence according to the running order for preprocessing based on the target log file.
[0114] In some embodiments, the preprocessing logic dynamic loading module is specifically used to re-parse the configuration metadata for dependencies when a change in the configuration metadata is detected to obtain the latest running order; and load and start the corresponding preprocessing logic unit based on the latest running order.
[0115] Each module in the above collector agent device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.
[0116] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store a preprocessing logic unit. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a log processing method is implemented.
[0117] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0118] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0119] After receiving a startup instruction for the collector agent device, loading at least one pre-processing logic unit defined by the user;
[0120] Performing encoding detection on the log file to be processed to obtain the original encoding type of the log file to be processed; performing encoding conversion on the log file to be processed based on the original encoding type to obtain a target log file of a target encoding type;
[0121] The target log files are preprocessed in sequence by at least one preprocessing logic unit to obtain log files to be output.
[0122] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0123] Read the byte data of the target part of the log file to be processed, where the target part is the part from the beginning of the file content to the Nth byte in the log file;
[0124] Detect the byte order mark BOM in the log file to be processed and obtain the possible candidate encoding type of the log file to be processed;
[0125] Perform byte statistics on the byte data to obtain byte distribution graph data; read the model byte distribution data corresponding to the candidate encoding type from the built-in trained character encoding model library;
[0126] Based on the byte distribution graph data and the model byte distribution data, the original encoding type of the log file to be processed is determined.
[0127] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0128] For each candidate encoding type, calculate the vector cosine between the byte distribution graph data and the model byte distribution data corresponding to the candidate encoding type, obtain the similarity corresponding to the candidate encoding type, and convert the similarity into a probability score;
[0129] The probability scores corresponding to each candidate encoding type are normalized to obtain the confidence of each candidate encoding type, and the candidate encoding type with the highest confidence is used as the original encoding type of the log file to be processed.
[0130] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0131] Searching for at least one target file to be processed according to the file index path configured by the collector; the file index path includes at least one of an absolute file path or a regular file path;
[0132] For each target file, the validity of the target file is confirmed. If the confirmation result is valid, the target file is used as a file to be processed, and the handle of the file to be processed is obtained by interacting with the operating system;
[0133] Perform encoding detection on the log files to be processed based on the handle.
[0134] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0135] After initializing at least one preprocessing logic unit defined by the user, read configuration metadata from the configuration file; perform dependency analysis on the configuration metadata to obtain the running order of at least one preprocessing logic unit;
[0136] Based on the target log file, the preprocessing logic units are started in sequence according to the running order to perform preprocessing.
[0137] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0138] When a change in the configuration metadata is detected, the dependency relationship of the configuration metadata is re-parsed to obtain the latest running order; and the corresponding preprocessing logic unit is loaded and started based on the latest running order.
[0139] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0140] After receiving a startup instruction for the collector agent device, loading at least one pre-processing logic unit defined by the user;
[0141] Performing encoding detection on the log file to be processed to obtain the original encoding type of the log file to be processed; performing encoding conversion on the log file to be processed based on the original encoding type to obtain a target log file of a target encoding type;
[0142] The target log files are preprocessed in sequence by at least one preprocessing logic unit to obtain log files to be output.
[0143] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0144] Read the byte data of the target part of the log file to be processed, where the target part is the part from the beginning of the file content to the Nth byte in the log file;
[0145] Detect the byte order mark BOM in the log file to be processed and obtain the possible candidate encoding type of the log file to be processed;
[0146] Perform byte statistics on the byte data to obtain byte distribution graph data; read the model byte distribution data corresponding to the candidate encoding type from the built-in trained character encoding model library;
[0147] Based on the byte distribution graph data and the model byte distribution data, the original encoding type of the log file to be processed is determined.
[0148] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0149] Read the byte data of the target part of the log file to be processed, where the target part is the part from the beginning of the file content to the Nth byte in the log file;
[0150] Detect the byte order mark BOM in the log file to be processed and obtain the possible candidate encoding type of the log file to be processed;
[0151] Perform byte statistics on the byte data to obtain byte distribution graph data; read the model byte distribution data corresponding to the candidate encoding type from the built-in trained character encoding model library;
[0152] Based on the byte distribution graph data and the model byte distribution data, the original encoding type of the log file to be processed is determined.
[0153] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0154] Searching for at least one target file to be processed according to the file index path configured by the collector; the file index path includes at least one of an absolute file path or a regular file path;
[0155] For each target file, the validity of the target file is confirmed. If the confirmation result is valid, the target file is used as a file to be processed, and the handle of the file to be processed is obtained by interacting with the operating system;
[0156] Perform encoding detection on the log files to be processed based on the handle.
[0157] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0158] After initializing at least one preprocessing logic unit defined by the user, read configuration metadata from the configuration file; perform dependency analysis on the configuration metadata to obtain the running order of at least one preprocessing logic unit;
[0159] Based on the target log file, the preprocessing logic units are started in sequence according to the running order to perform preprocessing.
[0160] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0161] When a change in the configuration metadata is detected, the dependency relationship of the configuration metadata is re-parsed to obtain the latest running order; and the corresponding preprocessing logic unit is loaded and started based on the latest running order.
[0162] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0163] After receiving a startup instruction for the collector agent device, loading at least one pre-processing logic unit defined by the user;
[0164] Performing encoding detection on the log file to be processed to obtain the original encoding type of the log file to be processed; performing encoding conversion on the log file to be processed based on the original encoding type to obtain a target log file of a target encoding type;
[0165] The target log files are preprocessed in sequence by at least one preprocessing logic unit to obtain log files to be output.
[0166] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0167] Read the byte data of the target part of the log file to be processed, where the target part is the part from the beginning of the file content to the Nth byte in the log file;
[0168] Detect the byte order mark BOM in the log file to be processed and obtain the possible candidate encoding type of the log file to be processed;
[0169] Perform byte statistics on the byte data to obtain byte distribution graph data; read the model byte distribution data corresponding to the candidate encoding type from the built-in trained character encoding model library;
[0170] Based on the byte distribution graph data and the model byte distribution data, the original encoding type of the log file to be processed is determined.
[0171] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0172] Read the byte data of the target part of the log file to be processed, where the target part is the part from the beginning of the file content to the Nth byte in the log file;
[0173] Detect the byte order mark BOM in the log file to be processed and obtain the possible candidate encoding type of the log file to be processed;
[0174] Perform byte statistics on the byte data to obtain byte distribution graph data; read the model byte distribution data corresponding to the candidate encoding type from the built-in trained character encoding model library;
[0175] Based on the byte distribution graph data and the model byte distribution data, the original encoding type of the log file to be processed is determined.
[0176] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0177] Searching for at least one target file to be processed according to the file index path configured by the collector; the file index path includes at least one of an absolute file path or a regular file path;
[0178] For each target file, the validity of the target file is confirmed. If the confirmation result is valid, the target file is used as a file to be processed, and the handle of the file to be processed is obtained by interacting with the operating system;
[0179] Perform encoding detection on the log files to be processed based on the handle.
[0180] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0181] After initializing at least one preprocessing logic unit defined by the user, read configuration metadata from the configuration file; perform dependency analysis on the configuration metadata to obtain the running order of at least one preprocessing logic unit;
[0182] Based on the target log file, the preprocessing logic units are started in sequence according to the running order to perform preprocessing.
[0183] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0184] When a change in the configuration metadata is detected, the dependency relationship of the configuration metadata is re-parsed to obtain the latest running order; and the corresponding preprocessing logic unit is loaded and started based on the latest running order.
[0185] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0186] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0187] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A log processing method, characterized in that: The method comprises: After receiving a startup instruction for the collector agent device, loading at least one pre-processing logic unit defined by the user; Performing encoding detection on the log file to be processed to obtain the original encoding type of the log file to be processed; performing encoding conversion on the log file to be processed based on the original encoding type to obtain a target log file of a target encoding type; The target log files are preprocessed in sequence by the at least one preprocessing logic unit to obtain log files to be output.
2. The method according to claim 1, characterized in that The encoding detection is performed on the log file to be processed to obtain the original encoding type of the log file to be processed, including: Read the byte data of the target part of the log file to be processed, where the target part is the part from the beginning of the file content in the log file to the Nth byte; Detecting the byte order mark BOM in the log file to be processed, and obtaining possible candidate encoding types of the log file to be processed; Perform byte statistics on the byte data to obtain byte distribution graph data; read the model byte distribution data corresponding to the candidate encoding type from the built-in trained character encoding model library; Based on the byte distribution graph data and the model byte distribution data, an original encoding type of the log file to be processed is determined.
3. The method according to claim 2, characterized in that There are multiple candidate encoding types; determining the original encoding type of the log file to be processed based on the byte distribution map data and the model byte distribution data includes: For each candidate encoding type, calculating the vector cosine between the byte distribution graph data and the model byte distribution data corresponding to the candidate encoding type, obtaining the similarity corresponding to the candidate encoding type, and converting the similarity into a probability score; The probability scores corresponding to the candidate encoding types are normalized to obtain the confidence of each candidate encoding type, and the candidate encoding type with the highest confidence is used as the original encoding type of the log file to be processed.
4. The method according to claim 1, characterized in that: Before performing encoding detection on the log file to be processed, the method further includes: Searching for at least one target file to be processed according to a file index path configured by the collector; the file index path includes at least one of an absolute file path or a regular file path; For each target file, the validity of the target file is confirmed. If the confirmation result is valid, the target file is used as a file to be processed, and a handle of the file to be processed is obtained by interacting with the operating system; The encoding detection of the log file to be processed includes: Perform encoding detection on the log file to be processed based on the handle.
5. The method according to claim 1, characterized in that The loading of at least one preprocessing logic unit defined by a user comprises: After initializing at least one user-defined preprocessing logic unit, read configuration metadata from the configuration file; perform dependency analysis on the configuration metadata to obtain the running order of at least one preprocessing logic unit; The preprocessing of the target log files in sequence by the at least one preprocessing logic unit includes: Based on the target log file, the preprocessing logic units are started in sequence according to the running order to perform preprocessing.
6. The method according to claim 5, characterized in that The method further comprises: When a change in the configuration metadata is detected, the configuration metadata is reparsed for dependencies to obtain a latest running order; and a corresponding preprocessing logic unit is loaded and started based on the latest running order.
7. A collector agent device, characterized in that: The device comprises: An input module, used for receiving a startup instruction for a collector agent device; A preprocessing logic dynamic loading module, used to load at least one preprocessing logic unit defined by a user; A file encoding recognition module is used to perform encoding detection on the log file to be processed to obtain the original encoding type of the log file to be processed; based on the original encoding type, the log file to be processed is encoded and converted to obtain a target log file of a target encoding type; The preprocessing logic dynamic loading module is further used to preprocess the target log file in sequence through the at least one preprocessing logic unit to obtain a log file to be output; The output module is used to output the log files to be output to the log collector.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.