Flume-based GC log acquisition method

Abstract

The invention provides a Flume-based GC log acquisition method. The Flume-based GC log acquisition method comprises the following steps: determining and configuring a server and a Java program which need to acquire GC logs as acquisition sources; detailed information acquired by Flume is set to realize comprehensive and efficient log acquisition; aiming at the characteristics of the multi-line records in the GC log, designing and realizing a multi-line log processing mechanism; configuring a file name segmentation rule, and adding information such as a host name and a node name into a log message; storing the processed GC log data in a storage pool, and pushing the processed GC log data to a data display area; a sealing processing mechanism is designed for different machine nodes and city information, and the accuracy and traceability of log data are ensured. The method has the beneficial effects that intelligent identification and processing of the multi-line GC logs are realized through accurate regular expression and rule matching, and the integrity and accuracy of data are ensured; and the combination of the buffer area and the message queue is used, so that the log processing efficiency and the expandability of the system are improved.

Description

Technical Field

[0001] The present invention belongs to the field of computers, and in particular relates to a GC log collection method based on Flume. Background Art

[0002] As business systems become increasingly sophisticated, more and more Java programs are deployed on servers. GC (garbage collection) logs are generated during the running of Java programs. Garbage collection is part of automatic memory management in Java. It is responsible for identifying and releasing objects that are no longer in use to free up memory for new objects. In the GC log, very important information is recorded, including the heap memory usage before and after garbage collection, such as total memory, used memory, remaining memory, etc. When the available memory usage is lower than a certain threshold, it actually indicates the availability of the program.

[0003] Therefore, a set of tools is urgently needed to collect and classify these GC log information in real time, record GC water level information, etc. Summary of the invention

[0004] In view of this, the present invention aims to propose a GC log collection method based on Flume to at least solve one problem in the background technology.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] A GC log collection method based on Flume, including:

[0007] Determine and configure the server and Java program that need to collect GC logs as the collection source;

[0008] Set the detailed information of Flume collection to achieve comprehensive and efficient log collection;

[0009] Design and implement a multi-line log processing mechanism based on the multi-line record feature in GC logs;

[0010] Configure the file name segmentation rule to append information such as host name and node name to the log message;

[0011] Store the processed GC log data in the storage pool and push it to the data display area;

[0012] A sealing processing mechanism is designed for different machine nodes and city information to ensure the accuracy and traceability of log data.

[0013] Furthermore, in the process of determining and configuring the server and Java program that need to collect GC logs as collection sources, the location and access rights of each collection source are specified through the Flume plug-in to ensure stable acquisition of log data, specifically including:

[0014] Determine the collection target: List all Java programs and corresponding servers for which GC logs need to be collected;

[0015] Install Flume plug-in: Install and configure the Flume plug-in on each target server to ensure that it can access the GC log files;

[0016] Configure the collection path: In the Flume configuration file, specify the path of the GC log file to ensure that Flume can monitor and read the updates of the log file in real time.

[0017] Furthermore, the detailed information collected by Flume includes the collection time, collection method, and multi-program and multi-file collection strategies, specifically including:

[0018] Set the collection time: define the collection frequency and time interval of Flume to ensure the real-time and integrity of log data;

[0019] Select the collection method: select the appropriate collection method according to the needs to optimize resource usage;

[0020] Multi-program and multi-file collection: In the Flume configuration, add multiple collection sources.

[0021] Furthermore, the design and implementation of a multi-line log processing mechanism for the multi-line log feature in the GC log includes:

[0022] Identify multi-line features: Define the features of multi-line logs;

[0023] Continuously read logs: develop a custom processor for Flume to continuously monitor and read multiple lines of logs; encapsulate log messages: encapsulate multiple lines of logs into a complete message, put it in the message queue, and prepare for subsequent processing.

[0024] Furthermore, to facilitate subsequent log parsing, configure the file name segmentation rules to append information such as the host name and node name to the log message, including:

[0025] Define segmentation rules: According to the format of the log file name, formulate segmentation rules and extract key information such as host name and node name;

[0026] Modify Flume configuration: In the Flume processing flow, add segmentation rules to append the extracted information to each log message;

[0027] Verify segmentation results: Ensure that the host name and node name are correctly included in the log message to facilitate subsequent data parsing and analysis.

[0028] Furthermore, the processing of storing the GC log data in the storage pool and pushing it to the data display area specifically includes:

[0029] Configure storage pool: select appropriate storage media and configure the data storage path and method in Flume;

[0030] Data push: Set Flume's Sink to push the data in the storage pool to the data display area regularly or in real time;

[0031] Monitoring and maintenance: Establish a monitoring mechanism for data storage and push.

[0032] Furthermore, a sealing processing mechanism is designed for different machine nodes and city information to ensure the accuracy and traceability of log data, including:

[0033] Identify coverslip conditions: define trigger conditions for coverslip processing, such as when the log file reaches a certain size or time threshold;

[0034] Combined summary modification: dynamically adjust the film sealing strategy according to the machine node and city information;

[0035] Testing and Optimization: Testing of coverslip handling mechanisms to optimize their performance and accuracy.

[0036] Furthermore, the present solution discloses an electronic device, including a processor and a memory connected to the processor for storing instructions executable by the processor, wherein the processor is used to execute a GC log collection method based on Flume.

[0037] Furthermore, the present solution discloses a server, comprising at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes a GC log collection method based on Flume.

[0038] Furthermore, the present solution discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, a GC log collection method based on Flume is implemented.

[0039] Compared with the prior art, the GC log collection method based on Flume described in the present invention has the following beneficial effects:

[0040] (1) The GC log collection method based on Flume described in the present invention realizes intelligent recognition and processing of multi-line GC logs through precise regular expressions and rule matching, thereby ensuring the integrity and accuracy of the data; and improves the efficiency of log processing and the scalability of the system by using a combination of a buffer and a message queue;

[0041] (2) The Flume-based GC log collection method described in the present invention combines machine nodes and city information to design a flexible and efficient sealing strategy to improve the system's adaptability and data management capabilities; introduces dynamic adjustment and intelligent optimization mechanisms to ensure that the sealing operation can be performed efficiently and stably under different environments and loads; and measures such as log classification, archiving, compression and encryption after sealing improve the standardization and security of data management and support subsequent in-depth analysis and tracing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0043] Figure 1 The present invention is a schematic diagram of a GC log collection method based on Flume according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0045] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0046] A GC log collection method based on Flume, including:

[0047] Determine and configure the server and Java program that need to collect GC logs as the collection source;

[0048] Set the detailed information of Flume collection to achieve comprehensive and efficient log collection;

[0049] Design and implement a multi-line log processing mechanism based on the multi-line record feature in GC logs;

[0050] Configure the file name segmentation rule to append information such as host name and node name to the log message;

[0051] Store the processed GC log data in the storage pool and push it to the data display area;

[0052] A sealing processing mechanism is designed for different machine nodes and city information to ensure the accuracy and traceability of log data.

[0053] Furthermore, in the process of determining and configuring the server and Java program that need to collect GC logs as collection sources, the location and access rights of each collection source are specified through the Flume plug-in to ensure stable acquisition of log data, specifically including:

[0054] Determine the collection target: List all Java programs and corresponding servers for which GC logs need to be collected;

[0055] Install Flume plug-in: Install and configure the Flume plug-in on each target server to ensure that it can access the GC log files;

[0056] Configure the collection path: In the Flume configuration file, specify the path of the GC log file to ensure that Flume can monitor and read the updates of the log file in real time.

[0057] Furthermore, the detailed information collected by Flume includes the collection time, collection method, and multi-program and multi-file collection strategies, specifically including:

[0058] Set the collection time: define the collection frequency and time interval of Flume to ensure the real-time and integrity of log data;

[0059] Select the collection method: select the appropriate collection method according to the needs to optimize resource usage;

[0060] Multi-program and multi-file collection: In the Flume configuration, add multiple collection sources.

[0061] Furthermore, the design and implementation of a multi-line log processing mechanism for the multi-line log feature in the GC log includes:

[0062] Identify multi-line features: Define the features of multi-line logs;

[0063] Continuously read logs: Develop a custom processor for Flume to continuously monitor and read multiple lines of logs;

[0064] Encapsulate log messages: Encapsulate multiple lines of logs into a complete message and put it into the message queue for subsequent processing.

[0065] Specifically, when executing "Design and implement a multi-line log processing mechanism for the multi-line record feature in GC logs", GC logs usually contain multiple lines of information, and single-line processing may result in incomplete data or parsing errors. This step designs and implements a multi-line log processing mechanism to ensure the integrity and parsability of each GC log, including:

[0066] Identify multi-line log features:

[0067] Timestamp detection: Each GC log starts with a timestamp;

[0068] Continuous lines without timestamp: Subsequent lines do not start with a timestamp, indicating that they belong to the same log as the previous line;

[0069] Specific keyword ending: Certain keywords (such as [GC, [Full GC) mark the beginning of the log, and the specific ending marks the end of the log;

[0070] Continuous reading and buffering:

[0071] Buffer design: Set up a buffer to temporarily store the read log lines until the complete log record is confirmed;

[0072] Reading strategy: Read line by line: Use Flume's Source component to read log files line by line; Matching rule: Use regular expressions to match timestamps and keywords to determine the start and end of log lines.

[0073] Encapsulates a complete log message:

[0074] Message assembly: When a complete GC log (from the start keyword to the end keyword) is detected, all lines in the buffer are merged into a complete log message;

[0075] Formatting: Format the merged logs to ensure that they are structured for easy subsequent parsing and analysis;

[0076] Put into the message queue: Push the formatted complete log message to the Flume Channel and wait for subsequent processing.

[0077] Furthermore, to facilitate subsequent log parsing, configure the file name segmentation rules to append information such as the host name and node name to the log message, including:

[0078] Define segmentation rules: According to the format of the log file name, formulate segmentation rules and extract key information such as host name and node name;

[0079] Modify Flume configuration: In the Flume processing flow, add segmentation rules to append the extracted information to each log message;

[0080] Verify segmentation results: Ensure that the host name and node name are correctly included in the log message to facilitate subsequent data parsing and analysis.

[0081] Furthermore, the processing of storing the GC log data in the storage pool and pushing it to the data display area specifically includes:

[0082] Configure storage pool: select appropriate storage media and configure the data storage path and method in Flume;

[0083] Data push: Set Flume's Sink to push the data in the storage pool to the data display area regularly or in real time;

[0084] Monitoring and maintenance: Establish a monitoring mechanism for data storage and push.

[0085] Furthermore, a sealing processing mechanism is designed for different machine nodes and city information to ensure the accuracy and traceability of log data, including:

[0086] Identify coverslip conditions: define trigger conditions for coverslip processing, such as when the log file reaches a certain size or time threshold;

[0087] Combined summary modification: dynamically adjust the film sealing strategy according to the machine node and city information;

[0088] Testing and Optimization: Testing of coverslip handling mechanisms to optimize their performance and accuracy.

[0089] Specifically, during the log collection process, you may encounter situations where the log file reaches a certain size or time threshold and needs to be sealed. This step designs a sealing mechanism to ensure the accuracy and traceability of log data based on different machine nodes and city information, including:

[0090] Identify the sealing conditions:

[0091] File size threshold: Set the log file to be capped when it reaches a certain size (such as 100MB).

[0092] Time threshold: Set the log file writing time to trigger the sealing when it reaches a specific interval (such as midnight every day).

[0093] Specific event trigger: events such as system restart, abnormal termination, etc. trigger the sealing of the film.

[0094] Design of sealing strategy:

[0095] Based on node information: Customize different sealing strategies according to the performance and log generation rate of different machine nodes.

[0096] Based on city information: According to the different needs of cities, adjust the time and size thresholds of sealing to adapt to the business characteristics of different regions.

[0097] Combined summary modification:

[0098] Dynamically adjust the cover film parameters: Dynamically adjust the cover film size and time threshold according to the current machine node and city information.

[0099] Data archiving and classification: The sealed log files are classified and archived by nodes and cities to facilitate subsequent data analysis and tracing.

[0100] Implementing the sealing mechanism:

[0101] Flume configuration adjustment: Add sealing processing logic to the Source and Channel of Flume to ensure that the sealing operation is automatically performed when the sealing conditions are met.

[0102] Script automation: Write scripts or use Flume's custom components to automatically monitor and execute sealing tasks.

[0103] Tested and verified:

[0104] Simulate sealing scenarios: Simulate different sealing conditions in a test environment to verify the accuracy and reliability of the sealing mechanism.

[0105] Performance evaluation: Evaluate the impact of coverslipping operations on system performance to ensure stable operation under high load conditions.

[0106] Formula and algorithm:

[0107] Seal trigger formula:

[0108] Trigger condition = (file size ≥ threshold size) ∨ (time interval ≥ threshold time)

[0109] Dynamic adjustment formula:

[0110]

[0111] Furthermore, the present solution discloses an electronic device, including a processor and a memory connected to the processor for storing instructions executable by the processor, wherein the processor is used to execute a GC log collection method based on Flume.

[0112] Furthermore, the present solution discloses a server, comprising at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes a GC log collection method based on Flume.

[0113] Furthermore, the present solution discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, a GC log collection method based on Flume is implemented.

[0114] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0115] In the several embodiments provided in the present application, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the division of the units described above is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The above-mentioned units may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

[0117] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A GC log collection method based on Flume, characterized in that: include: Determine and configure the server and Java program that need to collect GC logs as the collection source; Set the detailed information of Flume collection to achieve comprehensive and efficient log collection; Design and implement a multi-line log processing mechanism based on the multi-line record feature in GC logs; Configure the file name segmentation rule to append information such as host name and node name to the log message; Store the processed GC log data in the storage pool and push it to the data display area; A sealing processing mechanism is designed for different machine nodes and city information to ensure the accuracy and traceability of log data.

2. A GC log collection method based on Flume according to claim 1, characterized in that: In the process of determining and configuring the server and Java program that need to collect GC logs as collection sources, the location and access rights of each collection source are specified through the Flume plug-in to ensure stable acquisition of log data, including: Determine the collection target: List all Java programs and corresponding servers for which GC logs need to be collected; Install Flume plug-in: Install and configure the Flume plug-in on each target server to ensure that it can access the GC log files; Configure the collection path: In the Flume configuration file, specify the path of the GC log file to ensure that Flume can monitor and read the updates of the log file in real time.

3. According to a Flume-based GC log collection method according to claim 1, it is characterized in that: The detailed information of Flume collection includes the collection time, collection method, and multi-program and multi-file collection strategy, specifically including: Set the collection time: define the collection frequency and time interval of Flume to ensure the real-time and integrity of log data; Select the collection method: select the appropriate collection method according to the needs to optimize resource usage; Multi-program and multi-file collection: In the Flume configuration, add multiple collection sources.

4. A GC log collection method based on Flume according to claim 1, characterized in that: Aiming at the characteristics of multi-line records in GC logs, the processing mechanism of multi-line logs is designed and implemented, including: Identify multi-line features: Define the features of multi-line logs; Continuously read logs: Develop a custom processor for Flume to continuously monitor and read multiple lines of logs; Encapsulate log messages: Encapsulate multiple lines of logs into a complete message and put it into the message queue for subsequent processing.

5. The GC log collection method based on Flume according to claim 1 is characterized in that: To facilitate subsequent log parsing, configure the file name segmentation rules to append information such as the host name and node name to the log message, including: Define segmentation rules: According to the format of the log file name, formulate segmentation rules and extract key information such as host name and node name; Modify Flume configuration: In the Flume processing flow, add segmentation rules to append the extracted information to each log message; Verify segmentation results: Ensure that the host name and node name are correctly included in the log message to facilitate subsequent data parsing and analysis.

6. A GC log collection method based on Flume according to claim 1, characterized in that: The processing of GC log data is stored in the storage pool and pushed to the data display area, specifically including: Configure storage pool: select appropriate storage media and configure the data storage path and method in Flume; Data push: Set Flume's Sink to push the data in the storage pool to the data display area regularly or in real time; Monitoring and maintenance: Establish a monitoring mechanism for data storage and push.

7. A GC log collection method based on Flume according to claim 1, characterized in that: The sealing processing mechanism is designed for different machine nodes and city information to ensure the accuracy and traceability of log data, including: Identify coverslip conditions: define trigger conditions for coverslip processing, such as when the log file reaches a certain size or time threshold; Combined summary modification: dynamically adjust the film sealing strategy according to the machine node and city information; Testing and Optimization: Testing of coverslip handling mechanisms to optimize their performance and accuracy.

8. An electronic device, comprising a processor and a memory connected to the processor for storing instructions executable by the processor, characterized in that: The processor is used to execute the Flume-based GC log collection method described in any one of claims 1 to 7.

9. A server, characterized in that: It includes at least one processor and a memory in communication with the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes a Flume-based GC log collection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the Flume-based GC log collection method according to any one of claims 1 to 7 is implemented.

Images