Logstash-based ORC format log management method and device, equipment and medium

Abstract

The invention provides a log management method and device based on an ORC format of Logstash, equipment and a medium. The method comprises the steps that a Logstash service is started, and an ORC conversion Ruby script and multi-pipeline configuration are loaded; real-time data receiving and Schema dynamic mapping are carried out; according to the method, the problems of long link, poor real-time performance, large resource consumption and the like of traditional log processing are fundamentally solved, the architecture is simplified, the delay is reduced, the resource utilization efficiency is improved, and the real-time data processing capability is enhanced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a log management method, apparatus, device, and medium based on Logstash and ORC format. Background Technology

[0002] Currently, mainstream technical solutions in the log processing field generally adopt a processing system centered on the traditional ELK technology stack (Elasticsearch, Logstash, Kibana), combined with the Apache Hadoop ecosystem or big data processing frameworks such as Apache Spark / Flink, to achieve end-to-end support from log collection and storage to analysis. Specifically, Logstash, as a lightweight data collection and processing engine, is responsible for real-time parsing and initial transformation of logs, but its output is usually limited to unstructured or semi-structured formats such as text or JSON, lacking native support for efficient columnar storage formats. On the other hand, ORC (Optimized Row Columnar) format, as a high-performance columnar storage format widely used in big data scenarios, usually requires the use of heavyweight data processing frameworks such as Hive and Spark, and a complex ETL process to perform secondary transformation on the text data already written to disk before it can be generated.

[0003] This technical architecture has revealed several significant problems in practical applications:

[0004] First, the system architecture exhibits a clear separation of technology stacks. Log processing and format conversion rely on the collaboration of multiple independent systems, which not only introduces additional component dependencies and interface complexity but also significantly increases system integration and maintenance costs.

[0005] Secondly, the data processing chain is lengthy. Log data needs to go through multiple stages: "real-time acquisition → intermediate text storage → batch ETL processing → ORC format conversion", resulting in high overall latency and making it difficult to meet the needs of real-time or near real-time analysis.

[0006] Finally, resource utilization is inefficient. Because both log processing clusters and big data processing clusters need to be deployed and maintained simultaneously, the overall system resource consumption is high, and there is redundant storage and computing overhead between the data processing stages.

[0007] Furthermore, existing solutions cannot directly generate ORC format during the log collection phase, resulting in a delay in the release of data value and limited real-time performance. At the same time, the industry has yet to find a direct technical solution that effectively integrates lightweight log processing tools with columnar storage formats, making such systems technically challenging. Implementation teams need to master multiple technology stacks, including Logstash, Hadoop, and Spark, further increasing the difficulty of implementation.

[0008] Therefore, under the current technological background, there is an urgent need for a log processing solution that can directly support efficient columnar storage format during the data acquisition stage, in order to simplify the architecture, reduce latency, improve resource utilization efficiency, and enhance real-time data processing capabilities. Summary of the Invention

[0009] The technical problem to be solved by this invention is to provide a log management method, device, equipment and medium based on Logstash and ORC format, which fundamentally improves the problems of long traditional log processing links, poor real-time performance and high resource consumption, simplifies the architecture, reduces latency, improves resource utilization efficiency and enhances real-time data processing capabilities.

[0010] In a first aspect, the present invention provides a log management method based on Logstash in ORC format, characterized by comprising the following steps:

[0011] Step 1: Start the Logstash service and load the ORC conversion Ruby script and the multi-pipeline configuration. The multi-pipeline configuration is used for ORC conversion and file upload. The ORC conversion Ruby script embeds the Java ORC core library and adopts a Ruby-Java hybrid programming mode.

[0012] Step 2: Receive raw log data in JSON format in real time through the Logstash input plugin, parse the raw log data using a JSON filter, extract fields and verify the data structure, and map the JSON fields to the ORC Schema.

[0013] Step 3: Directly integrate and call the ORC Writer API in the Logstash data processing pipeline to perform real-time conversion of log data to ORC format, complete the mapping from JSON data type to ORC data type, and generate ORC format log files;

[0014] Step 4: Monitor the completed ORC format log files in real time, trigger the upload task, and upload the ORC files to the designated server.

[0015] Secondly, the present invention provides a log management device based on Logstash in ORC format, comprising:

[0016] The initialization module starts the Logstash service and loads the ORC conversion Ruby script and the multi-pipeline configuration, which is used for ORC conversion and file upload. The ORC conversion Ruby script embeds a Java ORC core library and adopts a Ruby-Java hybrid programming mode.

[0017] The dynamic mapping module receives raw log data in JSON format in real time through the Logstash input plugin, parses the raw log data using a JSON filter, extracts fields and verifies the data structure, and maps JSON fields to ORCSchema.

[0018] The ORC format conversion module is directly integrated into the Logstash data processing pipeline and calls the ORC Writer API to convert log data to ORC format in real time, complete the mapping from JSON data type to ORC data type, and generate ORC format log files.

[0019] The file upload module monitors completed ORC format log files in real time, triggers upload tasks, and uploads the ORC files to the designated server.

[0020] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0021] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0022] One or more technical solutions provided by this invention have at least the following technical effects or advantages:

[0023] This invention realizes the integrated processing mode of "Logstash+ORC", which effectively solves the problems of complex traditional log processing process, low real-time performance and insufficient storage efficiency, and brings significant improvements in technology, performance and business.

[0024] Technically, this invention achieves direct integration of Logstash and the ORC format, breaking down the long-standing technical barrier between lightweight log processing tools and efficient columnar storage formats. Through a Ruby-Java hybrid programming model and a dual-pipeline concurrent processing mechanism, it not only significantly simplifies the processing architecture and reduces system complexity by over 70%, but also achieves millisecond-level real-time ORC format conversion, reducing latency by more than 10 times compared to traditional batch processing methods.

[0025] In terms of performance improvement, this invention fully leverages the advantages of ORC columnar storage, saving 60%–80% of storage space compared to text format and improving query performance by 5–10 times. The dual-pipeline concurrency mechanism brings a 2–3 times improvement in data processing efficiency, while increasing resource utilization by more than 50%, achieving a dual optimization of high efficiency and low resource consumption.

[0026] In terms of commercial value, this invention significantly lowers the technical barrier, allowing ordinary developers to implement it without needing to master multiple big data technology stacks, reducing deployment costs by over 60%. The system boasts a high degree of automation, saving 40%–50% on operation and maintenance costs, and its unique technology combination creates a clear competitive advantage in the market, possessing broad industry application prospects.

[0027] This invention not only fundamentally improves the problems of long processing chains, poor real-time performance, and high resource consumption in traditional log processing, but also provides a new solution for the fields of log processing and big data storage with an innovative technical architecture, which has both technological breakthrough and practical application value.

[0028] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0030] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention;

[0031] Figure 2 This is a schematic diagram of the device in Embodiment 2 of the present invention. Detailed Implementation

[0032] This application provides a log management method, apparatus, device, and medium based on Logstash and the ORC format.

[0033] The overall concept of the technical solution in this application is as follows:

[0034] This embodiment provides an ORC format converter and cloud storage upload system based on Logstash, which innovatively realizes a direct conversion mode of "Logstash + ORC", including the following main steps:

[0035] Step 1: Innovative Architecture Initialization: Start the Logstash service and load the ORC conversion Ruby script. This script embeds a Java ORC library, realizing an innovative Ruby-Java hybrid programming mode, effectively breaking through the limitation of traditional Logstash only supporting text format output;

[0036] Step 2, Real-time Data Reception and Schema Mapping: The Logstash input plugin receives raw log data and innovatively implements a dynamic mapping mechanism from JSON to ORC Schema, automatically identifying the data structure and generating the corresponding ORC column definitions;

[0037] Step 3: Direct ORC format conversion:

[0038] The ORC Writer API is directly called in the Logstash data processing pipeline to achieve real-time conversion. An innovative batch processing buffer mechanism is used to balance memory usage and write efficiency.

[0039] Implement intelligent type inference, automatically mapping JSON data types to data types supported by ORC;

[0040] Integrated file rotation logic, supporting multi-dimensional rotation strategies based on size, time, and number of events;

[0041] Step 4: Concurrent and secure file management:

[0042] An innovative multi-level locking mechanism: locks are created to prevent multi-process conflicts, and round-robin locks ensure file rotation safety;

[0043] Atomicity operations: ensure the integrity of the file generation process and avoid generating incomplete ORC files;

[0044] Directory management: Automatically manages the working directory and completion directory, achieving a clear division of file status;

[0045] Step 5: Smart Cloud Storage Upload:

[0046] File monitoring: Real-time monitoring of ORC file generation; once the file is completed, the upload process is triggered.

[0047] Intelligent path construction: Automatically constructs OSS storage paths based on timestamps and business rules;

[0048] Multiple upload guarantees: Supports both Alibaba Cloud official SDK and REST API upload methods to ensure upload success rate;

[0049] Local file cleanup: After successful upload, you can choose to delete local files to save storage space;

[0050] Step 6: Comprehensive anomaly handling and monitoring:

[0051] Real-time status monitoring: Monitor key indicators such as conversion efficiency, file generation speed, and upload success rate;

[0052] Intelligent error recovery: Employs appropriate recovery strategies based on different error types;

[0053] Detailed logging: Records detailed information for each processing step, facilitating problem diagnosis and performance optimization.

[0054] Example 1

[0055] like Figure 1 As shown, this embodiment provides a log management method based on Logstash's ORC format, including the following steps:

[0056] Step 1: Start the Logstash service and load the ORC conversion Ruby script and the multi-pipeline configuration. The multi-pipeline configuration is used for ORC conversion and file upload. The ORC conversion Ruby script embeds the Java ORC core library and adopts a Ruby-Java hybrid programming mode.

[0057] Step 2: Receive raw log data in JSON format in real time through the Logstash input plugin, parse the raw log data using a JSON filter, extract fields and verify the data structure, and map the JSON fields to the ORC Schema.

[0058] Step 3: Directly integrate and call the ORC Writer API in the Logstash data processing pipeline to perform real-time conversion of log data to ORC format, complete the mapping from JSON data type to ORC data type, and generate ORC format log files;

[0059] Step 4: Monitor the completed ORC format log files in real time, trigger the upload task, and upload the ORC files to the designated server.

[0060] In this embodiment, preferably, step 3 specifically involves: directly integrating and calling the ORC Writer API in the Logstash data processing pipeline to perform real-time conversion of log data to ORC format, completing the mapping from JSON data type to ORC data type, and generating an ORC format log file;

[0061] Step 4 specifically involves: real-time monitoring of completed ORC format log files, triggering an upload task to upload the ORC files to the designated server; constructing an object storage path based on timestamps and business rules; integrating both SDK and REST API upload methods, and automatically cleaning up local files and releasing storage space after successful upload.

[0062] In this embodiment, preferably, a file rotation strategy is set to control the generation, segmentation and management of ORC files; specifically, when the file size exceeds the file threshold, the time interval exceeds the time threshold, the batch processing is abnormal or a set signal is triggered, the ORC file is closed.

[0063] It sets up creation locks and rotation locks, and sets a lock timeout mechanism. When the set time is reached, the lock is released. The creation lock is used to ensure that only one ORC file is generated at the same time. The rotation lock is used to ensure that only one ORC file is written to at the same time.

[0064] In this embodiment, preferably, it also includes step 5: real-time monitoring of key performance indicators, including file size, number of times, file generation speed, upload latency, failure type, upload success rate, and hardware resource utilization, and storing the key performance indicators.

[0065] Based on the same inventive concept, this application also provides an apparatus corresponding to the method in Embodiment 1, as detailed in Embodiment 2.

[0066] Example 2

[0067] like Figure 2 As shown, this embodiment provides a log management device based on Logstash and in ORC format, including:

[0068] The module initializes the Logstash service and loads the ORC conversion Ruby script and the multi-pipeline configuration, which is used for ORC conversion and file upload. The ORC conversion Ruby script embeds a Java ORC core library and adopts a Ruby-Java hybrid programming mode.

[0069] The dynamic mapping module receives raw log data in JSON format in real time through the Logstash input plugin, parses the raw log data using a JSON filter, extracts fields and verifies the data structure, and maps JSON fields to ORCSchema.

[0070] The ORC format conversion module is directly integrated into the Logstash data processing pipeline and calls the ORC Writer API to convert log data to ORC format in real time, complete the mapping from JSON data type to ORC data type, and generate ORC format log files.

[0071] The file upload module monitors completed ORC format log files in real time, triggers upload tasks, and uploads the ORC files to the designated server.

[0072] In this embodiment, preferably, the ORC format conversion module specifically integrates and calls the ORC Writer API directly in the Logstash data processing pipeline to perform real-time conversion of log data to ORC format, complete the mapping from JSON data type to ORC data type, and generate log files in ORC format.

[0073] The file upload module specifically monitors completed ORC format log files in real time, triggers an upload task, and uploads the ORC files to a designated server; constructs an object storage path based on timestamps and business rules; integrates both SDK and REST API upload methods; and automatically cleans up local files and releases storage space after successful upload.

[0074] In this embodiment, preferably, a file rotation strategy is set to control the generation, segmentation and management of ORC files; specifically, when the file size exceeds the file threshold, the time interval exceeds the time threshold, the batch processing is abnormal or a set signal is triggered, the ORC file is closed.

[0075] It sets up a creation lock and a rotation lock, and sets a lock timeout mechanism. When the set time is reached, the lock is released. The creation lock is used to ensure that only one ORC file is generated at the same time. The rotation lock is used to ensure that only one ORC file is written to at the same time.

[0076] In this embodiment, preferably, it also includes a monitoring and recording module to monitor key performance indicators in real time, including file size, number of times, file generation speed, upload latency, failure type, upload success rate, and hardware resource utilization, and to store the key performance indicators.

[0077] Since the apparatus described in Embodiment 2 of the present invention is an apparatus used to implement the method of Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in Embodiment 1 of the present invention, and therefore will not be described again here. All apparatuses used in the method of Embodiment 1 of the present invention fall within the scope of protection of the present invention.

[0078] Based on the same inventive concept, this application provides an electronic device embodiment corresponding to Embodiment 1, as detailed in Embodiment 3.

[0079] Example 3

[0080] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement any of the implementation methods in Embodiment 1.

[0081] Since the electronic device described in this embodiment is the device used to implement the method in Embodiment 1 of this application, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the method described in Embodiment 1 of this application. Therefore, how the electronic device implements the method in the embodiment of this application will not be described in detail here. Any device used by those skilled in the art to implement the method in the embodiment of this application falls within the scope of protection of this application.

[0082] Based on the same inventive concept, this application provides a storage medium corresponding to Embodiment 1, as detailed in Embodiment 4.

[0083] Example 4

[0084] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it can implement any of the implementation methods in Embodiment 1.

[0085] Example 5

[0086] This embodiment provides an ORC format conversion and cloud storage upload system based on Logstash, which specifically includes the following steps:

[0087] 1. System Architecture Initialization

[0088] 1.1 Start the Logstash service and load the multi-pipeline configuration (logstash.yml) for ORC conversion and OSS upload respectively.

[0089] 1.2 Configure environment variables (such as OSS_ACCESS_KEY_ID, ORC_SCHEMA, etc.) to avoid hardcoding sensitive information.

[0090] 1.3 Load Ruby scripts and embed Java ORC libraries through JRuby to achieve Ruby-Java hybrid programming.

[0091] 2. Data Reception and Parsing

[0092] 2.1 Receive raw log data in JSON format via Logstash input plugins (such as file, stdin, or kafka).

[0093] 2.2 Use a JSON filter to parse JSON data, extract fields, and validate the data structure.

[0094] 2.3 Dynamically map JSON fields to ORC Schema, supporting custom column names and types (such as struct).<time:string,message:string> ).

[0095] 3. ORC format conversion

[0096] 3.1 Call a Ruby script in the Logstash filter pipeline to convert JSON data to ORC format in real time via the ORC Writer API.

[0097] 3.2 Implement a batch processing mechanism (batch_size defaults to 1000) to balance memory usage and write efficiency.

[0098] 3.3 Smart Type Inference: Automatically maps JSON data types (such as strings, numbers, and booleans) to ORC-compatible types.

[0099] 3.4 File rotation strategy (triggered when any condition is met):

[0100] The file size exceeds max_file_size_mb (default 64MB).

[0101] The time interval exceeds rotation_interval_seconds (default 300 seconds).

[0102] Graceful shutdown triggered by batch processing exception or system signal (TERM / INT)

[0103] 3.5 Directory Management:

[0104] Working directory: / usr / share / logstash / output_orc / (stores ORC files being written)

[0105] Completed directories: Within the same directory, files are distinguished by their status (managed through a locking mechanism).

[0106] 4. Concurrency safety and file locking mechanism

[0107] 4.1 Use multi-level locks to ensure multi-process safety:

[0108] Create a lock: / usr / share / logstash / output_orc / .create_lock

[0109] Rotate lock: / usr / share / logstash / output_orc / .rotate_lock

[0110] Lock timeout mechanism: Automatically releases after 30 seconds to prevent deadlock.

[0111] 4.2 Atomicity operations: Ensure that no incomplete files are generated during file generation and rotation.

[0112] 5. OSS Upload Module

[0113] 5.1 File monitoring: Monitor the output directory ( / usr / share / logstash / output_orc / *.orc) using a Ruby filter script, and use start_position => "beginning" to ensure that no files are missed.

[0114] 5.2 Path Construction: Generate OSS storage path (e.g., orc / 2023 / 05 / 20 / 14 / logstash.orc) based on path_prefix and date_format (default %Y / %m / %d / %H).

[0115] 5.3 Upload methods: Supports both Alibaba Cloud official SDK and REST API to ensure high availability.

[0116] 5.4 Post-upload processing: Determine whether to delete local files based on delete_after_upload.

[0117] 6. Anomaly Handling and Monitoring

[0118] 6.1 Error Recovery: Retry or log failures for network anomalies, permission errors, format errors, etc.

[0119] 6.2 Log Recording:

[0120] Logstash runtime logs: / usr / share / logstash / logs / logstash-plain.log

[0121] OSS upload result log: / usr / share / logstash / logs / oss_result_YYYY-MM-DD.log

[0122] File completion log: / usr / share / logstash / logs / file_completed.log

[0123] 6.3 Monitoring Indicators:

[0124] ORC file generation rate, size, and number of events

[0125] Distribution of OSS upload success rate, latency, and failure types

[0126] System resource utilization (CPU, memory, disk I / O)

[0127] 7. Safety and Operations Recommendations

[0128] 7.1 Use RAM sub-accounts to authorize OSS access permissions, following the principle of least privilege.

[0129] 7.2 Rotate access keys regularly and enable OSS access log auditing.

[0130] 7.3 Deploy network isolation and data encryption (such as OSS server encryption).

[0131] 7.4 Regularly clean up temporary files and lock files to avoid accumulation.

[0132] 8. Troubleshooting Process

[0133] 8.1 Check Logstash logs and OSS upload logs to locate errors.

[0134] 8.2 Verify environment variables, configuration file format, and permissions.

[0135] 8.3 Use the ORC tool to check file integrity.

[0136] 8.4 Test network connectivity and OSS credential validity.

[0137] 8.5 Restart the service and clean up temporary files before re-verifying.

[0138] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0139] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0140] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0141] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0142] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A log management method based on an ORC format of Logstash, characterized by: Comprise the following steps: Step 1, start the Logstash service and load ORC conversion Ruby script and multi-pipeline configuration, the multi-pipeline configuration is used for ORC conversion and file upload; the ORC conversion Ruby script is embedded with Java ORC core library, and adopts a Ruby-Java hybrid programming mode; Step 2, real-time receiving JSON format original log data through the Logstash input plug-in, parsing the original log data using the JSON filter, extracting fields and verifying the data structure, and mapping the JSON field segment to the ORC Schema; Step 3, directly integrating and calling the ORC Writer API in the Logstash data processing pipeline, real-time converting log data into ORC format, completing the mapping of JSON data type to ORC data type, and generating an ORC format log file; Step 4, real-time monitoring the completed ORC format log file, triggering an upload task, and uploading the ORC file to a designated server.

2. The log management method based on the Logstash ORC format according to claim 1, characterized in that: The step 3 is specifically: directly integrating and calling the ORC Writer API in the Logstash data processing pipeline, real-time converting log data into ORC format, completing the mapping of JSON data type to ORC data type, and generating an ORC format log file; The step 4 is specifically: real-time monitoring the completed ORC format log file, triggering an upload task, and uploading the ORC file to a designated server; According to the timestamp and business rules, an object storage path is constructed; Integrating a set of SDK and RESTAPI dual upload methods, automatically cleaning local files after successful upload, and releasing storage space.

3. The log management method based on the Logstash ORC format according to claim 2, characterized in that: A file rotation strategy is set to control the generation, segmentation and management of ORC files; specifically: when the file size exceeds the file threshold, the time interval exceeds the time threshold, the batch processing is abnormal or a set signal is triggered, the ORC file is closed; And set the creation lock and the rotation lock, set the lock timeout mechanism, when the set time is reached, the lock is released; the creation lock is used to ensure that only one ORC file is generated at the same time; the rotation lock is used to ensure that only one ORC file is written at the same time.

4. The log management method based on the Logstash ORC format of claim 1, wherein: It also includes step 5, real-time monitoring key performance indicators, including file size, time quantity, file generation speed, upload delay time, failure type, upload success rate and hardware resource usage, and storing the key performance indicators. 5.A log management apparatus based on an ORC format of Logstash, characterized by: Comprise: An initialization module, starting the Logstash service and loading the ORC conversion Ruby script and the multi-pipeline configuration, the multi-pipeline configuration is used for ORC conversion and file upload; the ORC conversion Ruby script is embedded with Java ORC core library, and adopts a Ruby-Java hybrid programming mode; A dynamic mapping module, real-time receiving JSON format original log data through the Logstash input plug-in, parsing the original log data using the JSON filter, extracting fields and verifying the data structure, and mapping the JSON field segment to the ORC Schema; An ORC format conversion module is directly integrated and ORC Writer API is called in a Logstash data processing pipeline to convert log data into an ORC format in real time, complete mapping of a JSON data type to an ORC data type, and generate a log file in an ORC format. A file uploading module monitors an ORC format log file in real time, triggers an uploading task, and uploads the ORC file to a set server.

6. The log management apparatus based on Logstash and ORC format of claim 5, wherein: The ORC format conversion module is directly integrated and ORC Writer API is called in a Logstash data processing pipeline to convert log data into an ORC format in real time, complete mapping of a JSON data type to an ORC data type, and generate a log file in an ORC format. The file uploading module monitors an ORC format log file in real time, triggers an uploading task, and uploads the ORC file to a set server. An object storage path is constructed according to a timestamp and a business rule; A set SDK and a REST API dual uploading mode are integrated, and local files are automatically cleaned up after successful uploading to release storage space.

7. The log management apparatus based on Logstash and ORC format of claim 6, wherein: A file rotation strategy is set to control generation, segmentation, and management of an ORC file. When a file size exceeds a file threshold, a time interval exceeds a time threshold, batch processing is abnormal, or a set signal is triggered, an ORC file is closed. A creation lock and a rotation lock are set, and a lock timeout mechanism is set. When a set time is reached, the lock is released. The creation lock is used to ensure that only one ORC file is generated at the same time. The rotation lock is used to ensure that only one ORC file is written at the same time.

8. The log management apparatus based on Logstash ORC format of claim 5, wherein: A monitoring recording module monitors key performance indicators in real time, including a file size, a time quantity, a file generation speed, an uploading delay time, a failure type, an uploading success rate, and a hardware resource usage rate. The key performance indicators are stored.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method of any one of claims 1 to 4.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1 to 4.

Images