Log management method and device, storage medium and electronic equipment

Abstract

The invention provides a log management method and device, a storage medium and electronic equipment, and is applied to the technical field of financial data processing. According to the method, original log data streams are consumed in real time from a message queue, dynamic grouping is carried out according to log traffic of message themes, large-traffic themes are distributed to independent groups, and small-traffic themes are combined to a shared group; starting an independent compression task for each group, and dynamically adjusting the degree of parallelism and memory resource configuration of the tasks; using a time window to group logs of the same theme according to time, and compressing the logs into a compressed data packet in real time; according to the method and the system, the real-time compression and ordered archiving of the logs are realized through dynamic grouping and resource adjustment, and the efficient archiving of massive logs is realized while the real-time query performance is ensured.

Description

A log management method, apparatus, storage medium, and electronic device Technical Field

[0001] This invention relates to the field of financial data processing technology, and in particular to a log management method, apparatus, storage medium, and electronic device. Background Technology

[0002] ELK (ElasticSearch, Logstash, Kibana) is a complete log collection, analysis, and visualization solution widely adopted by most IT companies.

[0003] However, financial industry logs need to be stored long-term, typically exceeding one year, to meet the needs of customer service, auditing, and legal proceedings. But due to the high cost of Elasticsearch (ES) storage and the significant impact of massive log volumes on ES performance, currently only about 60TB of logs per day are stored in ES for 20 days, with the remainder needing to be archived to a Hadoop cluster. Existing archiving methods, such as ES SNAPSHOT file backups, require hundreds of servers, consuming substantial hardware resources.

[0004] Therefore, how to achieve efficient archiving of large-scale logs while ensuring real-time log query performance has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, the present invention provides a log management method, apparatus, storage medium, and electronic device that overcomes or at least partially solves the above problems, the technical solution of which is as follows:

[0006] A log management method, comprising:

[0007] Consume raw log data streams in real time from the message queue;

[0008] Based on the log traffic volume of each message topic, the topics in the original log data stream are dynamically grouped, with high-traffic topics assigned to independent groups and multiple low-traffic topics merged and assigned to a shared group to obtain topic grouping results;

[0009] An independent compression task is started for each group in the topic grouping results, and the parallelism and memory resource configuration of the compression task are dynamically adjusted according to the total log traffic of each group.

[0010] During the execution of any of the compression processing tasks according to the corresponding parallelism configuration and memory resource configuration, the original log data belonging to the same message topic are grouped by time using a time window, and the original log data after time grouping is compressed into compressed data packets in real time.

[0011] The compressed data packets obtained from each compression processing task are aggregated according to predefined file naming rules, and the aggregated compressed data packets are sequentially written into the distributed file storage system for archiving.

[0012] Optionally, the step of dynamically grouping the topics in the original log data stream according to the log traffic size of each message topic, assigning high-traffic topics to independent groups, and merging multiple low-traffic topics into a shared group to obtain topic grouping results includes:

[0013] Monitor real-time log traffic for each message topic;

[0014] The real-time log traffic is input into the traffic prediction model to obtain future log traffic data predicted by the traffic prediction model based on the real-time log traffic and according to the historical log traffic change pattern.

[0015] Based on the future log traffic data, message topics with traffic exceeding a preset threshold are divided into high-traffic topics, and these high-traffic topics are grouped separately. Message topics with traffic equal to or less than the preset threshold are divided into low-traffic topics, and these low-traffic topics are merged into a single group to obtain the topic grouping results.

[0016] Optionally, the step of initiating an independent compression processing task for each group in the topic grouping results, and dynamically adjusting the parallelism and memory resource configuration of the compression processing task according to the total log traffic of each group, includes:

[0017] The total log traffic of each group in the topic grouping results is obtained according to a preset period.

[0018] By using the changes in total log traffic and the pre-defined correspondence between traffic resources, the number of parallel execution units and memory allocation parameters required for the corresponding group are calculated;

[0019] Based on the number of parallel execution units, the parallelism configuration of the corresponding group is dynamically adjusted;

[0020] Based on the memory allocation parameters, the memory resource configuration of the corresponding group is dynamically adjusted.

[0021] Optionally, the step of compressing the time-grouped raw log data into compressed data packets in real time includes:

[0022] During the execution of the compression task, a memory buffer is created for the raw log data belonging to the same message topic within each time window;

[0023] Write the original log data into the corresponding memory buffer;

[0024] The raw log data in the memory buffer is compressed using a compression algorithm to generate compressed data blocks.

[0025] The compressed data block is encapsulated into a compressed data packet of a specified format.

[0026] Optionally, the step of aggregating the compressed data packets obtained from each compression processing task according to a predefined file naming rule includes:

[0027] Generate a filename containing the message topic name and window time range for each of the compressed data packets;

[0028] Multiple compressed data packets generated in parallel with the same filename are sent to the same file writer node for processing via a convergence function.

[0029] Optionally, the method further includes:

[0030] Receive a log recovery request, wherein the log recovery request includes a target message topic and a target time range;

[0031] Based on the target message topic and target time range, locate and read one or more corresponding compressed data packets from the distributed file storage system;

[0032] Decompress the read compressed data packet to recover the original log data;

[0033] The recovered raw log data stream is written to the specified real-time retrieval system.

[0034] Optionally, the method further includes:

[0035] Monitor the message queue cluster to detect in real time whether there are new message topics or changes in the number of partitions of existing topics. If so, execute the step of consuming the raw log data stream from the message queue in real time.

[0036] A log management device includes: a log data stream consumption unit, a topic grouping result acquisition unit, a task configuration dynamic adjustment unit, a data grouping compression unit, and a data aggregation and archiving unit;

[0037] The log data stream consumption unit is used to consume raw log data streams from the message queue in real time;

[0038] The topic grouping result obtaining unit is used to dynamically group the topics in the original log data stream according to the log traffic size of each message topic, assign high-traffic topics to independent groups, and merge multiple low-traffic topics into a shared group to obtain topic grouping results;

[0039] The task configuration dynamic adjustment unit is used to start an independent compression processing task for each group in the topic grouping result, and dynamically adjust the parallelism configuration and memory resource configuration of the compression processing task according to the total log traffic of each group.

[0040] The data grouping and compression unit is used to group the original log data belonging to the same message topic by time using a time window during the execution of any compression processing task according to the corresponding parallelism configuration and memory resource configuration, and compress the original log data after time grouping into compressed data packets in real time.

[0041] The data aggregation and archiving unit is used to aggregate the compressed data packets obtained from each compression processing task according to a predefined file naming rule, and sequentially write the aggregated compressed data packets into the distributed file storage system for archiving.

[0042] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the log management method described above.

[0043] An electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the log management method.

[0044] By employing the above technical solutions, this invention provides a log management method, apparatus, storage medium, and electronic device that consumes raw log data streams from a message queue in real time; dynamically groups topics in the raw log data stream according to the log traffic size of each message topic, allocating high-traffic topics to independent groups and merging multiple low-traffic topics into a shared group to obtain topic grouping results; initiating an independent compression processing task for each group in the topic grouping results, and dynamically adjusting the parallelism and memory resource configuration of the compression processing task according to the total log traffic of each group; during the execution of any compression processing task according to the corresponding parallelism and memory resource configuration, using a time window to group raw log data belonging to the same message topic by time, and compressing the time-grouped raw log data into compressed data packets in real time; aggregating the compressed data packets obtained from each compression processing task according to predefined file naming rules, and sequentially writing the aggregated compressed data packets into a distributed file storage system for archiving. This invention achieves efficient and resource-optimized large-scale archiving of massive logs while ensuring real-time log query performance. It consumes log data streams from message queues in real time, dynamically groups them, and starts independent compression tasks for topics with different traffic volumes. Combined with time windows, it performs real-time compression and orderly archiving.

[0045] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0046] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0047] Figure 1 shows a flowchart of one embodiment of the log management method provided by the present invention;

[0048] Figure 2 shows a flowchart of a specific implementation of step S110 in the log management method provided by the present invention;

[0049] Figure 3 shows a flowchart of a specific implementation of step S120 in the log management method provided by the present invention;

[0050] Figure 4 shows a schematic diagram of the log management device provided in an embodiment of the present invention;

[0051] Figure 5 shows a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0052] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0053] With the rapid development of information technology, log data plays an increasingly important role in enterprise information management. ELK (ElasticSearch, Logstash, Kibana), as a complete log collection, analysis, and visualization solution provided by the ElasticSearch open-source ecosystem, has been widely adopted by most IT companies and has become the industry standard solution for building log clouds. By adopting the ELK solution for log collection and visualization, unified log management and real-time querying are achieved.

[0054] However, application system logs in the financial industry have unique characteristics. Many logs need to be stored long-term, typically for a year or more, to meet the needs of customer service, auditing, supervision, and legal proceedings. Although the ELK solution performs excellently in log collection and retrieval, the high cost and performance limitations of Elasticsearch (ES) storage make it difficult to meet the long-term storage requirements of massive log volumes. Currently, financial institutions generate approximately 60TB of log data daily. Storing all of this data on an ES cluster would severely impact its performance. To ensure real-time log availability, the ES cluster only stores log data from the most recent 20 days; logs older than this period are periodically archived to a Hadoop cluster.

[0055] Current methods, using traditional backup techniques and archiving via Elasticsearch SNAPSHOT files, while enabling long-term storage, require hundreds of servers, significantly increasing hardware resources and operational costs. Therefore, achieving efficient and reliable archiving of large-scale logs while maintaining real-time log query performance has become a pressing technical challenge.

[0056] Based on this, this embodiment of the invention provides a log management method that consumes raw log data streams from a message queue in real time, dynamically groups logs based on the log traffic of each message topic, assigns high-traffic topics to independent groups, and merges low-traffic topics into shared groups; starts an independent compression task for each group, dynamically adjusts the parallelism and memory resource configuration of the task; uses time windows to group logs of the same topic by time, compresses them into compressed data packets in real time; and aggregates the compressed packets according to predefined naming rules, sequentially writes them to a distributed file storage system for archiving. Thus, through dynamic grouping and resource adjustment, real-time compression and orderly archiving of logs are achieved, ensuring real-time query performance while realizing efficient archiving of massive logs.

[0057] Figure 1 shows a flowchart of one embodiment of the log management method provided by this invention. The method may include:

[0058] S100: Consume raw log data streams in real time from the message queue.

[0059] In this context, a message queue refers to middleware that provides an asynchronous communication mechanism. In this embodiment of the invention, it refers to Kafka, which is used to receive, buffer, and distribute massive log data streams generated by different application systems; it serves as the source and transmission channel for log data.

[0060] The raw log data stream refers to a sequence of log data consumed continuously and in real time from the message queue, without any processing or compression. Each data entry may contain information such as log content, generation time, and the system or application identifier to which it belongs.

[0061] Specifically, in this embodiment of the invention, Flink tasks can be deployed on a Hadoop cluster to consume raw log data streams from a Kafka message queue in real time. The log data in the raw log data stream is divided into different topics at the system level, and the Flink task continuously receives log messages from Kafka, ensuring the real-time performance and integrity of the data.

[0062] As examples, embodiments of the present invention may first establish a connection with a Kafka cluster. Each independent Flink compression task acts as a consumer group, initiating consumption by subscribing to one or more specified topics. Flink's Kafka connector continuously pulls log data from the Kafka Broker and transforms it into a processable data stream within Flink, thereby consuming the raw log data stream in real time and streaming. This ensures that logs can be consumed quickly once they enter Kafka, avoiding data backlog.

[0063] S110. Based on the log traffic size of each message topic, dynamically group the topics in the original log data stream, assign high-traffic topics to independent groups, and merge multiple low-traffic topics into a shared group to obtain the topic grouping results.

[0064] In this context, a message topic refers to a logical channel or category used to classify messages within a message queue. In this embodiment of the invention, a message topic corresponds to a TOPIC, and each TOPIC corresponds to a specific business system or log type, serving as the basic unit for log archiving and recovery.

[0065] Log traffic volume refers to the amount of log data transmitted per unit of time through a specific message topic.

[0066] The topic grouping result refers to the logical grouping set formed after analysis and calculation based on the log traffic volume of each message topic. The topic grouping is based on the following criteria: high-traffic topics occupy a separate group, while multiple low-traffic topics are merged and shared into one group.

[0067] Specifically, embodiments of the present invention can continuously monitor and analyze the real-time log traffic of each message topic, predict its future traffic level through a traffic prediction model, and divide topics predicted to have high traffic into independent groups according to preset traffic threshold standards, while aggregating multiple topics predicted to have low traffic into a shared group, thereby constructing a dynamic topic grouping result that adapts to traffic fluctuations.

[0068] Furthermore, embodiments of the present invention can monitor the real-time log traffic volume of each TOPIC based on a pre-trained traffic model, and dynamically group the topics in the original log data stream. TOPICs with high traffic are grouped separately to ensure their exclusive use of resources; multiple TOPICs with lower traffic are merged into a shared group to save computing resources and achieve dynamic adjustment of topic grouping results.

[0069] As examples, embodiments of the present invention can continuously monitor the real-time data production rate (i.e., log traffic size) of all pending archived topics. The monitoring data is input into a pre-trained traffic prediction model, which combines historical "tidal characteristics" (e.g., high during the day, low at night) to predict the future traffic trend of each topic. Based on the prediction results and preset traffic thresholds (e.g., "high traffic threshold" and "low traffic threshold"), decisions are made: topics whose predicted traffic consistently exceeds the high traffic threshold are marked as "high traffic topics" and assigned to an independent consumer group; multiple topics whose predicted traffic consistently falls below the low traffic threshold are aggregated together, marked as a "low traffic topic set," and assigned to a shared consumer group. Finally, a dynamic "topic-group" mapping table is output, i.e., the topic grouping result.

[0070] S120. Start an independent compression processing task for each group in the topic grouping results, and dynamically adjust the parallelism configuration and memory resource configuration of the compression processing task according to the total log traffic of each group.

[0071] The compression task refers to a specific job instance running on a real-time computing framework. In this embodiment of the invention, it refers to a Flink job whose core function is to consume logs from a message queue and perform windowed grouping and data compression by topic and time in memory.

[0072] The parallelism configuration refers to the number of concurrent execution instances set for the compression processing task.

[0073] Among them, memory resource configuration refers to parameters such as the heap memory size allocated to the compression processing task execution instance (TaskManager).

[0074] Specifically, embodiments of the present invention can periodically collect and summarize the real-time total log traffic of each group, calculate the required concurrent computing scale and memory resource quota for each group based on a preset traffic and resource mapping relationship, and dynamically expand or shrink the parallelism of the compression processing task of the corresponding group and elastically allocate memory resources according to the calculation results.

[0075] Furthermore, embodiments of the present invention can launch independent Flink compression processing tasks for each topic group. Based on the total log traffic of the group, the parallelism and memory resource configuration of the task are dynamically adjusted to ensure efficient operation of the task under different traffic loads, achieving flexible resource allocation and balanced computing power.

[0076] As examples, embodiments of the present invention can create or configure an independent Flink job (i.e., compression task) for each independent group and shared group based on the topic grouping results. Each task loads a list of all topics within its corresponding group as a data source. During task execution, the scheduler continuously monitors the total inflow traffic to each group. Based on a preset "traffic-resource" mapping rule (e.g., allocating 1 degree of parallelism and 2GB of memory for every 100MB / s of traffic), the scheduler dynamically issues adjustment instructions to running tasks via the REST API of Flink or the cluster manager: increasing task parallelism and TaskManager heap memory configuration during traffic peaks; and reducing resources accordingly during traffic troughs, thereby achieving elastic scaling and peak-shifting of resources.

[0077] S130. During the execution of any compression processing task according to the corresponding parallelism configuration and memory resource configuration, the original log data belonging to the same message topic is grouped by time using a time window, and the original log data after time grouping is compressed into compressed data packets in real time.

[0078] In this context, a time window refers to a mechanism used in real-time computing to divide an unbounded data stream into finite data blocks for processing. In this embodiment of the invention, Flink's time window is used to group log data on the same topic according to a fixed time length, serving as a unit for subsequent memory compression.

[0079] Here, a compressed data packet refers to a data unit output by a compression processing task after compressing log data within a time window in memory. The compressed data packet provided in this embodiment can be a compressed file package generated using efficient compression algorithms such as bz2, and is the basic unit ultimately written to the storage system, significantly reducing storage usage.

[0080] Specifically, in this embodiment of the invention, a memory buffer can be established for each time window and topic in the compression processing task, the original log data belonging to that window and topic can be received in real time, the buffered data can be compressed in memory to form a compressed block, and the compressed block can be encapsulated into a compressed data packet that meets the archiving requirements.

[0081] Furthermore, in this embodiment of the invention, during the execution of each independent compression task, the Flink window mechanism can be used to aggregate and group the raw log data of the same message topic by time period. Then, the log data within the time window is compressed directly in memory to generate compressed data packets in bz2 format, achieving real-time and efficient data compression.

[0082] As examples, within each Flink compression task, the data stream is first partitioned by KeyBy according to the TOPIC field in the log, ensuring that data from the same TOPIC is sent to the same processing subtask. A rolling or sliding time window (e.g., 5 minutes) is defined for the data stream of each TOPIC. When a window is triggered, all raw log data belonging to the same TOPIC accumulated within that window is collected. The processing operator allocates a buffer in the TaskManager's heap memory, reads this batch of data, and immediately calls the bz2 compression algorithm library to compress it in memory, generating a complete bz2 format compressed data packet. This process is completed entirely in memory, avoiding intermediate disk writes and greatly improving processing efficiency.

[0083] S140. The compressed data packets obtained from each compression processing task are aggregated according to the predefined file naming rules, and the aggregated compressed data packets are sequentially written into the distributed file storage system for archiving.

[0084] Among them, the file naming rules refer to the specifications for defining the file names of the generated compressed data packets. They can include information such as the source message subject, window start time, and window end time to ensure the uniqueness and readability of the file names, which facilitates subsequent aggregation, storage, and retrieval.

[0085] In this context, a distributed file storage system refers to a software system capable of storing ultra-large-scale datasets across multiple physical servers. For example, the distributed file storage system provided in this embodiment of the invention can refer to the Hadoop Distributed File System (HDFS), which is used for persistent storage of compressed and archived log data packets, providing highly reliable and scalable storage services.

[0086] Specifically, embodiments of the present invention can generate filenames containing a topic and a time window identifier for compressed data packets, and direct the same-named data packets generated by different processing paths to a unified file writing node, thereby realizing the aggregation processing of compressed data packets according to naming rules.

[0087] Furthermore, embodiments of the present invention can use a window aggregation function to aggregate compressed data packets generated by multiple parallel tasks according to predefined file naming rules, thereby resolving the write conflict problem of files with the same name. The aggregated compressed packets are then written sequentially to a distributed file storage system, completing the ordered archiving and persistent storage of logs.

[0088] As examples, embodiments of the present invention can assign a unique and descriptive filename to each compressed bz2 data packet, typically following the naming convention: "{TOPIC name}_{window start time}_{window end time}.bz2". Since Flink tasks are executed in parallel, the same time window and the same TOPIC may generate multiple file fragments on different parallel instances. Utilizing Flink's window functions (such as reduce or process) and a custom partitioning strategy, all data packets with the same filename are aggregated into a designated downstream "file write slot". In a single write instance of this slot, these data packets are sorted according to their generation order and then serially written to a fixed directory path in HDFS (e.g., / archive-logs / {topic} / {date} / ) in append mode. This embodiment of the present invention, through a "parallel compression, serial writing" model, leverages the advantages of parallel computing while resolving file conflicts and performance degradation issues that may arise from parallel writing to HDFS, ultimately achieving ordered and stable archiving of compressed log packets on HDFS.

[0089] This invention provides a log management method, which includes: consuming raw log data streams from a message queue in real time; dynamically grouping topics in the raw log data stream according to the log traffic volume of each message topic, allocating high-traffic topics to independent groups, and merging multiple low-traffic topics into a shared group to obtain topic grouping results; starting an independent compression processing task for each group in the topic grouping results, and dynamically adjusting the parallelism and memory resource configuration of the compression processing task according to the total log traffic of each group; during the execution of any compression processing task according to the corresponding parallelism and memory resource configuration, using a time window to group the raw log data belonging to the same message topic by time, and compressing the time-grouped raw log data into compressed data packets in real time; aggregating the compressed data packets obtained from each compression processing task according to a predefined file naming rule, and sequentially writing the aggregated compressed data packets into a distributed file storage system for archiving. This invention, by consuming log data streams from a message queue in real time, dynamically grouping and starting independent compression tasks for topics with different traffic volumes, and combining time windows for real-time compression and orderly archiving, achieves efficient and resource-optimized large-scale archiving of massive logs while ensuring real-time log query performance.

[0090] Optionally, based on the method shown in Figure 1, and as shown in Figure 2, a flowchart illustrating a specific implementation of step S110 in the log management method provided by this embodiment of the invention, step S110 may specifically include:

[0091] S200 monitors real-time log traffic for each message topic.

[0092] Specifically, embodiments of the present invention can, in the Kafka consumer or Flink task, count the number of log messages or bytes per unit time for each message topic (TOPIC). Traffic data is collected through periodic polling or event-driven methods, and this data is aggregated to form real-time traffic metrics for each TOPIC at the current moment.

[0093] As examples, embodiments of this invention can, within a Flink task, for each Kafka message topic consumed, use the built-in Flink Metrics system or a custom traffic statistics calculator to statistically analyze and output the amount of log data flowing in within a unit time window in real time. Simultaneously, a corresponding intelligent operation and maintenance monitoring program periodically retrieves the partition consumption progress (Offset) of each topic in the cluster via the Kafka Admin API or other monitoring interfaces (such as JMX) and calculates the difference, thereby summarizing the actual consumption traffic of each topic within the monitoring period. This real-time traffic data is collected and sent to a central monitoring storage, providing real-time data input for subsequent traffic prediction and scheduling decisions.

[0094] S210. Input the real-time log traffic into the traffic prediction model to obtain the future log traffic data predicted by the traffic prediction model based on the real-time log traffic and the historical log traffic change pattern.

[0095] Specifically, in this embodiment of the invention, real-time traffic data for each TOPIC can be input into a pre-trained traffic prediction model. This traffic prediction model, based on historical traffic curves and current real-time data, uses time-series analysis, machine learning, and other methods to predict the log traffic trend of each TOPIC over a future period. The prediction results include the expected traffic for each TOPIC in future time windows, providing a basis for adaptive adjustments to topic grouping and resource scheduling.

[0096] As examples, embodiments of the present invention can periodically retrieve the latest real-time traffic data for each topic from monitoring storage via a traffic scheduler. This real-time data, along with features such as traffic data for the same period in the past and recent trend data extracted from a historical traffic database, are input into a pre-trained traffic prediction model. This traffic prediction model can be built based on time series analysis and can learn and memorize the inherent "tidal characteristics" (such as high during the day and low at night) and periodic patterns of log traffic. The model combines real-time traffic with historical patterns to perform inference calculations and output predicted values ​​for the log traffic of each topic within a future period (such as the next scheduling cycle or the next few hours), forming future log traffic data.

[0097] S220. Based on future log traffic data, message topics with traffic exceeding a preset threshold are divided into high-traffic topics, and high-traffic topics are grouped separately. Message topics with traffic equal to or less than the preset threshold are divided into low-traffic topics, and low-traffic topics are merged into a single group to obtain topic grouping results.

[0098] Specifically, this embodiment of the invention can set a traffic threshold based on future log traffic data, classify TOPICs with expected traffic exceeding the threshold into high-traffic topics, and process each high-traffic topic as a separate group; TOPICs with expected traffic below or equal to the threshold are classified into low-traffic topics, and multiple low-traffic topics are merged into one or more shared groups. The dynamically optimized TOPIC grouping results are output, providing a basis for subsequent parallelism adjustments and resource allocation, achieving efficient utilization and adaptive scheduling of computing resources.

[0099] As examples, embodiments of the present invention can read future log traffic data output by a traffic prediction model. By presetting a traffic threshold, for each message topic, its predicted future average or peak traffic is compared with this threshold: if the predicted traffic is greater than the threshold, the topic is marked as a "high-traffic topic," and each such high-traffic topic is assigned an independent consumption group to ensure it exclusively uses computing resources to avoid performance bottlenecks. Topics with predicted traffic less than or equal to the threshold are marked as "low-traffic topics." Based on topic-based business relevance or load balancing algorithms, multiple low-traffic topics are merged and allocated to a shared consumption group, which is consumed uniformly by a single Flink task instance to improve resource utilization. The output is a "topic grouping result" containing the mapping relationships of all topics and their respective groups.

[0100] This invention continuously monitors the real-time log traffic of each message topic and uses a traffic prediction model to predict future log traffic based on historical patterns. This allows for the scientific and accurate division of high-traffic and low-traffic topics, enabling dynamic grouping. Separating high-traffic topics ensures they receive sufficient computing resources, avoiding performance degradation due to resource contention. Conversely, merging multiple low-traffic topics into shared groups effectively improves resource utilization. This not only achieves reasonable allocation and load balancing of computing resources but also allows for flexible adjustment of task configurations based on the tidal characteristics of log traffic, significantly saving overall computing costs, improving system archiving efficiency and stability, and meeting the needs for efficient compression and archiving of massive logs.

[0101] Optionally, based on the method shown in Figure 1, and as illustrated in Figure 3, a flowchart of a specific implementation of step S120 in the log management method provided by this embodiment of the invention is shown. Step S120 may specifically include:

[0102] S300: Obtain the total log traffic of each group in the topic grouping results according to the preset period.

[0103] Specifically, in this embodiment of the invention, real-time log traffic data of all TOPICs in each topic group can be collected and statistically analyzed at preset intervals, and the total log traffic of the group in that period can be calculated.

[0104] As examples, embodiments of the present invention can utilize a periodic scheduler. At the start of each scheduling cycle, the currently active "topic grouping results" are first read to determine which specific Kafka message topics are included in each group. Subsequently, the real-time traffic data of these topics recorded in a central monitoring storage (such as a time-series database) for the most recent complete monitoring cycle is queried. For a group, the traffic data of all member topics within that group (only one topic for an exclusive group, multiple topics for a shared group) are summed to obtain the "total log traffic" metric representing the current or recent actual load of that group.

[0105] S310. Calculate the number of parallel execution units and memory allocation parameters required for the corresponding group by using the changes in total log traffic and the corresponding relationship of preset traffic resources.

[0106] Specifically, embodiments of the present invention can automatically calculate the required Flink task parallelism (i.e., the number of execution units) and memory usage parameters for each group based on the collected total log traffic change trend and a traffic-resource mapping model that reflects the pre-defined traffic-resource correspondence. This resource mapping model can comprehensively consider factors such as throughput, task performance, and fault tolerance requirements to ensure that resource configuration meets performance requirements while avoiding resource waste.

[0107] As examples, embodiments of the present invention compare the total log traffic of each group with the traffic of its previous historical period to identify whether the traffic is increasing, decreasing, or remaining stable. Simultaneously, a preset "traffic-resource" mapping model or configuration table is maintained. This model, based on performance stress testing and empirical data, defines the optimal traffic range that each "parallel execution unit" (i.e., a TaskManager instance or Slot of a Flink task) can efficiently handle under different traffic intervals to ensure processing timeliness and stability, as well as the recommended memory size for each execution unit. Based on the changing trend and current absolute value of the group's total traffic, calculations are performed by querying this model: dividing the total traffic by the optimal processing capacity of a single execution unit and rounding up to obtain the theoretically required number of parallel execution units for that group. Simultaneously, based on the total traffic scale and processing complexity, the recommended memory allocation parameters for each execution unit in that group are mapped from the model.

[0108] S320. Based on the number of parallel execution units, dynamically adjust the parallelism configuration of the corresponding group.

[0109] Specifically, embodiments of the present invention can call Flink's dynamic scaling interface to automatically adjust the parallelism of Flink tasks in the corresponding group based on the number of parallel execution units, increasing or decreasing the number of execution units to match the current traffic load demand, realize elastic resource allocation, and ensure the stable and efficient operation of compression tasks.

[0110] As examples, embodiments of the present invention can compare the number of parallel execution units with the actual parallelism of the corresponding consumer task (i.e., the Flink Job processing the group) in the Flink cluster. If there is a difference between the target number and the current value (e.g., increased traffic requiring scaling up, or decreased traffic requiring scaling down), an instruction to update the task parallelism is submitted to the Flink cluster by calling the Flink REST API or cluster management tools (such as Apache Flink's flink command-line tool or Kubernetes Operator). This instruction triggers the Flink Job to dynamically adjust the parallelism of specific operators (such as the data source Kafka Consumer and core processing operators) to increase or decrease the number of TaskManager Slots actually participating in data processing, thereby achieving elastic scaling of computing power and ensuring that resource supply matches the group's traffic load.

[0111] S330: Dynamically adjust the memory resource configuration of the corresponding group based on the memory allocation parameters.

[0112] Specifically, embodiments of the present invention can dynamically adjust the memory configuration of Flink tasks by combining memory allocation parameters, including JVM heap memory size and the management memory and network buffer used by Flink, to ensure that tasks adapt to traffic changes in memory resource allocation, reduce the risk of memory overflow, and improve processing efficiency and system stability.

[0113] As examples, embodiments of the present invention can compare the target memory allocation parameters with the actual memory configuration of the currently running task instances (TaskManagers) in the Flink cluster. If the target parameters do not match the current configuration (e.g., a larger heap memory is needed to handle larger window caching and compression operations), corresponding dynamic resource configuration actions are taken to apply the new memory parameters, thereby applying the new memory resource configuration to all Flink execution units processing that group, ensuring the stability and performance of task execution.

[0114] This invention, through periodic monitoring of the total log traffic of each group in the topic grouping results and combining this with a preset traffic-resource correspondence, can scientifically calculate the number of parallel execution units and memory allocation parameters required for each group, thereby dynamically adjusting the parallelism and memory configuration of the compression tasks for the corresponding group. This dynamic resource adjustment mechanism not only ensures a high degree of matching between computing resources and actual log traffic, avoiding resource waste and bottleneck risks, but also guarantees efficient and stable operation of group tasks with different traffic scales. It achieves elastic utilization and load balancing of computing resources, significantly improving overall compression processing efficiency and system stability, while reducing operational complexity and costs.

[0115] Optionally, based on one or more embodiments corresponding to Figure 1 above, in another optional embodiment provided by this invention, the original log data after time grouping is compressed into compressed data packets in real time, which may specifically include:

[0116] During the compression processing task, a memory buffer is created for the raw log data belonging to the same message topic within each time window; the raw log data is written to the corresponding memory buffer; the compression algorithm is applied to the raw log data in the memory buffer to compress it and generate compressed data blocks; the compressed data blocks are then encapsulated into compressed data packets of a specified format.

[0117] Specifically, in the execution of the compression processing task, this embodiment of the invention first dynamically creates an independent memory buffer for the raw log data belonging to the same message topic within each time window based on the Flink time window mechanism, ensuring that the data is aggregated in an orderly manner according to topic and time; the raw log data consumed in real time is written into the corresponding memory buffer according to its topic and time classification; when the window ends or the buffer reaches the preset capacity, the compression operation is triggered, and the efficient bz2 compression algorithm is used to compress the log data in the memory buffer in real time to generate compressed data blocks; the compressed data blocks are encapsulated into standardized compressed data packets according to the predefined data packet format.

[0118] As examples, embodiments of the present invention can allocate a dynamically expanding byte buffer as a memory buffer in the TaskManager's heap memory for each logical group jointly determined by the TOPIC and time window (e.g., 1 minute) during the execution of the Flink compression program. Before the window is triggered, each piece of raw log data consumed from Kafka belonging to the TOPIC and the current window is first serialized into a byte array and written into its corresponding buffer for accumulation. When the window triggering condition is met, the BZip2 compression algorithm is called to compress the raw byte data accumulated in the entire buffer, generating a compact compressed data block. The compressed data block is then encapsulated into a complete ".bz2" format compressed data packet that can be directly written to HDFS according to a predefined format (such as a packet header containing metadata such as TOPIC, window start time, and compression algorithm identifier).

[0119] This invention creates a dedicated memory buffer for the raw log data of the same message topic within each time window during the compression processing task, and writes the data to this buffer in real time. This efficiently utilizes memory resources, avoids frequent disk I / O, and enables rapid data aggregation and processing. Subsequently, an efficient compression algorithm is applied to compress the buffer data in real time, generating smaller compressed data blocks. This not only significantly reduces storage space requirements but also improves data transmission efficiency. Finally, the compressed data blocks are encapsulated into standardized compressed data packets, facilitating subsequent unified management and sequential writing to distributed storage. The entire process ensures the real-time performance, resource utilization, and storage efficiency of log data compression, effectively meeting the needs for low-cost archiving and rapid access to massive logs.

[0120] Optionally, based on one or more embodiments corresponding to Figure 1 above, in another optional embodiment provided by the present invention, the compressed data packets obtained from each compression processing task are aggregated according to a predefined file naming rule, which may specifically include:

[0121] Generate a filename containing the message topic name and window time range for each compressed data packet; send multiple compressed data packets generated in parallel with the same filename to the same file writer node for processing via a convergence function.

[0122] Specifically, in this embodiment of the invention, a unique and standardized filename can be dynamically generated based on the message topic (TOPIC) name corresponding to each compressed data packet and the start and end times of its Flink time window, ensuring that the filename accurately reflects the topic and time range of the data packet. For compressed data packets with the same name that may be generated in multiple parallel compression tasks, the Flink window aggregation function is used to aggregate these data packets with the same filename into a specified file and write them to a slot, centrally manage and process write requests sequentially, and avoid concurrent write conflicts and data overwrite issues. The aggregated compressed data packets are sequentially written to the distributed storage HDFS by a unified file write node, achieving efficient, stable and orderly log archiving storage.

[0123] As examples, embodiments of the present invention can generate a unique filename after the compressed data packet is generated, based on the embedded Kafka message topic name and the start and end timestamps of its corresponding time window, according to a preset naming rule (such as "{TOPIC_NAME}{START_TIMESTAMP}{END_TIMESTAMP}.bz2"). When the data flows downstream to write to HDFS, a custom KeyedProcessFunction or Flink's KeyedStream combined with ProcessFunction is used as an aggregation function to key-by partition all the compressed data packets generated by the parallel subtasks according to their filenames, ensuring that all data packets with the same filename are routed to the same downstream "file write node" (i.e., the same TaskManager Slot). On this node, through a single-threaded writer or locking mechanism, these data packets with the same name are received and processed in an orderly manner, and finally merged serially and written to the same final file in HDFS, thereby completely resolving the parallel write conflict problem.

[0124] This invention generates a unique filename containing the message topic name and window time range for each compressed data packet, which clearly distinguishes log data of different topics and time periods, ensuring the accuracy and traceability of archived information. At the same time, by using an aggregation function to send multiple compressed data packets with the same filename generated in parallel to the same file writing node, it avoids concurrent file writing conflicts and data overwrite problems, and realizes the orderly aggregation and unified management of compressed data packets. This not only improves the writing efficiency and stability of the distributed file system, but also ensures the integrity and consistency of archived data, thus effectively supporting the efficient archiving and subsequent rapid retrieval of massive log data.

[0125] Optionally, based on one or more embodiments corresponding to Figure 1 above, another optional embodiment provided by the present invention may further include:

[0126] Receive a log recovery request, which includes a target message topic and a target time range; locate and read one or more corresponding compressed data packets from the distributed file storage system based on the target message topic and target time range; decompress the read compressed data packets to recover the original log data; and write the recovered original log data stream to the specified real-time retrieval system.

[0127] Specifically, embodiments of the present invention can receive log recovery requests containing a target message topic and a target time range; based on the log recovery request, the corresponding compressed data packet filename rules in the distributed file storage system are queried to quickly locate and batch read the relevant bz2 compressed data packets written to disk. The read compressed data packets are decompressed in real time using a supporting Flink decompression task to recover the complete original log data stream. The recovered original log data stream is then written in real time to a specified real-time retrieval system (such as Elasticsearch) through an efficient data writing interface, enabling rapid recovery and querying of archived logs and providing users with timely and accurate log access capabilities.

[0128] As examples, after the accompanying Flink decompression program is started, this embodiment of the invention receives a log recovery request containing the target message topic and target time range. After parsing the request, based on the topic name and time range in the request, a corresponding HDFS file path pattern is generated. Then, using Flink's FileSource or a custom HDFS scanner, all ".bz2 format" compressed data packets matching this pattern are located and read from the distributed file storage system. Subsequently, the BZip2 decompression algorithm is applied to each data packet to decompress it, restoring the compressed byte stream to the original log text lines, thereby recovering the original log data. The decompressed original log data is organized into a data stream and written to a specified real-time retrieval system (such as Elasticsearch) through Flink's Sink operator (such as Elasticsearch Sink), completing the rapid recovery and reconstruction of the log's queryable state.

[0129] This invention, by receiving log recovery requests containing the target message topic and time range, can accurately locate and efficiently read the corresponding compressed data packets in a distributed file storage system, enabling rapid access to archived logs. The read compressed data packets are decompressed in real time to restore the original log data, ensuring data integrity and accuracy. The recovered log data stream is written to a designated real-time retrieval system, supporting users to flexibly and quickly retrieve and analyze historical log information, thereby effectively improving the utilization value of archived logs and meeting the needs for rapid log recovery and real-time querying.

[0130] Optionally, based on one or more embodiments corresponding to Figure 1 above, another optional embodiment provided by the present invention may further include:

[0131] Monitor the message queue cluster to detect in real time whether there are new message topics or changes in the number of partitions of existing topics. If so, execute the step of consuming the raw log data stream from the message queue in real time.

[0132] Specifically, this embodiment of the invention can continuously monitor the message topics and their partition status in the message queue cluster in real time. It utilizes the metadata interface of message queues such as Kafka to periodically or event-drivenly obtain information on all current topics and partitions. Once the creation of a new topic or a change in the number of partitions in an existing topic is detected, a notification mechanism is immediately triggered, driving Flink consumption tasks to dynamically adjust or start new consumption instances. This ensures seamless real-time consumption of newly added or changed log data streams, thereby achieving automatic detection and rapid response in conjunction with intelligent operation and maintenance management, guaranteeing the integrity and timeliness of log collection. It is understood that if there are no new message topics or changes in the number of partitions in existing topics, the message queue cluster will continue to be monitored without further processing.

[0133] As examples, embodiments of the present invention can utilize a supporting intelligent operation and maintenance monitoring component to periodically or through a listening mechanism monitor the metadata of a message queue cluster (such as Kafka) and perceive its topic list and partition configuration in real time. When a new message topic is detected or the number of partitions of an existing topic changes, the component will trigger a preset rule engine or call a management API. If the change meets preset conditions (e.g., a topic within the archiving scope), the component will automatically generate or update the consumption configuration of the corresponding Flink compression task (e.g., adding a topic subscription or adjusting the parallelism), and execute the dynamic update or restart of the task through the task management interface, thereby ensuring that the steps of consuming raw log data streams from the message queue in real time can be seamlessly connected and continuously performed, achieving unattended automated operation and maintenance.

[0134] This invention, through monitoring a message queue cluster, can detect changes in the number of newly added message topics or partitions in real time, ensuring timely coverage and seamless adaptation of the log archiving process across all data sources. This automatically initiates real-time consumption and archiving of the new data stream upon any adjustment to a new log topic or partition, preventing data loss or delays and significantly improving the completeness of log collection and the system's dynamic adaptability. Simultaneously, combined with dynamic grouping and adaptive resource allocation, it ensures efficient compression and archiving of log streams with newly added or changed topics, effectively enhancing the overall system's intelligence and operational efficiency.

[0135] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0136] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0137] Corresponding to the above method embodiments, the present invention also provides a log management device, the structure of which is shown in FIG4, and may include: a log data stream consumption unit 10, a topic grouping result acquisition unit 20, a task configuration dynamic adjustment unit 30, a data grouping compression unit 40, and a data aggregation and archiving unit 50.

[0138] Log data stream consumption unit 10 is used to consume raw log data streams from the message queue in real time.

[0139] The topic grouping result acquisition unit 20 is used to dynamically group topics in the original log data stream according to the log traffic size of each message topic, assign high-traffic topics to independent groups, and merge multiple low-traffic topics into a shared group to obtain topic grouping results.

[0140] The task configuration dynamic adjustment unit 30 is used to start an independent compression processing task for each group in the topic grouping results, and dynamically adjust the parallelism configuration and memory resource configuration of the compression processing task according to the total log traffic of each group.

[0141] The data grouping and compression unit 40 is used to group the original log data belonging to the same message topic by time using a time window during the execution of any compression processing task according to the corresponding parallelism configuration and memory resource configuration, and compress the original log data after time grouping into compressed data packets in real time.

[0142] The data aggregation and archiving unit 50 is used to aggregate the compressed data packets obtained from each compression processing task according to predefined file naming rules, and sequentially write the aggregated compressed data packets into the distributed file storage system for archiving.

[0143] Optionally, the topic grouping result acquisition unit 20 can be specifically used to monitor the real-time log traffic of each message topic; input the real-time log traffic into the traffic prediction model to obtain future log traffic data predicted by the traffic prediction model based on the real-time log traffic and according to the historical log traffic change pattern; based on the future log traffic data, message topics with traffic greater than a preset threshold are divided into high-traffic topics and high-traffic topics are grouped separately, message topics with traffic equal to or less than the preset threshold are divided into low-traffic topics and low-traffic topics are merged into a group to obtain the topic grouping result.

[0144] Optionally, the task configuration dynamic adjustment unit 30 can be specifically used to obtain the total log traffic of each group in the topic grouping results according to a preset period; calculate the number of parallel execution units and memory allocation parameters required for the corresponding group by using the change in total log traffic and the preset traffic resource correspondence; dynamically adjust the parallelism configuration of the corresponding group according to the number of parallel execution units; and dynamically adjust the memory resource configuration of the corresponding group according to the memory allocation parameters.

[0145] Optionally, the data grouping compression unit 40 can be specifically used to create a memory buffer for raw log data belonging to the same message topic within each time window during the execution of the compression processing task; write the raw log data into the corresponding memory buffer; apply a compression algorithm to the raw log data in the memory buffer to compress it and generate compressed data blocks; and encapsulate the compressed data blocks into a compressed data packet of a specified format.

[0146] Optionally, the data aggregation and archiving unit 50 can be specifically used to generate a filename containing the message topic name and window time range for each compressed data packet; multiple compressed data packets generated in parallel with the same filename are sent to the same file write node for processing through an aggregation function.

[0147] Optionally, the log management device may also include a log recovery unit.

[0148] The log recovery unit is used to receive log recovery requests, which include a target message topic and a target time range; locate and read one or more corresponding compressed data packets from the distributed file storage system according to the target message topic and target time range; decompress the read compressed data packets to recover the original log data; and write the recovered original log data stream to the specified real-time retrieval system.

[0149] Optionally, the log management device may also include an intelligent operation and maintenance unit.

[0150] The intelligent operation and maintenance unit is used to monitor the message queue cluster and detect in real time whether there are new message topics or changes in the number of partitions of existing topics. If so, it triggers the log data stream consumption unit 10.

[0151] This invention provides a log management device for: consuming raw log data streams from a message queue in real time; dynamically grouping topics in the raw log data stream according to the log traffic volume of each message topic, allocating high-traffic topics to independent groups and merging multiple low-traffic topics into a shared group to obtain topic grouping results; initiating an independent compression processing task for each group in the topic grouping results, and dynamically adjusting the parallelism and memory resource configuration of the compression processing task according to the total log traffic of each group; during the execution of any compression processing task according to the corresponding parallelism and memory resource configuration, using a time window to group the raw log data belonging to the same message topic by time, and compressing the time-grouped raw log data into compressed data packets in real time; aggregating the compressed data packets obtained from each compression processing task according to predefined file naming rules, and sequentially writing the aggregated compressed data packets into a distributed file storage system for archiving. This invention, by consuming log data streams from a message queue in real time, dynamically grouping and initiating independent compression tasks for topics with different traffic volumes, and combining time windows for real-time compression and orderly archiving, achieves efficient and resource-optimized large-scale archiving of massive logs while ensuring real-time log query performance.

[0152] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0153] The log management device includes a processor and a memory. The log data stream consumption unit 10, the topic grouping result acquisition unit 20, the task configuration dynamic adjustment unit 30, the data grouping compression unit 40, and the data aggregation and archiving unit 50 are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.

[0154] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and kernel parameters can be adjusted to consume raw log data streams from the message queue in real time. Log traffic is dynamically grouped based on message topics, allocating high-traffic topics to independent groups and merging low-traffic topics into shared groups. An independent compression task is initiated for each group, dynamically adjusting the task's parallelism and memory resource configuration. Logs on the same topic are grouped by time using time windows and compressed into compressed data packets in real time. These compressed packets are then aggregated according to predefined naming rules and sequentially written to a distributed file storage system for archiving. Through dynamic grouping and resource adjustment, real-time compression and orderly archiving of logs are achieved, ensuring real-time query performance while enabling efficient archiving of massive amounts of logs.

[0155] This invention provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the log management method.

[0156] This invention provides a processor for running a program, wherein the program executes the log management method during runtime.

[0157] As shown in Figure 5, this embodiment of the invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003. The processor 1001 and the memory 1002 communicate with each other via the bus 1003. The processor 1001 is used to call program instructions in the memory 1002 to execute the aforementioned log management method. The electronic device in this document can be a server, PC, PAD, mobile phone, etc.

[0158] The present invention also provides a computer program product that, when executed on an electronic device, is suitable for executing a program with initialization steps for a log management method.

[0159] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, electronic devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0160] In a typical configuration, an electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input / output interfaces, network interfaces, etc.

[0161] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0162] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0163] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0164] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0165] In the description of this invention, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0166] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0167] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0168] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the present invention.

Claims

1. A log management method, characterized in that, include: Consume raw log data streams in real time from the message queue; Based on the log traffic volume of each message topic, the topics in the original log data stream are dynamically grouped, with high-traffic topics assigned to independent groups and multiple low-traffic topics merged and assigned to a shared group to obtain topic grouping results; An independent compression processing task is started for each group in the topic grouping result, and the parallelism configuration and memory resource configuration of the compression processing task are dynamically adjusted according to the total log traffic of each group; during the execution of any compression processing task according to the corresponding parallelism configuration and memory resource configuration, the original log data belonging to the same message topic is grouped by time using a time window, and the original log data after time grouping is compressed into compressed data packets in real time. The compressed data packets obtained from each compression processing task are aggregated according to predefined file naming rules, and the aggregated compressed data packets are sequentially written into the distributed file storage system for archiving.

2. The method according to claim 1, characterized in that, The step of dynamically grouping topics in the original log data stream based on the log traffic volume of each message topic, allocating high-traffic topics to independent groups, and merging multiple low-traffic topics into a shared group to obtain topic grouping results, includes: monitoring the real-time log traffic of each message topic; inputting the real-time log traffic into a traffic prediction model to obtain future log traffic data predicted by the traffic prediction model based on the real-time log traffic and historical log traffic change patterns; and, based on the future log traffic data, classifying message topics with traffic exceeding a preset threshold as high-traffic topics and grouping them into a separate group, and classifying message topics with traffic equal to or less than the preset threshold as low-traffic topics and merging them into a single group to obtain topic grouping results.

3. The method according to claim 1 or 2, characterized in that, The step of initiating an independent compression processing task for each group in the topic grouping results, and dynamically adjusting the parallelism and memory resource configuration of the compression processing task according to the total log traffic of each group, includes: obtaining the total log traffic of each group in the topic grouping results according to a preset period; calculating the number of parallel execution units and memory allocation parameters required for the corresponding group using the changes in the total log traffic and the preset traffic resource correspondence; dynamically adjusting the parallelism configuration of the corresponding group according to the number of parallel execution units; and dynamically adjusting the memory resource configuration of the corresponding group according to the memory allocation parameters.

4. The method according to claim 1, characterized in that, The step of compressing the raw log data after time grouping into compressed data packets in real time includes: during the execution of the compression processing task, creating a memory buffer for raw log data belonging to the same message topic within each time window; writing the raw log data into the corresponding memory buffer; applying a compression algorithm to the raw log data in the memory buffer to compress it and generate compressed data blocks; and encapsulating the compressed data blocks into compressed data packets of a specified format.

5. The method according to claim 1, characterized in that, The step of aggregating the compressed data packets obtained from each compression processing task according to a predefined file naming rule includes: generating a filename containing a message topic name and a window time range for each compressed data packet; and sending multiple compressed data packets generated in parallel with the same filename to the same file writing node for processing through an aggregation function.

6. The method according to claim 1, characterized in that, Also includes: Receive a log recovery request, wherein the log recovery request includes a target message topic and a target time range; locate and read one or more corresponding compressed data packets from the distributed file storage system according to the target message topic and target time range; decompress the read compressed data packets to recover the original log data; and write the recovered original log data stream into the specified real-time retrieval system.

7. The method according to claim 1, characterized in that, Also includes: Monitor the message queue cluster to detect in real time whether there are new message topics or changes in the number of partitions of existing topics. If so, execute the step of consuming the raw log data stream from the message queue in real time.

8. A log management device, characterized in that, include: The system includes a log data stream consumption unit, a topic grouping result acquisition unit, a task configuration dynamic adjustment unit, a data grouping compression unit, and a data aggregation and archiving unit; the log data stream consumption unit is used to consume raw log data streams from the message queue in real time. The topic grouping result acquisition unit is used to dynamically group topics in the original log data stream according to the log traffic size of each message topic, assigning high-traffic topics to independent groups and merging multiple low-traffic topics into a shared group to obtain topic grouping results. The task configuration dynamic adjustment unit is used to start an independent compression processing task for each group in the topic grouping result, and dynamically adjust the parallelism configuration and memory resource configuration of the compression processing task according to the total log traffic of each group. The data grouping compression unit is used to use a time window to group the original log data belonging to the same message topic by time during the execution of any compression processing task according to the corresponding parallelism configuration and memory resource configuration, and compress the time-grouped original log data into compressed data packets in real time. The data aggregation and archiving unit is used to aggregate the compressed data packets obtained by each compression processing task according to predefined file naming rules, and sequentially write the aggregated compressed data packets into a distributed file storage system for archiving.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the log management method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the log management method as described in any one of claims 1 to 7.

Images