Log processing method, system, and readable storage medium

Abstract

The application provides a log processing method, system and readable storage medium. A log file is acquired. In the process of sending the log file to a message queue, based on the negative correlation between the consumption lag and the sending rate, the consumption lag is acquired in real time and the sending rate is dynamically adjusted, so that the sending behavior of the production end automatically adapts to the real-time processing capability of the consumption end without manual intervention or static configuration. This means that in the process of sending log events to the message queue based on the adjusted sending rate, when the processing capability of the consumption end fluctuates, the sending rate is adjusted smoothly, avoiding queue accumulation or resource waste caused by rate mismatch, thereby maintaining the stable operation of the system in a dynamic environment and improving the stability of the system. At the same time, through the adaptive log event sending mechanism, higher data processing throughput can be achieved under the same hardware resources, thereby improving the log processing efficiency.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a log processing method, system, and readable storage medium. Background Technology

[0002] With the rapid development of intelligent connected vehicle technology, the number of connected vehicles has surged, generating massive amounts of vehicle operation log data from vehicle-side components (such as in-vehicle terminals, sensors, and control systems). These logs are crucial for fault diagnosis, performance optimization, user experience improvement, and safety monitoring. Common log processing methods include the ELK Stack (Elasticsearch, Logstash, Kibana) or its evolved form, the Elastic Stack, both based on their respective log analysis platforms. However, both the ELK Stack and Elastic Stack have significant drawbacks: while message queues are used as buffers during log transmission, the rate at which log producers send data to these queues is often statically configured, limiting processing efficiency. Understandably, slow log production leads to idle downstream resources and low overall throughput. Conversely, when the log volume surges and log production speeds exceed downstream consumption capacity, message backlogs or even memory overflows occur in the message queue, further limiting processing efficiency and impacting system stability. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this specification provides a log processing method, system, and readable storage medium.

[0004] Firstly, a log processing method is provided, the method comprising: Obtain the log file and read log events from the log file; The sending rate of the log events is dynamically adjusted based on the change in the consumption lag of the message queue. The message queue is used to receive the log events. The consumption lag is determined by the number of log events in the message queue that have not been consumed by the consumer. The consumption lag is negatively correlated with the sending rate. The log events are sent to the message queue at the dynamically adjusted sending rate, so that the consumer associated with the message queue can consume the log events.

[0005] According to a log processing method provided in this application, the sending rate is determined by the batch size and / or the maximum waiting time, and the step of sending the log events to the message queue according to the dynamically adjusted sending rate includes: The log events are encapsulated into messages in a message queue, and the messages are added to the message buffer. When the number of messages cached in the message buffer reaches the preset batch size, and / or the time between the last time a batch of messages was sent and the current time reaches the preset maximum waiting time, all messages currently cached in the message buffer are sent to the message queue as a batch.

[0006] According to a log processing method provided in this application, the step of dynamically adjusting the sending rate of the log events based on changes in the consumption lag of the message queue includes: Based on the changes in the consumption lag, the batch size and / or the maximum waiting time are dynamically adjusted, wherein the consumption lag is negatively correlated with the batch size and positively correlated with the maximum waiting time.

[0007] According to a log processing method provided in this application, the step of obtaining a log file and reading log events from the log file includes: Obtain a log file containing unstructured log text; Read the log text line by line. For each line read, perform the following steps: If the starting time of the original timestamp in the timestamp information is inconsistent with the preset reference time, then the absolute time of the starting time relative to the reference time is determined. Based on the absolute time, the original timestamp is converted into an absolute timestamp calculated from the base time to obtain a log event carrying the absolute timestamp.

[0008] According to the log processing method provided in this application, the step of reading the log text line by line further includes performing the following steps for each line read: Determine whether the current row is a continuation of the previous row based on the preset merging rules; If so, the current line and the previous line are merged into a complete log event; otherwise, the current line is used as the starting line of a new log event.

[0009] According to a log processing method provided in this application, the step of determining whether the current line is a continuation of the previous line based on a preset merging rule includes: Check if the beginning of the current line matches a preset timestamp format; if not, it is considered a continuation of the previous line; or... Check if the current line begins with a space or tab; if so, it is considered a continuation of the previous line; or... Detect whether the current line starts with a preset keyword; if so, it is considered a continuation of the previous line.

[0010] According to a log processing method provided in this application, after sending the log event to the message queue according to the dynamically adjusted sending rate, the method further includes: Consume the log events in the message queue; For each log event that has been consumed by the consumer, a preset regular expression is matched according to the type of the log event; The matched regular expression is invoked to parse the unstructured text in the log event to obtain structured data; The structured data is stored in the search engine.

[0011] According to a log processing method provided in this application, storing the structured data in a search engine includes: The structured data is accumulated according to a preset batch size; When the amount of accumulated structured data reaches the specified batch size, all the accumulated structured data is written in batches to the index specified in the search engine through the application programming interface of the search engine.

[0012] In a second aspect, an electronic device is provided, the device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the log processing method as described above.

[0013] Thirdly, a computer-readable storage medium is provided, on which a log processing program is stored, wherein the log processing program, when executed, implements the steps of any of the log processing methods described above.

[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the log processing method as described in any of the above.

[0015] This application provides a log processing method, system, and readable storage medium, which have the following advantages compared with the current low efficiency of log processing: The process of acquiring log files and sending them to the message queue leverages the inverse correlation between consumption lag and sending rate. By dynamically adjusting the sending rate based on real-time consumption lag, the production end's sending behavior automatically adapts to the consumer's real-time processing capabilities without manual intervention or static configuration. This means that by sending log events to the message queue at the adjusted sending rate, the sending rate smoothly adjusts to fluctuations in consumer processing capacity, avoiding queue backlogs or resource waste caused by rate mismatches. This maintains stable system operation in dynamic environments and improves system stability. Simultaneously, this adaptive log event sending mechanism enables higher data processing throughput with the same hardware resources, thereby improving log processing efficiency.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0018] Figure 1 This is a schematic flowchart illustrating a log processing method according to an exemplary embodiment of this specification; Figure 2 This is a schematic diagram of the log text preprocessing process in a log processing method according to an exemplary embodiment of this specification; Figure 3 This is a flowchart illustrating a specific embodiment of a log processing method according to an exemplary embodiment of this specification; Figure 4 This is a schematic block diagram of a log processing device illustrated in this specification according to an exemplary embodiment. Detailed Implementation

[0019] The technical solutions in the embodiments (or "implementations") of this application will be clearly and completely described herein with reference to the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0020] If the embodiments of this application contain terms relating to directional indications or positional relationships (such as up, down, left, right, front, back, inside, outside, top, bottom, center, vertical, horizontal, longitudinal, transverse, length, width, counterclockwise, clockwise, axial, radial, circumferential, etc.), such terms are only used to explain the relative positional relationships and movement of the components in a specific posture (as shown in the attached figures); if the specific posture changes, the directional indications or positional relationships will also change accordingly. Furthermore, the terms "first" and "second" used in the embodiments of this application are only for descriptive convenience and should not be construed as indicating or implying relative importance.

[0021] This application provides a log processing method, system, and readable storage medium. The application will be described in detail below with reference to the accompanying drawings. The features described in the embodiments and implementations can be combined with each other.

[0022] To address the aforementioned technical problems, this application provides a log processing method.

[0023] The aim is to build a multi-stage, collaborative processing flow for vehicle-side log processing. This flow decouples log downloading and decompression, fine-grained preprocessing, controllable rate transmission, and high-performance consumption and storage. It then adopts a relatively optimized solution for each link / module in the chain, thereby improving the overall log processing speed while ensuring system stability, so as to meet actual needs.

[0024] In other words, this log processing method involves a log producer and a log consumer. Logs are cached between the producer and consumer via a message queue, thus decoupling the components and allowing them to operate independently. The sending rate of the producer can be dynamically adjusted based on the processing capacity of the consumer. The producer is the software module or service responsible for obtaining raw log data from the log source, processing it, encapsulating it into messages, and sending it to the message queue. Its core functions include: reading log files, formatting and normalizing log events (such as timestamp conversion and line break merging), and sending the processed log events to the message queue at a controllable rate. The consumer is the software module or service that pulls log messages from the message queue, parses the unstructured log text in the messages, converts it into structured data, and finally writes the structured data to the target storage system (such as a search engine). Its core functions include: continuously monitoring log topics in the message queue, matching preset parsing rules according to log type, extracting structured fields, and batch writing to the data storage system.

[0025] This application provides an embodiment of a log processing method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a log processing method provided in an embodiment of this specification.

[0026] The log processing method specifically includes the following steps S101 to S103: In step S101, a log file is obtained, and log events are read from the log file.

[0027] The production end retrieves log files from a specified path via a file system interface or object storage SDK, and reads the file content line by line using a streaming method. Each line of raw text is treated as an unprocessed log record. The file offset can be maintained during the reading process, and breakpoints can be resumed.

[0028] In some feasible implementations, obtaining the log file and reading log events from the log file includes steps S1 to S2: Step S1: Obtain a log file containing unstructured log text; Step S2: Read the log text line by line. For each line read, perform the following steps: Step S31: Identify the timestamp information in the current row; Step S32: If the starting time of the original timestamp in the timestamp information is inconsistent with the preset reference time, then determine the absolute time of the starting time relative to the reference time; based on the absolute time, convert the original timestamp into an absolute timestamp calculated from the reference time to obtain a log event carrying the absolute timestamp.

[0029] First, the production service listens for external calls. Upon receiving a request to process a log compression package, it uses the SDK to download the specified compressed package from storage to a local temporary folder. Next, it calls the decompression library to decompress the compressed package in the temporary folder, obtaining the required plain text log file.

[0030] Each decompressed log file is read, and its log text is retrieved. This log text refers to the raw, unprocessed single-line string read from the log file, containing unstructured log text. Traditional ELK Stack-based solutions lack the capability to process unstructured data, resulting in low data parsing accuracy and impacting the reliability of subsequent analysis and decision-making. To address this issue, as a feasible real-time approach, the log text is preprocessed before being sent, enhancing the data cleaning capabilities of the entire process chain through a preprocessing module.

[0031] Unstructured log text includes text with non-standard timestamps. Vehicle-side log timestamps are often non-standard, with two common scenarios: either the time written in the log is the number of seconds or milliseconds calculated from program startup, rather than an absolute standard time, or there are diverse time formats. This leads to low data parsing accuracy. (See reference...) Figure 2 , Figure 2 This is a schematic flowchart illustrating the log text preprocessing in a log processing method provided in the embodiments of this specification. The specific preprocessing process for this type of log text is as follows: The production end reads each line of the log file sequentially via a file streaming interface, temporarily storing each line of log text as a string in memory. Regular expression matching or position-based scanning is then performed on the current line to extract the timestamp field. In some examples, multiple timestamp patterns are pre-defined; timestamp information can be identified through pattern matching. If a match is successful, the type of timestamp is determined; otherwise, the line is skipped or marked as having no timestamp.

[0032] If the starting time of the original timestamp in the timestamp information of the log text is inconsistent with the preset base time, and is determined relative to a certain custom starting time as the starting point for timing, then in order to ensure the consistency of the time base and the standardization of the timestamp, the absolute time of the starting time relative to the standard base time is first calculated, and then the absolute timestamp of the current log line is obtained by adding the offset of the timestamp in the timestamp information relative to the starting time to the absolute time of the starting time.

[0033] In this case, the unit of offset must be consistent, such as seconds or milliseconds.

[0034] Here, the starting time refers to the time starting point for recording the timestamp of the current line of log text. This time starting point can be, but is not limited to, the system startup time, the log module initialization time, or the set time when the file begins recording. In this embodiment, the starting time can be obtained through external input, parsing from the filename, or reading from the first line of the log.

[0035] Then, the original timestamp in the current log line is replaced with an absolute timestamp to generate a new log line string. This new log line is the processed log event, which carries an absolute timestamp.

[0036] In other examples, the conversion result of absolute timestamps needs to be formatted into a uniform standard format for easy subsequent parsing. As an example, if the identified timestamps are in absolute time format but are disorganized, the program will pre-configure a batch of commonly used time format templates and attempt to match them for each log line. Once a match is successful, it is uniformly converted into a standard format. If no match is found, it is left as is and marked with an unknown format. For example, if a log text has timestamps of "3 minutes later," "2:30 PM," and "2:30 PM," then all times need to be converted to absolute time (e.g., by adding the current time) before being uniformly written in a standard format like "14:30:00."

[0037] Furthermore, unstructured log text also includes line-broken text. For example, some vehicle-side logs may split a complete log event into multiple lines. If these lines are not merged, subsequent parsing will treat the second, third, and other subsequent lines as independent error logs, causing confusion. A feasible implementation method for preprocessing this type of log text is proposed. The log text is read line by line, and for each line read, the following steps are further performed: Step S33: Determine whether the current row is a continuation of the previous row according to the preset merging rules; Step S34: If yes, merge the current line with the previous line into a complete log event; otherwise, use the current line as the starting line of a new log event.

[0038] Continue to refer to Figure 2 The log text preprocessing process is as follows: When the production server reads log text line by line, it maintains a buffer to temporarily store the log events currently being built. For the currently read line, a preset set of merging rules is applied for judgment.

[0039] As an example, merging rules include, but are not limited to, rules for missing timestamps, rules for line indentation, rules for keywords, and rules for line endings.

[0040] Based on this, the system determines whether the current line is a continuation of the previous line according to preset merging rules. For example, it checks whether the beginning of the current line matches a preset timestamp format; if not, it is determined to be a continuation line. Another example is checking whether the beginning of the current line is a space or a tab; if so, it is determined to be a continuation line. Yet another example is checking whether the current line begins with a preset keyword; if so, it is determined to be a continuation line. And yet another example is determining that the current line is a continuation line if the previous line ends with a preset continuation identifier.

[0041] The above rules can be matched sequentially according to priority based on actual needs, and any matching of any rule is considered a continuation line.

[0042] If the line is determined to be a continuation line, the string content of the current line is appended to the end of the current log event in the buffer. The appending process is configured according to the original log format requirements to maintain readability; for example, a space or newline character may be inserted. After merging, the data is not output immediately; the system continues reading the next line.

[0043] If the line is determined to be non-continuous, the complete log event already existing in the buffer is first output to the subsequent processing flow, then the buffer is cleared, and the current line is stored in the buffer as the starting line of the new log event.

[0044] This embodiment ensures that multiple log lines can be correctly restored into a single complete log event, providing structured input for subsequent parsing.

[0045] In some feasible implementations, the above-described preprocessing of log text is carried out using a general-purpose programming language. Preferably, Python is used, as Python excels at complex file operations and logical judgments, making it suitable for handling messy raw data.

[0046] Through the above embodiments, this Python preprocessing module integrates a content repair and transformation mechanism tailored to the specific characteristics of vehicle-side logs, enhancing the data cleaning capabilities of the data chain. It possesses the ability to extend its capabilities to parse various complex unstructured logs (such as those with special timestamp formats or abnormal line breaks), directly resolving the accuracy issues caused by non-standard raw log data. Timestamp standardization facilitates subsequent time-based analysis and investigation; abnormal line break repair ensures the integrity of each log event, laying a solid foundation for subsequent regular expression parsing, thereby improving the quality of the imported data.

[0047] In step S102, the sending rate of the log events is dynamically adjusted according to the change in the consumption lag of the message queue. The message queue is used to receive the log events, and the consumption lag is determined by the number of log events in the message queue that have not been consumed by the consumer. The consumption lag is negatively correlated with the sending rate.

[0048] In this embodiment, the message queue acts as a buffer layer to receive logs, temporarily storing messages sent by the producer until they are consumed by the consumer. Kafka, as a distributed streaming processing platform, possesses the core functions of a message middleware, enabling system decoupling, asynchronous communication, and traffic smoothing. Therefore, Kafka is preferably used as the message queue. Thus, pre-created Kafka groups and topics receive and temporarily store messages sent by the producer, awaiting consumption by the consumer.

[0049] To address the issue of static configuration on the log producer side leading to message backlog and even memory overflow when the number of logs surges, thus limiting processing efficiency and affecting system stability, a feasible implementation method is proposed. This method introduces an adaptive flow control strategy based on the producer side and oriented towards the message queue, namely, adaptive flow control that dynamically adjusts the producer sending rate parameter based on the consumer lag.

[0050] Specifically, by monitoring metrics such as Consumer Lag in the message queue, these parameters are dynamically fine-tuned to achieve adaptive sending rate. Consumer Lag reflects the backlog status on the consumer side. If Consumer Lag continues to increase, it indicates that the producer is sending too fast, exceeding the downstream consumption capacity, and the consumer cannot process messages in time, causing a backlog in Kafka. In this case, the program reduces the sending rate. Conversely, if Consumer Lag decreases, it indicates that the producer is sending too slowly, leading to idle downstream resources and low overall throughput. In this case, the program increases the sending rate, thus achieving dynamic adjustment of the sending rate.

[0051] In this embodiment, the sending rate is determined by the batch size (batch.size) and / or the maximum waiting time (linger.ms). The batch size refers to the number of log events included in each sending batch, and the maximum waiting time refers to the maximum time allowed for log events to accumulate before sending a batch. For example, configuring the batch size (batch.size) to 1000 messages means that 1000 log events are accumulated before sending a batch. As another example, configuring the maximum waiting time (linger.ms) to 100 milliseconds means that the currently accumulated batch of log events is sent after 100 milliseconds. By accumulating messages and sending them in batches to the message queue, higher throughput can be achieved.

[0052] To achieve dynamic adjustment of the sending rate by monitoring the backlog status of the consumer, an embodiment is proposed. The dynamic adjustment of the log event sending rate based on changes in the consumption lag of the message queue includes: Based on the changes in the consumption lag, the batch size and / or the maximum waiting time are dynamically adjusted, wherein the consumption lag is negatively correlated with the batch size and positively correlated with the maximum waiting time.

[0053] In this embodiment, by monitoring the consumption lag of the message queue, the batch size and maximum waiting time are dynamically adjusted to match the sending rate with the downstream consumption capacity. It can be understood that increasing the consumption lag increases the maximum waiting time to allow for longer waiting periods before sending, and can also reduce the batch size, accumulating smaller batches, but at a lower frequency. Conversely, decreasing the consumption lag reduces the maximum waiting time to shorten the waiting time, increasing the sending frequency, and can also increase the batch size.

[0054] Based on the above embodiments, an adaptive flow control mechanism ensures smooth data flow throughout the production, buffering, and consumption stages, preventing message backlog in the message queue and idle waiting at the consumer end. Compared to current technologies that use fixed-rate sending or simple producer-consumer models, this solution achieves higher data processing throughput with the same hardware resources while ensuring system stability, resulting in superior system throughput and resource utilization.

[0055] In step S103, the log event is sent to the message queue according to the dynamically adjusted sending rate, so that the consumer associated with the message queue can consume the log event.

[0056] In this embodiment, topics and consumer groups have been pre-created in the message queue. All log messages sent by the producer are first stored in the message queue. After a consumer (Vector) retrieves a message from the message queue, the message queue records the retrieval offset to ensure that each message is processed at least once.

[0057] In some feasible implementations, sending the log events to the message queue according to the dynamically adjusted sending rate includes: The log events are encapsulated into messages in a message queue, and the messages are added to the message buffer. When the number of messages cached in the message buffer reaches the preset batch size, and / or the time between the last time a batch of messages was sent and the current time reaches the preset maximum waiting time, all messages currently cached in the message buffer are sent to the message queue as a batch.

[0058] The specific process is as follows: First, message encapsulation.

[0059] Each time the producer receives a preprocessed log event, it calls the message construction interface provided by the message queue client to generate a message object. The Value field of this message object is set to the string content of the log event, and the Key field is optional (e.g., set to device ID, log source, or left blank). After construction, the message is passed into memory by calling the message queue producer's interface. A message buffer is maintained in memory according to the target topic and partition. After each message is encapsulated, it is added to the end of the message batch buffer.

[0060] Next, determine the triggering condition.

[0061] The producer maintains the current message count and the timestamp of the last send for each message buffer. For each buffer, two conditions are checked simultaneously: Batch size condition: If the number of messages in the current message buffer has reached the preset batch size, then sending is triggered.

[0062] Maximum waiting time condition: If the waiting time between the current time and the time of the last batch message sent has reached the preset linger.ms, and the message buffer is not empty, then sending is triggered.

[0063] Then, all currently cached messages are sent to the message queue as a batch.

[0064] When any of the above conditions are met, the producer packages all messages in the current message buffer as a batch and sends them to the specified topic and partition of the message queue server by invoking the underlying network protocol. After sending, the message buffer is cleared, the message count is reset, and the last sending time is updated to the current time. If sending fails, it can be retried according to the configuration, and the failed message is retained in the buffer or transferred to a specific queue.

[0065] If none of the above conditions are met, continue to wait for further information.

[0066] The batch sending mechanism in this example reduces the number of network requests by batch aggregation. When adjusting the sending rate, the total number of messages sent per unit time can be changed by dynamically modifying the batch.size and linger.ms parameters, thereby achieving precise control over the sending rate.

[0067] In some feasible implementations, after sending the log event to the message queue according to the dynamically adjusted sending rate, the method further includes: The log events are consumed in the message queue; for each log event consumed by the consumer, a preset regular expression is matched according to the type of the log event; the matched regular expression is used to parse the unstructured text in the log event to obtain structured data; the structured data is stored in the search engine.

[0068] Message queues allow consumers to independently control their read progress without affecting the producer's continued writing. Therefore, the consumer continuously listens to the message queue topic and pulls messages from the specified topic. Specifically, the consumer configures the message queue connection information (address, topic, consumer group ID) when it starts. It subscribes to the specified topic through the consumer API and continuously pulls messages using long polling. The value field of each message is the log event string written by the producer. The consumer automatically commits the consumption offset or commits manually.

[0069] The consumer maintains a rule mapping table, where each rule includes, but is not limited to, a log type identifier and its corresponding regular expression pattern. For each consumed message, its type characteristics are extracted sequentially, and a matching regular expression is searched in the mapping table. If no matching rule is found, parsing is skipped, and the original log is recorded in a backup index. As an example, the type of a log event can be determined by extracting a fixed-position field from the log content, or it can be determined based on configured static tags.

[0070] The log event string is input into a regular expression engine for pattern matching. If a match is successful, the consumer combines these fields into a structured object, such as JSON. If a match fails, the consumer can store the entire log as a raw field or discard it. This process parses unstructured log text into structured data. The structured data is then stored in a search engine for subsequent log querying and analysis.

[0071] In some feasible implementations, storing the structured data in the search engine includes: The structured data is accumulated according to a preset batch size; When the amount of accumulated structured data reaches the specified batch size, all the accumulated structured data is written in batches to the index specified in the search engine through the application programming interface of the search engine.

[0072] On the consumer side, structured data is cached in batches and sent to a specified index through the search engine's batch write API (such as Elasticsearch's API).

[0073] In some cases, index names can be dynamically generated by date, facilitating data management and deletion of expired data. Upon successful write, an acknowledgment is returned; if the write fails, a retry is initiated or an error is logged.

[0074] This concludes the description of an embodiment of the log processing method.

[0075] This application provides a log processing method, system, and readable storage medium, which have the following advantages compared with the current low efficiency of log processing: The process of acquiring log files and sending them to the message queue leverages the inverse correlation between consumption lag and sending rate. By dynamically adjusting the sending rate based on real-time consumption lag, the production end's sending behavior automatically adapts to the consumer's real-time processing capabilities without manual intervention or static configuration. This means that by sending log events to the message queue at the adjusted sending rate, the sending rate smoothly adjusts to fluctuations in consumer processing capacity, avoiding queue backlogs or resource waste caused by rate mismatches. This maintains stable system operation in dynamic environments and improves system stability. Simultaneously, this adaptive log event sending mechanism enables higher data processing throughput with the same hardware resources, thereby improving log processing efficiency.

[0076] Based on the same application concept as the above method, this application also proposes a log processing system.

[0077] To make the above embodiments easier to understand, a specific embodiment is given below. A log processing system includes a download and decompression module, a fine-grained preprocessing module, a controllable-rate transmission module, and a high-performance consumption and storage module.

[0078] The download and decompression module and the fine-grained preprocessing module are executed by a first processing module implemented in the general-purpose programming language Python; the high-performance consumption and storage module is executed by a second processing module using the high-performance data pipeline component Vector, which parses log events into structured data; and the message queue is the distributed stream processing platform Kafka. This combination leverages the strengths of each component, improving overall system efficiency from an architectural perspective and resolving issues of architectural decoupling and resource consumption.

[0079] This is because Python excels at complex file operations and logical judgments, making it ideal for handling messy raw data. Kafka, as a distributed streaming platform, possesses the core functionality of a message middleware, enabling system decoupling, asynchronous communication, and traffic shaping. The consumer side uses the Vector service and supports unstructured log parsing based on regular expressions. Vector, as a high-performance log collection tool, excels at high-performance data ingestion, transformation, and output for standardized parsing and data storage. From a deployment perspective, Vector operates efficiently with minimal memory usage. Compared to current log collection components, Vector consumes less CPU and memory, processes faster, and offers superior overall performance. This significantly improves the accuracy and availability of log data stored in Elasticsearch, providing a reliable guarantee for accurate data analysis.

[0080] Based on this specific embodiment, the log processing procedure is described. (Refer to...) Figure 3 , Figure 3 This is a flowchart illustrating a specific embodiment of a log processing method provided in this specification. The log processing process includes: downloading and decompressing the log package, reading log events from the decompressed log file using Python, and preprocessing them. In the Python sending module, the confluent_kafka library is used to finely configure parameters such as batch size (batch.size) and latency (linger.ms), and monitor the Kafka cluster status to achieve self-adjustment of the sending rate, dynamically matching it with the processing capacity of the downstream Vector consumer, thus solving the problem of uneven resource utilization. For each processed complete log line, it is sent to the message queue at a controllable rate. By controlling the sending behavior of the producer, it can sense the pressure on the consumer, avoiding congestion or idleness in the link, thereby maximizing the overall data processing speed while ensuring system stability. Subsequently, the Vector service continuously listens to the Kafka topic and consumes log messages. For each consumed message, it matches a pre-defined regular expression pattern based on its source or characteristics to parse the unstructured log text into structured data. Finally, Vector writes the parsed structured data in batches to the specified index through the Elasticsearch API.

[0081] The specific implementation process of the functions and roles of each of the above modules can be found in the implementation process of the corresponding steps in the above method, which can achieve the same technical effect, and will not be repeated here.

[0082] Figure 4 An example is a schematic diagram of the physical structure of a log processing device, such as... Figure 4 As shown, the log processing device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840. The processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions stored in the memory 830 to execute log processing methods.

[0083] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the log processing methods provided by the above methods.

[0085] In another aspect, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the log processing methods provided by the methods described above.

[0086] It should be noted that the technical solutions or features described in the above embodiments can be combined or supplemented with each other without conflict. The scope of protection of this application is not limited to the precise structures described in the above embodiments and shown in the accompanying drawings; all modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A log processing method, characterized in that, The method includes: Obtain the log file and read log events from the log file; The sending rate of the log events is dynamically adjusted based on the change in the consumption lag of the message queue. The message queue is used to receive the log events. The consumption lag is determined by the number of log events in the message queue that have not been consumed by the consumer. The consumption lag is negatively correlated with the sending rate. The log events are sent to the message queue at the dynamically adjusted sending rate, so that the consumer associated with the message queue can consume the log events.

2. The log processing method as described in claim 1, characterized in that, The sending rate is determined by the batch size and / or the maximum waiting time. Sending the log events to the message queue according to the dynamically adjusted sending rate includes: The log events are encapsulated into messages in a message queue, and the messages are added to the message buffer. When the number of messages cached in the message buffer reaches the preset batch size, and / or the time between the last time a batch of messages was sent and the current time reaches the preset maximum waiting time, all messages currently cached in the message buffer are sent to the message queue as a batch.

3. The log processing method as described in claim 2, characterized in that, The step of dynamically adjusting the sending rate of the log events based on changes in the message queue's consumption lag includes: Based on the changes in the consumption lag, the batch size and / or the maximum waiting time are dynamically adjusted, wherein the consumption lag is negatively correlated with the batch size and positively correlated with the maximum waiting time.

4. The log processing method as described in claim 1, characterized in that, The step of obtaining the log file and reading log events from the log file includes: Obtain a log file containing unstructured log text; Read the log text line by line. For each line read, perform the following steps: Identify the timestamp information in the current row; If the starting time of the original timestamp in the timestamp information is inconsistent with the preset reference time, then the absolute time of the starting time relative to the reference time is determined. Based on the absolute time, the original timestamp is converted into an absolute timestamp calculated from the base time to obtain a log event carrying the absolute timestamp.

5. The log processing method as described in claim 4, characterized in that, The step of reading the log text line by line further includes performing the following steps for each line read: Determine whether the current row is a continuation of the previous row based on the preset merging rules; If so, the current line and the previous line are merged into a complete log event; otherwise, the current line is used as the starting line of a new log event.

6. The log processing method as described in claim 5, characterized in that, The step of determining whether the current row is a continuation of the previous row according to preset merging rules includes: Check if the beginning of the current line matches a preset timestamp format; if not, determine that the current line is a continuation of the previous line; or, Check if the current line begins with a space or tab; if so, determine that the current line is a continuation of the previous line; or... Detect whether the current line begins with a preset keyword; if so, determine that the current line is a continuation of the previous line.

7. The log processing method as described in claim 1, characterized in that, After sending the log event to the message queue according to the dynamically adjusted sending rate, the method further includes: Consume the log events in the message queue; For each log event that has been consumed by the consumer, a preset regular expression is matched according to the type of the log event; The matched regular expression is invoked to parse the unstructured text in the log event to obtain structured data; The structured data is stored in the search engine.

8. The log processing method as described in claim 7, characterized in that, The step of storing the structured data into the search engine includes: The structured data is accumulated according to a preset batch size; When the amount of accumulated structured data reaches the specified batch size, all the accumulated structured data is written in batches to the index specified in the search engine through the application programming interface of the search engine.

9. An electronic device, characterized in that, The system includes a memory, a processor, and a log processing program stored in the memory and executable on the processor, wherein the processor, when executing the log processing program, implements the steps of the log processing method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a log processing program, which, when executed, implements the steps of the log processing method as described in any one of claims 1 to 8.

Images