Log processing method and device for memory optimization, equipment and medium

Abstract

The invention provides a log processing method and device for memory optimization, equipment and a medium, the method is applied to the technical field of log processing, and the method comprises the steps that a container management platform starts a main application container and a collection container, and mounts a shared storage volume to a predetermined directory of the main application container and the collection container; the collection container reads an environment variable to obtain a socket path, creates a Unix domain socket file in the shared storage volume and monitors the Unix domain socket file; the main application container creates a symbolic link from the system log interface to the socket file through the life cycle hook; log data output by the application program is redirected to the Unix domain socket through a symbolic link; and the collection container receives the log and writes the log into an annular buffer area, in-situ analysis is carried out in the buffer area to generate a structured log character string, and the structured log character string is packaged into a JSON format to be output to a standard output interface. According to the method, the log data can be subjected to in-situ analysis through multiplexing of the annular buffer area, so that resource consumption of the buffer area is reduced, and frequent data copying is reduced.

Description

Technical Field

[0001] This application relates to the field of log processing technology, and more specifically, to a memory-optimized log processing method, apparatus, device, and medium. Background Technology

[0002] In modern microservice architectures, application log management plays a crucial role, serving as an essential component for ensuring stable application operation, troubleshooting, and meeting compliance requirements.

[0003] In related technologies, servers need to allocate and release buffers when receiving messages. When processing log data, data needs to be copied multiple times. However, when there is a lot of log data, frequent memory allocation and release operations will affect performance degradation. Frequent copying of log data will lead to limited container resources and significant overhead. Summary of the Invention

[0004] This application provides a memory-optimized log processing method, apparatus, device, and medium. The method can reuse the circular buffer and parse the log data in place, thereby reducing buffer resource consumption and frequent data copying.

[0005] Firstly, a memory-optimized log processing method is provided, which includes: The container management platform starts the main application container and the collection container, and mounts the shared storage volume to the predetermined directories in the main application container and the collection container, respectively. After the collection container starts, it reads the pre-configured environment variables to obtain the socket path, creates a Unix domain socket file under the socket path of the shared storage volume, binds to and listens to the Unix domain socket file, and sets the access permissions of the Unix domain socket file. After the main application container starts, it executes a pre-defined lifecycle hook function to create a symbolic link within the main application container from the predefined system log interface to the socket path on the shared storage volume. The application in the main application container calls the system logging function to output log data, and the log data is redirected to the Unix domain socket file via the symbolic link; The collection container receives the log data through the Unix domain socket file and writes it into a circular buffer for reuse; the collection container parses the log data in-place within the circular buffer to generate a structured log string. The collection container encapsulates the structured log string into JSON format and outputs it to its standard output interface.

[0006] Using the above method, a symbolic link can be established within the main application container through the container lifecycle management mechanism, from the standard system log interface to the Unix domain socket based on the shared storage volume. This builds a transmission channel for application logs without changing the application code or operating mode. Furthermore, the collection container uses a circular buffer to receive log data and performs in-situ parsing and structuring processing on the log data. That is, by reusing the circular buffer and cyclically using pre-allocated contiguous memory blocks, frequent creation and destruction of temporary memory are avoided. Data is processed directly at the original memory address in the buffer, eliminating the need for multiple copies of data and thus reducing resource consumption. Finally, the structured log string is encapsulated into JSON format and output to its standard output interface, thereby making the log data compatible with different container platforms.

[0007] In conjunction with the first aspect, in some possible implementations, before the collection container receives the log data via the Unix domain socket file and writes it to a circular buffer for reuse, the method further includes: Scan the byte array of the log data and traverse the byte array; If the byte values ​​of the byte array are within a preset range, then the variable-length character encoding of the log data is deemed valid. If the byte value of the byte array is outside the preset range, then the variable-length character encoding of the log data is deemed invalid.

[0008] In conjunction with the first aspect, in some possible implementations, the method further includes: If variable-length character encoding is invalid in the log data, the invalid bytes in the log data are replaced with preset byte fragments, and the log data in the buffer is copied.

[0009] In conjunction with the first aspect, in some possible implementations, the collection container performs in-situ parsing of the log data within the circular buffer to generate a structured log string, including: The log data is read by partitioning based on the streaming protocol of the container management platform. The memory size is dynamically set according to the memory influencing factors of the streaming protocol, including memory usage, system page size, and cache size. A structured log string is generated based on the log data; The log string is stored in the memory.

[0010] In conjunction with the first aspect, in some possible implementations, after the collection container encapsulates the structured log string into JSON format and outputs it to its standard output interface, it further includes: Log strings are collected from each node of the container management platform, and the log strings are mapped in the configuration file; Selectively monitor log strings in the configuration file based on tags; The log string is parsed, and log fields are extracted from the log string. The management unit name, container name, node name, and management unit tag are added to the log fields.

[0011] In conjunction with the first aspect, in some possible implementations, after adding the management unit name, container name, node name, and management unit tag to the log field, the method further includes: Send the log fields to multiple backend storages simultaneously; Detect whether the backend response time exceeds a first threshold; If the backend response time exceeds a first threshold, the initial sending rate is reduced.

[0012] In conjunction with the first aspect, in some possible implementations, sending the log fields to multiple backend storages simultaneously includes: Create a time-series index based on the log field's received time; The log fields are stored on disk in correspondence with the time series index.

[0013] Secondly, a memory-optimized log processing apparatus is provided, the apparatus comprising: The startup and mounting module is used by the container management platform to start the main application container and the collection container, and to mount the shared storage volume to the predetermined directories in the main application container and the collection container, respectively. The module is read and created after the collection container starts, to read pre-configured environment variables to obtain the socket path, create a Unix domain socket file under the socket path of the shared storage volume, bind and listen to the Unix domain socket file, and set the access permissions of the Unix domain socket file. Execute and create a module for executing a pre-defined lifecycle hook function after the main application container starts, and creating a symbolic link within the main application container from a predefined system log interface to the socket path on the shared storage volume; The module is invoked and redirected for applications in the main application container to invoke system logging functions to output log data, which is then redirected to the Unix domain socket file via the symbolic link. A receiving and parsing module is used for the collection container to receive the log data through the Unix domain socket file and write it into a circular buffer for reuse; the collection container parses the log data in-place within the circular buffer to generate a structured log string; The output module is used by the collection container to encapsulate the structured log string into JSON format and output it to its standard output interface.

[0014] Thirdly, an electronic device is provided, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, causing the electronic device to perform the method executed by the memory-optimized log processing method described above.

[0015] Fourthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to execute the method described above for memory-optimized logging.

[0016] Fifthly, a computer-readable storage medium is provided that stores computer program code, which, when run on a computer, causes the computer to perform the method executed by the memory-optimized log processing method described above. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the implementation environment of a memory-optimized log processing method provided in an embodiment of this application; Figure 2 This is a schematic flowchart illustrating a memory-optimized log processing method provided in an embodiment of this application; Figure 3 This is a schematic flowchart illustrating a log collection and processing method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a memory-optimized log processing device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0019] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0020] In the following description of the embodiments of this application, it is used as... Figure 1 Taking an example, the implementation environment of the embodiments of this application will be introduced.

[0021] For example, such as Figure 1 As shown, the implementation environment includes a container management platform 110, a main application container 120, and a collection container 130.

[0022] Kubernetes (110) is an open-source system for automating the deployment, scaling, and management of containerized applications. It manages workloads and networks through a series of abstract objects. For example, the smallest unit of deployment is the Pod. A Pod can contain one or more closely related containers, such as the main application container (120) and the collection container (130).

[0023] The main application container 120 is the business container where the user's application resides. The main application container 120 is used to generate logs. In some embodiments, the user's application includes rtpengine, Asterisk, HAProxy, BIND, PostgreSQL, etc.

[0024] The collection container 130 is a sidecar container specifically responsible for receiving, processing, and outputting logs. In some embodiments, a circular buffer is provided within the collection container 130, from which the collection container 130 can read logs and process them.

[0025] In related technologies, the collection container creates a new circular buffer after each log reception. This necessitates calling the memory allocator each time the log is processed. However, frequent allocation and deallocation of the memory allocator leads to accumulated overhead, increased memory bandwidth consumption, and the inability to handle extremely large log messages using a fixed-size buffer. Furthermore, messages exceeding the buffer size are truncated, resulting in information loss. To address at least one of the above technical problems, this application provides a memory-optimized log processing method. This method can reuse the circular buffer and perform in-place parsing of log data, thereby reducing buffer resource consumption and frequent data copying.

[0026] Figure 2 This is a schematic flowchart of a memory-optimized log processing method provided in an embodiment of this application.

[0027] For example, such as Figure 2 As shown, taking a container management platform as the execution subject as an example, this application describes a memory-optimized log processing method, which includes the following steps.

[0028] Step 201: The container management platform starts the main application container and the collection container, and mounts the shared storage volume to the predetermined directories in the main application container and the collection container, respectively.

[0029] The main application container is the business container where the user's application resides. The collection container is a dedicated sidecar container responsible for receiving, processing, and outputting logs. A shared storage volume (emptyDir) is a temporary storage volume shared between containers. A predefined directory is a directory that both the main application container and the collection container can read. For example, this predefined directory could be / app.

[0030] In some embodiments, the container management platform (Kubernetes) is responsible for creating and starting a Pod. A Pod contains two containers: a main application container and a collection container. The container management platform creates a shared storage volume.

[0031] It should be understood that the main application container and the collection container cannot directly access files or communicate with each other, but logs need to be transferred from the main application container to the collection container. Therefore, a communication channel must be established. In this case, the container management platform mounts a shared storage volume to the two containers to establish a cross-container communication channel, enabling the logs of the main application container to be efficiently transferred to the collection container through Unix Domain Socket, achieving zero-intrusion, high-performance, and highly reliable log collection.

[0032] Step 202: After the collection container starts, it reads the pre-configured environment variables to obtain the socket path, creates a Unix domain socket file under the socket path of the shared storage volume, binds to and listens to the Unix domain socket file, and sets the access permissions of the Unix domain socket file.

[0033] It should be understood that the main application container and the collection container within a Pod cannot directly communicate in memory or share files. Logs from the main application container cannot be directly transmitted to the collection container; they can only be transmitted over the network (TCP / UDP), increasing network overhead. In this situation, a communication bridge between the two containers is implemented based on a shared storage volume, thereby overcoming the isolation limitations between containers.

[0034] In some embodiments, when the collection container starts, it reads the creation path of the Unix domain socket from pre-configured environment variables, such as SOCKET_PATH. A Unix domain socket (UDS) file is created in a shared storage volume shared across containers, and its own log receiving container is bound to that Unix domain socket file.

[0035] It should be understood that Unix domain sockets are a form of local inter-process communication (IPC), unlike TCP / UDP network sockets. They do not rely on a network protocol stack but instead directly implement process communication based on files in the file system. Binding establishes a unique association between the collection container's communication process and this UDS file. After binding, the collection container enters a listening state, continuously waiting for the main application container to send log data through this UDS file.

[0036] In some embodiments, access permissions for Unix domain socket files are configured by setting the access permissions to 0666.

[0037] This access permission is used to remove permission restrictions, allowing the main application container to write log data to the Unix domain socket file. The first digit 0 indicates octal permission format; the second digit 6 indicates that the collection container has read and write permissions; the third digit 6 indicates that the user of the file's group has read and write permissions; the fourth digit 6 indicates that all other users have read and write permissions, for example, the main application container has read and write permissions.

[0038] Step 203: After the main application container starts, it executes a pre-defined lifecycle hook function to create a symbolic link within the main application container from the predefined system log interface to the socket path on the shared storage volume.

[0039] The hook function is used to create symbolic links within the main application container from a predefined system log interface to a Unix domain socket file path on the shared storage volume. The predefined system log interface is the / dev / log interface.

[0040] It should be understood that symbolic links need to be created after the main application container starts. If no corresponding hook is executed before the container starts, the directory inside the main application container may not be initialized and the link may not be created. Therefore, the link creation needs to be completed before the application outputs logs to avoid log loss due to writing logs to the default / dev / log.

[0041] In some embodiments, a symbolic link is created from / dev / log to / app / log.

[0042] In step 204, the application in the main application container calls the system logging function to output log data, which is redirected to the Unix domain socket file via the symbolic link.

[0043] Redirection is used to change the default flow of data through system-level path mapping. The system logging function is syslog().

[0044] It should be understood that the business applications within the main application container can call the Linux system's built-in syslog logging functions and output logs to the system's default logging interface, such as / dev / log. Since the main application container has already created a symbolic link through lifecycle hooks after startup, log data written to / dev / log by the application does not remain on the container's local / dev / log, but is automatically forwarded by the symbolic link to the / app / log path on the shared storage volume.

[0045] Step 205: The collection container receives the log data through the Unix domain socket file and writes it to a circular buffer for reuse; the collection container parses the log data in-place within the circular buffer to generate a structured log string.

[0046] The collection container receives log data. The circular buffer is a reused memory buffer block. A circular buffer is a temporary storage area in computer memory, primarily used as a data transfer station or waiting area between two components with mismatched speeds or different operating rhythms, to smooth data flow and improve overall system efficiency and performance. The collection container can read logs from the circular buffer, where the head and tail pointers are reused cyclically through write and read pointers. In-place parsing means parsing the log directly at the existing memory address in the circular buffer; that is, no data is copied.

[0047] It should be understood that in related technologies, a new buffer is created each time a message is received. For example, processing 1000 messages per second requires 1000 memory allocations and 1000 memory releases, meaning that frequent memory allocation / release leads to severe memory fragmentation and CPU overhead. However, in the embodiments of this application, a circular buffer is created before receiving information, meaning that the same memory space is reused throughout the entire service lifecycle. For example, processing 1000 messages per second requires only 1 memory allocation and 1 memory release, which greatly improves performance and reduces memory management overhead.

[0048] It should be understood that in related technologies, when variable-length character encoding is performed on the log data, the collection container copies the entire log data; for example, for 1MB of log data, 1MB of data needs to be copied. However, in this embodiment, the log data is parsed in-situ to generate a structured log string. This means that the collection container no longer copies the log data in the buffer to other memory areas, but directly parses the raw byte data at the original memory address of the circular buffer, ultimately generating a structured log string such as JSON. In other words, the entire parsing process does not require copying the log data to other memory areas, achieving zero-copy data processing and thus saving bandwidth.

[0049] It should be understood that in related technologies, multiple memory copies can lead to memory fragmentation and double memory usage, for example, copying the same data multiple times. If the log data volume is large and memory allocation or deallocation is frequent, processing latency can spike. However, since the circular buffer is a pre-allocated fixed-size memory block, it does not need to be frequently created and destroyed, avoiding memory fragmentation. In-place parsing eliminates the need to copy data, and each piece of data occupies only one memory space, thus reducing memory waste. Combined with the container management platform's streaming protocol for "partition reading" of log data, structured log strings such as JSON are generated.

[0050] In one possible implementation, the byte array of the log data is scanned and traversed; if the byte values ​​of the byte array are within a preset range, it is determined that the log data is validly encoded using variable-length character encoding; if the byte values ​​of the byte array are outside the preset range, it is determined that the log data is invalidly encoded using variable-length character encoding.

[0051] Variable-length character encoding (UTF-8, 8-bit Universal Character Set / Unicode Transformation Format) is a variable-length character encoding for Unicode. Variable-length character encoding can represent any character in the Unicode standard, and its first byte is compatible with ASCII, allowing software that originally processed ASCII characters to continue using it with little or no modification.

[0052] The preset range is automatically determined by the container management platform, and this application embodiment does not limit it.

[0053] It should be understood that scanning the log byte array in the circular buffer checks whether the byte value is within a preset range and whether the variable-length character encoding is valid, thus avoiding garbled characters during parsing. For example, the range of UTF-8 encoding is 0x00~0xF7.

[0054] In some embodiments, if the read byte value is in the range of 0x00~0x7F, it is determined to be a 1-byte UTF-8 character, and no further byte verification is required. The traversal pointer moves forward 1 bit and continues to verify the next byte. If the read byte value is in the range of 0xC0~0xDF, it is determined to be the first byte of a 2-byte UTF-8 character, triggering the verification of the next 1 byte. If the read byte value is in the range of 0xE0~0xEF, it is determined to be the first byte of a 3-byte UTF-8 character, triggering the verification of the next 2 bytes. If the read byte value is in the range of 0xF0~0xF7, it is determined to be the first byte of a 4-byte UTF-8 character, triggering the verification of the next 3 bytes. If the read byte value exceeds the above range, the byte is directly determined to be invalid UTF-8 encoding, the current verification process is terminated, and an encoding error is marked.

[0055] In this implementation, log sources in the container environment are complex. If the logs are parsed directly without verification, garbled logs are likely to occur, causing log analysis to fail. However, the embodiments of this application verify the log data before parsing, thereby reducing resource waste.

[0056] Optionally, after performing the above steps, the following steps may also be performed.

[0057] In one possible implementation, if the variable-length character encoding of the log data is valid, the log data is parsed in place; if the variable-length character encoding of the log data is invalid, invalid bytes in the log data are replaced with preset byte fragments, and the log data in the buffer is copied.

[0058] It should be understood that in practical applications, directly parsing log data containing invalid bytes can easily lead to the collection container restarting or log collection being interrupted. Therefore, it is necessary to process invalid bytes.

[0059] The preset byte fragment is a fragment automatically determined by the container management platform, and this application embodiment does not limit this.

[0060] For example, if variable-length character encoding is invalid for the log data, replace the invalid bytes in the log data with U+FFFD and copy the log data from the buffer.

[0061] In this implementation, valid encoded bytes are parsed to achieve zero-copy data processing, which avoids copying log data to different areas multiple times, thereby reducing resource consumption. Invalid bytes are uniformly replaced with standard placeholders, thus avoiding container restarts or log collection interruptions, which improves the readability and accuracy of structured logs.

[0062] In one possible implementation, the log data is read by partitioning based on the streaming protocol of the container management platform; the memory size is dynamically set according to the memory influencing factors of the streaming protocol, including memory usage, system page size, and cache size; a structured log string is generated based on the log data; and the log string is stored in the memory.

[0063] Among them, streaming protocols include UDP, Unix Socket, and TCP.

[0064] It should be understood that the circular buffer stores the full log. If the full log is read at once, it will consume a large amount of memory instantly, causing lag. In this case, it is necessary to read the log data in sections.

[0065] For example, in the UDP protocol, the circular buffer is multiplexed to 4MB, and the maximum message reception limit of the UDP protocol is 64KB. However, if the log data is 4MB, reading the log data all at once will cause memory lag. Therefore, the log data is read in sections.

[0066] In some embodiments, the memory capacity is set to a second threshold, and the log data is read based on that memory capacity.

[0067] For example, the second threshold could be 8KB.

[0068] It should be understood that the default page size in Linux systems is 4KB or 8KB, with 8KB being an integer multiple of the page size. This allows for direct use of system memory pages when reading data, reducing the additional memory addressing overhead caused by cross-page reads. The core memory size of a central processing unit is typically 32-64KB, so an 8KB region is much smaller than the size of a circular cache region. If the region is too small, it will lead to too many loop reads; if the region is too large, it will consume too much memory. Therefore, a default memory capacity of 8KB balances memory usage and read efficiency.

[0069] In some embodiments, the memory capacity is expanded from a second threshold to a third threshold based on the memory usage of the container management platform, the system page size, and the cache size.

[0070] It should be understood that in practical applications, the second threshold of memory capacity may not be sufficient to meet the needs of reading log data, so it is necessary to expand the memory capacity.

[0071] In some embodiments, the second threshold is expanded by a power of 2, that is, the memory capacity is doubled.

[0072] In this implementation, when the remaining capacity is insufficient, the capacity is expanded on demand rather than allocating a large amount of memory at once, thereby reducing memory waste.

[0073] Step 206: The collection container encapsulates the structured log string into JSON format and outputs it to its standard output interface.

[0074] The standard output interface is the stdout interface.

[0075] It should be understood that because the format of log strings may be inconsistent and the adaptability is relatively poor, the collection container encapsulates the processed log strings into JSON format and outputs them to the container's standard output interface for native collection by the container management platform, thereby obtaining log data in a unified format.

[0076] This application provides a memory-optimized log processing method. This method establishes a symbolic link within the main application container, from the standard system log interface to a Unix domain socket based on a shared storage volume, through the container lifecycle management mechanism. This constructs a transmission channel for application logs without altering the application code or runtime behavior. Furthermore, the collection container uses a circular buffer to receive log data and performs in-situ parsing and structuring processing on the log data. Specifically, by reusing the circular buffer and cyclically using pre-allocated contiguous memory blocks, frequent creation and destruction of temporary memory are avoided. Data is processed directly at the original memory address in the buffer, eliminating the need for multiple data copies and reducing resource consumption. Finally, the structured log string is encapsulated into JSON format and output to its standard output interface, thus ensuring log data compatibility across different container platforms.

[0077] Figure 3 This is a schematic flowchart of a log collection and processing method provided in an embodiment of this application.

[0078] For example, such as Figure 3 As shown, taking a container management platform as the execution subject as an example, this application describes a log collection and processing method 300, which includes the following steps.

[0079] Step 301: Collect log strings.

[0080] In one possible implementation, log strings are collected from various nodes of the container management platform and mapped to a configuration file; log strings in the configuration file are selectively monitored based on tags; the log strings are parsed, and log fields are extracted from them, with the management unit name, container name, node name, and management unit tag added to the log fields.

[0081] These include Pod tags, node tags, and container tags.

[0082] It should be understood that the cluster nodes and the number of containers in the container management platform are large. Collecting and processing all logs would consume a lot of resources. In this case, monitoring target logs can be filtered by mapping configuration files and using tags to reduce invalid data processing and resource overhead.

[0083] In some embodiments, logs that conform to specified tag rules are selectively monitored based on the tag attributes of the logs in the configuration file. Examples include Pod tags, node tags, and container tags. The filtered log strings are then parsed for formatting; metadata is proactively added to the parsed log fields to form a complete structured log.

[0084] Among them, the management unit name, container name, node name, and management unit label are metadata.

[0085] In this implementation, selective monitoring using tags reduces a large amount of invalid log processing. Since the log formats output by different management units and containers are chaotic, the core fields are extracted by parsing the logs first, and then data such as management unit name, container name, node name and management unit tag are added to achieve log format standardization.

[0086] Step 302: Store the log string.

[0087] In one possible implementation, the log field is sent to multiple backend storages simultaneously; the response time of the backend is detected to exceed a first threshold; if the response time of the backend exceeds the first threshold, the initial sending rate is reduced.

[0088] The backend includes Elasticsearch, Kafka, and Loki. The first threshold is automatically determined by the container management platform, and this embodiment does not limit it.

[0089] In some embodiments, a time series index is created according to the time the log field is received; the log field and the time series index are stored on disk.

[0090] The time series index is used to query logs for different time periods.

[0091] It should be understood that a single backend cannot meet the needs of multiple scenarios. Therefore, the log field is sent to multiple backend storages at the same time to avoid duplicate collection and improve the adaptability of the log field.

[0092] In some embodiments, the log aggregator records the response time of each backend when sending logs, such as HTTP request time, Kafka production time, etc.

[0093] It should be understood that if the backend response time exceeds the first threshold, the backend will be overloaded. To avoid the above problems, the initial sending rate needs to be reduced.

[0094] In some embodiments, multiple log entries are accumulated and then sent in batches to reduce network requests. The batch logs are compressed using gzip / snappy, thereby saving bandwidth.

[0095] In this implementation, log fields are sent to multiple backends simultaneously, and the sending rate is dynamically adjusted based on the backend response time to avoid backend overload and improve log transmission reliability. Multi-backend distribution meets different business needs, avoids duplicate data collection, and improves overall system stability and log processing efficiency.

[0096] This application provides a log collection and processing method. This method collects logs from various nodes and maps them to configuration files. It performs selective monitoring based on tags, parses logs, and supplements them with metadata such as management units, containers, and nodes, achieving structured and traceable processing. Logs are simultaneously sent to multiple backends, and the processing speed is dynamically reduced based on response time to avoid backend overload and improve transmission reliability and system stability.

[0097] It should be understood that the embodiments of this application have verified the performance advantages under different load conditions through test scenarios.

[0098] In high-frequency, low-message scenarios, related technologies result in high CPU usage and large memory peaks due to frequent memory allocation and data copying. However, the embodiments of this application reduce the number of memory allocations from 600,000 to 1, and the amount of data copying to zero through buffer reuse and zero-copy technology. This reduces CPU utilization and memory peaks by 57% and 93% respectively, significantly improving resource efficiency.

[0099] In large message processing scenarios, related technologies often result in message truncation due to a fixed buffer size that is too small. This application embodiment uses a dynamic buffer mechanism to receive large messages completely while maintaining low CPU utilization, thus ensuring message integrity.

[0100] In mixed load scenarios, related technologies suffer from a 5.2% message loss rate and high latency due to large fluctuations in message size. The embodiments of this application demonstrate that zero loss is achieved through adaptive buffering and zero-copy processing, reducing average latency by 73% and increasing throughput by 5%, while maintaining stability and reliability under complex loads.

[0101] Figure 4 This is a schematic diagram of the structure of a memory-optimized log processing device provided in an embodiment of this application.

[0102] For example, the device 400 includes: Start and mount module 401, which is used by the container management platform to start the main application container and the collection container, and mount the shared storage volume to the predetermined directories in the main application container and the collection container respectively. Module 402 is used to read pre-configured environment variables to obtain the socket path after the collection container starts, create a Unix domain socket file under the socket path of the shared storage volume, bind and listen to the Unix domain socket file, and set the access permissions of the Unix domain socket file. Execute and create module 403, which, after the main application container starts, executes a pre-defined lifecycle hook function to create a symbolic link within the main application container from the predefined system log interface to the socket path on the shared storage volume; Module 404 is invoked and redirected so that the application in the main application container can call the system logging function to output log data, which is then redirected to the Unix domain socket file via the symbolic link. The receiving and parsing module 405 is used by the collection container to receive the log data through the Unix domain socket file and write it into a circular buffer for reuse; the collection container parses the log data in-place within the circular buffer to generate a structured log string. Output module 406 is used by the collection container to encapsulate the structured log string into JSON format and output it to its standard output interface.

[0103] In one possible implementation, the receiving and parsing module 405 is further configured to: Scan the byte array of the log data and iterate through the byte array; If the byte values ​​of the byte array are within the preset range, then the log data is determined to be validly encoded using variable-length character encoding. If the byte value of the byte array is outside the preset range, it is determined that the log data is invalid for variable-length character encoding.

[0104] In one possible implementation, the receiving and parsing module 405 is further configured to: If variable-length character encoding is invalid for the log data, replace the invalid bytes in the log data with a preset byte fragment and copy the log data in the buffer.

[0105] In one possible implementation, the receiving and parsing module 405 is further configured to: The log data is read by partition based on the streaming protocol of the container management platform; The memory size is dynamically set according to the memory-influencing factors of the streaming protocol, including memory usage, system page size, and cache size. A structured log string is generated based on this log data; Store the log string in this memory.

[0106] In one possible implementation, the output module 406 is further configured to: Log strings are collected from various nodes of the container management platform and mapped to configuration files. Selectively monitor log strings in this configuration file based on tags; Parse the log string and extract the log fields from it, then add the management unit name, container name, node name, and management unit label to the log fields.

[0107] In one possible implementation, the output module 406 is further configured to: Send this log field to multiple backend storages simultaneously; Check if the backend response time exceeds the first threshold; If the backend response time exceeds the first threshold, reduce the initial sending rate.

[0108] In one possible implementation, the output module 406 is further configured to: Create a time-series index based on the time the log field was received; Store the log field and the corresponding time series index to disk.

[0109] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0110] For example, such as Figure 5 As shown, the electronic device 500 includes a memory 501 and a processor 502. The memory 501 stores executable program code 503, and the processor 502 is used to call and execute the executable program code 503 to perform a memory-optimized log processing method.

[0111] Furthermore, embodiments of this application also protect an apparatus that may include a memory and a processor, wherein the memory stores executable program code, and the processor is used to call and execute the executable program code to perform a memory-optimized log processing method provided in embodiments of this application.

[0112] This embodiment can divide the device into functional modules based on the above method example. For example, each module can correspond to a separate function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0113] It should be understood that the apparatus provided in this embodiment is used to execute the above-described memory-optimized log processing method, and therefore can achieve the same effect as the above-described implementation method.

[0114] When using integrated units, the device may include a processing module and a storage module. When applied to an electronic device, the processing module can be used to control and manage the operation of the electronic device. The storage module can be used to support the execution of relevant program code by the electronic device.

[0115] The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits shown in conjunction with the disclosure of this application. The processor may also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and microprocessors, etc., and the storage module may be a memory.

[0116] In addition, the apparatus provided in the embodiments of this application may specifically be a chip, component or module. The chip may include a connected processor and a memory. The memory is used to store instructions. When the processor calls and executes the instructions, the chip can execute a memory-optimized log processing method provided in the above embodiments.

[0117] This embodiment also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the aforementioned related method steps to implement a memory-optimized log processing method provided in the above embodiment.

[0118] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement a memory-optimized log processing method provided in the above embodiment.

[0119] In this embodiment, the device, computer-readable storage medium, computer program product, or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0120] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0121] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0122] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for memory-optimized log processing, the method comprising: The method comprises: The container management platform starts a main application container and a collection container, and mounts a shared storage volume to a predetermined directory in the main application container and the collection container respectively; After the collection container is started, a pre-configured environment variable is read to obtain a socket path, a Unix domain socket file is created at the socket path of the shared storage volume, the Unix domain socket file is bound and listened to, and the access permission of the Unix domain socket file is set; After the main application container is started, a preset life cycle hook function is executed, and a symbolic link from a predefined system log interface to the socket path on the shared storage volume is created in the main application container; An application program in the main application container calls a system log function to output log data, and the log data is redirected to the Unix domain socket file via the symbolic link; The collection container receives the log data through the Unix domain socket file, and writes the log data into a ring buffer for multiplexing; the collection container performs in-place parsing on the log data in the ring buffer to generate a structured log string; The collection container encapsulates the structured log string into a JSON format and outputs the structured log string to a standard output interface thereof.

2. The method of claim 1, wherein, Before the collection container receives the log data through the Unix domain socket file and writes the log data into a ring buffer for multiplexing, the method further comprises: scanning a byte array of the log data and traversing the byte array; if a byte value of the byte array is within a preset range, it is determined that the log data is valid for variable-length character encoding; if the byte value of the byte array is outside the preset range, it is determined that the log data is invalid for variable-length character encoding.

3. The method of claim 2, wherein, The method further comprises: in the case that the log data is invalid for variable-length character encoding, replacing an invalid byte in the log data with a preset byte segment and copying the log data in the buffer.

4. The method of claim 1, wherein, The collection container performs in-place parsing on the log data in the ring buffer to generate a structured log string, which comprises: reading the log data based on a partition of a streaming protocol of the container management platform; dynamically setting a size of memory according to a memory influencing factor of the streaming protocol, the memory influencing factor comprising memory occupation, system page size and cache size; generating a structured log string based on the log data; storing the log string into the memory.

5. The method of claim 1, wherein, After the collection container encapsulates the structured log string into a JSON format and outputs the structured log string to a standard output interface thereof, the method further comprises: collecting log strings from each node of the container management platform and mapping the log strings in a configuration file; selectively monitoring the log strings in the configuration file according to a tag; parsing the log strings and extracting a log field from the log strings, and adding a management unit name, a container name, a node name and a management unit tag to the log field.

6. The method of claim 5, wherein, After the management unit name, the container name, the node name, and the management unit tag are added to the log field, further comprising: simultaneously sending the log field to a plurality of backend storages; detecting whether the backend response time exceeds a first threshold; in a case where the backend response time exceeds the first threshold, reducing an initial sending rate.

7. The method of claim 6, wherein, The simultaneously sending the log field to a plurality of backend storages, comprising: creating a time series index according to the log field receiving time; storing the log field corresponding to the time series index to a disk.

8. A memory-optimized log processing apparatus, characterized by, The apparatus comprises: a starting and mounting module, configured to start a master application container and a collection container of a container management platform, and mount a shared storage volume to a predetermined directory in the master application container and the collection container respectively; a reading and creating module, configured to, after the collection container is started, read a pre-configured environment variable to obtain a socket path, create a Unix domain socket file under the socket path of the shared storage volume, bind and listen to the Unix domain socket file, and set an access permission of the Unix domain socket file; an executing and creating module, configured to, after the master application container is started, execute a pre-set life cycle hook function, and create a symbolic link from a predefined system log interface to the socket path on the shared storage volume in the master application container; a calling and redirecting module, configured to, when an application program in the master application container calls a system log function to output log data, the log data is redirected to the Unix domain socket file via the symbolic link; a receiving and parsing module, configured to, when the collection container receives the log data through the Unix domain socket file, write the log data to a ring buffer for multiplexing, and perform in-place parsing on the log data in the ring buffer to generate a structured log string; an output module, configured to, when the collection container encapsulates the structured log string into a JSON format, and outputs the structured log string to a standard output interface of the collection container.

9. An apparatus, comprising: The device comprises: a memory configured to store executable program code; a processor configured to invoke and run the executable program code from the memory, so that the device performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, when the computer program is executed, the method according to any one of claims 1 to 7 is realized. The computer readable storage medium stores a computer program, when the computer program is executed, the method according to any one of claims 1 to 7 is realized.

Images