Log file uploading method and device, computer equipment and storage medium

Abstract

Relates to the technical field of data processing, and discloses a log file uploading method and device, computer equipment and a storage medium, and the log file uploading method comprises the steps that a plurality of continuous original data blocks are read from log files of a disk in a streaming mode through a reading thread, and the streaming reading comprises the step of reading the log files with the data blocks as units in sequence; the multiple original data blocks are sequentially written into an input buffer area, corresponding processing threads are distributed to the original data blocks sequentially written into the input buffer area, parallel processing is conducted on the original data blocks through the processing threads, and candidate data blocks are obtained; the candidate data blocks are sequentially written into an output buffer area, the candidate data blocks in the output buffer area are uploaded to an object storage service through an uploading thread, and the uploading thread feeds back a control signal to a reading thread by monitoring the occupancy state of the buffer area so that the reading thread can dynamically adjust the data reading behavior. The problems of resource consumption centralization, disk space waste and high processing delay in a traditional log uploading scheme are solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a method, apparatus, computer equipment, and storage medium for uploading log files. Background Technology

[0002] In log collection scenarios, the transmission module needs to upload scattered log files to Object Storage Service (OSS). Traditional solutions employ a centralized preprocessing and full-package upload model: first, all log files in the target directory are compressed, and then the complete archive is uploaded to OSS. This approach requires completely reading the original files and performing intensive compression calculations before network transmission.

[0003] However, traditional solutions have significant shortcomings: the preprocessing stage involves concentrated compression calculations, resulting in high peak CPU utilization, which can easily lead to resource contention and service blockage; double the disk space needs to be reserved to store the original files and archived files, resulting in low disk utilization and wasted resources; and the preprocessing stage becomes a performance bottleneck, as uploading requires waiting for the archiving to complete, resulting in high end-to-end latency and poor scalability. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a log file uploading method, apparatus, computer equipment, and storage medium to solve the problems of centralized resource consumption, wasted disk space, and high processing latency in traditional log uploading schemes.

[0005] In a first aspect, embodiments of the present invention provide a log file uploading method, the method comprising: A read thread streams multiple consecutive raw data blocks from a disk log file, wherein the streaming read includes reading the log file sequentially in units of data blocks until the end of the file; Multiple raw data blocks are sequentially written into an input buffer, and a corresponding processing thread is assigned to each raw data block sequentially written into the input buffer. The raw data blocks are then processed in parallel by the processing thread to obtain candidate data blocks. The candidate data blocks are written sequentially to the output buffer, and the candidate data blocks in the output buffer are uploaded to the object storage service through the upload thread. The upload thread feeds back control signals to the read thread by monitoring the occupancy status of each buffer, so that the read thread can dynamically adjust its data reading behavior.

[0006] Furthermore, the step of allocating corresponding processing threads to the original data blocks written sequentially in the input buffer, and performing parallel processing on the original data blocks through the processing threads to obtain candidate data blocks includes: The data block type of the original data block is identified; Based on the data block type, a corresponding processing thread is assigned to the original data block, and the original data block is processed according to the processing strategy associated with the processing thread to obtain candidate data blocks.

[0007] Furthermore, the step of allocating a corresponding processing thread to the original data block according to the data block type, and processing the original data block according to the processing strategy associated with the processing thread to obtain candidate data blocks includes: If the original data block is the starting block, a first processing thread is allocated to the original data block, and an archive file header is dynamically generated according to the first processing strategy associated with the first processing thread. The archive file header is then concatenated with the original data block to obtain a first candidate data block. If the original data block is an intermediate block, a second processing thread is allocated to the original data block, and the original data block is used as a second candidate data block according to the second processing strategy associated with the second processing thread. If the original data block is the end block, a third processing thread is allocated to the original data block, and the original data block is compressed according to the third processing strategy associated with the third processing thread to obtain a compressed data block. An archive file tail and padding data of a specific length are appended to the compressed data block to generate a third candidate data block.

[0008] Furthermore, the method also includes: Monitor the storage status of the output buffer; According to the storage state, a control signal corresponding to the storage state is sent to the reading thread so that the reading thread performs the corresponding reading operation according to the control signal.

[0009] Furthermore, the step of sending a control signal corresponding to the storage state to the reading thread based on the storage state, so that the reading thread performs a corresponding read operation according to the control signal, includes: If the storage state is idle, a first control signal is sent to the reading thread so that the reading thread reads the original data block at a specific rate based on the first control signal; If the storage state is an intermediate state, then the target read mode is determined according to the storage state, and a second control signal corresponding to the target read mode is sent to the read thread, so that the read thread reads the original data block according to the target read mode based on the second control signal; If the storage state is saturated, a third control signal is sent to the read thread to cause the read thread to pause reading the original data block based on the third control signal.

[0010] Furthermore, the step of uploading the candidate data blocks in the output buffer to the object storage service via the upload thread includes: A corresponding checksum is appended to the candidate data block to generate the target data block; The target data blocks are uploaded to the object storage service block by block, and the upload progress of the target data blocks is obtained; When the upload process is in a successful phase, the offset and checksum of the target data block are stored; or, when the upload process is in a failed phase, a retransmission operation is performed on the target data block.

[0011] Furthermore, the reading thread, the processing thread, and the uploading thread execute in parallel and exchange data through a data buffer, wherein the data buffer is an input buffer connecting the reading thread and the processing thread, and / or an output buffer connecting the processing thread and the uploading thread.

[0012] Secondly, embodiments of the present invention provide a log file uploading device, the device comprising: A read module is used to stream multiple consecutive raw data blocks from a log file on a disk via a read thread, wherein the streaming read includes reading the log file sequentially in units of data blocks until the end of the file; The processing module is used to write multiple raw data blocks into an input buffer in sequence, and to allocate corresponding processing threads to the raw data blocks written into the input buffer in sequence, and to process the raw data blocks in parallel through the processing threads to obtain candidate data blocks; An upload module is used to sequentially write the candidate data blocks into an output buffer, and upload the candidate data blocks in the output buffer to an object storage service through an upload thread. The upload thread monitors the occupancy status of each buffer and feeds back control signals to the read thread so that the read thread can dynamically adjust its data reading behavior.

[0013] Thirdly, embodiments of the present invention provide a computer device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method described in the first aspect or any corresponding embodiment thereof.

[0014] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions that cause a computer to perform the method described in the first aspect or any corresponding embodiment thereof.

[0015] The method provided in this application has the following beneficial effects: The method provided in this application reads multiple consecutive raw data blocks from a disk log file using a read thread, achieving block-based streaming data supply. This avoids the disk space waste and initial waiting delays caused by the need to pre-read the entire file in traditional solutions. By sequentially writing multiple raw data blocks into an input buffer and assigning corresponding processing threads to process the sequentially written raw data blocks in parallel, the traditional centralized high-intensity compression calculation is decomposed into multiple refined data block-level tasks. This achieves parallelization and pipelining of processing tasks, effectively distributing CPU load, eliminating resource consumption peaks, and avoiding service blocking. By sequentially writing candidate data blocks into an output buffer and having an upload thread upload the candidate data blocks in the output buffer to an object storage service, data processing and network transmission are decoupled at the operational level. This allows data packaging and upload operations to overlap, thereby significantly reducing end-to-end latency. In addition, by monitoring the occupancy status of each buffer and feeding back control signals to the reading thread, the reading thread dynamically adjusts its data reading behavior, thus constructing a dynamic flow control mechanism based on the buffer status. This effectively prevents buffer overflow or empty reads, ensuring stable operation and resource utilization efficiency under continuous high load, and achieving low latency, high throughput, and smooth resource consumption in log uploads. Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a log file upload method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating another log file upload method according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating another log file upload method according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a log file upload system according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the workflow of a log file upload system according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the workflow of another log file upload system according to an embodiment of the present invention; Figure 7This is a structural block diagram of a log file uploading device according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] According to embodiments of the present invention, a log file uploading method, apparatus, computer device, and storage medium are provided. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0020] This embodiment provides a method for uploading log files. Figure 1 This is a flowchart of a log file upload method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: A series of raw data blocks are streamed from the log file on the disk by a read thread. The streaming read includes reading the log file sequentially in units of data blocks until the end of the file.

[0021] In this embodiment, the read thread, as an independent thread in the pipelined parallel processing architecture, is responsible for sequentially reading log file data from the disk and dividing the file into multiple raw data blocks using a configurable data block size (default 2MB) as the basic processing unit. These raw data blocks are the unprocessed log content. The read thread optimizes performance by reducing I / O operations and ensuring real-time processing. Simultaneously, the read thread has the ability to perceive downstream control signals in real time, continuously monitor the status of the input buffer (such as a memory circular buffer), and dynamically adjust its reading behavior (such as normal rate, intermittent reading, or pause) according to the control signals, ensuring a smooth data stream injection into the pipeline. The read thread uses a lock-free mechanism to exchange data with subsequent modules, providing a stable data source for the entire streaming upload process.

[0022] In step S102, multiple raw data blocks are written into the input buffer in sequence, and corresponding processing threads are assigned to the raw data blocks written into the input buffer in sequence. The raw data blocks are processed in parallel by the processing threads to obtain candidate data blocks.

[0023] In this embodiment, multiple processing threads operate in parallel as independent computing units, with the core function being the implementation of a streaming packaging / compression engine. Threads in the processing thread pool are dynamically allocated by the scheduler. Each thread independently obtains a raw data block from the input buffer and then determines whether it is a start block, intermediate block, or end block based on its position in the source file. The processing strategy corresponding to the data block type is then applied: if it is a start block, a 512-byte archive header conforming to the POSIX TAR standard is dynamically generated, and this header is then concatenated with the raw data block; if it is an intermediate block, the raw data block is directly used as a candidate data block without additional encapsulation; if it is an end block, the raw data block is first stream-compressed, and then an archive header and padding data of a specific length (e.g., 1024 bytes) are appended to the end of the compressed data block to form a complete TAR archive format. By employing this strategy of differentiated processing by block type combined with multi-threaded parallel execution, multiple data blocks can be processed simultaneously, thereby effectively distributing the concentrated high-intensity computational load. This also achieves the key advantage of starting subsequent pipeline transmissions without waiting for the entire file, generating candidate data blocks that can be directly used for uploading.

[0024] In step S103, candidate data blocks are written to the output buffer in sequence, and the candidate data blocks in the output buffer are uploaded to the object storage service through the upload thread. The upload thread feeds back control signals to the read thread by monitoring the occupancy status of each buffer, so that the read thread can dynamically adjust the data reading behavior.

[0025] In this embodiment, the upload thread, as an independent network I / O unit, first calculates and appends a checksum (such as a CRC64 checksum) to each candidate data block obtained from the output buffer, generating a complete target data block. This checksum is used to verify the integrity of data transmission on the server side. Next, using an HTTP-based chunked encoding mechanism, each target data block is uploaded to OSS as an independent HTTP chunk. During this process, the upload thread continuously monitors network bandwidth and round-trip time (RTT) and dynamically adjusts the size of the HTTP chunk accordingly (e.g., using a large chunk to increase throughput when bandwidth is high and latency is low, and a small chunk to reduce latency when bandwidth is low) to optimize transmission efficiency. Simultaneously, the upload process of each target data block is acquired and monitored: if the upload is successful, the offset of the target data block in the file and its checksum are recorded locally, enabling the ability to resume interrupted uploads; if the upload fails (e.g., the server checksum verification fails), a retransmission operation is triggered only for the specific failed target data block, rather than retransmitting the entire file, greatly improving retransmission efficiency.

[0026] Furthermore, the core responsibility of the upload thread also includes performing dynamic flow control: it continuously monitors the occupancy status of the input and output buffers (such as idle, critical, or fully loaded). Based on this status, the upload thread generates corresponding control signals (such as "accelerate," "decelerate," or "pause") and feeds them back to the read thread. The read thread then dynamically adjusts its data reading rate or behavior accordingly, thereby preventing buffer overflows at the pipeline source and ensuring smooth transitions and speed matching between data processing and network transmission. This entire process ensures the reliability, efficiency, and resilience of network transmission, and through feedback control mechanisms, guarantees stable operation and low-latency performance of the entire system under load fluctuations.

[0027] In this embodiment of the application, a corresponding processing thread is allocated to the original data blocks written sequentially in the input buffer. The original data blocks are processed in parallel by the processing thread to obtain candidate data blocks, including the following steps A1-A2: Step A1: Identify the data block type of the original data block.

[0028] Specifically, after each processing thread retrieves a raw data block from the input buffer, it internally maintains a processing state machine (or a simple block counter) for the current file. It dynamically determines the data block's type by tracking whether it is the first, middle, or last block of the file. There are three specific data block types: start block (the first block of data in the file), middle block (a continuous block of data in the file that is neither the first nor the last block), and end block (the last block of data in the file, which may be smaller than a preset block size). This identification mechanism is performed in real-time, without needing to know the size or content of the entire file beforehand. This provides a crucial basis for subsequently allocating corresponding processing threads according to type, supporting the parallel and differentiated application of different processing strategies based on type (such as dynamically generating TAR headers or appending TAR tails), achieving pipelined operation and avoiding the delay of waiting for the entire file to be read before processing.

[0029] Step A2: Assign a corresponding processing thread to the original data block according to the data block type, and process the original data block according to the processing strategy associated with the processing thread to obtain candidate data blocks.

[0030] Based on the data block type, corresponding first, second, or third specially configured processing threads are assigned to the starting block, intermediate block, and ending block for execution. Identifying the data block type allows the system to adaptively distinguish the characteristics of different data blocks, providing a foundation for precise scheduling and differentiated processing. Furthermore, assigning corresponding processing threads according to the data block type and executing their associated strategies to process the original data blocks ensures that each type of data receives optimal processing. This achieves precise allocation and load balancing of computing resources, reduces unnecessary computational overhead, and thus improves overall processing speed and quality, making the generation of candidate data blocks more accurate and efficient.

[0031] In this embodiment of the application, a corresponding processing thread is allocated to the original data block according to the data block type, and the original data block is processed according to the processing strategy associated with the processing thread to obtain candidate data blocks, including the following steps A21-A23: Step A21: If the original data block is the starting block, then allocate a first processing thread to the original data block, dynamically generate an archive file header according to the first processing strategy associated with the first processing thread, and concatenate the archive file header with the original data block to obtain the first candidate data block.

[0032] Specifically, when the current raw data block is determined to be the starting block of a file, a first processing thread is allocated to the raw data block, and the first processing strategy associated with the first processing thread is triggered. The core of the first processing strategy is to dynamically generate an archive file header. That is, the first processing thread constructs a 512-byte file header structure conforming to the POSIX TAR standard in memory in real time. This structure is filled with metadata from the file system, including filename, file size, timestamp (last modified time), and file permissions. Subsequently, the thread calculates the checksum of the header structure content and fills the result into a specific field in the header. The header generation process does not require knowing the total file size or waiting for the file to be read completely, achieving real-time processing. Next, the first processing thread performs a concatenation operation, using the dynamically generated archive file header (512 bytes) as a prefix and the raw data content of the starting block as a suffix, sequentially concatenating them in memory to form the first candidate data block.

[0033] Step A22: If the original data block is an intermediate block, then a second processing thread is allocated to the original data block, and the original data block is used as a second candidate data block according to the second processing strategy associated with the second processing thread.

[0034] Specifically, when the system determines, based on its maintained internal state (such as a block counter), that the raw data block currently retrieved from the input buffer is an intermediate block (i.e., the data block is neither the first nor the last block of the file), a second processing thread is allocated to the raw data block, and a second processing strategy associated with the second processing thread is applied. The core of this second processing strategy is pass-through or zero-addition processing: the second processing thread does not dynamically generate any header information as the first processing thread does with the starting block; instead, it directly treats the raw data block, without structural modification, as a candidate data block. This indicates that the content and length of the data block remain unchanged; it is only re-marked or encapsulated into a candidate data block format recognizable by downstream pipeline modules (such as the compression engine and upload module), and then immediately written to the output buffer.

[0035] Step A23: If the original data block is the end block, then a third processing thread is allocated to the original data block, and the original data block is compressed according to the third processing strategy associated with the third processing thread to obtain a compressed data block. An archive file tail and padding data of a specific length are appended to the compressed data block to generate a third candidate data block.

[0036] Specifically, when the current raw data block is determined to be the end of the file, a third processing thread is allocated to it, triggering the associated third processing strategy. The core of this strategy lies in appending tail information and compression: First, the raw data block is stream-compressed, using a mode like Z_BEST_SPEED from the zlib library, with a fixed 32KB sliding dictionary window for incremental compression and hash table updates. After compression, a compressed data block is obtained. Next, the third processing thread appends an archive file tail to the compressed data block, consisting of two consecutive 512-byte blocks of all zeros, as a standard marker for the end of the file in TAR format. Then, padding data of a specific length (e.g., 1024 bytes of all zeros) is appended to ensure the entire archive file conforms to the POSIXTAR standard requirement (i.e., a total length that is a multiple of 512 bytes). After this series of operations, the final candidate data block not only contains the compressed content of the last part of the file but also has a complete archive format tail, indicating that the file has been processed in the streaming pipeline and can be finally transmitted by the upload thread.

[0037] By assigning corresponding processing threads to different types of data blocks and implementing differentiated strategies: for the starting block, the step of dynamically generating and concatenating the archive file header ensures the integrity and recognizability of the file, allowing streaming processing to begin immediately without waiting for the entire file, thereby reducing end-to-end latency; for intermediate blocks, they are directly used as candidate data blocks, and the second processing thread performs pass-through processing, minimizing computational overhead and improving overall processing efficiency and parallelism; for the ending block, the third processing thread compresses the data and appends the archive file tail and padding data, effectively reducing data volume, saving storage space and upload bandwidth, while ensuring the correct format of the archive file.

[0038] In the embodiments of this application, such as Figure 2 As shown, the method also includes: Step S201: Monitor the storage status of the output buffer.

[0039] In this embodiment, the monitoring mechanism periodically (e.g., every millisecond) or event-drivenally checks the current write and read positions of the circular queue to calculate the ratio of the queue's used capacity to its total capacity, thereby accurately quantifying its storage status. This status is typically categorized into three levels: idle (data volume below the low watermark, indicating strong downstream consumption and potential for upstream acceleration), intermediate (data volume between the high and low watermarks, indicating a basic balance between supply and demand), and saturated (data volume reaching or exceeding the high watermark, indicating insufficient downstream consumption and a risk of congestion). The monitoring process considers not only the queue's static fill rate but also real-time system metrics such as network bandwidth fluctuations, CPU load on upload threads, and network round-trip latency to dynamically assess and fine-tune the status. For example, even if the queue is not full, a sudden drop in network speed may trigger the monitoring module to preemptively determine that the status is approaching saturation. This continuous monitoring activity provides a basis for triggering precise control signals and is fundamental to the effective operation of the entire control mechanism.

[0040] Step S202: Send a control signal corresponding to the storage state to the reading thread according to the storage state, so that the reading thread can perform the corresponding reading operation according to the control signal.

[0041] By monitoring the storage status, the system can perceive the buffer load in real time and adjust the data flow accordingly. Based on the status, control signals are sent to the read thread, enabling the read thread to dynamically respond to the buffer status and perform corresponding read operations. This ensures the smoothness of the data flow and the stability of the system, and improves resource utilization and overall efficiency.

[0042] In this embodiment of the application, a control signal corresponding to the storage state is sent to the reading thread according to the storage state, so that the reading thread performs the corresponding reading operation according to the control signal, including the following steps B1-B3: Step B1: If the storage state is idle, a first control signal is sent to the read thread so that the read thread reads the original data block at a specific rate based on the first control signal.

[0043] Specifically, when the monitoring module (usually concurrently performed by the upload thread) determines that the output buffer is in an idle state (i.e., the amount of data in the buffer is below the set low watermark, indicating smooth downstream network transmission and data consumption capacity far exceeding supply), it immediately generates and asynchronously sends a first control signal (this signal corresponds to the normal operating mode; the first control signal is an acceleration / demand signal). The read thread continuously listens for such signals. Upon receiving the first control signal, it interprets it as an instruction that the downstream processing capacity is sufficient and can run at full speed. Therefore, the read thread enters an efficient data ingestion mode, reading raw data blocks at a specific rate. This specific rate typically refers to its maximum achievable sustainable I / O rate (e.g., continuous sequential disk reads at a default block size of 2MB), injecting data into the pipeline as quickly as possible without introducing artificial delays or throttling.

[0044] Step B2: If the storage state is an intermediate state, the target read mode is determined according to the storage state, and the second control signal corresponding to the target read mode is sent to the read thread so that the read thread reads the original data block according to the target read mode based on the second control signal.

[0045] Specifically, when the monitoring module determines that the storage state of the output buffer is in an intermediate state (i.e., the data volume is between the preset low and high water level lines, indicating mild congestion downstream and a tightening supply-demand relationship), it does not send a single signal. Instead, it first determines the target read mode based on the storage state. This determination process is a dynamic decision-making process: the monitoring module further analyzes the trend of the storage state (such as whether the fill rate is rising rapidly or falling slowly) and the current system load (such as network latency and CPU utilization), thereby selecting a target read mode from predefined strategies (for example, this mode corresponds to the intermittent read mode at the WARNING level). Subsequently, a second control signal (the second control signal is an adjustment signal) precisely corresponding to this mode is generated and sent. After receiving the second control signal, the reading thread parses the target reading mode instruction contained therein and dynamically adjusts its behavior accordingly. For example, it switches from full-speed continuous reading to reading raw data blocks according to the target reading mode. The specific manifestation of this mode can be: actively pausing for a short time interval (such as several milliseconds) after reading a data block, and then reading the next block, thereby periodically and rhythmically injecting data into the pipeline and actively reducing the data supply rate to assist the downstream packaging / compression and upload modules in digesting the existing data backlog in the buffer.

[0046] Step B3: If the storage state is saturated, a third control signal is sent to the read thread so that the read thread suspends reading the original data block based on the third control signal.

[0047] Specifically, when the monitoring module (such as the upload thread) determines that the output buffer's storage state is saturated (i.e., the buffer's data volume has reached or exceeded the preset high watermark, indicating severe downstream congestion, and the data consumption capacity can no longer keep up with the production speed; continued writing will lead to data loss or memory exhaustion), it immediately generates and sends the highest priority third control signal (this signal corresponds to the CRITICAL severe congestion state; the third control signal is a back pressure signal). The read thread continuously listens to the signal channel. Upon receiving the third control signal, it interprets it as an emergency instruction to immediately stop production. Therefore, the read thread pauses reading raw data blocks: interrupting the currently ongoing or upcoming disk I / O operation, entering a blocked waiting state (e.g., waiting on a condition variable), and actively releasing the CPU time slice it occupies during this process. This pause behavior cuts off the data inflow pipeline at the source, giving downstream processing modules (packaging / compression, uploading) time to fully process the candidate data blocks accumulated in the output buffer. Only when the monitoring module detects that the buffer's data volume has dropped to a safe level and sends a recovery signal (such as the first control signal) will the read thread be awakened and resume its work.

[0048] By sending a first control signal during the idle state to make the reading thread read at a specific rate, the stability and continuity of data input are ensured; by sending a second control signal during the intermediate state to adjust the reading according to the target reading mode, adaptive rate control is achieved, optimizing the data flow; by sending a third control signal during the saturation state to pause reading, buffer overflow is prevented, ensuring system reliability, thereby improving responsiveness and resource allocation efficiency.

[0049] In this embodiment of the application, the candidate data block in the output buffer is uploaded to the object storage service via the upload thread, including the following steps C1-C3: Step C1: Append the corresponding checksum to the candidate data block to generate the target data block.

[0050] Specifically, the upload thread first uses a verification algorithm (such as CRC64 cyclic redundancy check) to perform real-time calculations on the binary content of the candidate data block, generating a fixed-length checksum (i.e., check value) with strong error correction capabilities. Then, the upload thread appends this calculated checksum as metadata to a specific location within the candidate data block (e.g., as a small header before the data block content, or as a tail after it), thus assembling a new, complete target data block. This target data block contains the payload (i.e., the packaged / compressed log data) and its corresponding checksum. This mechanism ensures the integrity of data throughout its transmission from leaving the output buffer to being received by the object storage service. On the server side, this checksum can be used to verify whether the data block has been corrupted or erroneous during transmission, providing a basis for potential retransmission mechanisms and achieving end-to-end data reliability assurance. The calculation and appending of the checksum itself has minimal overhead and has a negligible impact on pipeline performance.

[0051] Step C2: Upload the target data blocks one by one to the object storage service and obtain the upload progress of the target data blocks.

[0052] Specifically, the upload thread uses the HTTP protocol and employs HTTP Chunked Encoding for uploading. Each target data block is formatted as an independent HTTP Chunk. A key optimization is that the upload thread does not always use a fixed-size Chunk. Instead, it dynamically adjusts the size of each HTTP Chunk based on real-time monitoring of network bandwidth and round-trip latency: larger Chunks are used under favorable network conditions (high bandwidth, low latency) to reduce HTTP frame overhead and improve throughput; smaller Chunks are used under poor network conditions (low bandwidth, high latency) to ensure data reaches the server faster, reducing latency and providing more timely feedback. Furthermore, to further improve transmission efficiency, the upload thread may establish multiple HTTP connections with the server, implementing parallel multiplexing. For example, high-priority data blocks can be sent via dedicated connections to ensure low latency, while ordinary data blocks share other connections. Throughout the upload process, the upload thread continuously acquires and tracks the upload progress (i.e., success, in progress, or failure) of each target data block.

[0053] Step C3: If the upload process is in the successful phase, store the offset and checksum of the target data block; or, if the upload process is in the failed phase, perform a retransmission operation on the target data block.

[0054] Specifically, the upload thread determines its success based on the response received from the server (i.e., the result of the upload process for the target data block). If the response indicates a successful upload (success phase), the thread persistently stores two key metadata entries for the target data block locally: its offset in the original log file (i.e., the starting byte position of the data block) and its calculated and appended checksum. This record forms the cornerstone of resumable uploads; once transmission is interrupted, it can be used to determine the boundaries of successful transmissions during recovery. Conversely, if the response indicates a failed upload (failure phase), for example, if the server returns a checksum mismatch error via the response protocol, the upload thread immediately triggers a retransmission. This operation is highly targeted: it only retransmits the specific target data block that failed, not the entire file or the entire session. The upload thread retrieves a copy of the data block from its local cache and re-executes the upload process.

[0055] By generating target data blocks with additional checksums, data integrity and accuracy are ensured, and transmission errors are reduced. Uploading data block by block and tracking the upload progress improves the flexibility and manageability of the upload process. Storing offsets and checksums upon successful upload facilitates data tracking and verification, while retransmission is performed upon failure, ensuring the final successful upload of data and thus enhancing system stability and data consistency.

[0056] In this embodiment, the reading thread, processing thread, and uploading thread execute in parallel and exchange data through a data buffer, wherein the data buffer is an input buffer connecting the reading thread and the processing thread, and / or an output buffer connecting the processing thread and the uploading thread.

[0057] It should be noted that the data buffer can be implemented using a lock-free ring buffer, which typically contains a fixed-size buffer array, a head pointer indicating the head position, and a tail pointer indicating the tail position. Updates to these pointers are thread-safe through atomic operations. When a thread (such as a reading thread writing to the input buffer or a processing thread writing to the output buffer) wants to add data to the queue, it does so through atomic operations (e.g., using Compare-And-Swap, i.e., CAS). The instruction attempts to update the write pointer, compares the current write pointer value with the expected value, and if they are equal, updates it to the next position and writes the data. If the comparison fails, it means that another thread is writing, and that thread will retry or handle it according to the strategy. When a thread (such as a processing thread reading from the input buffer or an upload thread reading from the output buffer) wants to retrieve data from the queue, it will attempt to update the read pointer in a similar way through atomic operations and read the data from the position pointed to by the current read pointer. In this way, multiple producer threads and consumer threads can access the queue concurrently without using mutexes or other heavyweight synchronization primitives, avoiding the performance overhead caused by thread blocking and lock contention. Moreover, the empty and full states of the queue can be determined by the relative positions of the read pointer and the write pointer.

[0058] This invention is not limited to the preferred lock-free circular queue implementation described above. Those skilled in the art can implement the lock-free data buffer of this invention using various other equivalent lock-free concurrent data structures, depending on the actual application scenario and performance requirements. For example, a lock-free linked list is also a common choice, managing node insertion and deletion through atomic operations, and can also achieve efficient concurrent data exchange. Compared to a circular queue, a lock-free linked list theoretically has no fixed capacity limit, but may have higher memory allocation / reclaim overhead and slightly lower cache locality in some cases.

[0059] Specifically, in the pipelined parallel processing architecture, the read thread, processing thread, and upload thread are designed as independent execution units, executing in parallel at the operating system level. This fully utilizes multi-core CPU resources, achieving true overlap between read, processing, and upload operations. Data exchange between these threads is not achieved through traditional shared memory locking mechanisms, but rather through a high-performance data structure called a data buffer. Specifically, the data buffer is a fixed-size circular buffer whose core implementation relies on atomic operations, such as Compare-And-Swap (CAS), updating the head (read position) and tail (write position) pointers of the queue. This ensures that when acting as an input buffer between the read and processing threads, and as an output buffer between the processing and upload threads, producer and consumer threads can concurrently write and read data without locking. This design completely eliminates thread blocking, context switching overhead, and potential deadlock risks caused by lock contention. When the read thread writes multiple raw data blocks into the input buffer (data buffer), the processing thread can immediately read the data from the other end for processing; similarly, after the processing thread writes the generated candidate data blocks into the output buffer (another data buffer), the upload thread can immediately obtain and upload them.

[0060] By enabling the read, process, and upload threads to execute collaboratively and exchanging data through a data buffer, the system's concurrency performance and efficiency are significantly improved. The collaborative execution of the threads fully utilizes multi-core resources, increasing data processing throughput; using a data buffer avoids lock contention between threads, reduces context switching overhead, thereby achieving high-speed data exchange, reducing latency, ensuring smooth data transfer, and enhancing the overall responsiveness of the system.

[0061] As an example, such as Figure 3As shown, the log file upload process includes: First, the read thread streams 4MB data blocks 1 and 2 from the disk and writes them to the input buffer. Next, the corresponding processing thread compresses the original data blocks (data blocks 1 and 2) and constructs candidate data blocks (i.e., data packets 1 and 2), then writes data packets 1 and 2 to the output buffer. Then, the upload thread retrieves data packets 1 and 2 from the output buffer and uploads them block by block to the object storage service via HTTP. Furthermore, the process includes control measures: when the output buffer is saturated (i.e., the write buffer is full, data processing is paused), a corresponding control signal is sent to the read thread (i.e., control takes effect), causing the read thread to pause reading the original data blocks and enter a state of waiting for the buffer to be released. Once the output buffer has free space (after the buffer is released), the read thread continues reading data block N. The subsequent processes repeat the steps of compressing and constructing candidate data blocks, writing to the output buffer, and sending via HTTP until all data is uploaded. The object storage service returns an upload completion confirmation, and finally, the processes of each thread terminate sequentially.

[0062] This embodiment provides a log file upload system, such as... Figure 4 As shown, the system includes: a transmission module 100 and an object storage service 200; The transmission module 100 is used to upload log files to the object storage service 200 through pipelined parallel processing and to send processed candidate data blocks to the object storage service 200. The object storage service 200 is used to receive and store candidate data blocks sent by the transmission module 100.

[0063] Specifically, the transmission module 100 includes: a streaming reading module 101, a memory ring buffer 102, a streaming packaging / compression engine 103, a network transmission buffer 104, and a streaming upload module 105; The streaming read module 101 is used to read raw data blocks from the disk of the log file, adjust the read operation according to the received control signals, and send the raw data blocks to the memory circular buffer 102. The memory circular buffer 102 is used to receive raw data blocks sent by the streaming reading module 101 and to temporarily store data as a data buffer for the streaming packaging / compression engine 103 to read. The streaming packaging / compression engine 103 is used to read raw data blocks from the memory ring buffer 102, identify the data block type of the raw data blocks, process the raw data blocks according to the processing strategy corresponding to the data block type, obtain candidate data blocks, and send the candidate data blocks to the network transmission buffer 104. The network transmission buffer 104 is used to receive candidate data blocks sent by the streaming packaging / compression engine 103 and to temporarily store data as a data buffer for the streaming upload module 105 to read. The streaming upload module 105 is used to read candidate data blocks from the network transmission buffer 104, add a checksum to the candidate data blocks to generate target data blocks, upload the target data blocks one by one to the object storage service 200, monitor the storage status of the network transmission buffer 104 and send control signals to the streaming read module 101 according to the storage status, and store the offset and checksum of the target data block when the upload is successful, and perform a retransmission operation on the target data block when the upload fails.

[0064] Specifically, such as Figure 5 As shown, when uploading the target data block, the streaming upload module 105 first performs streaming compression on the data (corresponding to streaming compressed data); then, it uses an HTTP chunk encoder to encode the compressed data into chunks (corresponding to an HTTP chunk encoder); then, it sends the encoded data into the TCP socket send buffer; afterwards, the data is transmitted encrypted through a TLS encrypted channel (corresponding to a TLS encrypted channel) and sent to the OSS access point of the object storage service 200 via the network interface; finally, the object storage service 200 persists the received data.

[0065] As an example, such as Figure 6 As shown, the streaming packaging / compression engine 103 follows this process when processing raw data blocks to generate candidate data blocks: First, it starts processing the new file, generating a TAR file header, writing metadata such as filename and size, calculating the header checksum, and completing the header construction. Next, it determines whether the file data needs to be processed in blocks: if it's a starting block, it appends the initial data block, performs streaming compression on the data block, updates the compression dictionary, generates a compressed output block, and adds it to the network output stream (for subsequent uploads). If it's an intermediate block, it reads the next data block, appends a plain data block, and again performs streaming compression, updates the dictionary, generates a compressed output block, and adds it to the network output stream. If it's an ending block, it generates a TAR file tail, fills it with 1024 bytes of empty data, and finally ends the file stream. Through this process, the streaming packaging / compression engine 103 can process raw data blocks into compressed candidate data blocks that conform to the TAR format, so that the subsequent streaming upload module 105 can upload them.

[0066] This system employs a pipelined parallel processing architecture, consisting of a streaming read module, a memory circular buffer, a streaming packing / compression engine, a network transmission buffer, and a streaming upload module, to achieve efficient and stable uploading of log files from local disk to object storage services. Specifically, pipelined parallel processing decouples and parallelizes data reading, packing / compression, and network upload operations, significantly improving the overall system throughput and resource utilization. The use of lock-free circular buffers (memory circular buffer and network transmission buffer) as data exchange channels between modules avoids thread contention, enabling low-latency, high-concurrency data transfer. The streaming packing / compression engine performs differentiated processing based on data block type, optimizing storage efficiency and ensuring the integrity of archived files. The streaming upload module ensures data reliability through additional checksums, monitoring upload status, and implementing retransmission mechanisms. Simultaneously, by monitoring the network buffer status and feeding back control signals to the read module, dynamic flow control is implemented, effectively preventing buffer overflows or resource idleness and ensuring stable system operation under high load.

[0067] This embodiment also provides a log file uploading device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0068] This embodiment provides a log file upload device, such as... Figure 7 As shown, it includes: The reading module 71 is used to stream multiple consecutive raw data blocks from a log file on the disk via a reading thread, wherein the streaming reading includes reading the log file sequentially in units of data blocks until the end of the file; Processing module 72 is used to write multiple raw data blocks into the input buffer in sequence, and to allocate corresponding processing threads to the raw data blocks written into the input buffer in sequence. The raw data blocks are processed in parallel by the processing threads to obtain candidate data blocks. The upload module 73 is used to write candidate data blocks sequentially into the output buffer and upload the candidate data blocks in the output buffer to the object storage service through the upload thread. The upload thread monitors the occupancy status of each buffer and feeds back control signals to the read thread so that the read thread can dynamically adjust its data reading behavior.

[0069] In this embodiment of the application, the processing module 72 includes: The identification submodule is used to identify the data block type of the original data block; The processing submodule is used to allocate corresponding processing threads to the original data blocks according to the data block type, and process the original data blocks according to the processing strategies associated with the processing threads to obtain candidate data blocks.

[0070] In this embodiment, the processing submodule is configured to: if the original data block is a starting block, allocate a first processing thread to the original data block, dynamically generate an archive file header according to a first processing strategy associated with the first processing thread, and concatenate the archive file header with the original data block to obtain a first candidate data block; if the original data block is an intermediate block, allocate a second processing thread to the original data block, and use the original data block as a second candidate data block according to a second processing strategy associated with the second processing thread; if the original data block is an ending block, allocate a third processing thread to the original data block, compress the original data block according to a third processing strategy associated with the third processing thread to obtain a compressed data block, and append an archive file tail and padding data of a specific length to the compressed data block to generate a third candidate data block.

[0071] In this embodiment of the application, the apparatus further includes: The monitoring module is used to monitor the storage status of the output buffer; The execution module is used to send control signals corresponding to the storage state to the read thread, so that the read thread can perform the corresponding read operation according to the control signals.

[0072] In this embodiment, the execution module is configured to: if the storage state is idle, send a first control signal to the reading thread so that the reading thread reads the original data block at a specific rate based on the first control signal; if the storage state is intermediate, determine a target reading mode based on the storage state and send a second control signal corresponding to the target reading mode to the reading thread so that the reading thread reads the original data block according to the target reading mode based on the second control signal; if the storage state is saturated, send a third control signal to the reading thread so that the reading thread pauses reading the original data block based on the third control signal.

[0073] In this embodiment of the application, the upload module 73 is used to attach a corresponding check code to the candidate data block to generate a target data block; upload the target data block to the object storage service block by block and obtain the upload progress of the target data block; when the upload process is in the successful stage, store the offset and check code of the target data block, or when the upload process is in the failed stage, perform a retransmission operation on the target data block.

[0074] In this embodiment, the reading thread, processing thread, and uploading thread execute in parallel and exchange data through a data buffer, wherein the data buffer is an input buffer connecting the reading thread and the processing thread, and / or an output buffer connecting the processing thread and the uploading thread.

[0075] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 8 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system).

[0076] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0077] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0078] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device as shown by a landing page for an app. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0079] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0080] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0081] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0082] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for uploading log files, characterized in that, The method includes: A read thread streams multiple consecutive raw data blocks from a disk log file, wherein the streaming read includes reading the log file sequentially in units of data blocks until the end of the file; Multiple raw data blocks are sequentially written into an input buffer, and a corresponding processing thread is assigned to each raw data block sequentially written into the input buffer. The raw data blocks are then processed in parallel by the processing thread to obtain candidate data blocks. The candidate data blocks are written sequentially to the output buffer, and the candidate data blocks in the output buffer are uploaded to the object storage service through the upload thread. The upload thread feeds back control signals to the read thread by monitoring the occupancy status of each buffer, so that the read thread can dynamically adjust its data reading behavior.

2. The method according to claim 1, characterized in that, The step involves allocating corresponding processing threads to the raw data blocks written sequentially in the input buffer, and performing parallel processing on the raw data blocks through the processing threads to obtain candidate data blocks, including: The data block type of the original data block is identified; Based on the data block type, a corresponding processing thread is assigned to the original data block, and the original data block is processed according to the processing strategy associated with the processing thread to obtain candidate data blocks.

3. The method according to claim 2, characterized in that, The step of allocating a corresponding processing thread to the original data block according to the data block type, and processing the original data block according to the processing strategy associated with the processing thread to obtain candidate data blocks includes: If the original data block is the starting block, a first processing thread is allocated to the original data block, and an archive file header is dynamically generated according to the first processing strategy associated with the first processing thread. The archive file header is then concatenated with the original data block to obtain a first candidate data block. If the original data block is an intermediate block, a second processing thread is allocated to the original data block, and the original data block is used as a second candidate data block according to the second processing strategy associated with the second processing thread. If the original data block is the end block, a third processing thread is allocated to the original data block, and the original data block is compressed according to the third processing strategy associated with the third processing thread to obtain a compressed data block. An archive file tail and padding data of a specific length are appended to the compressed data block to generate a third candidate data block.

4. The method according to claim 1, characterized in that, The method further includes: Monitor the storage status of the output buffer; According to the storage state, a control signal corresponding to the storage state is sent to the reading thread so that the reading thread performs the corresponding reading operation according to the control signal.

5. The method according to claim 4, characterized in that, The step of sending a control signal corresponding to the storage state to the reading thread, so that the reading thread performs a corresponding reading operation according to the control signal, includes: If the storage state is idle, a first control signal is sent to the read thread so that the read thread reads the original data block at a specific rate based on the first control signal; If the storage state is an intermediate state, then the target read mode is determined according to the storage state, and a second control signal corresponding to the target read mode is sent to the read thread, so that the read thread reads the original data block according to the target read mode based on the second control signal; If the storage state is saturated, a third control signal is sent to the read thread to cause the read thread to pause reading the original data block based on the third control signal.

6. The method according to claim 1, characterized in that, The step of uploading candidate data blocks in the output buffer to the object storage service via the upload thread includes: A corresponding checksum is appended to the candidate data block to generate the target data block; The target data blocks are uploaded to the object storage service block by block, and the upload progress of the target data blocks is obtained; When the upload process is in a successful phase, the offset and checksum of the target data block are stored; or, when the upload process is in a failed phase, a retransmission operation is performed on the target data block.

7. The method according to claim 1, characterized in that, The reading thread, the processing thread, and the uploading thread execute in parallel and exchange data through a data buffer, wherein the data buffer is an input buffer connecting the reading thread and the processing thread, and / or an output buffer connecting the processing thread and the uploading thread.

8. A log file upload device, characterized in that, The device includes: A read module is used to stream multiple consecutive raw data blocks from a log file on a disk via a read thread, wherein the streaming read includes reading the log file sequentially in units of data blocks until the end of the file; The processing module is used to write multiple raw data blocks into an input buffer in sequence, and to allocate corresponding processing threads to the raw data blocks written into the input buffer in sequence, and to process the raw data blocks in parallel through the processing threads to obtain candidate data blocks; An upload module is used to sequentially write the candidate data blocks into an output buffer, and upload the candidate data blocks in the output buffer to an object storage service through an upload thread. The upload thread monitors the occupancy status of each buffer and feeds back control signals to the read thread so that the read thread can dynamically adjust its data reading behavior.

9. A computer device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 7.

Images