A double-buffered file writing method with elastic scalability

Abstract

The present invention discloses a double-buffered file writing method with elastic scalability. The present invention uses two memory buffers, CurrentBuffer block and SyncBuffer block, for buffering in the process of file writing. The client writes the file data into the CurrentBuffer block, then exchanges the memory into the SyncBuffer block, and finally writes to the disk. During the buffering and writing process, it is determined whether the memory buffer needs to be expanded or reduced, so as to realize adaptive adjustment of the buffer space size. At the same time, since the space size of the memory buffer can be automatically adjusted, multiple freezes are avoided during data transmission and writing, thereby improving the stability and efficiency of data writing.

Description

Technical Field

[0001] The present invention relates to the technical field of file data reading and writing, and in particular to a double-buffered file writing method capable of elastic expansion and contraction. Background Art

[0002] In the process of writing files, in order to improve the response speed, memory is used to read and write data. The block storage space of this part of memory is the buffer. However, in the scenario of high-concurrency writing files, the data is temporarily stored in the memory buffer, and the data is flushed to the disk after the buffer is full. This process is prone to jamming and seriously affects the performance of file writing. At the same time, since the buffer size is fixed, it cannot be automatically adjusted according to business peaks and valleys. When the business is low, the empty buffer will cause a waste of resources.

[0003] Therefore, the prior art has defects and needs to be improved. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a double-buffered file writing method that is elastically scalable.

[0005] The technical solution of the present invention is as follows: a double-buffered file writing method with elastic scalability is provided, comprising the following steps:

[0006] Step 1: Configure the initial size of the memory buffer, the expansion threshold, the reduction threshold, the lock wait times clearing cycle, and the maximum buffer space;

[0007] Step 2: The client writes data to the CurrentBuffer block of the memory buffer at high concurrency;

[0008] Step 3: When the CurrentBuffer block space is full or reaches the shrinking time, the memory is exchanged with the SyncBuffer block;

[0009] Step 4: During the exchange of CurrentBuffer blocks and SyncBuffer blocks, CurrentBuffer adds a synchronization lock, and the client write operation is blocked; after the memory data exchange is completed, CurrentBuffer releases the lock and the client write operation resumes;

[0010] Step 5: After the data exchange is completed, write the SyncBuffer block data to the disk and erase the SyncBuffer block data;

[0011] Step 6: After the data flushing operation is completed, determine whether the lock waiting times have reached the expansion threshold. If not, the process ends. If it has reached the expansion threshold, increase the expansion times by one, immediately trigger the memory swap process, and perform expansion.

[0012] Step 7: If the capacity expansion has occurred, the shrinking timer is started. When the cycle returns to step 6 again, if the capacity expansion is not needed, it is determined whether the shrinking threshold is reached. If the shrinking threshold is reached, the number of expansion times is reduced by one, and the memory swap process is immediately triggered and the shrinking is performed.

[0013] Furthermore, when the memory is swapped in step 3, if the SyncBuffer block is performing a data flushing operation at this time, it is necessary to wait for the data flushing operation to release the synchronization lock and increase the lock waiting times.

[0014] Furthermore, in step 5, during writing to disk, the SyncBuffer block is synchronized and no memory swapping operation is allowed.

[0015] Furthermore, in step 6, when expansion is required, the CurrentBuffer block is first locked, and the CurrentBuffer block and the SyncBuffer block exchange memory. After the memory exchange is completed, the space of the CurrentBuffer and SyncBuffer blocks is expanded.

[0016] Furthermore, in step 7, when shrinking is required, the CurrentBuffer block is first locked, and the CurrentBuffer block and the SyncBuffer block exchange memory. After the memory exchange is completed, the space of the CurrentBuffer and SyncBuffer blocks is shrunk.

[0017] By adopting the above scheme, the present invention uses two memory buffers, CurrentBuffer block and SyncBuffer block, for buffering in the process of writing files. The client writes the file data into the CurrentBuffer block, then exchanges the memory into the SyncBuffer block, and finally writes it to the disk. During the buffering and writing process, it is determined whether the memory buffer needs to be expanded or reduced, so as to realize the adaptive adjustment of the buffer space size. At the same time, since the space size of the memory buffer can be automatically adjusted, multiple jams in the process of data transmission and writing are avoided, thereby improving the stability and efficiency of data writing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a system schematic diagram of the present invention. DETAILED DESCRIPTION

[0019] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0020] See also Figure 1 The present invention provides a double-buffered file writing method with elastic scalability, comprising the following steps:

[0021] Step 1: Configure the initial size of the memory buffer, the expansion threshold, the reduction threshold, the lock wait times clearing cycle, and the maximum buffer space. The initial size of the memory buffer can meet the performance requirements most of the time; the expansion threshold is related to the lock wait times; the reduction threshold is related to time. If the buffer space is about to exceed the maximum value, the expansion will not continue. Similarly, if the expansion times are reduced to 0, the reduction will not continue.

[0022] Step 2: The client writes data to the CurrentBuffer block of the memory buffer at high concurrency.

[0023] Step 3: When the CurrentBuffer block is full or reaches the shrinking time, it exchanges memory with the SyncBuffer block. When exchanging memory, if the SyncBuffer block is performing a data disk flushing operation, it is necessary to wait for the data disk flushing operation to release the synchronization lock and increase the lock waiting times.

[0024] Step 4: During the exchange of CurrentBuffer blocks and SyncBuffer blocks, CurrentBuffer adds a synchronization lock and the client write operation is blocked; after the memory data exchange is completed, CurrentBuffer releases the lock and the client write operation resumes.

[0025] Step 5: After the data exchange is completed, the SyncBuffer block data is written to the disk and the SyncBuffer block data is erased. During the writing to disk, the SyncBuffer block is synchronized and memory exchange operations are not allowed.

[0026] Step 6: After the data flushing operation is completed, determine whether the lock waiting times have reached the expansion threshold. If not, the process ends. If it has reached the expansion threshold, increase the expansion times by one, trigger the memory swap process immediately, and perform the expansion. First, lock the CurrentBuffer block, and swap the memory between the CurrentBuffer block and the SyncBuffer block. After the memory swap is completed, expand the space of the CurrentBuffer and SyncBuffer blocks. During the writing to disk, the SyncBuffer is synchronized and memory swap operations are not allowed.

[0027] Step 7: If the capacity expansion has occurred, the shrinking timer is started. When the cycle returns to step 6 again, if the capacity expansion is not needed, it is determined whether the shrinking threshold is reached. If the shrinking threshold is reached, the number of expansions is reduced by one, and the memory swap process is immediately triggered and the capacity is reduced. First, the CurrentBuffer block is locked, and the CurrentBuffer block and the SyncBuffer block exchange memory. After the memory swap is completed, the space of the CurrentBuffer and SyncBuffer blocks is reduced. During the writing to disk, the SyncBuffer is synchronized and memory swap operations are not allowed.

[0028] In some embodiments, in the scenario of high-concurrency writing of large files, the user first configures the initial size of the memory buffer, the expansion threshold, the reduction threshold, the lock wait number clearing cycle, and the maximum space of the buffer in the system, or calls the user's original configuration file. The client writes data to the CurrentBuffer block of the memory buffer at high concurrency. When the CurrentBuffer block space is full, it exchanges memory with the SyncBuffer block. If the SyncBuffer block is performing a data disk flushing operation at this time, it is necessary to wait for SyncBuffer to release the synchronization lock before exchanging, and at the same time increase the lock wait number count. During the period when the CurrentBuffer block and the SyncBuffer block are performing a memory exchange, a synchronization lock is added to the CurrentBuffer, and the client's write operation is blocked. At this time, the client cannot write data to the CurrentBuffer block. After completing the exchange of memory data, the CurrentBuffer releases the lock and resumes the client's write operation.

[0029] After the memory swap is completed, the data in the SyncBuffer block is written to the disk, and the data in the SyncBuffer block is erased. During the writing period, the SyncBuffer adds a synchronization lock, and the memory swap operation is not allowed to occur, so as to avoid data confusion or errors caused by the swapped memory. After the disk flushing operation is completed, if the lock waiting times reach the expansion threshold, it means that during the memory swap process, when the memory swap between the two memory buffers occurs multiple times, the SyncBuffer block is in the disk flushing state, that is, the buffer space size is insufficient. Therefore, after the memory swap is completed, the CurrentBuffer block and the SyncBuffer block are expanded so that the space of the two memory buffers is expanded to twice the original size, so that the CurrentBuffer block can write more data before the memory swap is performed, ensuring that when the CurrentBuffer block is full, the SyncBuffer block has completed the disk flushing operation, thereby improving the efficiency of data writing.

[0030] If the CurrentBuffer block and the SyncBuffer block have been expanded, the expansion count will be increased by 1. If there is no need for expansion within the set time (reduction threshold), the system determines that the memory buffer can meet the needs of high-concurrency writing, the transmission is stable, and there is free space in the buffer. Therefore, in order to avoid waste of resources, the CurrentBuffer block and the SyncBuffer block are reduced. After the memory swap is completed, the space of the CurrentBuffer block and the SyncBuffer block is adjusted to half of the original size, reducing the free space occupied by the memory buffer and avoiding waste of resources.

[0031] In summary, the present invention uses two memory buffers, CurrentBuffer block and SyncBuffer block, for buffering in the process of writing files. The client writes the file data into the CurrentBuffer block, then exchanges the memory into the SyncBuffer block, and finally writes it to the disk. During the buffering and writing process, it is determined whether the memory buffer needs to be expanded or reduced, so as to realize adaptive adjustment of the buffer space size. At the same time, since the space size of the memory buffer can be automatically adjusted, multiple jams during data transmission and writing are avoided, thereby improving the stability and efficiency of data writing.

[0032] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A double-buffered file writing method with elastic scalability, characterized in that: The steps include: Step 1: Configure the initial size of the memory buffer, the expansion threshold, the reduction threshold, the lock wait times clearing cycle, and the maximum buffer space; the expansion threshold is associated with the lock wait times; the reduction threshold is associated with time; Step 2: The client writes data to the CurrentBuffer block of the memory buffer at high concurrency; Step 3: When the CurrentBuffer block space is full or the shrinking time is reached, the memory is exchanged with the SyncBuffer block. During the memory exchange, if the SyncBuffer block is performing a data disk flushing operation, it is necessary to wait for the data disk flushing operation to release the synchronization lock and increase the lock waiting times. Step 4: During the exchange of CurrentBuffer blocks and SyncBuffer blocks, CurrentBuffer adds a synchronization lock and the client write operation is blocked; After the memory data exchange is completed, CurrentBuffer releases the lock and the client write operation resumes; Step 5: After the data exchange is completed, write the SyncBuffer block data to the disk and erase the SyncBuffer block data; During the write to disk, the SyncBuffer block is synchronized and memory swapping is not allowed; Step 6: After the data flushing operation is completed, determine whether the lock waiting times have reached the expansion threshold. If not, the process ends. If it has reached the expansion threshold, increase the expansion times by one, trigger the memory swap process immediately, and perform expansion. When expansion is required, first lock the CurrentBuffer block, and then swap the memory between the CurrentBuffer block and the SyncBuffer block. After the memory swap is completed, expand the space of the CurrentBuffer and SyncBuffer blocks. Step 7: If expansion has occurred, the shrinking timer is started. When the cycle returns to step 6 again, if expansion is not needed, determine whether the shrinking threshold is reached. If the shrinking threshold is reached, the number of expansions is reduced by one, and the memory exchange process is immediately triggered and the shrinking is performed. When shrinking is required, the CurrentBuffer block is locked first, and the CurrentBuffer block and the SyncBuffer block exchange memory. After the memory exchange is completed, the CurrentBuffer and SyncBuffer block space is reduced.

Images