Buffer management method and system
By introducing buffer maintenance threads and write disk threads in the buffer management system, combining synchronous and asynchronous IO technologies, the logical numbering and sorting of buffers is realized, solving the problem of low buffer management efficiency in the existing technology, and improving the Realistic performance and data consistency in high concurrency scenarios.
Patent Information
- Application Number
- CN202311664573.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-06
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2043-12-06
AI Technical Summary
In the prior art, the buffer to the hard disk realistic method adopts single-block control, which cannot support the performance requirements in high concurrency scenarios, and lacks a trigger mechanism to predict insufficient space in advance, resulting in passive detection when the buffer is insufficient and cannot be effective. Manage data writing.
The buffer manager is initialized through the main thread, creating buffer maintenance threads and disk write threads, using synchronous and asynchronous IO methods to manage buffer data, implementing the logical numbering and sorting of buffers, optimizing the disk write triggering mechanism, ensuring data consistency and Write efficiency.
In high concurrency scenarios, the realistic speed is improved, the consistency of buffer and hard disk data is ensured, the disk seek time and waiting time is reduced, and the performance and real-time performance of disk writing is improved.
Smart Images

Figure CN117453145B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer buffer management, in particular to a buffer management method and system. Background Art
[0002] In data communication networks, data traffic is very large. The memory area used to temporarily store data to be processed is usually called a data buffer, or buffer for short. Various protocols and functional modules in the system protocol stack usually temporarily store the received data in a buffer queue composed of data buffers before processing the data. These data traffics complete the intercommunication between the source port and the destination port at the data exchange point (such as switches, routers, etc.). At each data exchange point, the data from the source port must be exchanged to the destination port in a quick and effective manner. The data transmitted on the network is usually transmitted in fixed-size packets or data frames with length restrictions. Usually, after the packets (or data frames) come in from the source port, they are required to be stored in the buffer.
[0003] In the prior art, the buffer writes to the hard disk in a single-block control manner without combining the design of the physical hard disk, which results in amplified disk writes and is unable to support performance requirements in high-concurrency scenarios. When the buffer is insufficient, the existing trigger mechanism is passive detection by the maintenance thread, which can only be triggered when the insufficient situation actually occurs, and cannot support early prediction of insufficient space. Summary of the invention
[0004] In order to solve the above problems, the present invention provides a buffer management method, comprising the following steps: S1. starting the system and initializing the buffer manager through the main thread; S2. creating a group of control variables, a buffer maintenance thread, one or more buffer write threads, and a group of support functions, and making the buffer maintenance thread enter a waiting state; S3. when the waiting timeout or an exit signal is received, the buffer maintenance thread leaves the waiting state and performs dirty data processing according to the exit flag of the buffer maintenance thread.
[0005] Furthermore, the S1 step includes the following sub-steps: S11. Initialize the headers of each queue, each counter, latch and semaphore object; S12. Apply for hash table memory and format the hash table; S13. Apply for buffer header memory and format it into several buffer structures, and link the buffer structure to the free buffer header chain; S14. Apply for data block memory, divide the data block memory into a system data block, and add the system data block to the back of the free block chain.
[0006] Furthermore, in the step S3, the dirty data processing according to the exit flag of the buffer maintenance thread specifically includes: if the exit flag is T, the dirty block is written to the disk in a synchronous manner; if the exit flag is not T, the asynchronous IO mode is enabled to write the buffer back to the disk.
[0007] Furthermore, if the exit flag is not T, the following sub-steps are specifically included: A31. Detect the number of free blocks of various sizes, and compare the detected number of free blocks with the lower limit ratio. If the number of free blocks is greater than the lower limit ratio, enter step A32; if the number of free blocks is less than the lower limit ratio, enter step A33; A32. Add part of the buffer in the dirty data queue to the disk write queue, enter the waiting state and wait for the exit signal in a timeout manner. When the exit signal is received, enter step S3 to perform data According to processing; A33. Select n buffer blocks from the LRU queue, and judge the dirty data of the selected n buffer blocks. If the selected buffer block has dirty data, enter step A34; if the selected buffer block does not have dirty data, the buffer is directly eliminated and added to the free queue, and step A32 is entered at the same time; A34. Lock the buffer block and set the handler pointer after writing to the disk, add the locked buffer block to the disk write back queue, start asynchronous IO, and enter step A32 after completing the write back of the buffer.
[0008] Furthermore, the A34 step includes the following sub-steps: A341. Locking the buffer with a latch; A342. Mapping the locked buffer to the file handle where the buffer is located and its position in the file according to the table entity and block number of the buffer; A343. Filling the pointer of the post-processing program after writing the buffer to the disk into the callback function pointer of the asynchronous IO control structure to start asynchronous IO.
[0009] Furthermore, it also includes a step: using the disk write-back thread to simulate enabling asynchronous IO for buffer write-back, and its implementation process specifically includes the following sub-steps: B1. When the system does not support the asynchronous IO mechanism or the asynchronous IO function provided by the system is not enabled, first add an appropriate latch to the buffer, set the post-write disk processing program pointer, and add the locked buffer to the write disk queue; B2. Start the disk write-back thread, check the exit flag of the buffer maintenance thread, if the exit flag is T, go to step B3; if the exit flag is not T, go to step B4; B3. The serial number of the disk write-back thread is judged. If the current disk write-back thread is the disk write-back thread No. 1, the current disk write-back thread directly exits the operation; if the current disk write-back thread is not the disk write-back thread No. 1, all buffers in the queue are written to the disk, and the disk write-back thread exits the operation after completion; B4. Get a group of buffers in the same table space. If the buffer is not obtained, the current disk write-back thread sleeps for one millisecond and returns to step B2; if the buffer is obtained, the obtained buffer is sorted according to the block number, and the write command is called to write the sorted buffer to the table space, and after completion, enter step B2.
[0010] A buffer management system comprises a buffer disk writing module, a buffer maintenance module, a memory module, a control module and a support function module; the buffer disk writing module is used to take out the buffer to be written from the disk writing queue and sort it, and write the sorted buffer to the disk; the buffer maintenance module is used to manage and maintain the number of each free buffer block, and use the idle time requested by the buffer to write the dirty block back to the disk; the memory module is used to store the data in the buffer and the data that needs to be called; the control module is used to provide the buffer lock and unlock function, and control the buffer disk writing module to write back data through synchronous IO or asynchronous IO; the support function module is used to support the management and operation of the buffer management system and provide functions for the system to call.
[0011] Furthermore, the states of the buffer block in the system include: locked (not dirty) state, locked (dirty) state, unlocked (valid, dirty) state, unlocked (data valid, not dirty) state, in disk write queue state, and in buffer state.
[0012] Furthermore, the buffer zone maintenance module is in an intermittent circulation state.
[0013] The present invention provides a buffer management method and system, which have the following beneficial effects:
[0014] The present invention supports a single or multiple maintenance threads for the buffer, realizes flexible configuration, improves the writing speed in high concurrency scenarios, and ensures the consistency of the buffer changes with the hard disk data; the present invention logically numbers the buffer, and by sorting the buffer blocks, allows the disk to perform sequential writing to the maximum extent, reduces the disk seek time, and improves the writing speed; the disk writing trigger mechanism provided by the present invention can better control the disk writing timing of the buffer, realizes the real-time writing of the disk, reduces the waiting time, and does not affect other operations, thereby improving performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0016] Figure 1 is a flow chart of the method of the present invention;
[0017] Figure 2 It is a system function module diagram of the present invention;
[0018] Figure 3 It is a buffer maintenance thread workflow diagram of the present invention;
[0019] Figure 4 It is a workflow diagram of the disk write-back thread of the present invention;
[0020] Figure 5 It is a schematic diagram of buffer memory distribution of the present invention. DETAILED DESCRIPTION
[0021] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0022] The following is a detailed description of the implementation method of the present invention in conjunction with the accompanying drawings. Only some embodiments are described, not all embodiments. For the purpose of clarity, representations and descriptions that are not related to the present invention are omitted in the drawings and descriptions.
[0023] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below. Obviously, the implementation cases described are part of the embodiments of the present invention, not all of the embodiments, and cannot be understood as limiting the scope of the implementation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.
[0024] like Figure 1 As shown, the present invention provides a buffer management method, comprising the following steps: S1. starting the system and initializing the buffer manager through the main thread; S2. creating a group of control variables, a buffer maintenance thread, one or more buffer disk writing threads, and a group of support functions, and making the buffer maintenance thread enter a waiting state; S3. when the waiting timeout or an exit signal is received, the buffer maintenance thread leaves the waiting state and performs dirty data processing according to the exit flag of the buffer maintenance thread.
[0025] Among them, step S1 includes the following sub-steps: S11. Initialize the headers of each queue, each counter, latch and semaphore object; S12. Apply for hash table memory and format the hash table; S13. Apply for buffer header memory and format it into several buffer structures, and link the buffer structure to the free buffer header chain; S14. Apply for data block memory, divide the data block memory into a system data block, and add the system data block to the back of the free block chain.
[0026] like Figure 3 As shown, in step S3, dirty data processing according to the exit flag of the buffer maintenance thread specifically includes: if the exit flag is T, the dirty block is written to the disk in a synchronous manner; if the exit flag is not T, the asynchronous IO mode is enabled to write the buffer back to the disk.
[0027] If the exit flag is not T, the following sub-steps are specifically included: A31. Detect the number of free blocks of various sizes, and judge the size of the detected number of free blocks with the lower limit ratio. If the number of free blocks is greater than the lower limit ratio, enter step A32; if the number of free blocks is less than the lower limit ratio, enter step A33; A32. Add part of the buffer in the dirty data queue to the disk write queue, enter the waiting state and wait for the exit signal in a timeout manner, and enter step S3 for data processing when the exit signal is received; A33. Select n buffer blocks from the LRU queue, and perform dirty data judgment on the selected n buffer blocks. If the selected buffer block has dirty data, enter step A34; if the selected buffer block does not have dirty data, directly eliminate the buffer, add it to the free queue, and enter step A32 at the same time; A34. Lock the buffer block and set the handler pointer after writing to the disk, add the locked buffer block to the disk write back queue, start asynchronous IO, and enter step A32 after completing the buffer write back.
[0028] Step A34 includes the following sub-steps: A341. Use a latch to lock the buffer; A342. Map the locked buffer to the file handle where the buffer is located and its position in the file according to the table entity and block number of the buffer; A343. Fill the pointer of the post-processing program after the buffer is written to the disk into the callback function pointer of the asynchronous IO control structure to start asynchronous IO.
[0029] Also includes a step such as Figure 4 As shown: Use the disk write-back thread to simulate enabling asynchronous IO for buffer write-back, and its implementation process specifically includes the following sub-steps: B1. When the system does not support the asynchronous IO mechanism or the asynchronous IO function provided by the system is not enabled, first add an appropriate latch to the buffer, set the post-write disk processing program pointer, and add the locked buffer to the write disk queue; B2. Start the disk write-back thread, check the exit flag of the buffer maintenance thread, if the exit flag is T, go to step B3; if the exit flag is not T, go to step B4; B3. Write back to the current disk thread . If the current disk write-back thread is the No. 1 disk write-back thread, the current disk write-back thread directly exits; if the current disk write-back thread is not the No. 1 disk write-back thread, all buffers in the queue are written to the disk, and the disk write-back thread exits after completion; B4. Get a group of buffers in the same table space. If no buffer is obtained, the current disk write-back thread sleeps for one millisecond and returns to step B2; if the buffer is obtained, the obtained buffer is sorted according to the block number, and the write command is called to write the sorted buffer into the table space, and then enter step B2 after completion.
[0030] like Figure 2 As shown, a buffer management system includes a buffer disk writing module, a buffer maintenance module, a memory module, a control module, and a support function module; the buffer disk writing module is used to take out the buffer to be written from the disk writing queue and sort it, and write the sorted buffer to the disk; the buffer maintenance module is used to manage and maintain the number of each free buffer block, and use the idle time requested by the buffer to write the dirty block back to the disk; the memory module is used to store the data in the buffer and the data that needs to be called; the control module is used to provide the buffer lock and unlock function, and control the buffer disk writing module to write back data through synchronous IO or asynchronous IO; the support function module is used to support the management and operation of the buffer management system and provide functions for the system to call. The buffer maintenance module is in an intermittent cycle state.
[0031] The states of the buffer block in the system include: locked (not dirty) state, locked (dirty) state, unlocked (valid, dirty) state, unlocked (data valid, not dirty) state, in the disk write queue state, and in the buffer state.
[0032] Example:
[0033] In an embodiment of the present invention, the buffer management system is mainly composed of: one or several large memory areas, a group of control variables (including synchronization control), a buffer maintenance thread, one or several buffer disk writing threads, and a group of support functions. When the service program is started, the buffer manager is initialized and each thread is started. Then, the system provides function call support such as buffer locking and unlocking.
[0034] A buffer consists of a header and a data block. The data block is separate from the header, and a pointer member in the header points to the data block to combine the two. The variables in the header reflect the various states of the buffer. A system pointer member links the buffer to multiple queues to implement buffer operations.
[0035] Buffer header: Buffer; supports Windows and POSIX asynchronous IO.
[0036] Part of the code for this structure is as follows:
[0037] / *prev and next pointers are used to link the buffer into a bucket of the hash table, and the next pointer is also used to link the buffer header into the free buffer header queue* /
[0038] struct Buffer * volatile prev;
[0039] struct Buffer * volatile next;
[0040] / *prev_lru and next_lru pointers are used to link the buffer into the lru queue* /
[0041] struct Buffer * volatile prev_lru;
[0042] struct Buffer * volatile next_lru;
[0043] / *prev_dirty and next_dirty pointers are used to link the buffer into the "dirty" queue* /
[0044] struct Buffer * volatile prev_dirty;
[0045] struct Buffer * volatile next_dirty;
[0046] / * prev_entity_buff and prev_entity_buff pointers are used to link the buffer to the corresponding table entity. When the table entity is closed or destroyed, all buffers of the table entity can be found through this queue, so as to cancel them* /
[0047] struct Buffer * volatile prev_entity_buff;
[0048] struct Buffer * volatile next_entity_buff;
[0049] / * prev_entity_buff and prev_entity_buff pointers are used to link the buffer into the dirty buffer queue of the corresponding table entity. When the table entity is closed or flushed, all dirty buffers of the table entity can be found through the queue, so that they can be written to disk* /
[0050] struct Buffer * volatile prev_entity_dirty;
[0051] struct Buffer * volatile next_entity_dirty;
[0052] / *aftger_io is a function pointer, which is called after the disk writing thread writes the buffer to disk for post-processing (such as putting it into the free queue, etc.)* /
[0053] AfterIO volatile after_io;
[0054] / *Since multiple threads may operate on a block at the same time, when a thread is operating the buffer, due to incompatible locking methods, other threads may need to wait for the thread currently locking the buffer to unlock before continuing to execute. first_waiter and last_waiter together describe the queue of threads waiting for the buffer to be unlocked* /
[0055] struct ServiceThd * volatile first_waiter;
[0056] struct ServiceThd * volatile last_waiter;
[0057] struct TableEntity * volatile p_entity
[0058] Data block: BlkData; this structure only occupies the first few bytes of the data block, is valid for free data blocks, and is used to link the data block into the free data block queue. The code of this part of the structure is as follows:
[0059] typedef struct BlkData
[0060] {
[0061] struct BlkData * volatile prev;
[0062] struct BlkData * volatile next;
[0063] }BlkData
[0064] The buffer memory distribution is shown as follows Figure 5 As shown, from Figure 5 It can be seen that the buffer header structure memory is separate from the data block memory. The data block memory can be composed of several large memory areas, and each area is divided into several data blocks.
[0065] When the system starts, the buffer manager is initialized by the main thread. The initialization process is: initialize the heads of each queue; initialize each counter; initialize objects such as latches and semaphores; apply for hash table memory and format the hash table; apply for buffer header memory and format it into several Buffer structures, linking them to the free buffer header chain, i.e. buff_head_list; apply for data block memory, divide them into a system data block, and hang them behind free block chains of various sizes, free_blk_stub.
[0066] The buffer maintenance thread is created by the system main thread together with other threads when creating each work thread. In a server instance, there is only one buffer maintenance thread. The role of the buffer maintenance thread is to keep the number of various free buffer blocks at a reasonable level and use the idle time of the buffer request to write dirty blocks back to the disk. The buffer maintenance thread is always in an intermittent loop, that is, it keeps working in a loop when there is something to do, and enters the state of waiting for the semaphore need_buff_sema when there is nothing to do. The waiting process has a timeout value. When the service thread needs a free buffer and the free buffer has been used up, the semaphore need_buff_sema will be incremented. At this time, the buffer maintenance thread is awakened and enters the maintenance work loop. In addition, due to the timeout of the waiting process, the buffer maintenance thread itself will also leave the waiting state and re-enter the next workpiece cycle.
[0067] The implementation process of writing the buffer back to the disk includes: if the operating system provides an asynchronous IO mechanism and the configuration item in the system configuration file is set to use the asynchronous IO mode, the writing of the buffer will enable the asynchronous IO mode. The specific implementation method is: (1) First, add an appropriate lock to the buffer to prevent the buffer from being accidentally modified during disk writing; (2) According to the table entity and block number of the buffer, map it to the file handle where the buffer is located and its position in the file; (3) Create an asynchronous IO structure, fill the buffer's file handle, position in the file, address pointer of the data block, data block size, etc. into the asynchronous IO control structure; (4) Fill the pointer of the post-processing program after the buffer is written to the disk into the callback function pointer of the asynchronous IO control structure; (5) Start asynchronous IO.
[0068] After that, the buffer writing process will be completed by the operating system. After completing the asynchronous writing of the buffer to disk, the operating system will call the callback function to implement the subsequent processing after the buffer is written to disk.
[0069] If the asynchronous IO function provided by the operating system is not enabled, the buffer write-back task will be completed by the disk write-back thread. For the buffer maintenance thread, writing the buffer in this way is still asynchronous, but this asynchronous function is simulated by the disk write-back thread. In this case, the buffer maintenance thread will start the disk write process of a buffer with the following steps: first, add an appropriate lock to the buffer to prevent the buffer from being accidentally modified during the disk write; set the disk write post-processor pointer; add the buffer to the disk write queue. After that, the buffer will be processed by the disk write-back thread. When the buffer is written to the disk by the disk write-back thread, the disk write-back thread will call the post-processor to implement the subsequent processing after writing.
[0070] The disk write-back thread is created together with the buffer maintenance thread. Its behavior is: it continuously takes out the buffers to be written from the write-disk queue. Usually, it takes out several buffers belonging to the same table space from the queue at a time, sorts them by block number, and then writes them to the disk. The sorting process makes the order of writing to the disk more reasonable. After each buffer completes the disk write operation, the disk write-back thread will immediately call the buffer's post-write processing program.
[0071] The disk write-back thread uses synchronous IO when writing to disk. It will wait for one buffer to be written before writing another buffer. Therefore, if the system has multiple disks, multiple disk write-back threads should be started to enable each disk to run in parallel.
[0072] The system support function module mainly uses the following functions:
[0073] (1) flush_space_buffs(TableSpace *p_space), this function flushes all buffers associated with a tablespace to disk: *p_space is the tablespace pointer.
[0074] (2) flush_entity_buffs(TableEntity *p_entity), this function flushes all buffers associated with a table entity to disk: *p_entity is the table entity pointer.
[0075] (3) flush_all_buff(); This function flushes all buffers to disk.
[0076] (4) yield_buff(TableEntity *p_entity, int32 blk_no): This function is used to invalidate a buffer. That is, if the buffer exists, it will be eliminated and will not be written back regardless of whether the buffer is dirty or not. This function will be called when deleting a table or clearing a table. *p_entity is the table entity pointer; blk_no is the block number.
[0077] (5) yield_space_buffs(TableSpace *p_space). This function is used to invalidate all buffers associated with the specified tablespace. That is, if the buffer exists, it will be eliminated, and the buffer will not be written back regardless of whether it is dirty. This function will be called when the database is closed or deleted, or the tablespace is deleted: *p_space is the tablespace pointer.
[0078] (6) yield_entity_buffs(TableEntity *p_entity). This function is used to invalidate all buffers related to the specified table entity. That is, if the buffer exists, it will be eliminated, and the buffer will not be written back regardless of whether it is dirty. This function will be called when the database is closed or deleted, or the table space is deleted: *p_entity is the table entity pointer.
[0079] (7) yield_all_buff(), this function invalidates all buffers.
[0080] (8) *lock_buff(TableEntity *p_entity, int32 blk_no, LockType l_type) This function is used to lock a data block in the specified table entity and return the buffer corresponding to the data block: *p_entity is the table entity pointer; blk_no is the block number; l_type is the lock type (S_LOCK or X_LOCK, that is, shared lock or exclusive lock).
[0081] (9) *lock_sys_buff(TableSpace *p_space, int32 blk_no, LockType l_type) This function is used to lock a control block of the specified tablespace and return the buffer corresponding to the data block: *p_space is the tablespace pointer; blk_no is the block number; l_type is the lock type (S_LOCK or X_LOCK, that is, shared lock or exclusive lock).
[0082] (10) *lock_lob_buff(TableEntity *p_entity, int32 blk_no, LockType l_type); This function is used to lock a data block of the specified large object table entity and return the buffer corresponding to the data block: *p_entity is the pointer to the large object storage entity; blk_no is the block number; l_type is the lock type (S_LOCK or X_LOCK, that is, shared lock or exclusive lock).
[0083] (11) *check_lock_buff(TableEntity *p_entity, int32 blk_no, LockType l_type), this function is used to change the lock level of the buffer: *p_entity is the table entity pointer; blk_no is the block number; l_type is the lock type (S_LOCK or X_LOCK, that is, shared lock or exclusive lock).
[0084] (12) *lock_buff_new(TableEntity *p_entity, int32 blk_no, LockType l_type). This function creates a buffer for a new data block in a table entity. Unlike lock_buff, 1) the returned buffer must not exist before the call; 2) there is no need to read data from disk. The data in the buffer is initialized to blank: *p_entity is the table entity pointer; blk_no is the block number; l_type is the lock type (S_LOCK or X_LOCK, that is, shared lock or exclusive lock).
[0085] (13) unlock_buff(Buffer *p_buff, int pri_level), this function is used to unlock the buffer locked by the current thread: *p_buff is the buffer pointer; pri_level is the priority. If no other thread has locked the buffer when unlocking, the buffer will be placed in the LRU queue. pri_level determines the level of the LRU queue the buffer should be placed in. The value range is: 0~3. Level 0 means that the buffer is almost never eliminated. In other cases, it will be automatically eliminated by the buffer maintenance thread, but the smaller the value, the less likely it is to be eliminated.
[0086] The present invention supports a single or multiple maintenance threads for the buffer, realizes flexible configuration, improves the writing speed in high concurrency scenarios, and ensures the consistency of the buffer changes with the hard disk data; the present invention logically numbers the buffer, and by sorting the buffer blocks, allows the disk to perform sequential writing to the maximum extent, reduces the disk seek time, and improves the writing speed; the disk writing trigger mechanism provided by the present invention can better control the disk writing timing of the buffer, realizes the real-time writing of the disk, reduces the waiting time, and does not affect other operations, thereby improving performance.
[0087] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.
Claims
1. A buffer management method, It is characterized in that The following steps are involved: S1. Start the system and initialize the buffer manager through the main thread; S2. Create a set of control variables, a buffer maintenance thread, one or more buffer write threads, a set of support functions, and put the buffer maintenance thread into a waiting state; S3. When the waiting timeout or the exit signal is received, the buffer maintenance thread leaves the waiting state and processes the dirty data according to the exit flag of the buffer maintenance thread; In the step S3, the dirty data processing according to the exit flag of the buffer maintenance thread specifically includes: if the exit flag is T, the dirty block is written to the disk in a synchronous manner; if the exit flag is not T, the asynchronous IO mode is enabled to write the buffer back to the disk; If the exit flag is not T, the following sub-steps are specifically included: A31. Detect the number of free blocks of various sizes, and judge the size of the detected number of free blocks with the lower limit ratio. If the number of free blocks is greater than the lower limit ratio, proceed to step A32; if the number of free blocks is less than the lower limit ratio, proceed to step A33; A32. Add part of the buffer in the dirty data queue to the disk write queue, enter the waiting state and wait for the exit signal in a timeout manner, and enter step S3 for data processing when the exit signal is received; A33. Select n buffer blocks from the LRU queue, and perform dirty data judgment on the selected n buffer blocks. If the selected buffer block has dirty data, proceed to step A34; if the selected buffer block has no dirty data, directly eliminate the buffer, add it to the free queue, and proceed to step A32; A34 locks the buffer block and sets the pointer to the handler after writing to disk, adds the locked buffer block to the disk write-back queue, starts asynchronous IO, and enters step A32 after completing the buffer write-back; The A34 step includes the following sub-steps: A341. Use latch to lock the buffer; A342. Map the locked buffer to the file handle where the buffer is located and its position in the file according to the table entity and block number of the buffer; A343. Fill the pointer of the post-processing program after writing the buffer to the disk into the callback function pointer of the asynchronous IO control structure to start asynchronous IO.
2. The buffer management method according to claim 1, It is characterized in that The S1 step includes the following sub-steps: S11. Initialize the heads of each queue, each counter, latch and semaphore object; S12. Apply for hash table memory and format the hash table; S13. Apply for buffer header memory and format it into several buffer structures, and link the buffer structures to the free buffer header chain; S14. Apply for data block memory, divide the data block memory into a system data block, and add the system data block to the end of the free block chain.
3. The buffer management method according to claim 1, It is characterized in that There is also a step: using the disk write-back thread to simulate enabling asynchronous IO for buffer write-back. The implementation process specifically includes the following sub-steps: B1. When the system does not support the asynchronous IO mechanism or the asynchronous IO function provided by the system is not enabled, first add an appropriate latch to the buffer, set the post-write processing program pointer, and add the locked buffer to the write queue; B2. Start the disk write-back thread and detect the exit flag of the buffer maintenance thread. If the exit flag is T, proceed to step B3; if the exit flag is not T, proceed to step B4; B3. Determine the serial number of the current disk write-back thread. If the current disk write-back thread is the No. 1 disk write-back thread, the current disk write-back thread will directly exit. If the current disk write-back thread is not the No. 1 disk write-back thread, all buffers in the queue will be written to the disk. After completion, the disk write-back thread will exit. B4. Get a group of buffers in the same table space. If no buffer is obtained, the current disk write-back thread sleeps for one millisecond and returns to step B2. If the buffer is obtained, the obtained buffers are sorted according to the block number, and the write command is called to write the sorted buffers into the table space. After completion, go to step B2.
4. A buffer management system, used to implement the buffer management method according to any one of claims 1 to 3, It is characterized in that It includes buffer disk writing module, buffer maintenance module, memory module, control module and support function module; The buffer disk writing module is used to take out the buffers to be written from the disk writing queue and sort them, and write the sorted buffers to the disk; The buffer maintenance module is used to manage and maintain the number of free buffer blocks, and use the idle time of the buffer request to write dirty blocks back to the disk; The memory module is used to store the data in the buffer and the data that needs to be called; The control module is used to provide buffer lock and unlock functions, and control the buffer write module to write back data through synchronous IO or asynchronous IO; The support function module is used to support the management and operation of the buffer management system and provide functions for the system to call.
5. The buffer zone management system according to claim 4, It is characterized in that The states of the buffer block in the system include: locked and not dirty state, locked and dirty state, unlocked valid and dirty state, unlocked data valid and not dirty state, in disk write queue state, and in buffer state.
6. The buffer zone management system according to claim 4, It is characterized in that The buffer zone maintenance module is in an intermittent circulation state.
