Adaptive buffer storage method, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention discloses a self-adaptive buffer storage method, electronic equipment, a computer readable storage medium and a computer program product, and the method comprises the steps that multiple layers of memory pools are created in a buffer and at least comprise a first memory pool, a second memory pool and a third memory pool, the first memory pool comprises a plurality of memory blocks with the size of a, and the pre-allocation number is N1; the second memory pool comprises a plurality of memory blocks with the size of b, and the pre-allocation number is N2; the third memory pool comprises a plurality of memory blocks with the size of c, and the pre-allocation number is N3; a < b < c, N1 > N2 > N3; receiving a memory allocation request with buffer strategy configuration; the buffer strategy configuration comprises a strategy type and a buffer size; selecting a memory pool from a plurality of layers of memory pools according to the strategy type, reading file contents into a memory block according to the size of a buffer area, and creating a buffer area object; and directly accessing the file content through the buffer area object.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of buffer management, and particularly relates to a self-adaptive buffer storage method, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] A buffer is an important memory management technology in a computer system, refers to a continuous or scattered storage area specially divided in a main memory, and is mainly used for temporarily storing data transmitted between devices or processing units with different speeds. Its core role is to serve as a "transit station" in the data flow process, effectively alleviating the mismatch between producers (data sources) and consumers (data processing units) in terms of processing speed, data generation frequency and access timing. In addition, the buffer also has an important cache function, which can effectively avoid the performance bottleneck of the CPU due to waiting for slow disk I / O operations. At the same time, the buffer can intelligently aggregate multiple small-scale data requests into larger-scale batch operations, thereby significantly reducing the access frequency of the system to the disk.

[0003] The existing operating system provides a buffer for the I / O operation of a user process, uses a fixed-size memory page as the buffer, usually 4 KB, and relies on an LRU or other eviction algorithm to manage buffer data. Although this scheme can provide basic buffering services for most applications, it exposes obvious limitations when facing modern complex application scenarios: a fixed-size buffer cannot well adapt to the file size difference from KB to GB. If a larger buffer is fixedly used, accessing small files will result in low memory utilization and waste of memory space; if a smaller buffer is fixedly used, accessing large files will require managing a large number of scattered memory page mappings, increasing management overhead and memory fragmentation. In a high-concurrency scenario, these problems will be further exacerbated. SUMMARY

[0004] The application aims to solve the problems of low memory utilization, serious memory fragmentation and large management overhead of the operating system buffer in a high-concurrency scenario, and proposes a self-adaptive buffer storage method, an electronic device, a computer readable storage medium and a computer program product, which adaptively selects the optimal buffering strategy for different file sizes and access modes at runtime.

[0005] Technical solution: in a first aspect, the application proposes a self-adaptive buffer storage method, comprising the following steps:

[0006] A multi-layer memory pool is created in the buffer area, including at least a first memory pool, a second memory pool and a third memory pool, the first memory pool includes a plurality of memory blocks with a size of a, and a pre-allocated number of N1; the second memory pool includes a plurality of memory blocks with a size of b, and a pre-allocated number of N2; the third memory pool includes a plurality of memory blocks with a size of c, and a pre-allocated number of N3; a<b<c, N1>N2>N3;

[0007] receiving a memory allocation request with buffer policy configuration; the buffer policy configuration includes: policy type and buffer size;

[0008] According to the policy type, a memory pool is selected from the multi-layer memory pool, the file content is read into the memory block according to the buffer size, and the buffer object is created; the file content is directly accessed through the buffer object.

[0009] Further, the memory allocation request with buffer policy configuration is obtained according to the following steps:

[0010] Obtaining file information, including file size and access mode;

[0011] According to the file size, the policy type is determined; after determining the policy type, the buffer size is determined according to the access mode;

[0012] Based on the determined policy type and buffer size, the memory allocation request with buffer policy configuration is obtained.

[0013] Further, the policy type is determined according to the file size, including:

[0014] If the file size is less than or equal to a, the determined policy type is byte array strategy;

[0015] If a

[0016] If the file size is greater than or equal to b, the determined policy type is multi-buffer strategy;

[0017] Further, when the policy type is determined, the buffer size is determined according to the access mode, including:

[0018] For single-buffer strategy, if the access mode is sequential access, the buffer size is set to one quarter of the file size, and is rounded up to an integer multiple of a; if the access mode is random access, the buffer size is set to one half of the file size, and is rounded up to an integer multiple of a;

[0019] For multi-buffer strategy, the buffer size is determined according to the file size.

[0020] Further, the following steps are further included:

[0021] Check if there is available memory block in the corresponding memory pool: if there is no available memory block, use LRU strategy to evict and recycle the long time unused memory block.

[0022] Further, for single buffer strategy, when the read position is close to the end of the buffer, the next block of data is asynchronously pre-read.

[0023] Further, for multi-buffer strategy, according to the buffer size, the required buffer number is calculated, and the corresponding number of file contents are loaded into the large memory block in parallel using multi-threading, and a multi-buffer object is created

[0024] If the buffer strategy is multi-buffer strategy, allocate memory blocks from the large memory pool; that is, according to the request parameter to determine the buffer size BUFFER_SIZE, the required buffer number is calculated as 1 MB / BUFFER_SIZE, and the corresponding number of file contents are loaded into the corresponding memory block in parallel using multi-threading, and a multi-buffer object is created.

[0025] In a second aspect, the present application provides an electronic device, which comprises:

[0026] at least one processor;

[0027] and a memory in communication connection with the at least one processor;

[0028] wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute an adaptive buffer storage method.

[0029] In a third aspect, the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to execute an adaptive buffer storage method when executed.

[0030] In a fourth aspect, the present application provides a computer program product, which comprises a computer program, and the computer program is used to enable a processor to execute an adaptive buffer storage method when executed.

[0031] Beneficial effects: compared with the prior art, the present application selects the optimal buffer strategy and buffer size adaptively according to the file size and access mode at runtime, which avoids the waste of memory caused by using a too large buffer to carry a small file, and prevents a large number of fragmentation and management overhead caused by using a too small buffer to process a large file, thereby significantly improving the memory utilization and reducing the fragmentation and management overhead. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The small file byte array strategy processing flow chart is provided for the third embodiment of the application.

[0033] Figure 2 The medium file single buffer strategy processing flow chart is provided for the fourth embodiment of the application.

[0034] Figure 3 The large file multi-buffer strategy processing flow chart is provided for the fifth embodiment of the application. DETAILED DESCRIPTION

[0035] Embodiment one:

[0036] The embodiment of the application proposes an intelligent buffer strategy selection method, mainly including the following steps:

[0037] Receiving a file access request, obtaining the basic information of the file, including the file size FILE_SIZE and the access mode;

[0038] According to the preliminary judgment according to the file size FILE_SIZE: if FILE_SIZE ≤ 4 KB, the byte array strategy is selected; if 4 KB ≤ FILE_SIZE < 64 KB, the single buffer strategy is selected; if FILE_SIZE ≥ 64 KB, the multi-buffer strategy is selected;

[0039] For the single buffer strategy, if the access mode is sequential access, the buffer size is set to one fourth of the file size, and is rounded up to an integer multiple of 4 KB; if the access mode is random access, the buffer size is set to one half of the file size, and is rounded up to an integer multiple of 4 KB, so as to better support the random access scenario.

[0040] For the multi-buffer strategy, if the file size FILE_SIZE < 10 MB, the buffer size is set to 64 KB; if 10 MB ≤ FILE_SIZE ≤ 1 GB, the buffer size is set to 256 KB; if FILE_SIZE > 1 GB, the buffer size is set to 1 MB;

[0041] Generating the final buffer strategy configuration, including the strategy type, buffer size and other parameters, and sending a memory allocation request with the buffer strategy configuration to the memory pool manager.

[0042] Embodiment two:

[0043] The embodiment of the application proposes a memory pool management and allocation method, mainly including the following steps:

[0044] Step 1: When the system starts, initialize the buffer memory pool manager and create a three-level memory pool structure: small block memory pool, which manages 4 KB memory blocks with a pre-allocation quantity of 1000; medium block memory pool, which manages 64 KB memory blocks with a pre-allocation quantity of 100; and large block memory pool, which manages 1 MB memory blocks with a pre-allocation quantity of 10.

[0045] Step 2: Receive a memory allocation request with a buffer policy configuration and check if there are any available memory blocks in the corresponding pool. If there are no available memory blocks, use the LRU policy to evict and reclaim memory blocks that have not been accessed for a long time.

[0046] Step 3: If the buffering strategy is a byte array strategy, allocate memory blocks from the small memory pool, that is, read the entire file content into a 4KB small memory block at once, generate a byte array with an array size equal to the file size, and create a byte array buffer object; if the buffering strategy is a single buffer strategy, allocate memory blocks from the medium memory pool, that is, determine the buffer size according to the request parameters, read the corresponding size of the file content into a 64KB medium memory block, and create a buffer object; implement a pre-read mechanism: when the read position is close to the end of the buffer, the system will asynchronously pre-read the next block of data to improve the efficiency of subsequent accesses; if the buffering strategy is a multi-buffer strategy, allocate memory blocks from the large memory pool; that is, determine the buffer size BUFFER_SIZE according to the request parameters, calculate the required number of buffers as 1 MB / BUFFER_SIZE, and use multi-threading to load the corresponding number of file contents into the large memory block in parallel, and create multiple buffer objects.

[0047] Example 3:

[0048] The embodiments of the present invention are illustrated using the small file byte array strategy as an example.

[0049] Figure 1 This illustrates the complete process of an adaptive buffer storage system handling small files, specifically including the following steps:

[0050] Suppose there is a configuration file config.toml, which is 2 KB in size and needs to be read and accessed frequently.

[0051] The application sends a file access request to the system, passing in the file path " / config / config.toml" and the access mode RANDOM_ACCESS;

[0052] The system obtains basic file information, including file size FILE_SIZE=2KB and access mode ACCESS_PATTERN=RANDOM_ACCESS;

[0053] The policy selector receives the file information, judges that FILE_SIZE = 2 KB < 4 KB, and selects the byte array policy;

[0054] The memory pool manager receives the allocation request, allocates a 4 KB memory block from the small block memory pool, actually uses 2 KB, and the memory address is 0x7F8A12000000;

[0055] The file I / O module reads the complete content of config.toml into the allocated memory address 0x7F8A12000000, and creates a byte array buffer object;

[0056] The application directly accesses the configuration file data through the buffer object;

[0057] In the subsequent process of reading the configuration file, the buffer object is directly constructed from the existing buffer, and the config.toml file data is read through the buffer object.

[0058] Embodiment four:

[0059] The embodiment of the application takes the medium file single buffer strategy processing as an example for illustration.

[0060] Figure 2 The complete flow of the adaptive buffer storage system processing medium size file is shown, which specifically includes the following steps:

[0061] Suppose there is a data file data.csv, the size is 50 KB, and it needs to be sequentially read and processed.

[0062] The application initiates a file access request, and inputs the file path " / data / data.csv" and the access mode SEQUENTIAL_ACCESS;

[0063] The system obtains the file information, FILE_SIZE = 50 KB, and the access mode ACCESS_PATTERN = SEQUENTIAL_ACCESS;

[0064] The policy selector judges that 4 KB ≤ FILE_SIZE = 50 KB < 64 KB, selects the single buffer strategy, ACCESS_PATTERN = SEQUETIAL_ACCESS, sets the buffer size BUFFER_SIZE = FILE_SIZE / 4 = 12.5 KB, and rounds up to 16 KB;

[0065] The memory pool manager allocates a 64 KB memory block from the medium block memory pool, and creates a single buffer object;

[0066] The application program reads the request for the first time, loads the first 16 KB data of the file to the buffer, and triggers the pre-reading mechanism to load the next 16 KB data block asynchronously when the reading position approaches the end of the buffer;

[0067] When the subsequent process reads the data file, the buffer object is directly constructed from the existing single buffer, and the data of the data.csv file is read through the single buffer object.

[0068] Embodiment five:

[0069] The embodiment of the application takes the large file multi-buffer strategy processing as an example for illustration.

[0070] Figure 3 The complete flow of the adaptive buffer storage system processing a large file is shown, which specifically includes the following steps:

[0071] Suppose there is a large data file bigdata.bin, with a size of 500 MB, which needs to be randomly accessed.

[0072] The application program initiates a file access request, and inputs the file path " / data / bigdata.bin" and the access mode RANDOM_ACCESS;

[0073] The system obtains the file information, FILE_SIZE = 500 MB, and ACCESS_PATTERN = RANDOM_ACCESS;

[0074] The strategy selector judges that FILE_SIZE = 500 MB ≥ 64 KB, and selects the multi-buffer strategy;

[0075] According to the file size, the buffer size is determined, FILE_SIZE = 500 MB belongs to the range of 10MB-1GB, and BUFFER_SIZE = 256 KB is set, and the file is divided into 2000 data blocks in total;

[0076] The memory pool manager allocates a 1 MB memory block from the large memory pool to create a multi-buffer object:

[0077] The application program requests to access the data at the file offset 1MB position, calculates CHUNK_INDEX = 1 MB / 256 KB = 4, checks whether CHUNK_INDEX = 4 has been loaded, and finds that it has not been loaded, uses 4 threads to load the data of CHUNK_INDEX = [4, 5, 6, 7] in parallel to improve the loading efficiency;

[0078] When the subsequent process reads the data file, the multi-buffer object is constructed directly from the existing buffer, and the bigdata.bin file data is read through the multi-buffer object.

Claims

1. An adaptive buffer storage method, characterized in that: Includes the following steps: Create a multi-level memory pool in the buffer, including at least a first memory pool, a second memory pool, and a third memory pool. The first memory pool includes several memory blocks of size 'a', with a pre-allocated number of N1. The second memory pool includes several memory blocks of size 'b', with a pre-allocated number of N2. The third memory pool includes several memory blocks of size 'c', with a pre-allocated number of N3. a < b < c, N1 > N2 > N3. Receive a memory allocation request with a buffer policy configuration; the buffer policy configuration includes: policy type and buffer size; Based on the strategy type, a memory pool is selected from the multi-level memory pools. Based on the buffer size, the file content is read into the memory block and a buffer object is created. The file content is then accessed directly through the buffer object.

2. The adaptive buffer storage method according to claim 1, characterized in that: The memory allocation request with the attached buffer policy configuration is obtained according to the following steps: Obtain file information, including file size and access mode; Determine the policy type based on the file size; once the policy type is determined, determine the buffer size based on the access mode. Based on the determined policy type and buffer size, a memory allocation request with attached buffer policy configuration is obtained.

3. The adaptive buffer storage method according to claim 2, characterized in that: The determination of the strategy type based on file size includes: If the file size is less than or equal to a, then the determined strategy type is the byte array strategy. If a < file size < b, then the determined strategy type is a single buffer strategy; If the file size is greater than or equal to b, then the determined strategy type is a multi-buffer strategy.

4. The adaptive buffer storage method according to claim 3, characterized in that: The process of determining the buffer size based on the access mode after determining the policy type includes: For a single-buffer strategy, if the access mode is sequential access, the buffer size is set to one-quarter of the file size and rounded up to an integer multiple of 'a'; if the access mode is random access, the buffer size is set to half of the file size and rounded up to an integer multiple of 'a'. For a multi-buffer strategy, the buffer size is determined based on the file size.

5. The adaptive buffer storage method according to claim 1, characterized in that: It also includes the following steps: Check if there are any available memory blocks in the corresponding memory pool: if there are no available memory blocks, use the LRU policy to evict and reclaim memory blocks that have not been accessed for a long time.

6. The adaptive buffer storage method according to claim 4, characterized in that: For a single-buffer strategy, when the read position is close to the end of the buffer, the next block of data is read asynchronously.

7. The adaptive buffer storage method according to claim 4, characterized in that: For the multi-buffer strategy, the required number of buffers is calculated based on the buffer size, and the corresponding number of file contents are loaded into a large memory block in parallel using multiple threads, and multi-buffer objects are created. If the buffering strategy is a multi-buffer strategy, memory blocks are allocated from the large memory pool; that is, the buffer size BUFFER_SIZE is determined according to the request parameters, the required number of buffers is calculated to be 1 MB / BUFFER_SIZE, and the corresponding number of file contents are loaded into the corresponding memory blocks in parallel using multiple threads, and multi-buffer objects are created.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform an adaptive buffer storage method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute an adaptive buffer storage method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements an adaptive buffer storage method according to any one of claims 1-7.

Images