Memory data disk optimization method, system and equipment, computer readable medium and computer program product
By setting dynamic thresholds for the data size that triggers disk writes and the maximum number of files that can exist on the disk, combined with a merging strategy, the data write-to-disk process is optimized, solving the problem of storage system performance degradation caused by write amplification during data write-to-disk processing, and achieving reduced write amplification and improved system performance.
Patent Information
- Application Number
- CN202510882140.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-27
- Publication Date
- 2025-11-07
AI Technical Summary
In existing technologies, write amplification occurs during the data writing process to disk, which leads to a decrease in storage system performance. This is especially true in storage systems with high read/write concurrency, where written data needs to be merged and reordered with disk data, resulting in the actual amount of data written to disk far exceeding the amount written to memory, thus significantly impacting performance.
By setting dynamic thresholds for the data size that triggers disk write-in and the maximum number of files that can exist on the disk, combined with different merging strategies, including a merging module, a threshold increase module, and a disk file merging module, the data write-in process is optimized.
It effectively reduces write amplification in storage systems, ensuring that memory data is written to disk normally, improving system performance, reducing write amplification by several times, and enhancing system efficiency.
Smart Images
Figure CN120909500A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer storage, in particular to a memory data disk-falling optimization method, system, device, computer readable medium and computer program product. BACKGROUND
[0002] Data disk-falling refers to the process of writing temporary data in memory to a persistent storage medium such as a disk, ensuring that data can be retained after power failure or system restart, and is one of the core mechanisms of databases and file systems.
[0003] In general, in a disk-based storage system, especially in a storage system with high read-write concurrency, the data written will first be written to memory, and the read-write performance of memory is much higher than that of disk. When the data in memory reaches a certain threshold, a disk-falling operation is triggered to merge the data in memory with the data on disk.
[0004] In general, the merge operation needs to reorder and index the data in memory and the data on disk together, which leads to write amplification of the storage system. Assuming that 10GB of data has been written to the disk, and the merge is triggered when 1GB of data is written to the memory, at this time, 1GB and 10GB of data need to be merged and then written to the disk. Since merging requires reordering and indexing, the memory data cannot be directly appended to the disk file, and the actual data written to the disk will be close to 11GB, which is nearly 11 times the write amplification of the 1GB of memory data written, and as the disk-falling data continues to increase, it has a great impact on the performance of the system.
[0005] Therefore, how to reduce the write amplification phenomenon in the data disk-falling process is a technical problem to be solved. SUMMARY
[0006] The present application provides a memory data disk-falling optimization method, system, device, computer readable medium and computer program product, which sets the data size threshold for triggering disk-falling of memory data and the maximum number of files that can exist in the disk, to reduce the write amplification of the storage system and improve the performance of the system.
[0007] To achieve the above technical effects, one aspect of the present application provides a memory data disk-falling optimization method, comprising: S101, setting a first threshold of the size of memory data when triggering disk-falling according to the total amount of storage data on the disk, and writing the memory data to the disk to form a new storage file when the size of the memory data reaches the first threshold; S102, when the number of storage files in the disk reaches a preset second threshold, merging the memory data with the smallest storage file in the disk and writing it to the disk; S103, when the size difference between the memory data and the stored files in the disk exceeds a preset third threshold value, increasing the upper limit of the second threshold value and repeating steps S101 and S102; S104, when the number of stored files in the disk reaches a preset fourth threshold value, merging two stored files with the smallest size difference in the stored files in the disk, and repeating steps S101-S104.
[0008] According to the preferred embodiment of the present application, the step S102 further comprises: setting a fourth threshold value of the number of stored files in the disk; setting a second threshold value of the number of stored files in the disk according to the fourth threshold value, wherein the second threshold value is smaller than the fourth threshold value; when the number of stored files in the disk reaches the preset second threshold value, sorting the stored files in the disk from small to large, and merging the memory data with the smallest stored file in the disk and writing the memory data into the disk.
[0009] According to the preferred embodiment of the present application, the step of setting a fourth threshold value of the number of stored files in the disk further comprises: the fourth threshold value c satisfies the following formula: , wherein s is the first threshold value and t is the total amount of stored data of the disk.
[0010] According to the preferred embodiment of the present application, the step of setting a second threshold value of the number of stored files in the disk according to the fourth threshold value further comprises: the second threshold value n satisfies the following formula: , wherein t is the total amount of stored data of the disk, c is the fourth threshold value, and s is the first threshold value.
[0011] According to the preferred embodiment of the present application, the step S103 further comprises: setting a third threshold value of the size difference between the memory data and the smallest file data in the disk; when the size difference between the memory data to be written into the disk and the smallest file data in the disk does not exceed the third threshold value, merging the memory data to be written into the disk with the smallest stored file in the disk and writing the memory data into the disk; when the size difference between the memory data to be written into the disk and the smallest file data in the disk exceeds the third threshold value, sorting the stored files in the disk from small to large, comparing the size difference between the adjacent two stored files in the disk in sequence, merging the adjacent two stored files with the closest size and the size difference not exceeding the third threshold value in the disk, and writing the memory data into the disk; When the difference between the memory data to be written to the disk and the size of all file data in the disk exceeds the third threshold value, and the difference between the sizes of the two adjacent stored files in the disk after sorting also exceeds the third threshold value, the second threshold value is increased by one, and steps S101 and S102 are repeatedly executed.
[0012] According to a preferred embodiment of the present application, the step S104 further comprises: When the number of stored files in the disk reaches the fourth threshold value, the largest stored file F1 is selected from the stored files formed in step S102; The smallest stored file F2 is selected from the stored files newly formed after the upper limit of the second threshold value is increased in step S103; The stored file F1 and the stored file F2 are merged to form a new stored file, and all steps S101-S104 are repeatedly executed.
[0013] According to another aspect of the present application, a memory data disk writing optimization system is provided, comprising: A memory data writing module is configured to set a first threshold value of the size of memory data triggering disk writing according to the total amount of stored data in the disk, and write the memory data to the disk to form a new stored file when the size of the memory data reaches the first threshold value; A memory data merging module is configured to merge the memory data with the smallest stored file in the disk and then write the memory data to the disk when the number of stored files in the disk reaches a preset second threshold value; A threshold value increasing module is configured to increase the upper limit of the second threshold value when the difference between the size of the memory data and the size of the file data in the disk exceeds a preset third threshold value, and then repeatedly execute the work of the memory data writing module and the data merging module in sequence; A disk file merging module is configured to merge the two stored files with the smallest size difference in the stored files in the disk when the number of stored files in the disk reaches a preset fourth threshold value, and then repeatedly execute the work of the memory data writing module, the memory data merging module, the threshold value increasing module, and the disk file merging module in sequence.
[0014] According to another aspect of the present application, a memory data disk writing optimization device is provided, comprising: At least one processor; and A memory in communication connection with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method as described above.
[0015] In another aspect of the present application, a computer readable storage medium is provided, which stores computer program instructions executable by a processor to implement the method described above.
[0016] In another aspect of the present application, a computer program product is provided, which comprises a computer program executable by a processor to implement the method described above.
[0017] In the scheme provided by the embodiments of the present application, by setting the data size threshold of memory data triggering disk writing and the dynamic threshold of the maximum number of files that can exist in the disk, and combining different merging strategies, the storage system write amplification is reduced by several times, and in any case, the memory data can be normally written to the disk, thereby improving the system performance. BRIEF DESCRIPTION OF DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0019] Other features, objects and advantages of the present application will become more apparent from the following detailed description of the non-limiting embodiments, with reference to the accompanying drawings: Figure 1 A flowchart of a memory data disk writing optimization method provided by an embodiment of the present application; Figure 2 A flowchart of step S102 in a memory data disk writing optimization method provided by an embodiment of the present application; Figure 3 A flowchart of step S103 in a memory data disk writing optimization method provided by an embodiment of the present application; Figure 4 A flowchart of step S104 in a memory data disk writing optimization method provided by an embodiment of the present application; Figure 5 A function relationship diagram of different memory data disk writing times and disk writing amounts in the prior art simulation; Figure 6 A function relationship diagram of different memory data disk writing times and disk writing amounts in the simulation of the memory data disk writing optimization method of the present application; Figure 7 A schematic diagram of a memory data disk writing optimization system provided by an embodiment of the present application; Figure 8 A structural schematic diagram of a device suitable for implementing the scheme in the embodiments of the present application.
[0020] The same or similar reference numbers in the drawings represent the same or similar components. DETAILED DESCRIPTION
[0021] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts should fall into the scope of the present application.
[0022] In one typical configuration of the present application, the devices of the terminal and the service network each include one or more processors (CPUs), input / output interfaces, network interfaces and memories.
[0023] The memory can include non-persistent memory in computer-readable media, random access memories (RAMs) and / or non-volatile memories, such as read-only memory (ROM) or flash memory (flash RAM), and the like. The memory is an example of the computer-readable media.
[0024] The computer-readable media include non-transitory and transitory, removable and non-removable media, and can be implemented by any method or technology for storing information. The information can be computer program instructions, data structures, program modules or other data. Examples of the storage media of the computer include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact discs (CD-ROMs), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tape disks storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible to computing devices.
[0025] In actual scenarios, the execution subject of the method can be a user device, or a device integrated by a user device and a network device through a network, or an application program running on the above device, the user device includes but is not limited to computers, mobile phones, tablet computers, smart watches, wristbands and various terminal devices, and the network device includes but is not limited to network hosts, single network servers, multiple network server sets or computer sets based on cloud computing, which can be used to realize part of the processing function when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, wherein cloud computing is a kind of distributed computing, which is composed of a virtual computer set by a loose coupled computer group.
[0026] Term definition: write amplification: the ratio of the actual disk write amount to the memory data amount; Storage file: data block in the disk divided according to the data size threshold.
[0027] In order to reduce the write amplification phenomenon, the files of the disk are divided into blocks in the application, for example, the number of blocks of the disk is set to 10, the memory write reaches 1GB to trigger the disk drop, and 10GB data exists in 10 1GB files. At this time, when the 1GB data in the new memory is written to the disk, since the disk file has reached the set 10, the memory data and the 10 files on the disk find the smallest two to merge, so that the number of files on the disk does not exceed the set value 10. Here, it is assumed that the files on the disk are all 1GB, so the 1GB of memory is merged with one of the 1GB on the disk, and a new 2GB file is written. At this time, the write amplification is 2 times.
[0028] The above scheme can reduce the write amplification when the write times are small, but too many blocks will affect the disk read-write performance, so the number of blocks can only be in a limited range. When the number of blocks is limited, with the increasing number of merging times, the write amplification will also gradually increase.
[0029] For the convenience of description, specific numerical values are used in the above description, but in actual situations, the above numerical values are variable parameters, which can be optimized and adjusted according to the actual situation of the service. The specific parameters are: 1. Memory threshold size (Size, abbreviated as s) of memory write to disk, in order to facilitate description, 1G is used in the above description, in actual situations, it is generally 128MB, 256MB, etc. The value is generally an empirical value.
[0030] 2. Threshold value (Count, abbreviated as c) of the maximum number of data files that can exist on the disk, in order to facilitate description, it is assumed to be 10 in the above description, in actual situations, it needs to be adjusted according to the amount of data stored by the storage service node.
[0031] Principle: the smaller the size gap between the two files, the smaller the write amplification after the file merging and writing.
[0032] According to the principle, the embodiment of the application provides a memory data disk-falling optimization method, Figure 1 A flowchart of a memory data disk-falling optimization method provided by an embodiment of the application is shown in Figure 1 As shown, the method comprises at least the following processing steps: Step S101, according to the total amount of storage data of the disk, setting the first threshold value of the memory data size when the memory data triggers disk-falling, and writing the memory data into the disk to form a new storage file when the memory data size reaches the first threshold value.
[0033] Specifically, according to the above principle, first, according to the total amount of storage data of the disk, set the threshold value of the memory data size when the memory data is written into the disk as the first threshold value, for example, in order to compare with the prior art, set the first threshold value as 1G to trigger the disk-falling operation.
[0034] Step S102, when the number of storage files in the disk reaches a preset second threshold value, merging the memory data with the smallest storage file in the disk and writing into the disk.
[0035] Specifically, Figure 2 A flowchart of step S102 in the memory data disk-falling optimization method provided by an embodiment of the application is shown in Figure 2 As shown, step S102 comprises: S1021, setting the fourth threshold value of the number of storage files in the disk.
[0036] First, set the fourth threshold value of the maximum number of data files that can exist on the disk, for example, set the fourth threshold value of the maximum number of data files that can exist on the disk as 10, so that when the disk file does not reach 10, the memory data is directly written into the disk every time it reaches the first threshold value of 1G, and the first 10 times are directly written into the disk.
[0037] The fourth threshold value c satisfies the following formula: , Where s is the first threshold value, and t is the total amount of storage data of the disk. For example, the total amount of storage data t required is 500G, and the first threshold value of the memory written into the disk is 1G, then the minimum number of files c is 8, and in order to facilitate calculation in the example, it is set to 10, which also meets the condition.
[0038] When the number of files in the disk reaches the fourth threshold value of 10, at this time, the memory data and the 11 data blocks in the disk file will be merged, and the two data blocks with the smallest size will be merged, and the memory data is written to reach 1GB to trigger the merging, for example, after continuous merging, the following situations will occur: Memory: 1G Hard disk: 2G 4G 8G 16G 32G 64G 128G 256G 512G 1024G.
[0039] At this time, the memory data differs from the smallest storage file in the disk by 1 times, and the size difference between the smallest and largest files in the disk is very large, and if the merging continues, it will cause a large write amplification. Obviously, this method has defects, and at this time, a second threshold of the number of storage files in the disk needs to be set.
[0040] S1022, a second threshold of the number of storage files in the disk is set according to the fourth threshold, wherein the second threshold is less than the fourth threshold.
[0041] The second threshold n satisfies the following formula: , Where t is the total amount of storage data of the disk, c is the fourth threshold, and s is the first threshold. The total amount of storage data t is 500G, the first threshold of the memory writing to the disk is 1G, the file number fourth threshold c is set to 10, and the minimum second threshold n is 7.
[0042] S1023, when the number of storage files in the disk reaches the preset second threshold, the storage files in the disk are sorted from small to large, and the memory data is merged with the smallest storage file in the disk and written to the disk.
[0043] When the number of storage files in the disk reaches the preset second threshold 7, the storage files in the disk are sorted from small to large after continuous merging, and the following situations will occur: Memory: 1G Hard disk: 1G 2G 4G 8G 16G 32G 64G.
[0044] After merging the memory data 1G with the smallest storage file 1G in the disk and writing it to the new 1G memory, the following situations finally occur: Memory: 1G Hard disk: 2G 4G 8G 16G 32G 64G 128G At this time, the memory data differs from the smallest storage file in the disk by 1 times, and the size difference between the smallest and largest files in the disk is much smaller than when the fourth threshold is set to 10, but if the merging continues, it will still cause a large write amplification. Therefore, step S103 is performed to further improve the method.
[0045] Step S103, when the size difference between the memory data and the storage file in the disk exceeds the preset third threshold, the upper limit of the second threshold is increased, and steps S101 and S102 are repeatedly executed.
[0046] Specifically, Figure 3A memory data disk writing optimization method provided by an embodiment of the present application includes step S103, as shown in the flowchart of step S103. Figure 3 As shown in step S103, step S103 includes: S1031, setting a third threshold value of a difference between the memory data and the minimum file data size in the disk.
[0047] In order to avoid the case that the memory data is too different from the minimum storage file in the disk, a third threshold value of a difference between the memory data and the minimum file data size in the disk is set, for example, 10%, which can be adjusted as needed.
[0048] S1032, when the difference between the memory data to be written to the disk and the minimum file data in the disk does not exceed the third threshold value, merging the memory data to be written to the disk and the minimum storage file in the disk and writing to the disk.
[0049] When the merging is triggered, the seven data blocks of the disk file are sorted from small to large, and if the memory data is not more than 10% different from the minimum file block size, the memory data is directly merged with the minimum data file.
[0050] S1033, when the difference between the memory data to be written to the disk and the minimum file data in the disk exceeds the third threshold value, sorting the storage files in the disk from small to large, comparing the size difference between the adjacent two storage files in the disk in turn, merging the adjacent two storage files in the disk with the closest size and the difference not exceeding the third threshold value, and writing the memory data to the disk.
[0051] For example, when the merging is triggered, the following cases occur: Memory: 1G Hard disk: 16G 2G 4G 8G 2G 64G 32G, at this time, when the difference between the memory data 1G to be written to the disk and the minimum file data 2G in the disk exceeds the third threshold value 10%, the merging cannot be triggered again, at this time, the storage files in the disk are sorted from small to large, and the sorted hard disk is: 2G 2G 4G 8G 16G 32G 64G, the size difference between the adjacent two storage files in the disk is compared in turn, the adjacent two 2G storage files in the disk with the closest size and the difference not exceeding the third threshold value 10% are merged, and then the memory data is written to the disk, and after writing, the memory data is: Memory: 1G Hard disk: 1G 4G 4G 8G 16G 32G 64G, at this time, the merging operation of the memory and the hard disk file can be performed again.
[0052] S1034, when the difference between the memory data to be written to disk and the size of all file data in the disk exceeds the third threshold value, and the difference between the sizes of the adjacent two stored files in the disk after sorting also exceeds the third threshold value, the second threshold value is increased by one, and steps S101 and S102 are repeatedly executed.
[0053] For example, the following situations occur when the merging is triggered: Memory: 1G Hard disk: 2G 4G 8G 16G 32G 64G 128G, at this time, the difference between the memory data 1G to be written to disk and the size of all file data in the disk exceeds the third threshold value 10%, and the difference between the sizes of the adjacent two stored files in the disk after sorting also exceeds the third threshold value 10%, so the merging cannot be triggered again, at this time, the second threshold value is set as a dynamic threshold value, the eighth file is enabled, that is, the second threshold value is increased by one, and then step S101 is repeatedly executed to directly write the memory data to the disk without merging, at this time, the files are in the order from small to large as follows: Hard disk: 1G 2G 4G 8G 16G 32G 64G 128G, step S102 is executed again, at this time, the second threshold value is 8, and the merging operation is triggered.
[0054] Step S104, when the number of stored files in the disk reaches a preset fourth threshold value, the two stored files with the smallest size difference in the stored files in the disk are merged, and steps S101-S104 are repeatedly executed.
[0055] Specifically, Figure 4 A flowchart of step S104 in a memory data writing to disk optimization method provided by an embodiment of the present application is shown in FIG. 4. Figure 4 As shown in FIG. 4, step S104 includes: S1041, when the number of stored files in the disk reaches the fourth threshold value, the largest stored file F1 is selected from the stored files formed in step S102.
[0056] The operations of steps S101-S103 are repeated until the number of stored files in the disk reaches the fourth threshold value, that is, the second threshold value 7 reaches the maximum, which is equal to the fourth threshold value 10, at this time, the sizes of the files in the hard disk are: 2G 4G 8G 16G 32G 64G 128G(7) 128G(8) 128G(9) 128G(10), because the third threshold value is not met, the merging cannot be triggered according to steps S101-S103, at this time, the largest stored file F1 is selected from the stored files formed in step S102, that is, the largest 128G stored file F1 is selected from 2G 4G 8G 16G 32G 64G 128G(7).
[0057] S1042, selecting the smallest storage file F2 from the newly formed storage file after increasing the upper limit of the second threshold in step S103.
[0058] The upper limit of the second threshold is increased to 8, 9, and 10 in step S103, and the smallest storage file F2 is selected from the newly formed storage files 128G(8) 128G(9) 128G(10) after increasing the upper limit of the second threshold. Since the sizes of the three files are the same in the example, and they are sorted from small to large, the 128G file formed when the upper limit of the second threshold is 8 is considered the smallest.
[0059] S1043, merging the storage file F1 and the storage file F2 to form a new storage file, and repeating all steps S101-S104.
[0060] The storage file F1 and the storage file F2 are the two storage files with the smallest size difference, and the two storage files are merged to form a new 256G storage file. At this time, the file sizes in the hard disk are: 2G 4G 8G 16G 32G 64G 128G(9) 128G(10) 256G, a total of 9 files. The new 1G memory data can be directly written by repeating step S101, and the hard disk is obtained: 1G 2G 4G 8G 16G 32G 64G 128G 128G 256G. The second threshold and the fourth threshold are equal to 10 by repeating steps S102-S104.
[0061] By this operation, compared with the hard disk without setting the second threshold: 2G 4G 8G 16G 32G 64G 128G 256G 512G 1024G, the write amplification is reduced by several times.
[0062] Through the above steps S101-S104, it can be ensured that the memory data can be normally written to the disk under any condition.
[0063] Figure 5 The existing technology simulates the function relationship graph of different memory data disk landing times and disk write amount, Figure 6 The memory data disk landing optimization method of the present application simulates the function relationship graph of different memory data disk landing times and disk write amount, as shown in Figure 5 and 6 The horizontal coordinate is the memory data disk landing times, and the vertical coordinate is the disk write amount. In order to facilitate comparison, the first threshold is set to 1GB, and the number of storage files in the disk is set to 10.
[0064] Figure 5 It can be seen that the total amount of disk writing combined in the prior art increases exponentially with the number of disk landing, and when the disk landing is 1000 times, the total amount of disk writing is about 50,000, and the write amplification is more than 50 times. Figure 6 It can be seen that the total amount of disk writing combined in the prior art increases exponentially with the number of disk landing, and when the disk landing is 1000 times, the total amount of disk writing is about 50,000, and the write amplification is more than 50 times.
[0065] With the continuous increase of the number of disk landing, the effect is more obvious.
[0066] The method embodiment provides a scheme in which a dynamic threshold of the maximum number of files that can exist in the disk and a data size threshold of the memory data triggering disk landing are set, different merging strategies are combined, the write amplification of the storage system is reduced by several times, and the memory data can be normally written to the disk under any condition, thereby improving the system performance.
[0067] Figure 7 A memory data disk landing optimization system provided by an embodiment of the application is shown in FIG. 1. Figure 7 As shown in the figure, the system comprises: The memory data writing module 11 is configured to set a first threshold of the memory data size triggering disk landing according to the total amount of storage data of the disk, and write the memory data to the disk to form a new storage file when the memory data size reaches the first threshold. The memory data merging module 22 is configured to merge the memory data with the smallest storage file in the disk and write the merged data to the disk when the number of storage files in the disk reaches a preset second threshold. The threshold increasing module 33 is configured to increase the upper limit of the second threshold when the difference between the size of the memory data and the size of the file data in the disk exceeds a preset third threshold, and then repeatedly execute the work of the memory data writing module and the data merging module. The disk file merging module 44 is configured to merge the two storage files with the smallest size difference in the storage files in the disk when the number of storage files in the disk reaches a preset fourth threshold, and then repeatedly execute the work of the memory data writing module, the memory data merging module, the threshold increasing module and the disk file merging module.
[0068] The system can execute the memory data disk landing optimization method in the foregoing embodiments, wherein, The memory data writing module 11 executes step S101.
[0069] The memory data merging module 22 performs step S102, sets a fourth threshold value of the number of stored files in the disk; sets a second threshold value of the number of stored files in the disk according to the fourth threshold value, wherein the second threshold value is smaller than the fourth threshold value; when the number of stored files in the disk reaches the preset second threshold value, sorts the stored files in the disk from small to large, and writes the memory data into the disk after merging the memory data with the smallest stored file in the disk.
[0070] wherein the fourth threshold value c satisfies the following formula: wherein s is the first threshold value and t is the total amount of stored data in the disk.
[0071] The second threshold value n satisfies the following formula: wherein t is the total amount of stored data in the disk, c is the fourth threshold value, and s is the first threshold value.
[0072] The threshold value increasing module 33 performs step S103, sets a third threshold value of the difference between the memory data and the size of the smallest file data in the disk; when the difference between the memory data to be written into the disk and the size of the smallest file data in the disk does not exceed the third threshold value, writes the memory data to be written into the disk after merging the memory data with the smallest stored file in the disk; when the difference between the memory data to be written into the disk and the size of the smallest file data in the disk exceeds the third threshold value, sorts the stored files in the disk from small to large, compares the size difference between adjacent two stored files in the disk in turn, merges the adjacent two stored files with the closest size and the difference not exceeding the third threshold value, and writes the memory data into the disk; when the difference between the memory data to be written into the disk and the size of all file data in the disk exceeds the third threshold value, adds one to the second threshold value, and then repeatedly performs the work of the memory data writing module 11 and the data merging module 22 in turn.
[0073] The disk file merging module 44 performs step S104, when the number of stored files in the disk reaches the fourth threshold value, screens the largest stored file F1 from the stored files formed in step S102; screens the smallest stored file F2 from the stored files newly formed after increasing the upper limit of the second threshold value in step S103; merges the stored file F1 with the stored file F2 to form a new stored file, and then repeatedly performs the work of the memory data writing module 11, the memory data merging module 22, the threshold value increasing module 33 and the disk file merging module 44 in turn.
[0074] Based on the same inventive concept, the electronic device provided in the embodiments of the present application corresponds to the memory data flushing optimization method in the foregoing embodiments, and the problem solving principle thereof is similar to that of the method. The electronic device provided in the embodiments of the present application includes at least one processor, and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method and / or technical solution of the foregoing embodiments of the present application.
[0075] The electronic device can be a user device, or a device integrated by a user device and a network device through a network, or also can be an application program running on the above device, the user device includes but is not limited to computers, mobile phones, tablet computers, smart watches, wristbands and various terminal devices, and the network device includes but is not limited to network hosts, single network servers, multiple network server sets or computer sets based on cloud computing, which can be used to realize part of the processing function when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, wherein the cloud computing is a kind of distributed computing, which is composed of a virtual computer set by a loose coupling computer group.
[0076] Figure 8 The structure of a device suitable for implementing the method and / or technical solution in the embodiments of the present application is shown, the device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In the RAM 1203, various programs and data required for system operation are also stored. The CPU 1201, the ROM 1202 and the RAM 1203 are connected to each other through a bus 1204. The input / output (I / O, Input / Output) interface 1205 is also connected to the bus 1204.
[0077] The following components are connected to the I / O interface 1205: an input part 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, and the like; an output part 1207 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, and the like, and a speaker, and the like; a storage part 1208 including one or more computer readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, and the like; and a communication part 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication part 1209 performs communication processing via a network such as the Internet.
[0078] In particular, the methods and / or embodiments in the present application can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods shown in the flowcharts. When the computer program is executed by a central processing unit (CPU) 1201, the above-mentioned functions defined in the methods of the present application are performed.
[0079] Another embodiment of the present application also provides a computer readable storage medium having stored thereon computer program instructions, which can be executed by a processor to implement the method and / or technical solutions of any one or more embodiments of the present application.
[0080] In particular, the embodiments can employ any combination of one or more computer readable media. The computer readable media can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.
[0081] A computer readable signal medium can include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that can be involved in
[0082] The computer readable program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0083] Another embodiment of the application also provides a computer program product, including a computer program, which, when executed by a processor, implements a method for optimizing memory data flushing.
[0084] Computer program code for carrying out operations of the application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).
[0085] The flow diagrams and block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of devices, methods and computer program products according to various embodiments disclosed. In this regard, each block in the flow diagrams and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and
[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.
[0087] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or page components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0088] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.
[0089] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.
[0090] The integrated unit implemented in the form of software function unit can be stored in a computer readable storage medium. The software function unit is stored in a storage medium, and includes a plurality of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of steps of the method described in various embodiments of the present application. The storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage media that can store program codes.
[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
[0092] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The plurality of units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and not to indicate any particular order.
Claims
1. A method for optimizing memory data persistence to disk, characterized in that, The method comprises the following steps: S101, setting a first threshold value of memory data size when memory data is triggered to be written to a disk according to the total amount of storage data of the disk, and writing the memory data to the disk to form a new storage file when the memory data size reaches the first threshold value; S102, when the number of storage files in the disk reaches a preset second threshold value, merging the memory data with the smallest storage file in the disk and then writing the merged data to the disk; S103, when the size difference between the memory data and the storage file in the disk exceeds a preset third threshold value, increasing the upper limit of the second threshold value and repeating steps S101 and S102; S104, when the number of storage files in the disk reaches a preset fourth threshold value, merging the two storage files with the smallest size difference in the storage files in the disk, and repeating steps S101-S104.
2. The method of claim 1, wherein, The step S102 further comprises: setting a fourth threshold value of the number of storage files in the disk; setting a second threshold value of the number of storage files in the disk according to the fourth threshold value, wherein the second threshold value is smaller than the fourth threshold value; when the number of storage files in the disk reaches the preset second threshold value, sorting the storage files in the disk from small to large, merging the memory data with the smallest storage file in the disk, and then writing the merged data to the disk.
3. The method of claim 2, wherein, The step of setting the fourth threshold value of the number of storage files in the disk further comprises: The fourth threshold value c satisfies the following formula: ， Where s is the first threshold value, and t is the total amount of storage data of the disk.
4. The method of claim 3, wherein, The step of setting the second threshold value of the number of storage files in the disk according to the fourth threshold value further comprises: The second threshold value n satisfies the following formula: ， Where t is the total amount of storage data of the disk, c is the fourth threshold value, and s is the first threshold value.
5. The method of claim 1, wherein, The step S103 further comprises: setting a third threshold value of the size difference between the memory data and the smallest file data in the disk; when the size difference between the memory data to be written to the disk and the smallest file data in the disk does not exceed the third threshold value, merging the memory data to be written to the disk with the smallest storage file in the disk and then writing the merged data to the disk; when the size difference between the memory data to be written to the disk and the smallest file data in the disk exceeds the third threshold value, sorting the storage files in the disk from small to large, comparing the size difference between the adjacent two storage files in the disk in turn, merging the adjacent two storage files with the closest size and the size difference not exceeding the third threshold value, and then writing the memory data to the disk; when the size difference between the memory data to be written to the disk and all file data in the disk exceeds the third threshold value, and the size difference between the adjacent two storage files after sorting in the disk also exceeds the third threshold value, increasing the second threshold value by one, and repeating steps S101 and S102.
6. The method of claim 1, wherein, The step S104 further comprises: when the number of storage files in the disk reaches the fourth threshold value, selecting the largest storage file F1 from the storage files formed in step S102; selecting the smallest storage file F2 from the newly formed storage files after increasing the upper limit of the second threshold value in step S103; merging the storage file F1 and the storage file F2 to form a new storage file, and repeating all steps S101-S104.
7. A memory data flushing optimization system, comprising: The method comprises the following steps: The memory data writing module is configured to set a first threshold of the memory data size when triggering the memory data to be written to the disk to form a new storage file according to the total amount of the storage data of the disk; The memory data merging module is configured to merge the memory data with the smallest storage file in the disk and then write the merged data to the disk when the number of the storage files in the disk reaches a preset second threshold; The threshold increasing module is configured to increase the upper limit of the second threshold when the difference between the memory data and the file data in the disk exceeds a preset third threshold, and then repeatedly execute the memory data writing module and the data merging module in sequence; The disk file merging module is configured to merge the two storage files with the smallest size difference in the storage files in the disk when the number of the storage files in the disk reaches a preset fourth threshold, and then repeatedly execute the memory data writing module, the memory data merging module, the threshold increasing module and the disk file merging module in sequence.
8. A memory data flushing optimization device, comprising: The device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.
9. A computer readable medium having stored thereon computer program instructions, characterized in that, The computer program instructions can be executed by the processor to implement the method of any one of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 6.