Disk refreshing method and device based on nonvolatile memory, equipment and medium

Abstract

The invention relates to the technical field of data storage, and discloses a disk refreshing method and device based on a nonvolatile memory, equipment and a medium. Write-in threads are grouped according to the core number of a central processing unit to obtain a plurality of thread groups; and writing data corresponding to each target write-in thread in the target thread group into the target disk brushing buffer area, and suspending the target thread group. And when the target disk refreshing buffer receives data sent by all the corresponding target write-in threads, synchronously writing the data in the target disk refreshing buffer into the solid state disk, and releasing the target thread group. By dividing the thread groups and distributing a disk refreshing buffer area for each thread group, a message storage mode of sequential writing and random reading can be realized, and the capacity limitation of the current file storage scheme is solved. When disk refreshing is executed, the data corresponding to the multiple write-in threads in one thread group are synchronously aggregated for disk refreshing, so that frequent switching between a user mode and a kernel mode is reduced, and the message write-in speed is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a disk flashing method, device, equipment and medium based on non-volatile memory. Background Art

[0002] With the rapid development of computer and network technology, in extreme scenarios where large amounts of data are continuously read and written, cache pollution caused by insufficient cache will seriously affect the efficiency of disk flushing. The so-called disk flushing is to write and save the messages of multiple writing threads to the solid state disk (SSD). The program will dynamically group the writing threads, aggregate them and write them to the disk flushing buffer, and then write them to the SSD in a unified manner. This is a synchronous disk flushing process that can ensure the power-off security of data.

[0003] The message itself is designed with attributes such as topic, queueId, offset, etc. A queue is a two-dimensional array. Once the topic and queueId are determined, a unique queue can be determined. Queues do not store individual messages, but the indexes of individual messages. The offset can be used to uniquely determine the index of a message and thus find the message. Since a pair of topic and queueId can determine a unique queue, if there are N topics and each topic contains M queueIds, there will be N*M queues. The current file storage solution is that each queue corresponds to a file. This storage solution writes randomly and reads continuously. However, the random write performance of both SSDs and disks is very poor, which will affect the writing speed of messages. In addition, N*M files will be created under this solution. The Linux operating system defaults to a maximum of 1024 file handles that a single process can open. When N*M is very large, the limit will be exceeded.

[0004] It can be seen that how to improve the message writing speed is a problem that technical personnel in this field need to solve. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a disk flushing method, device, equipment and medium based on non-volatile memory, which can solve the problem of slow message writing speed.

[0006] In order to solve the above technical problems, an embodiment of the present invention provides a disk flashing method based on non-volatile memory, comprising:

[0007] The writing threads are grouped according to the number of cores of the central processing unit to obtain a plurality of thread groups; each thread group corresponds to a disk flushing buffer; the disk flushing buffer is a non-volatile memory;

[0008] Writing the data corresponding to each target writing thread in the target thread group into the target flush buffer, and suspending the target thread group; wherein the target thread group is any thread group among all thread groups;

[0009] When the target disk flushing buffer receives data sent by all corresponding target writing threads, the data in the target disk flushing buffer is synchronously written to the solid state drive, and the target thread group is released.

[0010] On the one hand, when the target flush buffer receives data sent by all corresponding target write threads, synchronously writing the data in the target flush buffer to the solid state drive and releasing the target thread group includes:

[0011] When all target write threads included in the target thread group reach the barrier, all data in the target disk flush buffer are uniformly and forcibly written to the solid-state drive based on the asynchronous thread, and the suspended target thread group is controlled to resume operation.

[0012] On the one hand, when all target writing threads included in the target thread group reach the barrier, all data in the target flush buffer are uniformly and forcibly written to the solid state drive based on the asynchronous thread, including:

[0013] Get the number of threads in the target thread group that have reached the barrier;

[0014] Determine whether the number of threads reaches the number of barrier threads;

[0015] When the number of threads reaches the number of barrier threads, all data in the target disk flush buffer are forcibly written to the solid state drive based on the asynchronous thread.

[0016] On the one hand, with respect to the capacity setting of the disk flushing buffer, the method further includes:

[0017] Get the number of active threads in the target thread group during program execution;

[0018] Based on the set single message data volume, sector byte number, original offset and the number of active threads, the capacity of the target flush buffer corresponding to the target thread group is determined; wherein the target flush buffer is the flush buffer corresponding to the target thread group;

[0019] Writing the data corresponding to each target writing thread in the target thread group into the target flush buffer includes:

[0020] Aggregate and write the data corresponding to all target writing threads in the target thread group to the target flush buffer;

[0021] After the data corresponding to all target writing threads in the target thread group are aggregated and written to the target flush buffer, the method further includes:

[0022] Determine whether the target thread group has a corresponding target persistent file; wherein the target persistent file is a storage area divided from the solid state drive;

[0023] In the case where the target thread group has a corresponding target persistent file, adjusting the file capacity of the target persistent file according to the total amount of all data in the target flush buffer;

[0024] In the case that the target thread group does not have a corresponding target persistent file, determining the file capacity for performing the disk flushing operation according to the total amount of all data in the target disk flushing buffer; and dividing the target persistent file from the solid state drive according to the file capacity;

[0025] Synchronously writing the data in the target flush buffer into the solid state drive comprises:

[0026] The data in the target flush buffer is written into the target persistent file.

[0027] On the one hand, based on the set single message data volume, sector byte number, original offset and the number of active threads, determining the capacity of the target flush buffer corresponding to the target thread group includes:

[0028] The disk flushing buffer calculation formula is called to process the set single message data volume, sector byte number, original offset and the number of active threads to determine the capacity of the target disk flushing buffer corresponding to the target thread group; the disk flushing buffer calculation formula is:

[0029] S=(20*number of active threads / 4+x);

[0030] Among them, S represents the capacity of the target disk flushing buffer, 20 is the amount of single message data, 4 is the number of sector bytes, and x represents the original offset.

[0031] On the one hand, it also includes:

[0032] Map each persistent file to the address space of the process to achieve the mapping relationship between the disk address of the persistent file and the virtual address of the process;

[0033] Writing the data in the target flush buffer into the target persistent file includes:

[0034] Read the data in the target disk flushing buffer through a pointer;

[0035] According to the mapping relationship, the data read from the target disk flushing buffer is written into the target persistent file.

[0036] On the one hand, the writing threads are grouped according to the number of cores of the central processing unit to obtain multiple thread groups including:

[0037] Divide all writing threads into multiple thread groups equal to the number of CPU cores.

[0038] The embodiment of the present invention also provides a disk flushing device based on non-volatile memory, comprising a grouping unit, a cache unit, a suspension unit, a writing unit and a releasing unit;

[0039] The grouping unit is used to group the writing threads according to the number of cores of the central processing unit to obtain a plurality of thread groups; wherein each thread group corresponds to a disk flushing buffer; the disk flushing buffer is a non-volatile memory;

[0040] The cache unit is used to write the data corresponding to each target writing thread in the target thread group into the target flushing buffer;

[0041] The suspending unit is used to suspend the target thread group; wherein the target thread group is any one of all thread groups;

[0042] The writing unit is used to synchronously write the data in the target flushing buffer into the solid state drive when the target flushing buffer receives the data sent by all the corresponding target writing threads;

[0043] The releasing unit is used to release the target thread group.

[0044] On the one hand, the writing unit is used to force all data in the target flush buffer to be written to the solid state drive uniformly based on the asynchronous thread after all target writing threads included in the target thread group reach the barrier;

[0045] The release unit is used to control the suspended target thread group to resume running.

[0046] On the one hand, the writing unit includes an acquisition subunit, a judgment subunit and a forced writing subunit;

[0047] The acquisition subunit is used to acquire the number of threads of the target thread group that have reached the barrier;

[0048] The judging subunit is used to judge whether the number of threads reaches the number of barrier threads;

[0049] The forced writing subunit is used to force all data in the target disk flushing buffer to be written uniformly to the solid state drive based on asynchronous threads when the number of threads reaches the number of barrier threads.

[0050] On the one hand, with respect to the capacity setting of the disk flushing buffer, the device further comprises an acquisition unit, a buffer determination unit, a file determination unit, an adjustment unit, a file capacity determination unit and a division unit;

[0051] The acquisition unit is used to acquire the number of active threads of the target thread group during program execution;

[0052] The buffer determination unit is used to determine the capacity of the target disk flushing buffer corresponding to the target thread group based on the set single message data volume, sector byte number, original offset and the number of active threads; wherein the target disk flushing buffer is the disk flushing buffer corresponding to the target thread group;

[0053] The cache unit is used to aggregate and write data corresponding to all target writing threads in the target thread group into the target flushing buffer;

[0054] The file determination unit is used to determine whether the target thread group has a corresponding target persistent file; wherein the target persistent file is a storage area divided from the solid state drive;

[0055] The adjustment unit is used to adjust the file capacity of the target persistent file according to the total amount of all data in the target flush buffer when the target thread group has a corresponding target persistent file;

[0056] The file capacity determination unit is used to determine the file capacity for performing the disk flushing operation according to the total amount of all data in the target disk flushing buffer when the target thread group does not have a corresponding target persistent file;

[0057] The partitioning unit is used to partition the target persistent file from the solid state drive according to the file capacity;

[0058] The writing unit is used to write the data in the target disk flushing buffer into the target persistent file.

[0059] On the one hand, the buffer determination unit is used to call the disk flush buffer calculation formula to process the set single message data volume, sector byte number, original offset and the number of active threads to determine the capacity of the target disk flush buffer corresponding to the target thread group; the disk flush buffer calculation formula is:

[0060] S=(20*number of active threads / 4+x);

[0061] Among them, S represents the capacity of the target disk flushing buffer, 20 is the amount of single message data, 4 is the number of sector bytes, and x represents the original offset.

[0062] On the one hand, it also includes a mapping unit;

[0063] The mapping unit is used to map each persistent file to the address space of the process to realize the mapping relationship between the disk address of the persistent file and the virtual address of the process;

[0064] The writing unit is used to read the data in the target disk flushing buffer through a pointer; and write the data read from the target disk flushing buffer into the target persistent file according to the mapping relationship.

[0065] On the one hand, the grouping unit is used to divide all writing threads into a plurality of thread groups having the same number of cores as the central processing unit.

[0066] The embodiment of the present invention further provides a disk-swiping device based on a non-volatile memory, comprising:

[0067] Memory for storing computer programs;

[0068] A processor is used to execute the computer program to implement the steps of the above-mentioned disk flashing method based on non-volatile memory.

[0069] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the disk flashing method based on non-volatile memory are implemented as described above.

[0070] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned disk flashing method based on non-volatile memory when executed by a processor.

[0071] It can be seen from the above technical solution that the write threads are grouped according to the number of cores of the central processing unit to obtain multiple thread groups; wherein each thread group corresponds to a disk flushing buffer; the disk flushing buffer is a non-volatile memory, and the non-volatile memory has the characteristics of persistence, which can ensure the security of data when power is off or the memory bar is damaged. The processing method of each thread group is similar. Taking any thread group among all thread groups, that is, the target thread group, as an example, the data corresponding to each target write thread in the target thread group can be written into the target disk flushing buffer. In order to avoid the target write thread being called by other applications during data caching, causing confusion in the data cache, the target thread group can be suspended at this time. When the target disk flushing buffer receives the data sent by all the corresponding target write threads, the data in the target disk flushing buffer is synchronously written to the solid-state hard disk, and the target thread group is released. The beneficial effect of the present invention is that by dividing the thread groups and allocating a disk flushing buffer to each thread group, a message storage method of sequential writing and random reading can be realized, which can effectively give play to the characteristics of fast random reading of non-volatile memory, and solve the capacity limitation of the current file storage solution. When performing disk flushing, the data corresponding to multiple writing threads in a thread group are synchronously aggregated for disk flushing, which reduces the frequent switching between user state and kernel state and improves the writing speed of messages. The present invention can ensure the validity of data power failure while the writing speed reaches the upper limit of the bandwidth of the solid-state hard disk, so that the data will not be lost during power failure, and solves the problem that the message middleware is read and written sequentially under the traditional storage structure and will only be read once, and is easily lost during power failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 A schematic diagram of a logical address of a message provided by the prior art;

[0074] Figure 2 A flowchart of a disk flashing method based on non-volatile memory provided by an embodiment of the present invention;

[0075] Figure 3 A schematic diagram of a gathering brush operation provided by an embodiment of the present invention;

[0076] Figure 4 A schematic diagram of file storage provided by an embodiment of the present invention;

[0077] Figure 5 A schematic diagram of a file channel mapping provided by an embodiment of the present invention;

[0078] Figure 6 A schematic diagram of the structure of a disk-swiping device based on non-volatile memory provided by an embodiment of the present invention;

[0079] Figure 7 A structural diagram of a disk-swiping device based on non-volatile memory provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0082] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0083] The message itself has attributes such as topic, queueId, offset, etc. Topic refers to a collection of messages of the same type, and queue is the physical entity that stores messages. A topic can contain multiple queues, and each queue has its own id number, i.e., queueId. The logical address of a message is Figure 1 As shown, there are multiple queues under each topic, and each queue has its own queue number. After the queue is uniquely identified by the topic and queue number, the specific message in the queue is found according to the offset. Figure 1 In the example, two topics, Topic 0 and Topic 1, are used. Each topic contains four queues, namely Queue 1 to Queue 4. Each queue can store four messages, and each message has its corresponding offset.

[0084] One current storage solution is to assign a file to each queue. This solution writes randomly and reads continuously. However, the random write performance of both SSDs and disks is very poor, which will affect the writing speed of messages. Another storage solution is to assign a file to each topic. For example, if there are N topics, N files need to be created. This solution is random reading and random writing, which will also cause a low writing speed. When the number of files is large, these two storage solutions will exceed the file limit. In addition, the storage method of assigning a file to each topic will result in data from multiple queues being written to the same file. However, due to the need to ensure data security in the event of power outages or memory damage, the disk needs to be manually flushed after each message is written. This will cause frequent switching between user mode and kernel mode, resulting in low efficiency.

[0085] In the embodiment of the present invention, considering the high cost performance, byte-addressable characteristics and persistence advantages of non-volatile memory (Persistent Memory Module, PMEM), a new choice is brought to the storage part of the message middleware. Among them, non-volatile memory can also be called persistent memory.

[0086] Message middleware is a supporting software system that provides synchronous or asynchronous, reliable message transmission for application systems in a network environment based on queue and message passing technology. Different from other systems, the messages of message middleware have unique data characteristics: messages are large amounts of small-capacity data; messages under the same queue are read one by one in order; all messages are read only once. These characteristics have brought great challenges to the introduction of non-volatile memory in the middleware storage system. Therefore, the embodiment of the present invention proposes a method for fast disk flushing based on non-volatile memory in view of the data characteristics of the message middleware, and designs a message storage solution with sequential writing and random reading, which can effectively give play to the characteristics of fast random reading speed of non-volatile memory. In the disk flushing strategy, a multi-threaded aggregation synchronous disk flushing method is adopted, which can ensure data power-off safety while reaching the upper limit of SSD write speed.

[0087] Next, a disk flashing method based on non-volatile memory provided by an embodiment of the present invention is described in detail. Figure 2 A flowchart of a disk flashing method based on non-volatile memory provided by an embodiment of the present invention, the method comprising:

[0088] S201: grouping writing threads according to the number of cores of a central processing unit to obtain a plurality of thread groups.

[0089] Each thread group corresponds to a disk flush buffer; the disk flush buffer is a non-volatile memory.

[0090] Flashing is to write the messages to be written into the SSD for storage, and to ensure the security of data when power is off or the memory stick is damaged while writing all the bandwidth. Although non-volatile memory has the characteristics of persistence, its capacity is still much smaller than that of SSD, and its cost is many times higher than that of SSD. Therefore, in the embodiment of the present invention, the non-volatile memory is still used as a buffer, and the persistence of data will be stored on the SSD. This is the most effective way to give full play to the characteristics of persistent memory.

[0091] In the embodiment of the present invention, in order to give full play to the performance of the central processing unit, all writing threads may be divided into a plurality of thread groups having the same number of cores as the central processing unit.

[0092] S202: Write the data corresponding to each target writing thread in the target thread group into the target disk flushing buffer, and suspend the target thread group.

[0093] The processing type of each thread group is described by taking any one thread group among all thread groups, namely, the target thread group, as an example in the embodiment of the present invention.

[0094] Each thread group corresponds to a disk flush buffer, and each thread group can contain multiple write threads. For easy distinction, the write thread contained in the target thread group can be called the target write thread. The disk flush buffer corresponding to the target thread group is called the target disk flush buffer.

[0095] In the disk flushing solution, the Thread Group Manager is an important component, which provides operations for flushing data to disk and writing to the flush buffer. The Thread Manager can divide the writing threads into N groups according to the system configuration. When the threads in each group execute the operation of writing data to the flush buffer by the write middleware, the writing threads will be blocked. The Thread Manager is responsible for gathering the threads in the same group for writing.

[0096] Assume that a thread group contains M writing threads. The data obtained by the M writing threads are first written to the corresponding flush buffer and then the thread group is suspended. The thread group is resumed after the data in the flush buffer is written to the disk. This storage method can ensure sequential writing and random reading. And the data written each time is sufficient, which can reduce the number of switches between kernel mode and user mode.

[0097] S203: When the target flush buffer receives data sent by all corresponding target write threads, the data in the target flush buffer is synchronously written to the solid state drive, and the target thread group is released.

[0098] In the embodiment of the present invention, in order to ensure that all data corresponding to a thread group can perform a disk flushing operation at the same time, a barrier (CyclicBarrier) can be set inside the thread manager.

[0099] When all target write threads in the target thread group reach the barrier, all data in the target flush buffer can be written to the SSD based on the asynchronous thread, and the suspended target thread group can be controlled to resume operation. This synchronous writing process also ensures data power-off security.

[0100] In the specific implementation, the thread manager can obtain the number of threads of the target thread group that have reached the barrier, and determine whether the number of threads has reached the barrier thread number. If the number of threads has reached the barrier thread number, it means that all target write threads contained in the target thread group have reached the barrier. At this time, all data in the target flush buffer can be uniformly forced to be written to the SSD based on the asynchronous thread.

[0101] Figure 3 A schematic diagram of a gathering brush operation provided by an embodiment of the present invention, Figure 3 In this example, a thread group contains n threads. After writing data to the flush buffer, the n threads will be blocked. When all threads in the same group reach the barrier, another asynchronous thread will flush the data written to the flush buffer. After the flush operation is completed, the blocked thread can be resumed, thereby waking up the thread to perform other operations.

[0102] It should be noted that there are two ways for n threads to write data into the flush buffer. One way is to write the data in a cluster after all n threads have obtained the corresponding data. Another way is to write the data directly into the flush buffer after each thread has obtained the data without waiting for other threads.

[0103] To sum up, the definition of the disk brushing module is as follows:

[0104] Input: int topic, int queueId, ByteBuffer buffer

[0105] Output: Disk file#Disk means writing data to the SSD

[0106] 1. write(topic,queueId,buffer)

[0107] 2. Get the thread group manager;

[0108] 3. buffer writes to the disk buffer

[0109] 4. barrier.await thread suspends;

[0110] 5. Get the number of threads that have reached the barrier in the current thread group;

[0111] 6. If the number of threads reaches the requirement then

[0112] 7. Forced disk flushing

[0113] 8. else

[0114] 9. Resume the current thread

[0115] 10. Messages are uniformly persisted in SSD

[0116] 11. End

[0117] The disk flushing module defines the (int topic, int queueId, ByteBuffer buffer) interface, specifies the topic and queueId of the message, and finds the corresponding Queue based on the topic and queueId. The disk flushing module obtains the Queue and the thread group manager. The data to be written will be written to the disk flushing buffer through this disk flushing module, and then persisted to the SSD through the disk flushing.

[0118] It can be seen from the above technical solution that the write threads are grouped according to the number of cores of the central processing unit to obtain multiple thread groups; wherein each thread group corresponds to a disk flushing buffer; the disk flushing buffer is a non-volatile memory, and the non-volatile memory has the characteristics of persistence, which can ensure the security of data when power is off or the memory bar is damaged. The processing method of each thread group is similar. Taking any thread group among all thread groups, that is, the target thread group, as an example, the data corresponding to each target write thread in the target thread group can be written into the target disk flushing buffer. In order to avoid the target write thread being called by other applications during data caching, causing confusion in the data cache, the target thread group can be suspended at this time. When the target disk flushing buffer receives the data sent by all the corresponding target write threads, the data in the target disk flushing buffer is synchronously written to the solid-state hard disk, and the target thread group is released. The beneficial effect of the present invention is that by dividing the thread groups and allocating a disk flushing buffer to each thread group, a message storage method of sequential writing and random reading can be realized, which can effectively give play to the characteristics of fast random reading of non-volatile memory, and solve the capacity limitation of the current file storage solution. When performing disk flushing, the data corresponding to multiple writing threads in a thread group are synchronously aggregated for disk flushing, which reduces the frequent switching between user state and kernel state and improves the writing speed of messages. The present invention can ensure the validity of data power failure while the writing speed reaches the upper limit of the bandwidth of the solid-state hard disk, so that the data will not be lost during power failure, and solves the problem that the message middleware is read and written sequentially under the traditional storage structure and will only be read once, and is easily lost during power failure.

[0119] Considering that in actual applications, multiple write threads contained in each thread group may not all be active, the number of write threads contained in each thread group when performing data flushing operations is a dynamically changing quantity and is not a fixed value.

[0120] The disk flush buffer is designed to facilitate the writing of multiple threads to the disk flush buffer before writing to disk. Its capacity is related to the number of writing threads. Taking a thread group as an example, the larger the number of currently active writing threads in the thread group, the larger the capacity of the disk flush buffer is required; the smaller the number of currently active writing threads in the thread group, the smaller the capacity of the disk flush buffer is required.

[0121] Therefore, in the embodiment of the present invention, the number of active threads of the target thread group can be obtained during program execution; based on the set single message data volume, sector byte number, original offset and active thread number, the capacity of the target flush buffer corresponding to the target thread group is determined. After the capacity of the target flush buffer is determined, the capacity of the target flush buffer can be adjusted to the currently determined capacity.

[0122] The target disk flushing buffer is the disk flushing buffer corresponding to the target thread group.

[0123] In the specific implementation, the disk flush buffer calculation formula can be called to process the set single message data volume, sector byte number, original offset and number of active threads to determine the capacity of the target disk flush buffer corresponding to the target thread group; the disk flush buffer calculation formula is:

[0124] S=(20*number of active threads / 4+x);

[0125] Among them, S represents the capacity of the target disk flushing buffer, 20 is the amount of single message data, 4 is the number of sector bytes, and x represents the original offset.

[0126] Since the size of messages in the test data of the embodiment of the present invention is less than 20 KB, the data size of a single message can be set to 20 KB.

[0127] In the embodiment of the present invention, a 4K alignment solution based on withheld write is adopted, so the number of sector bytes is set to 4KB.

[0128] The value of x is a dynamically adjustable value. In practical applications, its value can be determined according to actual needs, for example, it can be set to 40 KB by default.

[0129] When flushing the disk, the thread may be detained until the next batch to flush the disk, so the aggregation number of the thread group cannot be a constant. Therefore, a thread controller can be introduced in the embodiment of the present invention, and a polling thread inside it will obtain the number of active threads during the program running and dynamically determine the aggregation number. The aggregation number refers to the number of thread groups currently performing the disk flushing operation. After completing the adjustment of the target disk flushing buffer capacity, the data corresponding to all target write threads in the target thread group can be aggregated and written to the target disk flushing buffer.

[0130] In the embodiment of the present invention, in order to implement data flushing, a persistent file may be allocated to each thread group. The persistent file is a storage area divided from the solid state drive.

[0131] Figure 4 A schematic diagram of a file storage provided by an embodiment of the present invention, if all writing threads are divided into N groups, each group corresponds to a persistent file, there are a total of N persistent files. Each group has a thread group manager, each thread group manager has a disk flush buffer, and the disk flush buffer is a continuous area in the persistent memory. The thread group manager will force the message to be written to the persistent file after all threads in the same group arrive, such as Figure 4 This ensures data security after power failure or memory damage, because the write program will not return until the disk is flushed, which is a synchronous process.

[0132] For the sake of distinction, the persistent file corresponding to the target thread group may be referred to as the target persistent file. After the data buffering is completed, it may be determined whether the target thread group has a corresponding target persistent file.

[0133] In the case where the target thread group has a corresponding target persistent file, the file capacity of the target persistent file may be adjusted according to the total amount of all data in the target flush buffer.

[0134] When the target thread group does not have a corresponding target persistent file, the file capacity for performing the disk flushing operation can be determined based on the total amount of all data in the target disk flushing buffer; the target persistent file is divided from the solid-state drive according to the file capacity.

[0135] After the target persistent file capacity is adjusted, the data in the target flush buffer can be written to the target persistent file.

[0136] In the embodiment of the present invention, the capacity of the disk flushing buffer corresponding to the thread group is dynamically adjusted according to the number of active threads in the thread group, so that the setting of the disk flushing buffer capacity is more reasonable. The file capacity of the persistent file is adjusted according to the total amount of data contained in the disk flushing buffer, so that the persistent files divided by the solid-state drive are more in line with the actual storage needs.

[0137] The program used in the embodiment of the present invention is Java. The Application Programming Interface (API) in Java is implemented through FileInputStream / FileOutputStream and FileReader / FileWriter. The former reads and writes files through byte streams, and the latter reads and writes files through character streams.

[0138] However, Java introduced the FileChannel class in the Non-blocking Input / Output (NIO) package, which is used together with the ByteBuffer to read and write files. Compared with the FileInputStream and FileOutputStream classes, FileChannel provides bidirectional reading and writing capabilities and is easier to operate the reading and writing accuracy of data.

[0139] In the embodiment of the present invention, the mapping method of FileChannel can be used to map each persistent file to the address space of the process to realize the mapping relationship between the disk address of the persistent file and the virtual address of the process. The asynchronous thread can read the data in the target disk buffer through the pointer; according to the mapping relationship, the data read from the target disk buffer is written to the target persistent file.

[0140] Figure 5 A schematic diagram of a file channel mapping provided in an embodiment of the present invention maps a file to the address space of a process, and realizes the mapping between the disk address of the file and a section of the virtual address of the process. The process can read and write this section of memory through a pointer, and the operating system will automatically write back the dirty page to the corresponding file disk, so there is no need to call system call functions such as read (read) and write (write) to read and write files, reducing the number of switching between kernel mode and user mode. Therefore, in an embodiment of the present invention, FileChannel is selected as the API of the file.

[0141] After all writing threads are divided into a number of thread groups equal to the number of cores of the central processing unit, the number of thread groups can be dynamically adjusted according to the blocking state of the writing threads.

[0142] When a task performs an IO operation, its thread will be blocked, and the CPU can immediately perform a context switch to process other ready threads. If the thread group is blocked 50% of the time, the number of thread groups required can be adjusted to twice the number of processor cores.

[0143] In the embodiment of the present invention, the number of thread group groups is dynamically adjusted in combination with the blocking state of the write thread, so that the processing performance of the central processing unit can be fully mobilized.

[0144] Figure 6 A schematic diagram of the structure of a disk-swiping device based on a non-volatile memory provided by an embodiment of the present invention includes a grouping unit 61, a cache unit 62, a suspension unit 63, a writing unit 64 and a releasing unit 65;

[0145] The grouping unit 61 is used to group the writing threads according to the number of cores of the central processing unit to obtain a plurality of thread groups; wherein each thread group corresponds to a disk flushing buffer; the disk flushing buffer is a non-volatile memory;

[0146] The cache unit 62 is used to write the data corresponding to each target writing thread in the target thread group into the target flushing buffer;

[0147] A suspending unit 63, used for suspending a target thread group; wherein the target thread group is any thread group among all thread groups;

[0148] The writing unit 64 is used to synchronously write the data in the target flushing buffer into the solid state drive when the target flushing buffer receives the data sent by all the corresponding target writing threads;

[0149] The release unit 65 is used to release the target thread group.

[0150] In some embodiments, the write unit is used to force all data in the target flush buffer to be written to the solid-state drive uniformly based on the asynchronous thread after all target write threads included in the target thread group reach the barrier;

[0151] The release unit is used to control the suspended target thread group to resume running.

[0152] In some embodiments, the writing unit includes an acquisition subunit, a determination subunit, and a forced writing subunit;

[0153] Get the subunit to get the number of threads in the target thread group that have reached the barrier;

[0154] A judging subunit, used to judge whether the number of threads reaches the number of barrier threads;

[0155] The forced write subunit is used to force all data in the target disk flush buffer to be written to the solid-state drive based on asynchronous threads when the number of threads reaches the barrier thread number.

[0156] In some embodiments, for the capacity setting of the disk flushing buffer, the apparatus further includes an acquisition unit, a buffer determination unit, a file determination unit, an adjustment unit, a file capacity determination unit, and a division unit;

[0157] An acquisition unit is used to obtain the number of active threads of the target thread group during program execution;

[0158] A buffer determination unit, configured to determine the capacity of a target flush buffer corresponding to a target thread group based on a set single message data volume, a sector byte number, an original offset, and a number of active threads; wherein the target flush buffer is a flush buffer corresponding to the target thread group;

[0159] A cache unit, used to aggregate and write data corresponding to all target writing threads in the target thread group into a target flushing buffer;

[0160] A file determination unit, used to determine whether a target thread group has a corresponding target persistent file; wherein the target persistent file is a storage area divided from a solid state drive;

[0161] The adjustment unit is used to adjust the file capacity of the target persistent file according to the total amount of all data in the target flush buffer when the target thread group has a corresponding target persistent file;

[0162] A file capacity determination unit, configured to determine the file capacity for performing a disk flushing operation according to the total amount of all data in the target disk flushing buffer when the target thread group does not have a corresponding target persistent file;

[0163] A partitioning unit, used to partition a target persistent file from the solid state drive according to the file capacity;

[0164] The writing unit is used to write the data in the target flush buffer into the target persistent file.

[0165] In some embodiments, the buffer determination unit is used to call the disk flush buffer calculation formula to process the set single message data volume, sector byte number, original offset and active thread number to determine the capacity of the target disk flush buffer corresponding to the target thread group; the disk flush buffer calculation formula is:

[0166] S=(20*number of active threads / 4+x);

[0167] Among them, S represents the capacity of the target disk flushing buffer, 20 is the amount of single message data, 4 is the number of sector bytes, and x represents the original offset.

[0168] In some embodiments, a mapping unit is also included;

[0169] A mapping unit, used to map each persistent file to the address space of the process, so as to realize the mapping relationship between the disk address of the persistent file and the virtual address of the process;

[0170] The writing unit is used to read the data in the target disk flushing buffer through the pointer; and write the data read from the target disk flushing buffer into the target persistent file according to the mapping relationship.

[0171] In some embodiments, the grouping unit is used to divide all writing threads into a plurality of thread groups which is equal to the number of cores of the central processing unit.

[0172] Figure 6 The description of the features in the corresponding embodiments can be found in Figure 1 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0173] It can be seen from the above technical solution that the write threads are grouped according to the number of cores of the central processing unit to obtain multiple thread groups; wherein each thread group corresponds to a disk flushing buffer; the disk flushing buffer is a non-volatile memory, and the non-volatile memory has the characteristics of persistence, which can ensure the security of data when power is off or the memory bar is damaged. The processing method of each thread group is similar. Taking any thread group among all thread groups, that is, the target thread group, as an example, the data corresponding to each target write thread in the target thread group can be written into the target disk flushing buffer. In order to avoid the target write thread being called by other applications during data caching, causing confusion in the data cache, the target thread group can be suspended at this time. When the target disk flushing buffer receives the data sent by all the corresponding target write threads, the data in the target disk flushing buffer is synchronously written to the solid-state hard disk, and the target thread group is released. The beneficial effect of the present invention is that by dividing the thread groups and allocating a disk flushing buffer to each thread group, a message storage method of sequential writing and random reading can be realized, which can effectively give play to the characteristics of fast random reading of non-volatile memory, and solve the capacity limitation of the current file storage solution. When performing disk flushing, the data corresponding to multiple writing threads in a thread group are synchronously aggregated for disk flushing, which reduces the frequent switching between user state and kernel state and improves the writing speed of messages. The present invention can ensure the validity of data power failure while the writing speed reaches the upper limit of the bandwidth of the solid-state hard disk, so that the data will not be lost during power failure, and solves the problem that the message middleware is read and written sequentially under the traditional storage structure and will only be read once, and is easily lost during power failure.

[0174] Figure 7A structural diagram of a disk-swiping device based on non-volatile memory provided by an embodiment of the present invention, such as Figure 7 As shown, the non-volatile memory-based disk flashing device includes: a memory 70 for storing computer programs;

[0175] The processor 71 is used to implement the steps of the disk flushing method based on non-volatile memory in the above embodiment when executing the computer program.

[0176] The non-volatile memory-based flashing device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0177] Among them, the processor 71 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 71 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 71 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 71 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 71 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0178] The memory 70 may include one or more computer-readable storage media, which may be non-transitory. The memory 70 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 70 is at least used to store the following computer program 701, wherein, after the computer program is loaded and executed by the processor 71, it can implement the relevant steps of the disk-swiping method based on non-volatile memory disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 70 may also include an operating system 702 and data 703, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 702 may include Windows, Unix, Linux, etc. Data 703 may include, but is not limited to, data corresponding to the write thread, etc.

[0179] In some embodiments, the non-volatile memory-based disk flashing device may also include a display screen 72 , an input / output interface 73 , a communication interface 74 , a power supply 75 , and a communication bus 76 .

[0180] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the non-volatile memory-based disk-wiping device, and may include more or fewer components than those shown in the figure.

[0181] It is understandable that if the non-volatile memory-based disk flashing method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the contribution to the current technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk and other media that can store program codes.

[0182] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the disk flashing method based on non-volatile memory are implemented as described above.

[0183] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned disk flashing method based on non-volatile memory when executed by a processor.

[0184] The above is a detailed introduction to a disk flashing method, device, equipment, computer-readable storage medium and computer program product based on non-volatile memory provided by an embodiment of the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0185] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0186] The above is a detailed introduction to a disk flashing method, device, equipment, computer-readable storage medium and computer program product based on non-volatile memory provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A disk flashing method based on non-volatile memory, characterized in that: include: The writing threads are grouped according to the number of cores of the central processing unit to obtain a plurality of thread groups; each thread group corresponds to a disk flushing buffer; the disk flushing buffer is a non-volatile memory; Writing the data corresponding to each target writing thread in the target thread group into the target flush buffer, and suspending the target thread group; wherein the target thread group is any thread group among all thread groups; When the target disk flushing buffer receives data sent by all corresponding target writing threads, the data in the target disk flushing buffer is synchronously written to the solid state drive, and the target thread group is released.

2. The disk flashing method based on non-volatile memory according to claim 1, characterized in that: When the target flush buffer receives data sent by all corresponding target write threads, synchronously writing the data in the target flush buffer to the solid state drive and releasing the target thread group includes: When all target write threads included in the target thread group reach the barrier, all data in the target disk flush buffer are uniformly and forcibly written to the solid-state drive based on the asynchronous thread, and the suspended target thread group is controlled to resume operation.

3. The disk flashing method based on non-volatile memory according to claim 2, characterized in that: When all target write threads included in the target thread group reach the barrier, all data in the target flush buffer are forcibly written to the solid-state drive based on the asynchronous thread, including: Get the number of threads in the target thread group that have reached the barrier; Determine whether the number of threads reaches the number of barrier threads; When the number of threads reaches the number of barrier threads, all data in the target disk flush buffer are forcibly written to the solid state drive based on the asynchronous thread.

4. The disk flashing method based on non-volatile memory according to claim 1, characterized in that: Regarding the capacity setting of the disk flushing buffer, the method further includes: Get the number of active threads in the target thread group during program execution; Based on the set single message data volume, sector byte number, original offset and the number of active threads, the capacity of the target flush buffer corresponding to the target thread group is determined; wherein the target flush buffer is the flush buffer corresponding to the target thread group; Writing the data corresponding to each target writing thread in the target thread group into the target flush buffer includes: Aggregate and write the data corresponding to all target writing threads in the target thread group to the target flush buffer; After the data corresponding to all target writing threads in the target thread group are aggregated and written to the target flush buffer, the method further includes: Determine whether the target thread group has a corresponding target persistent file; wherein the target persistent file is a storage area divided from the solid state drive; In the case where the target thread group has a corresponding target persistent file, adjusting the file capacity of the target persistent file according to the total amount of all data in the target flush buffer; In the case that the target thread group does not have a corresponding target persistent file, determining the file capacity for performing the disk flushing operation according to the total amount of all data in the target disk flushing buffer; and dividing the target persistent file from the solid state drive according to the file capacity; Synchronously writing the data in the target flush buffer into the solid state drive comprises: The data in the target flush buffer is written into the target persistent file.

5. The disk flashing method based on non-volatile memory according to claim 4, characterized in that: Based on the set single message data volume, sector byte number, original offset, and the number of active threads, determining the capacity of the target flush buffer corresponding to the target thread group includes: The disk flushing buffer calculation formula is called to process the set single message data volume, sector byte number, original offset and the number of active threads to determine the capacity of the target disk flushing buffer corresponding to the target thread group; the disk flushing buffer calculation formula is: S=(20*number of active threads / 4+x); Among them, S represents the capacity of the target disk flushing buffer, 20 is the amount of single message data, 4 is the number of sector bytes, and x represents the original offset.

6. The disk flashing method based on non-volatile memory according to claim 4, characterized in that: Also includes: Map each persistent file to the address space of the process to achieve the mapping relationship between the disk address of the persistent file and the virtual address of the process; Writing the data in the target flush buffer into the target persistent file includes: Read the data in the target disk flushing buffer through a pointer; According to the mapping relationship, the data read from the target disk flushing buffer is written into the target persistent file.

7. The disk flashing method based on non-volatile memory according to claim 1, characterized in that: The write threads are grouped according to the number of cores of the central processing unit to obtain multiple thread groups including: Divide all writing threads into multiple thread groups equal to the number of CPU cores.

8. A disk-swiping device based on non-volatile memory, characterized in that: It includes a grouping unit, a cache unit, a suspension unit, a writing unit and a releasing unit; The grouping unit is used to group the writing threads according to the number of cores of the central processing unit to obtain a plurality of thread groups; wherein each thread group corresponds to a disk flushing buffer; the disk flushing buffer is a non-volatile memory; The cache unit is used to write the data corresponding to each target writing thread in the target thread group into the target flushing buffer; The suspending unit is used to suspend the target thread group; wherein the target thread group is any one of all thread groups; The writing unit is used to synchronously write the data in the target flushing buffer into the solid state drive when the target flushing buffer receives the data sent by all the corresponding target writing threads; The releasing unit is used to release the target thread group.

9. A disk-swiping device based on non-volatile memory, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the non-volatile memory-based disk flushing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the disk flashing method based on non-volatile memory as described in any one of claims 1 to 7 are implemented.

Images