A disk flashing method, device, equipment and medium based on non-volatile memory

Abstract

The present invention relates to the field of data storage technology, and discloses a disk flushing method, device, equipment and medium based on non-volatile memory, which groups write threads according to the number of cores of a central processing unit to obtain multiple thread groups; writes the data corresponding to each target write thread in the target thread group into a target disk flushing buffer, and suspends the target thread group. When the target disk flushing buffer receives the data sent by all the corresponding target write threads, the data in the target disk flushing buffer is synchronously written to the solid-state hard disk, and the target thread group is released. By dividing the thread groups and allocating a disk flushing buffer to each thread group, a message storage method of sequential writing and random reading can be implemented, and the capacity limitation of the current file storage solution is solved. When executing disk flushing, the data corresponding to multiple write threads in a thread group are synchronously aggregated for disk flushing, which reduces the frequent switching between user mode and kernel mode and improves the message writing speed.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a disk flashing method, device, equipment and medium based on non-volatile memory. Background Art

[0002] With the rapid development of computer and network technologies, in extreme scenarios involving continuous mixed reading and writing of large amounts of data, cache contamination caused by insufficient cache can seriously affect the efficiency of disk flushing. Disk flushing involves writing messages from multiple write threads to a solid-state disk (SSD) and saving them. The program dynamically groups the write threads, aggregates them into a disk flush buffer, and then writes them to the SSD. This synchronous flushing process ensures data security during power outages.

[0003] Messages are designed with attributes such as topic, queue ID, and offset. A queue is a two-dimensional array. Determining the topic and queue ID uniquely identifies a queue. Queues don't store individual messages, but rather their indexes. The offset uniquely identifies the message index and thus the message. Since a pair of topic and queue ID uniquely identifies a queue, if there are N topics and each topic contains M queue IDs, there will be N*M queues. Current file storage solutions assign one file to each queue. This storage solution uses random writes and sequential reads. However, random write performance on both SSDs and hard disks is poor, which affects message write speeds. Furthermore, this solution creates N*M files. The Linux operating system sets a default maximum file handle size of 1024 for a single process. When N*M is large, this limit may be exceeded.

[0004] It can be seen that how to improve the message writing speed is a problem that those skilled in the art need to solve. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide a disk flushing method, apparatus, device and medium based on non-volatile memory, which can solve the problem of slow message writing speed.

[0006] To solve the above technical problems, an embodiment of the present invention provides a disk flashing method based on non-volatile memory, comprising:

[0007] The write threads are grouped according to the number of cores of the central processing unit to obtain multiple thread groups; each thread group corresponds to a disk flush buffer; the disk flush buffer is a non-volatile memory;

[0008] Writing the data corresponding to each target writing thread in the target thread group into the target flush buffer, and suspending the target thread group; wherein the target thread group is any thread group among all thread groups;

[0009] When the target disk flushing buffer receives data sent by all corresponding target writing threads, the data in the target disk flushing buffer is synchronously written to the solid-state drive, and the target thread group is released.

[0010] On the one hand, when the target flush buffer receives data sent by all corresponding target write threads, synchronously writing the data in the target flush buffer to the solid-state drive and releasing the target thread group includes:

[0011] When all target write threads contained in the target thread group reach the barrier, all data in the target flush buffer are forcibly written to the solid-state drive based on the asynchronous thread, and the suspended target thread group is controlled to resume operation.

[0012] On the one hand, when all target write threads included in the target thread group reach the barrier, all data in the target flush buffer are forcibly written to the solid-state drive based on the asynchronous thread, including:

[0013] Get the number of threads in the target thread group that have reached the barrier;

[0014] Determine whether the number of threads reaches the barrier thread number;

[0015] When the number of threads reaches the barrier thread number, all data in the target flush buffer are forcibly written to the solid-state drive based on the asynchronous thread.

[0016] On the one hand, with respect to the capacity setting of the disk flush buffer, the method further includes:

[0017] Get the number of active threads in the target thread group during program execution;

[0018] Determine the capacity of the target flush buffer corresponding to the target thread group based on the set single message data volume, sector byte number, original offset, and the number of active threads; wherein the target flush buffer is the flush buffer corresponding to the target thread group;

[0019] Writing the data corresponding to each target writing thread in the target thread group into the target flush buffer includes:

[0020] Aggregate and write the data corresponding to all target writing threads in the target thread group to the target flush buffer;

[0021] After the data corresponding to all target writing threads in the target thread group are aggregated and written to the target flush buffer, the following steps are also included:

[0022] Determine whether the target thread group has a corresponding target persistent file; wherein the target persistent file is a storage area divided from the solid state drive;

[0023] In the case where the target thread group has a corresponding target persistent file, adjusting the file capacity of the target persistent file according to the total amount of all data in the target flush buffer;

[0024] If the target thread group does not have a corresponding target persistent file, determining a file capacity for performing a disk flushing operation based on the total amount of all data in the target disk flushing buffer; and dividing the target persistent file from the solid-state drive according to the file capacity;

[0025] Synchronously writing the data in the target flush buffer into the solid-state hard disk includes:

[0026] The data in the target flush buffer is written into the target persistent file.

[0027] On the one hand, based on the set single message data volume, sector byte number, original offset, and the number of active threads, determining the capacity of the target flush buffer corresponding to the target thread group includes:

[0028] The disk flush buffer calculation formula is called to process the set single message data volume, sector byte number, original offset, and the number of active threads to determine the target disk flush buffer capacity corresponding to the target thread group; the disk flush buffer calculation formula is:

[0029] S=(20*number of active threads / 4+x);

[0030] Where S represents the capacity of the target flush buffer, 20 is the amount of data in a single message, 4 is the number of sector bytes, and x represents the original offset.

[0031] On the one hand, it also includes:

[0032] Map each persistent file to the process's address space to achieve the mapping relationship between the persistent file's disk address and the process's virtual address;

[0033] Writing the data in the target flush buffer into the target persistent file includes:

[0034] Read the data in the target disk buffer through the pointer;

[0035] According to the mapping relationship, the data read from the target flush buffer is written into the target persistent file.

[0036] On the one hand, the write threads are grouped according to the number of cores of the central processing unit to obtain multiple thread groups including:

[0037] Divide all writing threads into multiple thread groups equal to the number of CPU cores.

[0038] The embodiment of the present invention further provides a disk flushing device based on non-volatile memory, comprising a grouping unit, a cache unit, a suspension unit, a writing unit, and a releasing unit;

[0039] The grouping unit is used to group the writing threads according to the number of cores of the central processing unit to obtain multiple thread groups; wherein each thread group corresponds to a disk flush buffer; the disk flush buffer is a non-volatile memory;

[0040] The cache unit is used to write data corresponding to each target writing thread in the target thread group into the target flush buffer;

[0041] The suspending unit is configured to suspend the target thread group; wherein the target thread group is any one of all thread groups;

[0042] The writing unit is configured to synchronously write the data in the target flush buffer to the solid-state hard disk when the target flush buffer receives data sent by all corresponding target write threads;

[0043] The releasing unit is configured to release the target thread group.

[0044] On the one hand, the writing unit is used to force all data in the target flush buffer to be written to the solid-state drive uniformly based on the asynchronous thread after all target writing threads included in the target thread group reach the barrier;

[0045] The release unit is used to control the suspended target thread group to resume running.

[0046] On the one hand, the writing unit includes an acquisition subunit, a judgment subunit and a forced writing subunit;

[0047] The acquisition subunit is used to obtain the number of threads of the target thread group that have reached the barrier;

[0048] The judging subunit is configured to judge whether the number of threads reaches the number of barrier threads;

[0049] The forced write subunit is used to force all data in the target disk flush buffer to be written to the solid state drive based on asynchronous threads when the number of threads reaches the barrier thread number.

[0050] On the one hand, for setting the capacity of the disk flushing buffer, the device further includes an acquisition unit, a buffer determination unit, a file determination unit, an adjustment unit, a file capacity determination unit, and a division unit;

[0051] The acquisition unit is used to obtain the number of active threads in the target thread group during program execution;

[0052] The buffer determination unit is configured to determine the capacity of a target flush buffer corresponding to the target thread group based on a set single message data volume, a sector byte number, an original offset, and the number of active threads; wherein the target flush buffer is the flush buffer corresponding to the target thread group;

[0053] The cache unit is used to aggregate and write data corresponding to all target writing threads in the target thread group into the target flush buffer;

[0054] The file determination unit is configured to determine whether the target thread group has a corresponding target persistent file; wherein the target persistent file is a storage area divided from the solid state drive;

[0055] The adjusting unit is configured to adjust the file capacity of the target persistent file according to the total amount of all data in the target flush buffer when the target thread group has a corresponding target persistent file;

[0056] The file capacity determination unit is configured to determine the file capacity for performing a disk flushing operation according to the total amount of all data in the target disk flushing buffer when the target thread group does not have a corresponding target persistent file;

[0057] The partitioning unit is configured to partition the target persistent file from the solid state drive according to the file capacity;

[0058] The writing unit is used to write the data in the target disk flushing buffer into the target persistent file.

[0059] On the one hand, the buffer determination unit is used to call the disk flush buffer calculation formula to process the set single message data volume, sector byte number, original offset and the number of active threads to determine the capacity of the target disk flush buffer corresponding to the target thread group; the disk flush buffer calculation formula is:

[0060] S=(20*number of active threads / 4+x);

[0061] Where S represents the capacity of the target flush buffer, 20 is the amount of data in a single message, 4 is the number of sector bytes, and x represents the original offset.

[0062] On the one hand, it also includes a mapping unit;

[0063] The mapping unit is used to map each persistent file to the address space of the process to achieve a mapping relationship between the disk address of the persistent file and the virtual address of the process;

[0064] The writing unit is configured to read the data in the target disk flushing buffer through a pointer; and write the data read from the target disk flushing buffer into the target persistent file according to the mapping relationship.

[0065] On the one hand, the grouping unit is used to divide all writing threads into a plurality of thread groups having the same number of cores as the central processing unit.

[0066] An embodiment of the present invention further provides a disk-swiping device based on a non-volatile memory, comprising:

[0067] memory for storing computer programs;

[0068] A processor is used to execute the computer program to implement the steps of the above-mentioned disk flashing method based on non-volatile memory.

[0069] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned disk flashing method based on non-volatile memory are implemented.

[0070] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned disk flashing method based on non-volatile memory when executed by a processor.

[0071] It can be seen from the above technical solution that the write threads are grouped according to the number of cores of the central processing unit to obtain multiple thread groups; wherein, each thread group corresponds to a disk flush buffer; the disk flush buffer is a non-volatile memory, and the non-volatile memory has the characteristic of persistence, which can ensure the security of data when the power is off or the memory stick is damaged. The processing method of each thread group is similar. Taking any thread group among all thread groups, that is, the target thread group, as an example, the data corresponding to each target write thread in the target thread group can be written to the target disk flush buffer. In order to avoid the target write thread being called by other applications when the data is cached, causing confusion in the data cache, the target thread group can be suspended at this time. When the target disk flush buffer receives the data sent by all the corresponding target write threads, the data in the target disk flush buffer is synchronously written to the solid-state drive, and the target thread group is released. The beneficial effect of the present invention is that by dividing the thread groups and allocating a disk flush buffer to each thread group, a message storage method of sequential write and random read can be realized, which can effectively give play to the fast random read characteristics of non-volatile memory and solve the capacity limitation of the current file storage solution. When flushing, the data corresponding to multiple write threads in a thread group are synchronously aggregated and flushed, reducing the frequent switching between user mode and kernel mode and improving the message writing speed. While achieving write speeds that reach the upper limit of the solid-state drive's bandwidth, this invention also ensures data validity during power outages, ensuring no data loss during power outages. This solves the problem of message middleware in traditional storage structures, where data is read and written sequentially and only read once, and is easily lost during power outages. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0073] Figure 1 A schematic diagram of a logical address of a message provided by the prior art;

[0074] Figure 2 A flowchart of a disk flashing method based on non-volatile memory provided by an embodiment of the present invention;

[0075] Figure 3 A schematic diagram of a gathering brush operation provided by an embodiment of the present invention;

[0076] Figure 4 A schematic diagram of file storage provided by an embodiment of the present invention;

[0077] Figure 5 A schematic diagram of file channel mapping provided by an embodiment of the present invention;

[0078] Figure 6 A schematic structural diagram of a disk-swiping device based on non-volatile memory provided by an embodiment of the present invention;

[0079] Figure 7 A structural diagram of a disk-swiping device based on non-volatile memory provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0081] The terms "including" and "having," as used in the present description and accompanying drawings, and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements and may include steps or elements that are not listed.

[0082] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0083] The message itself has attributes such as topic, queueId, offset, etc. Topic refers to a collection of messages of the same type, and queue is the physical entity that stores messages. A topic can contain multiple queues, and each queue has its own ID number, namely queueId. The logical address of the message is as follows: Figure 1 As shown, there are multiple queues under each topic, and each queue has its own queue number. After the queue is uniquely identified by the topic and queue number, the specific message in the queue is found according to the offset. Figure 1 The example below uses two topics, Topic 0 and Topic 1. Each topic contains four queues, Queue 1 to Queue 4. Each queue can store four messages, and each message has its own offset.

[0084] One current storage solution is to assign a file to each queue. This solution writes randomly and reads continuously. However, the random write performance of both SSDs and disks is very poor, which will affect the message writing speed. Another storage solution is to assign a file to each topic. For example, if there are N topics, N files need to be created. This solution is random reading and random writing, which will also result in a low writing speed. When the number of files is large, these two storage solutions will exceed the file limit. In addition, the storage method of assigning a file to each topic will result in data from multiple queues being written to the same file. However, due to the need to ensure data security in the event of power outages or memory damage, the disk needs to be manually flushed after each message is written. This will cause frequent switching between user mode and kernel mode, resulting in low efficiency.

[0085] In this embodiment, the high cost-effectiveness, byte-addressability, and persistence advantages of non-volatile memory (Persistent Memory Module, PMEM) are considered to provide a new storage option for message-based middleware. Non-volatile memory can also be referred to as persistent memory.

[0086] Message middleware is a supporting software system that provides synchronous or asynchronous, reliable message transmission for application systems in a network environment based on queue and message passing technology. Unlike other systems, the messages of message middleware have unique data characteristics: messages are large amounts of small-capacity data; messages under the same queue are read one by one in sequence; all messages are read only once. These characteristics pose a great challenge to the introduction of non-volatile memory in the middleware storage system. Therefore, the embodiment of the present invention proposes a method for fast disk flushing based on non-volatile memory based on the data characteristics of the message middleware, and designs a message storage solution with sequential writing and random reading, which can effectively give play to the characteristics of fast random reading speed of non-volatile memory. The disk flushing strategy adopts a multi-threaded aggregation synchronous disk flushing method, which can ensure data power-off safety while reaching the upper limit of SSD write speed.

[0087] Next, a disk flashing method based on non-volatile memory provided by an embodiment of the present invention is described in detail. Figure 2 A flowchart of a disk flashing method based on non-volatile memory provided in an embodiment of the present invention, the method comprising:

[0088] S201: Grouping writing threads according to the number of cores of a central processing unit to obtain a plurality of thread groups.

[0089] Each thread group corresponds to a disk flush buffer; the disk flush buffer is a non-volatile memory.

[0090] Flashing is the process of writing pending messages to the SSD for storage. This ensures data security in the event of power outages or memory card failure while fully utilizing the write bandwidth. While non-volatile memory offers persistence, its capacity is significantly smaller than that of an SSD and its cost is many times higher. Therefore, in this embodiment of the present invention, non-volatile memory is used as a buffer, while data persistence is fully stored on the SSD. This is the most effective way to leverage the characteristics of persistent memory.

[0091] In the embodiment of the present invention, in order to fully utilize the performance of the central processing unit, all writing threads may be divided into a plurality of thread groups having the same number of cores as the central processing unit.

[0092] S202: Writing data corresponding to each target writing thread in the target thread group into the target flush buffer, and suspending the target thread group.

[0093] The processing type of each thread group is described by taking any one thread group among all thread groups, namely the target thread group, as an example in the embodiment of the present invention.

[0094] Each thread group corresponds to a disk flush buffer. Each thread group can contain multiple write threads. For easy distinction, the write thread contained in the target thread group can be called the target write thread. The disk flush buffer corresponding to the target thread group is called the target disk flush buffer.

[0095] In the disk flushing solution, the Thread Group Manager is a key component. It provides operations for flushing data to disk and writing to the flush buffer. Based on the system configuration, the Thread Group Manager can divide writing threads into N groups. When each group's threads execute the write middleware's operation to write data to the flush buffer, they are blocked. The Thread Group Manager is responsible for grouping threads in the same group for writing.

[0096] Suppose a thread group contains M write threads. The data obtained by these M write threads is first written to their corresponding flush buffers, and then the thread group is suspended. The thread group resumes after the data in the flush buffers is flushed to disk. This storage method ensures sequential writes and random reads. Furthermore, each write contains sufficient data, which reduces the number of kernel and user mode switches.

[0097] S203: When the target flush buffer receives data sent by all corresponding target write threads, the data in the target flush buffer is synchronously written to the solid-state drive, and the target thread group is released.

[0098] In an embodiment of the present invention, in order to ensure that all data corresponding to a thread group can be flushed to disk at the same time, a barrier (CyclicBarrier) can be set inside the thread manager.

[0099] When all target write threads in the target thread group reach the barrier, the asynchronous thread can force all data in the target flush buffer to be written to the SSD, and the suspended target thread group can be controlled to resume operation. This synchronous writing process also ensures data security in the event of power failure.

[0100] In a specific implementation, the thread manager can obtain the number of threads in the target thread group that have reached the barrier and determine whether the number of threads has reached the barrier thread count. If the number of threads has reached the barrier thread count, it means that all target write threads in the target thread group have reached the barrier. At this time, all data in the target flush buffer can be forcibly written to the SSD based on the asynchronous thread.

[0101] Figure 3 A schematic diagram of a gathering brush operation provided by an embodiment of the present invention, Figure 3 In this example, a thread group consisting of n threads is blocked after writing data to the flush buffer. When all threads in the group reach the barrier, another asynchronous thread flushes the data written to the flush buffer. This completes the flush operation, resuming the blocked thread and waking it up to perform other operations.

[0102] It should be noted that there are two ways for n threads to write data to the flush buffer. One way is to write the data in a cluster after all n threads have obtained the corresponding data. The other way is to write the data directly to the flush buffer after each thread obtains the data, without waiting for other threads.

[0103] To sum up, the definition of the disk brushing module is as follows:

[0104] Input: int topic, int queueId, ByteBuffer buffer

[0105] Output: Disk file # Disk means writing data to the solid state drive

[0106] 1. write(topic,queueId,buffer)

[0107] 2. Get the thread group manager;

[0108] 3. buffer writes to the disk buffer

[0109] 4. barrier.await thread suspends;

[0110] 5. Get the number of threads in the current thread group that have reached the barrier;

[0111] 6. If the number of threads reaches the requirement then

[0112] 7. Forced disk flushing

[0113] 8. else

[0114] 9. Resume the current thread

[0115] 10. Messages are persisted uniformly to SSD

[0116] 11. End

[0117] The disk flushing module defines the (int topic, int queueId, ByteBuffer buffer) interface, specifies the topic and queueId of the message, and finds the corresponding queue based on the topic and queueId. The disk flushing module obtains the queue and the thread group manager. The data to be written will be written to the disk flushing buffer through this disk flushing module and then persisted to the SSD through the disk flushing.

[0118] It can be seen from the above technical solution that the write threads are grouped according to the number of cores of the central processing unit to obtain multiple thread groups; wherein, each thread group corresponds to a disk flush buffer; the disk flush buffer is a non-volatile memory, and the non-volatile memory has the characteristic of persistence, which can ensure the security of data when the power is off or the memory stick is damaged. The processing method of each thread group is similar. Taking any thread group among all thread groups, that is, the target thread group, as an example, the data corresponding to each target write thread in the target thread group can be written to the target disk flush buffer. In order to avoid the target write thread being called by other applications when the data is cached, causing confusion in the data cache, the target thread group can be suspended at this time. When the target disk flush buffer receives the data sent by all the corresponding target write threads, the data in the target disk flush buffer is synchronously written to the solid-state drive, and the target thread group is released. The beneficial effect of the present invention is that by dividing the thread groups and allocating a disk flush buffer to each thread group, a message storage method of sequential write and random read can be realized, which can effectively give play to the fast random read characteristics of non-volatile memory and solve the capacity limitation of the current file storage solution. When flushing, the data corresponding to multiple write threads in a thread group are synchronously aggregated and flushed, reducing the frequent switching between user mode and kernel mode and improving the message writing speed. While achieving write speeds that reach the upper limit of the solid-state drive's bandwidth, this invention also ensures data validity during power outages, ensuring no data loss during power outages. This solves the problem of message middleware in traditional storage structures, where data is read and written sequentially and only read once, and is easily lost during power outages.

[0119] Considering that in actual applications, the multiple write threads contained in each thread group may not all be active, the number of write threads contained in each thread group when performing data flushing operations is a dynamically changing quantity and is not a fixed value.

[0120] The flush buffer is designed to facilitate clustered writes from multiple threads, prioritizing writes to the flush buffer before flushing to disk. Its capacity is related to the number of write threads. For example, within a thread group, the more active write threads in the group, the larger the flush buffer capacity required; the fewer active write threads in the group, the smaller the flush buffer capacity required.

[0121] Therefore, in an embodiment of the present invention, the number of active threads in a target thread group can be obtained during program execution; based on the set single message data volume, sector byte count, raw offset, and number of active threads, the target flush buffer capacity corresponding to the target thread group can be determined. After the target flush buffer capacity is determined, the target flush buffer capacity can be adjusted to the currently determined capacity.

[0122] The target disk flushing buffer is the disk flushing buffer corresponding to the target thread group.

[0123] In the specific implementation, the disk flush buffer calculation formula can be called to process the set single message data volume, sector byte number, original offset and number of active threads to determine the target disk flush buffer capacity corresponding to the target thread group; the disk flush buffer calculation formula is:

[0124] S=(20*number of active threads / 4+x);

[0125] Where S represents the capacity of the target flush buffer, 20 is the amount of data in a single message, 4 is the number of sector bytes, and x represents the original offset.

[0126] Since the size of messages in the test data of the embodiment of the present invention is less than 20 KB, the data size of a single message can be set to 20 KB.

[0127] In the embodiment of the present invention, a 4K alignment solution based on withheld write is adopted, so the number of sector bytes is set to 4KB.

[0128] The value of x is dynamically adjustable. In practical applications, its value can be determined based on actual needs. For example, it can be set to 40 KB by default.

[0129] When flushing the disk, the thread may be detained until the next batch for flushing, so the aggregation number of the thread group cannot be a constant. Therefore, in the embodiment of the present invention, a thread controller can be introduced, which has a polling thread inside that obtains the number of active threads during program execution and dynamically determines the aggregation number. The aggregation number refers to the number of thread groups currently performing the disk flushing operation. After completing the adjustment of the target disk flushing buffer capacity, the data corresponding to all target write threads in the target thread group can be aggregated and written to the target disk flushing buffer.

[0130] In an embodiment of the present invention, in order to implement data flushing, a persistent file may be allocated to each thread group. The persistent file is a storage area divided from the solid-state drive.

[0131] Figure 4 A schematic diagram of a file storage provided by an embodiment of the present invention. If all writing threads are divided into N groups, each group corresponds to a persistent file, and there are a total of N persistent files. Each group has a thread group manager, and each thread group manager has a flush buffer, which is a continuous area in the persistent memory. The thread group manager will force the message to be written to the persistent file after all threads in the same group arrive, such as Figure 4 This ensures data security after a power outage or memory corruption, because the write program will not return until the disk is flushed, which is a synchronous process.

[0132] For the sake of distinction, the persistence file corresponding to the target thread group can be called the target persistence file. After completing the buffering of the data, it can be determined whether the target thread group has a corresponding target persistence file.

[0133] In the case that the target thread group has a corresponding target persistent file, the file capacity of the target persistent file can be adjusted according to the total amount of all data in the target flush buffer.

[0134] When the target thread group does not have a corresponding target persistent file, the file capacity for performing the disk flushing operation can be determined based on the total amount of all data in the target disk flushing buffer; the target persistent file is divided from the solid-state drive according to the file capacity.

[0135] After the target persistent file capacity is adjusted, the data in the target flush buffer can be written to the target persistent file.

[0136] In this embodiment of the present invention, the capacity of the flush buffer corresponding to a thread group is dynamically adjusted based on the number of active threads in the thread group, making the setting of the flush buffer capacity more reasonable. The file capacity of the persistent file is adjusted based on the total amount of data contained in the flush buffer, making the persistent files allocated by the solid-state drive more in line with actual storage needs.

[0137] The program used in the embodiments of the present invention is Java. The Application Programming Interface (API) in Java is implemented through FileInputStream / FileOutputStream and FileReader / FileWriter. The former reads and writes files through byte streams, while the latter reads and writes files through character streams.

[0138] However, Java introduced the FileChannel class in the Non-blocking Input / Output (NIO) package, which is used together with the ByteBuffer to read and write files. Compared with the FileInputStream and FileOutputStream classes, FileChannel provides bidirectional reading and writing capabilities and is easier to operate data reading and writing accuracy.

[0139] In this embodiment of the present invention, the FileChannel mapping method can be used to map each persistent file to the process's address space, thereby achieving a mapping relationship between the persistent file's disk address and the process's virtual address. The asynchronous thread can read data from the target flush buffer through a pointer and, based on the mapping relationship, write the data read from the target flush buffer to the target persistent file.

[0140] Figure 5 This diagram illustrates a file channel mapping scheme provided in an embodiment of the present invention. This scheme maps a file into a process's address space, mapping the file's disk address to a segment of the process's virtual address. The process can read and write this memory segment using pointers, and the operating system automatically writes dirty pages back to the corresponding file on disk. This eliminates the need to invoke system calls like read and write to the file, reducing the number of kernel-mode and user-mode transitions. Therefore, in this embodiment of the present invention, FileChannel is selected as the file API.

[0141] After all writing threads are divided into a number of thread groups equal to the number of cores of the central processing unit, the number of thread groups can be dynamically adjusted according to the blocking status of the writing threads.

[0142] When a task performs an I / O operation, its thread is blocked, and the CPU can immediately context switch to process other ready threads. If the thread group is blocked 50% of the time, the number of thread groups required can be adjusted to twice the number of cores on the processor.

[0143] In the embodiment of the present invention, the number of thread group groups is dynamically adjusted in combination with the blocking state of the write thread, thereby fully mobilizing the processing performance of the central processing unit.

[0144] Figure 6 A schematic structural diagram of a disk flushing device based on non-volatile memory provided by an embodiment of the present invention, comprising a grouping unit 61, a cache unit 62, a suspension unit 63, a writing unit 64, and a releasing unit 65;

[0145] The grouping unit 61 is used to group the writing threads according to the number of cores of the central processing unit to obtain multiple thread groups; wherein each thread group corresponds to a disk flush buffer; the disk flush buffer is a non-volatile memory;

[0146] The cache unit 62 is used to write the data corresponding to each target writing thread in the target thread group into the target flush buffer;

[0147] The suspending unit 63 is configured to suspend a target thread group, wherein the target thread group is any one of all thread groups;

[0148] The writing unit 64 is configured to synchronously write the data in the target flush buffer to the solid state drive when the target flush buffer receives data sent by all corresponding target write threads;

[0149] The releasing unit 65 is used to release the target thread group.

[0150] In some embodiments, the write unit is configured to forcefully write all data in the target flush buffer to the solid-state drive based on the asynchronous thread after all target write threads in the target thread group reach the barrier.

[0151] The release unit is used to control the suspended target thread group to resume running.

[0152] In some embodiments, the writing unit includes an acquiring subunit, a judging subunit, and a forced writing subunit;

[0153] Get the subunit to get the number of threads in the target thread group that have reached the barrier;

[0154] A judgment subunit, used to judge whether the number of threads reaches the barrier thread number;

[0155] The forced write subunit is used to force all data in the target flush buffer to be written to the solid-state drive based on asynchronous threads when the number of threads reaches the barrier thread number.

[0156] In some embodiments, for setting the capacity of the flush buffer, the apparatus further includes an acquisition unit, a buffer determination unit, a file determination unit, an adjustment unit, a file capacity determination unit, and a division unit;

[0157] The acquisition unit is used to obtain the number of active threads in the target thread group during program execution;

[0158] a buffer determination unit, configured to determine the capacity of a target flush buffer corresponding to a target thread group based on a set single message data volume, sector byte count, original offset, and number of active threads; wherein the target flush buffer is the flush buffer corresponding to the target thread group;

[0159] A cache unit, configured to aggregate and write data corresponding to all target writing threads in a target thread group into a target flush buffer;

[0160] A file determination unit, configured to determine whether a target thread group has a corresponding target persistent file; wherein the target persistent file is a storage area divided from the solid-state hard disk;

[0161] The adjustment unit is used to adjust the file capacity of the target persistent file according to the total amount of all data in the target flush buffer when the target thread group has a corresponding target persistent file;

[0162] A file capacity determination unit is used to determine the file capacity for performing a disk flushing operation according to the total amount of all data in the target disk flushing buffer when the target thread group does not have a corresponding target persistent file;

[0163] A partitioning unit is used to partition target persistent files from the solid-state drive according to file capacity;

[0164] The writing unit is used to write the data in the target flush buffer to the target persistent file.

[0165] In some embodiments, the buffer determination unit is used to call a disk flush buffer calculation formula to process the set single message data volume, sector byte number, original offset, and number of active threads to determine the capacity of the target disk flush buffer corresponding to the target thread group; the disk flush buffer calculation formula is:

[0166] S=(20*number of active threads / 4+x);

[0167] Where S represents the capacity of the target flush buffer, 20 is the amount of data in a single message, 4 is the number of sector bytes, and x represents the original offset.

[0168] In some embodiments, a mapping unit is further included;

[0169] A mapping unit is used to map each persistent file to the address space of the process to achieve a mapping relationship between the disk address of the persistent file and the virtual address of the process;

[0170] The writing unit is used to read the data in the target disk flushing buffer through the pointer; and write the data read from the target disk flushing buffer into the target persistent file according to the mapping relationship.

[0171] In some embodiments, the grouping unit is configured to divide all writing threads into a plurality of thread groups equal to the number of cores of the central processing unit.

[0172] Figure 6 The description of the features in the corresponding embodiment can be found in Figure 1 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0173] It can be seen from the above technical solution that the write threads are grouped according to the number of cores of the central processing unit to obtain multiple thread groups; wherein, each thread group corresponds to a disk flush buffer; the disk flush buffer is a non-volatile memory, and the non-volatile memory has the characteristic of persistence, which can ensure the security of data when the power is off or the memory stick is damaged. The processing method of each thread group is similar. Taking any thread group among all thread groups, that is, the target thread group, as an example, the data corresponding to each target write thread in the target thread group can be written to the target disk flush buffer. In order to avoid the target write thread being called by other applications when the data is cached, causing confusion in the data cache, the target thread group can be suspended at this time. When the target disk flush buffer receives the data sent by all the corresponding target write threads, the data in the target disk flush buffer is synchronously written to the solid-state drive, and the target thread group is released. The beneficial effect of the present invention is that by dividing the thread groups and allocating a disk flush buffer to each thread group, a message storage method of sequential write and random read can be realized, which can effectively give play to the fast random read characteristics of non-volatile memory and solve the capacity limitation of the current file storage solution. When flushing, the data corresponding to multiple write threads in a thread group are synchronously aggregated and flushed, reducing the frequent switching between user mode and kernel mode and improving the message writing speed. While achieving write speeds that reach the upper limit of the solid-state drive's bandwidth, this invention also ensures data validity during power outages, ensuring no data loss during power outages. This solves the problem of message middleware in traditional storage structures, where data is read and written sequentially and only read once, and is easily lost during power outages.

[0174] Figure 7A structural diagram of a non-volatile memory-based disk-swiping device provided in an embodiment of the present invention, such as Figure 7 As shown, the non-volatile memory-based disk flashing device includes: a memory 70 for storing computer programs;

[0175] The processor 71 is configured to implement the steps of the disk flushing method based on non-volatile memory in the above embodiment when executing a computer program.

[0176] The non-volatile memory-based flashing device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0177] The processor 71 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 71 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 71 may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 71 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing content required to be displayed on the display screen. In some embodiments, the processor 71 may also include an artificial intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0178] The memory 70 may include one or more computer-readable storage media, which may be non-transitory. The memory 70 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 70 is at least used to store the following computer program 701, wherein, after the computer program is loaded and executed by the processor 71, it can implement the relevant steps of the disk-swiping method based on non-volatile memory disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 70 may also include an operating system 702 and data 703, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 702 may include Windows, Unix, Linux, etc. The data 703 may include but is not limited to data corresponding to the write thread, etc.

[0179] In some embodiments, the non-volatile memory-based disk flashing device may further include a display screen 72 , an input / output interface 73 , a communication interface 74 , a power supply 75 , and a communication bus 76 .

[0180] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the non-volatile memory-based disk flashing device, and may include more or fewer components than shown in the figure.

[0181] It is understandable that if the non-volatile memory-based flashing method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the current technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and executes all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk, and other media that can store program code.

[0182] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned disk flashing method based on non-volatile memory are implemented.

[0183] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned disk flashing method based on non-volatile memory when executed by a processor.

[0184] The above describes in detail a non-volatile memory-based disk flashing method, device, equipment, computer-readable storage medium, and computer program product provided by the embodiments of the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method section.

[0185] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0186] The above is a detailed introduction to a disk-swiping method, device, equipment, computer-readable storage medium, and computer program product based on non-volatile memory provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A disk flashing method based on non-volatile memory, characterized in that: include: The write threads are grouped according to the number of cores of the central processing unit to obtain multiple thread groups; each thread group corresponds to a disk flush buffer; the disk flush buffer is a non-volatile memory; Writing the data corresponding to each target writing thread in the target thread group into the target flush buffer, and suspending the target thread group; wherein the target thread group is any thread group among all thread groups; When the target flush buffer receives data sent by all corresponding target write threads, the data in the target flush buffer is synchronously written to the solid-state drive, and the target thread group is released; The capacity setting for the disk flush buffer also includes: Get the number of active threads in the target thread group during program execution; Determine the capacity of the target flush buffer corresponding to the target thread group based on the set single message data volume, sector byte number, original offset, and the number of active threads; wherein the target flush buffer is the flush buffer corresponding to the target thread group; Writing the data corresponding to each target writing thread in the target thread group into the target flush buffer includes: Aggregate and write the data corresponding to all target writing threads in the target thread group to the target flush buffer; After the data corresponding to all target writing threads in the target thread group are aggregated and written to the target flush buffer, the following steps are also included: Determine whether the target thread group has a corresponding target persistent file; wherein the target persistent file is a storage area divided from the solid state drive; In the case where the target thread group has a corresponding target persistent file, adjusting the file capacity of the target persistent file according to the total amount of all data in the target flush buffer; If the target thread group does not have a corresponding target persistent file, determining a file capacity for performing a disk flushing operation based on the total amount of all data in the target disk flushing buffer; and dividing the target persistent file from the solid-state drive according to the file capacity; Synchronously writing the data in the target flush buffer into the solid-state hard disk includes: Writing the data in the target flush buffer into the target persistent file; Determining the capacity of the target flush buffer corresponding to the target thread group based on the set single message data volume, sector byte number, original offset, and the number of active threads includes: The disk flush buffer calculation formula is called to process the set single message data volume, sector byte number, original offset, and the number of active threads to determine the target disk flush buffer capacity corresponding to the target thread group; the disk flush buffer calculation formula is: S=(20*number of active threads / 4+x); Where S represents the capacity of the target flush buffer, 20 is the amount of data in a single message, 4 is the number of sector bytes, and x represents the original offset.

2. The disk flashing method based on non-volatile memory according to claim 1, characterized in that: When the target flush buffer receives data sent by all corresponding target write threads, synchronously writing the data in the target flush buffer to the solid-state drive and releasing the target thread group includes: When all target write threads contained in the target thread group reach the barrier, all data in the target flush buffer are forcibly written to the solid-state drive based on the asynchronous thread, and the suspended target thread group is controlled to resume operation.

3. The disk flashing method based on non-volatile memory according to claim 2, characterized in that: When all target write threads in the target thread group reach the barrier, all data in the target flush buffer are forcibly written to the solid-state drive based on the asynchronous thread, including: Get the number of threads in the target thread group that have reached the barrier; Determine whether the number of threads reaches the barrier thread number; When the number of threads reaches the barrier thread number, all data in the target flush buffer are forcibly written to the solid-state drive based on the asynchronous thread.

4. The disk flashing method based on non-volatile memory according to claim 1, characterized in that: Also includes: Map each persistent file to the process's address space to achieve the mapping relationship between the persistent file's disk address and the process's virtual address; Writing the data in the target flush buffer into the target persistent file includes: Read the data in the target disk buffer through the pointer; According to the mapping relationship, the data read from the target flush buffer is written into the target persistent file.

5. The disk flashing method based on non-volatile memory according to claim 1, characterized in that: The write threads are grouped according to the number of cores of the CPU to obtain multiple thread groups including: Divide all writing threads into multiple thread groups equal to the number of CPU cores.

6. A disk-swiping device based on non-volatile memory, characterized in that: It includes a grouping unit, a cache unit, a suspension unit, a writing unit and a releasing unit; The grouping unit is used to group the writing threads according to the number of cores of the central processing unit to obtain multiple thread groups; wherein each thread group corresponds to a disk flush buffer; the disk flush buffer is a non-volatile memory; The cache unit is used to write data corresponding to each target writing thread in the target thread group into the target flush buffer; The suspending unit is configured to suspend the target thread group; wherein the target thread group is any one of all thread groups; The writing unit is configured to synchronously write the data in the target flush buffer to the solid-state hard disk when the target flush buffer receives data sent by all corresponding target write threads; The releasing unit is used to release the target thread group; Regarding the capacity setting of the disk flushing buffer, the device further includes an acquisition unit, a buffer determination unit, a file determination unit, an adjustment unit, a file capacity determination unit, and a division unit; The acquisition unit is used to obtain the number of active threads in the target thread group during program execution; The buffer determination unit is configured to determine the capacity of a target flush buffer corresponding to the target thread group based on a set single message data volume, a sector byte number, an original offset, and the number of active threads; wherein the target flush buffer is the flush buffer corresponding to the target thread group; The cache unit is used to aggregate and write data corresponding to all target writing threads in the target thread group into the target flush buffer; The file determination unit is configured to determine whether the target thread group has a corresponding target persistent file; wherein the target persistent file is a storage area divided from the solid state drive; The adjusting unit is configured to adjust the file capacity of the target persistent file according to the total amount of all data in the target flush buffer when the target thread group has a corresponding target persistent file; The file capacity determination unit is configured to determine the file capacity for performing a disk flushing operation according to the total amount of all data in the target disk flushing buffer when the target thread group does not have a corresponding target persistent file; The partitioning unit is configured to partition the target persistent file from the solid-state drive according to the file capacity; The writing unit is configured to write the data in the target flush buffer into the target persistent file; The buffer determination unit is used to call a disk flush buffer calculation formula to process the set single message data volume, sector byte number, original offset, and the number of active threads to determine the capacity of the target disk flush buffer corresponding to the target thread group; the disk flush buffer calculation formula is: S=(20*number of active threads / 4+x); Where S represents the capacity of the target flush buffer, 20 is the amount of data in a single message, 4 is the number of sector bytes, and x represents the original offset.

7. A disk-swiping device based on non-volatile memory, characterized in that: include: memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the non-volatile memory-based disk flashing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the disk flashing method based on non-volatile memory as described in any one of claims 1 to 5 are implemented.

Images