Buffer management method, device, solid state hard disk and storage medium
By introducing a buffer manager in the solid-state drive and using continuous and discrete buffer modes to manage the buffer space, the problem of low read and write performance of SSD is solved, the buffer utilization efficiency and garbage collection efficiency are improved, and the overall performance is improved.
Patent Information
- Application Number
- CN202410823909.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-25
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-06-25
AI Technical Summary
Solid-state drives (SSDs) have shortcomings in read and write performance, especially the low buffer management efficiency, resulting in low read and write performance.
It adopts a buffer manager, supports continuous buffer mode and discrete buffer mode, manages buffer space through buffer identification list, optimizes data read and write operations, and improves buffer utilization efficiency and garbage collection efficiency.
It improves the efficiency and turnover of read and write buffers in the storage system, reduces read and write waiting time, improves the read and write performance of the solid-state disk, and optimizes the garbage collection efficiency.
Smart Images

Figure CN118377440B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of storage devices, and in particular to a buffer management method, device, solid-state hard disk and storage medium. Background Art
[0002] Storage devices can store data in storage space and perform corresponding read and write operations based on the read and write commands issued by the host. Solid State Drives (SSDs) are increasingly being used in various scenarios due to their excellent read and write performance.
[0003] To improve read and write performance, a portion of the storage space in the SSD will be reserved for use as a buffer. How to efficiently manage the allocation and release of the buffer to ensure SSD read and write performance is a technical problem that needs to be solved urgently. Summary of the invention
[0004] In view of this, the present invention provides a buffer management method, device, solid state drive and storage medium to solve the problem of low SSD read and write performance.
[0005] In a first aspect, the present invention provides a buffer management method, which is applied to a buffer manager, comprising:
[0006] Setting the working mode of the buffer manager; the working mode includes a continuous buffer mode and a discrete buffer mode;
[0007] When the buffer manager is in a continuous buffer mode, in response to a first read / write instruction, determining a plurality of continuous buffer spaces corresponding to the first read / write instruction, and performing continuous read / write operations according to the plurality of continuous buffer spaces;
[0008] When the buffer manager is in discrete buffer mode, in response to a second read / write instruction, a single buffer space corresponding to the second read / write instruction is determined, and discrete read / write operations are performed according to the single buffer space.
[0009] In some optional embodiments, the method further comprises:
[0010] At least one buffer identification list is configured according to the set working mode; the buffer identification list is used to record the buffer identification corresponding to the buffer space;
[0011] When the set working mode is the continuous buffer mode, each bit in the buffer identifier list corresponds to a corresponding buffer identifier;
[0012] When the set working mode is the discrete buffer mode, r bits in the buffer identifier list correspond to corresponding buffer identifiers; r>1.
[0013] In some optional implementations, 2^r is not less than the number of buffer identifiers in the buffer identifier list.
[0014] In some optional implementations, configuring at least one buffer identification list according to the set working mode includes:
[0015] In the case where the set working mode is the continuous buffer mode, the depth of the buffer identifier list is set to a first depth, and the buffer identifier list is set to be full of buffer identifiers;
[0016] When the set working mode is the discrete buffer mode, the depth of the buffer identification list is set to a second depth, and the buffer identification list is set to be empty.
[0017] In some optional implementations, the buffer identification list is provided with a head pointer and a tail pointer;
[0018] The configuring at least one buffer identification list according to the set working mode also includes:
[0019] When the set working mode is the continuous buffer mode, the head pointer of the buffer identification list is pointed to the tail of the buffer identification list, and the tail pointer of the buffer identification list is pointed to the head of the buffer identification list;
[0020] When the set working mode is the discrete buffer mode, the head pointer and the tail pointer of the buffer identification list are pointed to the same position.
[0021] In some optional implementations, configuring at least one buffer identification list according to the set working mode further includes:
[0022] When the set working mode is the continuous buffer mode, checking whether the highest bit of the buffer identification list is a preset value;
[0023] When the highest bit of the buffer identification list is a preset value, it is determined that the configuration of the buffer identification list is completed.
[0024] In some optional implementations, performing the continuous read and write operations according to the plurality of continuous buffer spaces comprises: determining a plurality of continuous buffer identifiers in the buffer identifier list corresponding to the plurality of buffer spaces currently required, and then releasing the plurality of continuous buffer identifiers in the buffer identifier list;
[0025] The performing discrete read and write operations according to the single buffer space includes: after a flash memory controller or a processor releases a target buffer identifier corresponding to the single buffer space, writing the target buffer identifier into the buffer identifier list.
[0026] In some optional implementations, the buffer identification list is provided with a head pointer and a tail pointer; the method further includes:
[0027] After releasing the plurality of consecutive buffer identifiers in the buffer identifier list, triggering the tail pointer of the buffer identifier list to be incremented by one;
[0028] After the target buffer identifier is written into the buffer identifier list, the head pointer of the buffer identifier list is triggered to increase by one.
[0029] In some optional embodiments, the method further comprises:
[0030] In response to a mode switching instruction, switch from the current first mode to another second mode; wherein the first mode is a continuous buffer mode and the second mode is a discrete buffer mode; or, the first mode is a discrete buffer mode and the second mode is a continuous buffer mode.
[0031] In some optional implementations, the switching from the current first mode to another second mode includes:
[0032] When the first mode is the continuous buffer mode, releasing all buffer identifiers in the buffer identifier list, and then switching to the discrete buffer mode;
[0033] When the first mode is the continuous buffering mode, all released buffer identifiers are added to the buffer identifier list, and then the continuous buffering mode is switched.
[0034] In some optional implementations, the mode switching instruction is generated based on the following method:
[0035] In the case where the first mode is the continuous buffer mode, when the current amount of junk data exceeds a first preset threshold, generating a mode switching instruction for switching to the discrete buffer mode;
[0036] In the case where the first mode is the discrete buffer mode, when the current amount of junk data is less than a second preset threshold, a mode switching instruction for switching to the continuous buffer mode is generated.
[0037] In some optional implementations, the buffer identification list is configured to be used exclusively by one processor core; or, the buffer identification list is configured to be shared by multiple processor cores.
[0038] In some optional implementations, the buffer identification list is stored in a corresponding buffer register, and the bit width of the buffer register is consistent with the bit width of the buffer identification list.
[0039] In some optional embodiments, the method further comprises:
[0040] When the buffer manager is in discrete buffer mode, collecting target data pages in the single buffer space;
[0041] Determine a data classification of the target data page; the data classification includes a cold data page and a hot data page;
[0042] When the target data page is a cold data page, aggregating the cold data page into a cold data block;
[0043] When the target data page is a hot data page, the hot data page is aggregated into a hot data block.
[0044] In some optional implementations, determining the data classification of the target data page includes:
[0045] Extracting features of the target data page to determine feature information of the target data page; the feature information includes: at least one of attribute information, time information, a logical block address, and a physical block address of the target data page;
[0046] The characteristic information of the target data page is input into a pre-trained classification prediction model, and the data classification of the target data page is determined according to the classification prediction result of the classification prediction model.
[0047] In some optional implementations, the classification prediction model is a decision tree model, and the t-th decision tree is generated based on the t-1-th decision tree; ,in, Represents the t-th decision tree for the sample The predicted output is Represents the minimum square loss of the residual corresponding to the jth sample, where J is the number of samples;
[0048] The step of inputting the characteristic information of the target data page into a pre-trained classification prediction model, and determining the data classification of the target data page according to the classification prediction result of the classification prediction model, comprises:
[0049] Preprocessing the characteristic information of the target data page to generate preprocessing information of the target data page; the preprocessing includes one or more of missing value processing, outlier processing, feature engineering processing, categorical feature processing, and standardization processing format conversion processing;
[0050] Inputting the preprocessed information into a pre-trained decision tree model to determine a predicted output of at least a portion of the decision tree;
[0051] The prediction outputs of at least part of the decision tree are weighted, and the data classification of the target data page is determined according to the weighted processing result.
[0052] In a second aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the buffer management method of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0053] In a third aspect, the present invention provides a solid-state hard drive, comprising: a flash memory conversion layer and a flash memory chip; the flash memory conversion layer comprises a processor, a flash memory controller and a buffer manager; wherein the buffer manager is used to execute the buffer management method of the above-mentioned first aspect or any corresponding embodiment thereof.
[0054] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the buffer management method of the first aspect or any corresponding embodiment thereof.
[0055] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the buffer management method of the first aspect or any corresponding embodiment thereof.
[0056] The buffer manager in the embodiment of the present invention supports two working modes, namely continuous buffer mode and discrete buffer mode, and can realize the management of continuous buffer space and single discrete buffer space, so as to optimize data read and write operations, and can be applied to more refined read and write operations, thereby improving the utilization efficiency and turnover of the read and write buffer in the storage system, while reducing the waiting time of the read and write buffer in the storage system, and improving the read and write performance of the solid-state disk; and the discrete buffer mode is more conducive to garbage collection and can improve the garbage collection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0058] Figure 1is a flow chart of a buffer management method according to an embodiment of the present invention;
[0059] Figure 2 is a flow chart of another buffer management method according to an embodiment of the present invention;
[0060] Figure 3 is a schematic diagram of an architecture of a flash translation layer according to an embodiment of the present invention;
[0061] Figure 4 is a schematic diagram of a buffer identification list according to an embodiment of the present invention;
[0062] Figure 5 is a schematic diagram of multiple buffer identification lists in a continuous buffer mode according to an embodiment of the present invention;
[0063] Figure 6 is a schematic diagram of multiple buffer identification lists in a discrete buffer mode according to an embodiment of the present invention;
[0064] Figure 7 is a flowchart of another buffer management method according to an embodiment of the present invention;
[0065] Figure 8 is a structural schematic diagram of a solid state hard disk according to an embodiment of the present invention;
[0066] Fig. 9 is a schematic diagram of a process of building a model according to an embodiment of the present invention;
[0067] Fig.10 is a structural block diagram of a buffer management device according to an embodiment of the present invention;
[0068] Fig.11 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0070] SSDs are equipped with flash memory chips, such as Nand Flash Memory chips. The read and write characteristics of NAND Flash are very different from those of disks. For example, NAND Flash has the limitation of "erase before write", and the granularity of erase operations in Flash is much larger than that of write operations. Due to the unique physical characteristics of Nand Flash, the FTL (Flash Translation Layer) layer is needed to map the LBA (Logical Block Address) to the PBA (Physical Block Address) inside the device; on SSDs, both HRW / HRR (Host Random Write / Host Random Read) performance and HSW / HSR (Host Sequential Write / Host Sequential Read) performance are highly dependent on the performance of the FTL layer algorithm.
[0071] Currently, part of the storage space in the SSD is reserved for use as a buffer to compensate for the physical characteristics of Nand flash memory. Buffer space management technology is a technology that manages the space of data stored in the buffer to optimize data read and write operations. When data needs to be accessed, it needs to be read from the flash chip to the buffer; when the data in the buffer is modified or no longer needed, it needs to be written back to the flash chip. In this process, the buffer needs to be allocated and released reasonably.
[0072] According to an embodiment of the present invention, a buffer management method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0073] In this embodiment, a buffer management method is provided, which can be applied to a buffer manager (BufferManager). The buffer manager is located in a flash translation layer (FTL) of a solid-state drive and cooperates with a flash controller (Nand Flash Controller, NFC) and a processor of the flash translation layer to achieve buffer management. Figure 1 is a flow chart of a method for managing a buffer zone according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps.
[0074] Step S101, setting the working mode of the buffer manager; the working mode includes a continuous buffer mode and a discrete buffer mode.
[0075] The buffer manager manages the buffer in the solid state drive to allocate and release the buffer. In this embodiment, the buffer is divided into multiple buffer spaces, each of which is generally based on a page to process a corresponding data page; the buffer manager performs different management methods on the buffer space in different working modes.
[0076] Specifically, the buffer manager has two working modes, namely, continuous buffer mode and discrete buffer mode. In the continuous buffer mode, multiple continuous buffer spaces can be managed uniformly; while in the discrete buffer mode, the buffer manager can manage a single buffer space, that is, it can allocate and release an independent buffer space.
[0077] Step S102: when the buffer manager is in a continuous buffer mode, in response to a first read / write instruction, determine a plurality of continuous buffer spaces corresponding to the first read / write instruction, and perform continuous read / write operations according to the plurality of continuous buffer spaces.
[0078] In this embodiment, the buffer manager is in different working modes and will execute different processing methods.
[0079] Specifically, if the current working mode of the buffer manager is the continuous buffer mode, that is, when the buffer manager is in the continuous buffer mode, the buffer manager can obtain the corresponding operation instruction, that is, the first read-write instruction; for example, the host can send the corresponding first read-write instruction to the solid-state drive, and the first read-write instruction can specifically be a read instruction or a write instruction. The buffer manager of the solid-state drive can respond to the first read-write instruction, and then determine a plurality of buffer spaces corresponding to the first read-write instruction, and these buffer spaces are continuous; for example, each buffer space can be provided with a unique buffer ID, and the buffer IDs corresponding to these buffer spaces are continuous. After determining these continuous buffer spaces, continuous read and write operations are performed on these continuous buffer spaces to complete the allocation of multiple continuous buffer spaces.
[0080] For example, when the host sends a first read / write instruction for writing data, the buffer manager can determine multiple continuous buffer spaces corresponding to the first read / write instruction based on the first read / write instruction, and then instruct the processor to write the corresponding data into the multiple continuous buffer spaces.
[0081] Step S103: when the buffer manager is in discrete buffer mode, in response to the second read / write instruction, determine a single buffer space corresponding to the second read / write instruction, and perform discrete read / write operations according to the single buffer space.
[0082] If the current working mode of the buffer manager is discrete buffer mode, that is, the buffer manager is in discrete buffer mode, the buffer manager can also obtain operation instructions suitable for discrete buffer mode, that is, second read and write instructions; for example, the host can send second read and write instructions to the solid state drive, and the second read and write instructions can specifically be read instructions or write instructions; or, in discrete buffer mode, the buffer manager can perform garbage collection (GC), and the second read and write instructions can also be corresponding garbage collection instructions.
[0083] The second read / write instruction is an instruction to use a certain buffer space. Specifically, the buffer manager may respond to the second read / write instruction, and then determine a single buffer space corresponding to the second read / write instruction; for example, each buffer space may be provided with a unique buffer ID, and the single buffer space corresponds to a buffer ID. After determining the single buffer space, continuous read / write operations may be performed on the single buffer space, and the allocation of the single buffer space is completed.
[0084] For example, when performing garbage collection, a read instruction (i.e., a second read / write instruction) for reading a data page in a buffer space can be generated. The buffer manager can determine the single buffer space corresponding to the read instruction and perform garbage collection on the single buffer space, thereby achieving refined garbage collection and facilitating erase operations.
[0085] The buffer management method provided in this embodiment supports two working modes: continuous buffer mode and discrete buffer mode. The buffer manager can realize the management of continuous buffer space and single discrete buffer space, so as to optimize data reading and writing operations. It can be applied to more refined reading and writing operations, improve the utilization efficiency and turnover of the read and write buffer in the storage system, and reduce the waiting time of the read and write buffer in the storage system, thereby improving the read and write performance of the solid-state disk; and the discrete buffer mode is more conducive to garbage collection and can improve the garbage collection efficiency.
[0086] In this embodiment, a buffer management method is provided, which can be applied to a buffer manager located in a flash memory conversion layer of a solid state drive. Figure 2 is a flow chart of a method for managing a buffer zone according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps.
[0087] Step S201, setting the working mode of the buffer manager; the working mode includes a continuous buffer mode and a discrete buffer mode.
[0088] For details, please see Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.
[0089] Step S202: configuring at least one buffer identification list according to the set working mode; the buffer identification list is used to record buffer identifications corresponding to the buffer spaces.
[0090] In this embodiment, the buffer manager manages the buffer space based on the buffer ID; specifically, the buffer manager is configured with a buffer ID list (Buffer ID List) for recording buffer IDs, and efficiently and reasonably allocates buffer IDs to front-end commands based on the buffer ID list, thereby fully utilizing the buffer space and achieving hardware acceleration.
[0091] The buffer manager cooperates with the flash controller and processor of the flash translation layer to manage the buffer. The processor is generally a microprocessor (Micro Controller Unit, MCU). Figure 3 The schematic diagram of the architecture of the flash translation layer is shown in FIG. Figure 3 As shown, the flash conversion layer of the solid-state drive can process the host commands (such as the first read and write commands, etc.) issued by the host; wherein the flash controller can release the corresponding buffer identifier as needed, the processor can obtain the required buffer identifier, and release certain buffer identifiers; the buffer manager is a key component responsible for receiving front-end commands, coordinating the interaction with the processor and the flash controller hardware module, and can allocate buffer identifiers so that the processor can perform read and write operations based on the buffer space corresponding to the allocated buffer identifier.
[0092] The buffer identifier list is related to the currently set working mode. When the set working mode is the continuous buffer mode, each bit in the buffer identifier list corresponds to a corresponding buffer identifier; when the set working mode is the discrete buffer mode, r bits in the buffer identifier list correspond to corresponding buffer identifiers; r>1. In order for the buffer identifier list to represent all buffer identifiers, 2^r must be not less than the number of buffer identifiers in the buffer identifier list.
[0093] In this embodiment, if the currently set working mode is the continuous buffer mode, since multiple continuous buffer spaces are managed at this time, that is, multiple continuous buffer identifiers are managed at this time, it is not necessary to fully represent each buffer identifier. Specifically, in the continuous buffer mode, each bit in the buffer identifier list corresponds to a buffer identifier, and multiple buffer identifiers can be used as an identifier block, and the corresponding initial address can be returned when the identifier block is allocated; for example, when multiple buffer identifiers are allocated, the initial addresses of the buffer blocks corresponding to the multiple buffer identifiers are used to represent the corresponding multiple buffer identifiers.
[0094] If the currently set working mode is discrete buffer mode, since a single buffer space is managed at this time, and the managed buffer space is random, that is, the buffer identifier list records random buffer identifiers, so the buffer identifier list needs to record complete buffer identifiers; in this embodiment, each r bits represents a buffer identifier. In addition, in order to make the buffer identifier list represent all buffer identifiers, 2^r is not less than the number of buffer identifiers in the buffer identifier list, that is, r≥log 2 n, where n is the number of buffer identifiers in the buffer identifier list.
[0095] In some optional implementations, the above step S202 "configuring at least one buffer identification list according to the set working mode" may specifically include the following steps A1 to A2.
[0096] Step A1: when the set working mode is the continuous buffer mode, the depth of the buffer identifier list is set to a first depth, and the buffer identifier list is set to be full of buffer identifiers.
[0097] In this embodiment, when entering the continuous buffering mode, the buffer manager needs to initialize the buffer identification list to configure the buffer identification list so that the buffer identification list can meet the requirements of the continuous buffering mode.
[0098] Specifically, it is necessary to set the depth of the buffer identifier list. In the continuous buffer mode, the depth of the buffer identifier list is set to the first depth, which can be determined based on actual applications; wherein, since in the continuous buffer mode, each bit in the buffer identifier list corresponds to a buffer identifier, the depth of the buffer identifier list can be determined based on the number of buffer identifiers that need to be represented, for example, the depth of the buffer identifier list is set to be consistent with the number of buffer identifiers.
[0099] Moreover, for the continuous buffer mode, after the buffer identifier list is initialized, the default buffer identifier list is full of buffer identifiers; at this time, the IDs in the buffer identifier list may all be 1. At this time, the buffer space corresponding to the buffer identifier in the buffer identifier list may be used, that is, data may be stored therein; specifically, when performing read and write operations on the buffer space based on the buffer identifier list, the corresponding buffer identifier may be read from the buffer identifier list, so that the corresponding buffer space may be used, and the buffer space may be released after use, and the corresponding buffer identifier may be released (the buffer identifier is marked from 1 to 0), and when the buffer identifier list is empty, all the buffer identifiers therein are 0.
[0100] Optionally, the buffer identification list is provided with a head pointer and a tail pointer; and the above step S202 "configuring at least one buffer identification list according to the set working mode" may further include step A3.
[0101] Step A3: when the set working mode is the continuous buffering mode, the head pointer of the buffer identification list is pointed to the tail of the buffer identification list, and the tail pointer of the buffer identification list is pointed to the head of the buffer identification list.
[0102] In this embodiment, the head pointer and tail pointer of the buffer identification list are used to indicate the use of the buffer identification list, where the buffer identification is recorded. For the continuous buffer mode, when the buffer identification list is set to be full of buffer identifications, the positions of the head pointer and tail pointer of the buffer identification list are also adaptively set, that is, the head pointer of the buffer identification list points to the tail of the buffer identification list, corresponding to the end address of the buffer identification list, and the tail pointer of the buffer identification list points to the head of the buffer identification list, corresponding to the start address of the buffer identification list.
[0103] For example, the buffer identification list is stored in a corresponding buffer register, and a corresponding head pointer and tail pointer may be set for the buffer register; and the bit width of the buffer register is consistent with the bit width of the buffer identification list, so that the two can correspond.
[0104] Figure 4 A schematic diagram of a buffer identifier list after initialization in a continuous buffer mode is shown, wherein each square corresponds to an identifier block formed by a plurality of continuous buffer identifiers; for example, if s continuous buffer identifiers need to be allocated each time, the identifier block corresponds to s buffer identifiers, i.e., a total of s bits.
[0105] In addition, multiple buffer identification lists can be optionally set; for example, the number of buffer identification lists can be set based on the number of processor cores of the processor, wherein the base address of the buffer identification list can be determined when the flash memory controller is initialized. In addition, the buffer identification list can be configured to be used exclusively by one processor core, for example, each processor core corresponds to one buffer identification list; or, the buffer identification list can also be configured to be shared by multiple processor cores.
[0106] If the buffer identifier list can be configured to be used exclusively by one processor core, each processor core can read the buffer register corresponding to the buffer identifier list to obtain the corresponding buffer identifier, that is, the starting address of multiple consecutive buffer identifiers. When multiple processor cores share a buffer identifier list, at a certain point in time, the buffer identifier list can only be used by one of the processor cores.
[0107] Figure 5 FIG. 2 shows a schematic diagram of multiple buffer identification lists set in the continuous buffer mode. Figure 5 As shown, there are 8 buffer identification lists, namely, list 0 to list 7; and after reading the buffer identification in the buffer identification list, the corresponding buffer identification is released, that is, the bit corresponding to the buffer identification is set to 0.
[0108] In this embodiment, the solid state drive is generally provided with a DRAM (dynamic random access memory) buffer space and an SRAM (static random access memory) buffer space, and the main responsibilities of the buffer manager include the management of the DRAM buffer space and the SRAM buffer space. When managing the DRAM buffer space and the SRAM buffer space, the buffer manager supports both random release and continuous release. This means that the buffer manager can flexibly manage these buffers and release a single or a series of continuous buffer spaces as needed to meet system requirements.
[0109] For example, Figure 5 As shown in the figure, for example, List 0 to List 3 represent the buffer identification lists of the SRAM buffer space, and List 4 to List 7 represent the buffer identification lists of the DRAM buffer space. Their base addresses are determined when NFC is initialized.
[0110] Step A2: when the set working mode is the discrete buffer mode, the depth of the buffer identification list is set to a second depth, and the buffer identification list is set to be empty.
[0111] In this embodiment, when entering the discrete buffer mode, the buffer manager also needs to initialize the buffer identification list to configure the buffer identification list so that the buffer identification list can meet the requirements of the discrete buffer mode.
[0112] Specifically, the depth of the buffer identifier list needs to be set. In discrete buffer mode, the depth of the buffer identifier list is set to the second depth; wherein, since in discrete buffer mode, each r bits in the buffer identifier list corresponds to a buffer identifier, the depth of the buffer identifier list can be determined based on the number of buffer identifiers that need to be represented, for example, the ratio of the number of buffer identifiers to r is used as the depth of the buffer identifier list.
[0113] Moreover, for the discrete buffer mode, after the buffer identification list is initialized, the default buffer identification list is empty, that is, it does not have a full buffer identification; in this embodiment, the IDs in the buffer identification list are also set to 1. At this time, when a certain buffer space needs to be used, the flash memory controller or the processor releases the corresponding buffer space, and then the buffer controller obtains the released buffer identification and records it in the buffer identification list to indicate that the buffer space corresponding to the buffer identification can be used.
[0114] In this embodiment, each buffer identifier list may also be limited by depth and width, where the depth is limited to a maximum depth depth and the width is width. In continuous buffer mode, each bit represents a buffer identifier, so each buffer identifier list can accommodate a maximum of (width × depth) buffer identifiers. In discrete buffer mode, each r bits represent a buffer identifier, so each buffer identifier list can accommodate a maximum of (width × depth) / r buffer identifiers, where (width × depth) / r is an integer. It can be understood that if the depth of the buffer identifier list is the same in different working modes, that is, the first depth is the same as the second depth, then the buffer identifier list in the continuous buffer mode can represent more buffer identifiers.
[0115] Optionally, the buffer identification list is provided with a head pointer and a tail pointer; and the above step S202 "configuring at least one buffer identification list according to the set working mode" may further include step A4.
[0116] Step A4: when the set working mode is the discrete buffer mode, the head pointer and the tail pointer of the buffer identification list are pointed to the same position.
[0117] In this embodiment, for the discrete buffer mode, when the buffer identifier list is set to be full of buffer identifiers, the positions of the head pointer and the tail pointer of the buffer identifier list are also adaptively set, that is, the head pointer and the tail pointer of the buffer identifier list point to the same position, for example, the head pointer and the tail pointer of the buffer identifier list both point to the head of the buffer identifier list. Figure 4The structure shown is similar, and the values therein are also all 1, except that the head pointer and the tail pointer of the buffer identification list point to the same position; wherein each block can specifically represent an r-bit buffer identification.
[0118] In discrete buffer mode, if a buffer space needs to be allocated currently, the buffer identifier corresponding to the buffer space needs to be released first, and then the obtained buffer identifier is stored in the buffer identifier list in order; among them, the buffer identifier can be stored to the position pointed to by the current head pointer, and the head pointer is increased by one after storage.
[0119] In addition, multiple buffer identification lists can be optionally set; for example, the number of buffer identification lists can be set based on the number of processor cores of the processor. In addition, the buffer identification list can be configured to be used exclusively by one processor core, for example, each processor core corresponds to a buffer identification list; or, the buffer identification list can also be configured to be shared by multiple processor cores. Among them, when multiple processor cores share a buffer identification list, at a certain point in time, the buffer identification list can only be used by one of the processor cores.
[0120] Figure 6 FIG. 2 shows a schematic diagram of multiple buffer identification lists set in discrete buffer mode. Figure 6 As shown, there are 8 buffer identification lists, namely, list 0 to list 7; and after the buffer identification is obtained, it is recorded in the corresponding buffer identification list. Figure 6 As shown, the positions where all the values are 1 (ie, FFFF) indicate that the buffer identifier is not recorded. In other words, in discrete buffer mode, if all the data in the buffer identifier list is 1, it indicates that the buffer identifier list is empty.
[0121] In addition, optionally, the above-mentioned step S202 "configure at least one buffer identification list according to the set working mode" can also include: when the set working mode is a continuous buffer mode, check whether the highest bit of the buffer identification list is a preset value; when the highest bit of the buffer identification list is a preset value, determine that the configuration of the buffer identification list is completed.
[0122] In this embodiment, since the buffer identification list after initialization records all buffer identifications in the continuous buffer mode, in order to ensure that the configuration of the buffer identification list can be completed accurately at this time, after setting the buffer identification list to store all buffer identifications, it is possible to further check whether the highest bit of the buffer identification list is a preset value, for example, the buffer manager reads the buffer register to check whether its highest bit is 1. If the highest bit of the buffer identification list is a preset value, it means that the configuration is correct, and the initialization is completed at this time. It can be determined that the configuration of the buffer identification list is completed, and then the configured buffer identification list can be used to manage the buffer identification.
[0123] In the discrete buffer mode, after setting the buffer identification list to empty, it can be directly determined that the configuration of the buffer identification list is completed, that is, the buffer manager can be used directly according to the discrete buffer mode and the specified depth.
[0124] Step S203, when the buffer manager is in continuous buffer mode, in response to a first read / write instruction sent by the host, a plurality of continuous buffer spaces corresponding to the first read / write instruction are determined, and continuous read / write operations are performed according to the plurality of continuous buffer spaces.
[0125] For details, please see Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.
[0126] Optionally, the above step of "performing continuous read and write operations according to multiple continuous buffer spaces" specifically includes: determining multiple continuous buffer identifiers in the buffer identifier list corresponding to the multiple buffer spaces currently required, and then releasing the multiple continuous buffer identifiers in the buffer identifier list.
[0127] In this embodiment, in the continuous buffer mode, since the default buffer identifier list is full of buffer identifiers after initialization, when multiple continuous buffer spaces need to be operated, it is necessary to first obtain the corresponding multiple continuous buffer identifiers and then release them. Among them, the continuous buffer identifier release is mainly that the DMA (Direct Memory Access) of the flash memory controller processes the corresponding data and releases the corresponding buffer identifiers one by one according to the buffer identifier list. The buffer manager releases according to the bits of the buffer identifier. Only after all bits of the entire identifier block are released can it be provided to the processor. The processor can obtain the starting buffer identifier by reading the buffer register, and realize the allocation of the starting addresses of multiple continuous buffer identifiers to the processor. It can be understood that in the continuous buffer mode, s buffer identifiers can be obtained at a time. If the s buffer identifiers obtained at the same time are not completely released, or the buffer identifiers obtained before are not completely released, these s buffer identifiers cannot be obtained.
[0128] In the continuous buffer mode, the continuous release buffer flag is mainly used to accelerate the continuous write performance, which can accelerate the release and acquisition of the buffer space. For example, it can accelerate the release and acquisition of the SRAM buffer space.
[0129] Optionally, if the buffer identifier list is provided with a head pointer and a tail pointer, the method may further include: after releasing a plurality of consecutive buffer identifiers in the buffer identifier list, triggering the tail pointer of the buffer identifier list to be incremented by one.
[0130] In this embodiment, the continuous buffer mode release involves obtaining a certain amount of buffer identifiers each time, which are represented by bits, and s buffer identifiers are combined into an identifier block, that is, the size of each identifier block is s bits. In the continuous buffer mode, the main goal is to obtain a series of buffer identifiers and return their starting addresses. After initialization, the head pointer of the buffer identifier list is located at the head and the tail pointer is located at the tail. Whenever an identifier block is released, the tail pointer is triggered once, so that the tail pointer is increased by 1; in addition, if the buffer identifier list is not full, when each identifier block is full, the head pointer of the buffer identifier list is increased by 1.
[0131] When the processor needs to occupy a buffer identifier, a certain amount of buffer identifiers can be obtained through the bus and allocated to the flash memory controller. After the processor configures the buffer controller to occupy the corresponding buffer space, the buffer space is automatically released, and the flash memory controller releases the corresponding buffer identifier, so that the position corresponding to the buffer identifier can be set to 0. In this way, through the management of this continuous buffer mode, the system can effectively allocate and release buffer identifiers to ensure efficient use of resources.
[0132] Step S204: when the buffer manager is in discrete buffer mode, in response to a second read / write instruction sent by the host, determine a single buffer space corresponding to the second read / write instruction, and perform discrete read / write operations according to the single buffer space.
[0133] For details, please see Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.
[0134] Optionally, the above step of "performing discrete read and write operations according to a single buffer space" may include: after the flash memory controller or processor releases the target buffer identifier corresponding to the single buffer space, writing the target buffer identifier into a buffer identifier list to allocate the target buffer identifier to the processor.
[0135] In this embodiment, in discrete buffer mode, since the default buffer identifier list is empty after initialization, when the buffer space needs to be operated, the corresponding buffer identifier needs to be released first and then acquired, and the actual buffer identifier of r bits is acquired. In discrete buffer mode, as long as the buffer identifier has been released before, it can be acquired. In addition, optionally, if the buffer identifier list is provided with a head pointer and a tail pointer, the method may also include: after writing the target buffer identifier into the buffer identifier list, triggering the head pointer of the buffer identifier list to be increased by one to ensure that the head and tail pointers in the buffer identifier list correspond to the recorded buffer identifier position.
[0136] In some optional implementations, the discrete buffer mode can be mainly used for garbage collection. When the solid-state drive is in use, its default working mode is the continuous buffer mode, that is, the buffer manager first enters the continuous buffer mode, and then switches to the discrete buffer mode when garbage collection is required, and can be restored to the continuous buffer mode after garbage collection is completed.
[0137] The working mode of the buffer manager can be controlled by a corresponding mode switching instruction, that is, the method can also include: in response to the mode switching instruction, switching from the current first mode to another second mode; wherein the first mode is a continuous buffer mode and the second mode is a discrete buffer mode; or the first mode is a discrete buffer mode and the second mode is a continuous buffer mode. That is, the mode switching instruction can be used to switch from the continuous buffer mode to the discrete buffer mode, or it can also be used to switch from the discrete buffer mode to the continuous buffer mode.
[0138] In this embodiment, the mode switching instruction can be generated based on the following method: when the first mode is a continuous buffer mode, when the current amount of garbage data exceeds a first preset threshold, a mode switching instruction for switching to a discrete buffer mode is generated; when the first mode is a discrete buffer mode, when the current amount of garbage data is less than a second preset threshold, a mode switching instruction for switching to a continuous buffer mode is generated.
[0139] Specifically, the buffer identifier released in the discrete buffer mode is mainly used for garbage collection, and the purpose is to quickly write the data read during garbage collection to the Nand flash memory; among them, the speed of obtaining the buffer identifier in the discrete buffer mode is generally the speed of the bus, that is, one read every two clocks.
[0140] For example, when configuring the Link of the flash controller, each buffer identifier occupies a Link unit; the release of random buffer identifiers is mainly that the flash controller will release the buffer identifiers one by one according to the flash controller Link after processing the corresponding data, and the buffer manager puts the released buffer identifiers into the corresponding buffer identifier list, and the buffer identifiers in the buffer identifier list are in disorder. The processor obtains the corresponding buffer identifier by reading the corresponding buffer register. In the discrete buffer mode, after initialization, the head and tail pointers can both be located at the head of the buffer identifier list. When the flash controller or processor releases the buffer identifier to the buffer manager, the buffer manager first stores the buffer identifier in a cache area. When the processor needs to obtain the buffer identifier, if the buffer space is not empty, that is, the head and tail pointers are not in the same identifier block (at this time, one buffer identifier corresponds to one identifier block), the processor will obtain the corresponding buffer identifier from the position of the tail pointer, and then the tail pointer will be increased by 1 and moved to the next identifier block. If the buffer space is empty, but there is a buffer identifier in the cache area, the processor can obtain the buffer identifier from the cache area. If the buffer space is empty and there is no buffer identifier in the cache area, a position of all 1 will be obtained, indicating that the obtained buffer identifier is invalid.
[0141] In this embodiment, compared with the traditional garbage collection that erases the entire data block, in the discrete buffer mode, a single buffer identifier can be released. The buffer identifier release accelerates the flow of the buffer, thereby improving the garbage collection efficiency.
[0142] Optionally, the above step of "switching from the current first mode to another second mode" specifically includes: when the first mode is a continuous buffer mode, releasing all buffer identifiers in the buffer identifier list, and then switching to the discrete buffer mode; when the first mode is a continuous buffer mode, adding all released buffer identifiers to the buffer identifier list, and then switching to the continuous buffer mode.
[0143] In this embodiment, if the continuous buffer mode is currently in operation and it is necessary to switch to the discrete buffer mode, that is, before entering the discrete buffer mode (for example, before the above step A2), it is necessary to clear the previously saved buffer identifiers, that is, it is necessary to release all buffer identifiers in the buffer identifier list, and then switch to the discrete buffer mode, so that initialization can be performed to reconfigure the depth of the buffer identifier list and make the buffer identifiers therein empty. After the reconfiguration is completed, it is necessary to restart the state of each buffer identifier list.
[0144] If the current mode is discrete buffer mode and needs to be switched to continuous buffer mode, that is, before entering the continuous buffer mode (for example, before the above step A1), all released buffer identifiers need to be acquired into the buffer identifier list, and then switch to discrete buffer mode so that initialization can be performed. In other cases, the configuration of mode and depth is not allowed.
[0145] In this embodiment, when the buffer identifier list needs to be reconfigured, it is ensured that all acquired buffer identifiers in the continuous buffer mode have been released, or that all released buffer identifiers in the discrete buffer mode have been acquired into the buffer identifier list, thereby avoiding the problem of buffer identifiers being repeatedly acquired and then reused, and ensuring the robustness and correctness of the buffer manager between different configurations.
[0146] In this embodiment, a buffer management method is provided, which can be applied to a buffer manager located in a flash memory conversion layer of a solid state drive. Figure 7 is a flow chart of a method for managing a buffer zone according to an embodiment of the present invention. Figure 7 As shown, the process includes the following steps.
[0147] Step S701, setting the buffer manager to continuous buffer mode.
[0148] In this embodiment, the default working mode of the buffer manager is the continuous buffer mode. After the machine is turned on, the buffer manager can automatically enter the continuous buffer mode. The process of setting the working mode of the buffer manager can be specifically referred to in Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.
[0149] Step S702, receiving a first read / write instruction sent by the host, and performing continuous read / write operations according to a plurality of continuous buffer spaces.
[0150] In this embodiment, the host can send a write command to the SSD, and the SSD executes the host write command after receiving the host command. The host and SSD begin to interact with each other in the write command and data. Among them, the flash conversion layer of the SSD can also include a host write unit. After the buffer manager turns on the continuous buffer mode, it can manage the buffer and mobilize the host write unit to process the write task sent by the host.
[0151] Step S703: when there is too much garbage data, the discrete buffer mode is enabled, and garbage collection is performed using a single buffer space.
[0152] When the current amount of garbage data exceeds a first preset threshold, the mode is switched to a discrete buffer mode; the buffer manager manages the buffer space and activates the garbage collection processing unit.
[0153] Figure 8 A structural diagram of a solid state hard disk is shown. Figure 8 As shown, the solid state drive includes a flash memory translation layer and a flash memory chip (eg, Nand Flash Memory), and the flash memory translation layer includes, in addition to the above-mentioned processor, flash memory controller and buffer manager, a host write processing unit and a garbage collection processing unit.
[0154] Among them, the solid-state drive communicates with the host through the host interface unit, can obtain the corresponding read and write commands, and return the read and write results. The processor mainly uses the buffer space to control and process the read and write tasks on the solid-state drive. The host write processing unit is used to process the host write tasks, including continuous writes and random writes on the host. The garbage collection processing unit is used to perform garbage collection; among them, when the solid-state drive writes new data to the flash memory chip, it must go through block erasure. Using the garbage collection processing unit, the valid data in one or more data blocks (blocks) can be read and written to the new data block. The data in the original data block is all invalid data. The original data block can be erased and can then be used to write new data.
[0155] Step S704: when the buffer manager is in discrete buffer mode, collect target data pages in a single buffer space.
[0156] In this embodiment, in the discrete buffer mode, since a single buffer identifier is managed, a data page corresponding to a buffer space can be read and written, so that a data page in a single buffer space, ie, a target data page, can be acquired.
[0157] Step S705 , determining the data classification of the target data page; the data classification includes cold data pages and hot data pages.
[0158] Among them, based on the frequency of data use, etc., it can be divided into cold data and hot data, and the corresponding data pages are cold data pages and hot data pages. In the traditional garbage collection mechanism, it is not possible to classify the fragmented cold and hot data in units of pages, and it can generally only manage data blocks with large data volumes; in this embodiment, for example, the feature of being able to read and write in units of data pages in discrete buffer mode can collect the corresponding target data pages, and then determine the classification of the target data page, that is, whether the target data page is a cold data page or a hot data page.
[0159] Step S706: when the target data page is a cold data page, aggregate the cold data page into a cold data block.
[0160] Step S707 : when the target data page is a hot data page, aggregate the hot data page into a hot data block.
[0161] In this embodiment, different types of data pages are classified and aggregated respectively, and aggregated into different data blocks, so that cold data blocks and hot data blocks can be processed separately, which facilitates subsequent wear leveling and other related processing on the SSD.
[0162] Optionally, a classification prediction model for classification prediction may be pre-built, and classification prediction may be performed on the target data page based on the classification prediction model to determine whether it is a cold data page or a hot data page.
[0163] Specifically, the above step S705 “determine the data classification of the target data page” may include the following steps B1 to B2.
[0164] Step B1, extracting features of a target data page to determine feature information of the target data page; the feature information includes: at least one of attribute information, time information, logical block address, and physical block address of the target data page.
[0165] Step B2: input the characteristic information of the target data page into a pre-trained classification prediction model, and determine the data classification of the target data page according to the classification prediction result of the classification prediction model.
[0166] In this embodiment, a model for classification prediction can be pre-trained, and then classification prediction can be performed on the target data page collected based on the model. Figure 8 As shown, the flash conversion layer may also include: a training data acquisition unit, a prediction data acquisition unit, a model training unit, a model prediction unit and a data aggregation unit.
[0167] Among them, the training data acquisition unit is used to collect the data pages corresponding to the discrete buffer identifier when the discrete buffer mode is turned on, extract the corresponding feature information from it as training data, and hand it over to the model training unit. The model training unit is used to learn and train the feature information of the data pages corresponding to the collected discrete buffer identifiers, and train a convergent classification prediction model. The prediction data acquisition unit is used to collect the feature information of the data page corresponding to the current discrete buffer identifier (i.e., the target data page) when the discrete buffer mode is turned on, and use it as prediction data to hand it over to the model prediction unit. The model prediction unit analyzes the feature information of the collected target data page and predicts whether the target data page is cold data or hot data. After the prediction, the data aggregation unit can be used to classify and aggregate the predicted fragmented cold and hot data pages into larger data blocks, which is convenient for subsequent wear leveling and other related processing on the SSD.
[0168] Specifically, because a discrete buffer identifier corresponds to a data page, when the discrete buffer mode is turned on, the data page corresponding to a buffer identifier will be written or read. In this embodiment, the feature information extracted from the data page corresponding to the discrete buffer identifier is labeled, so as to realize feature extraction for each data page; wherein, the feature information includes attribute information, time information, logical block address (LBA), physical block address (PBA), etc. of the data page.
[0169] When training or prediction is performed, the collected data page may be preprocessed so that the data page can be used for training or prediction. Fig. 9 A schematic diagram of the process of building a model is shown in FIG. Fig. 9 As shown, after data preprocessing and model building, the required classification prediction model can be trained.
[0170] Specifically, the collected real-time operation data (current data page) can be used as training data, and the historical operation data (historically collected data page) can be annotated with actual labels (indicating whether it is cold data or hot data), and the information extracted from these data pages can be subjected to data preprocessing model. Among them, if necessary, after multiple steps of processing such as processing missing values, processing outliers, feature engineering, categorical features, data standardization, category imbalance, dividing data sets, and data format conversion, the processing process is finally completed. If the information of data preprocessing exceeds the threshold required for training, a training set can be constructed, and after model construction processing, model training can be carried out.
[0171] In the model building process, the model can be initialized, and the model can be trained based on the initialized model. The generalization learning ability of the trained model can be evaluated, and whether the model has converged can be determined. If the model converges, the classification prediction model obtained by the final training is output. If the model does not converge, it is necessary to continue the model training based on the training data obtained after data preprocessing until the model converges, and a classification prediction model with convergence that can be used to predict hot and cold data pages is obtained.
[0172] Specifically, this embodiment uses a decision tree model for training and learning, such as an XGBoost model. Among them, processing missing values: interpolating and filling missing values, marking missing values, etc., so that they no longer affect the training process of the model; processing outliers: detecting and processing outliers in the data, which can be processed by statistical methods; feature engineering: creating new features and transforming original features; categorical features: processing categorical features, including methods such as One-Hot Encoding; data standardization: standardizing or normalizing numerical features to ensure that different features have similar scales; category imbalance: if there is category imbalance in the data set, oversampling, undersampling or weight adjustment can be used to balance the data; dividing the data set: dividing the data set into a training set and a test set; data format conversion: converting the data into a format unique to the decision tree model to improve the training efficiency of the model.
[0173] Since the present embodiment adopts a decision tree model, during the model construction process, the model is mainly composed of a classification tree, and a classification tree is constructed according to training features and training data to determine the prediction result of each data.
[0174] If it is the first time to enter the discrete buffer mode to start training, the initial model is a simple tree with only one node, and its prediction value is the average value of the entire sample. It can be expressed as: ; where N is the total sample size, y i is the corresponding predicted value.
[0175] Traverse and calculate the residual between the predicted value of the current model and each actual label. The residual is the difference between the real label and the predicted value of the current model. Build the first decision tree with the residual as the target. Add the prediction result of the first tree to the initial model to form an updated model. Start to build more decision trees. When updating the model, take into account the contribution of the first tree to the residual, so as to iteratively train more weak learners: calculate the new residual, calculate the residual of the current updated model as the training target of the next weak learner, and build more decision trees with the new residual as the target. Among them, generate the first The model of a tree is: ,in, Represents the t-th decision tree for the sample The predicted output is That is, the t-1th decision tree for the sample The predicted output is is the minimum squared loss of the residual corresponding to the jth sample, where J is the number of samples. For example, the corresponding residual can be minimized to determine the minimum squared loss .
[0176] Each new tree generated attempts to capture information that was not learned by the previous model. Prediction results of the tree and Model of a tree , we can deduce The prediction result of sample i after the iteration. The prediction result of the newly created decision tree is added to the existing model to form a more powerful model. The prediction result of sample i after iterations is: .
[0177] Combine the models by taking a weighted sum of the outputs of all the trees, with the weight of each tree proportional to its contribution to the model performance. If the model is found to be overfitting, regularization parameters can be introduced to prevent the model from overfitting and adjust the parameters to optimize the model performance. According to methods such as cross-validation, the depth and number of nodes of the tree can be adjusted to find the optimal model complexity. Using the trained model, select an appropriate threshold to map the output of the model to the classification of hot and cold data. Use the validation set to evaluate the model performance to ensure that the model can generalize and converge on new data, and not only perform well on the training set, but also converge and perform well on generalized data.
[0178] After the decision tree model is trained, classification prediction can be performed based on the decision tree model. Specifically, the above step B2 "inputting the characteristic information of the target data page into the pre-trained classification prediction model, and determining the data classification of the target data page according to the classification prediction result of the classification prediction model" can include steps B11 to B13.
[0179] Step B11, preprocessing the feature information of the target data page to generate preprocessing information of the target data page; the preprocessing includes one or more of missing value processing, outlier processing, feature engineering processing, categorical feature processing, and standardized processing format conversion processing.
[0180] Step B12, inputting the preprocessing information into a pre-trained decision tree model to determine the prediction output of at least part of the decision tree.
[0181] Step B13, weighting the prediction outputs of at least part of the decision trees, and determining the data classification of the target data page according to the weighted processing result.
[0182] In this embodiment, similar to the way of processing the collected samples during the training process, after the feature information of the target data page is obtained, it also needs to be preprocessed; among them, the recognition stage is irrelevant to the category imbalance and the division of the data set, so it is only necessary to perform missing value processing, outlier processing, feature engineering processing, category feature processing, standardization processing format conversion processing, etc. on the feature information, and finally generate preprocessing information suitable for the decision tree model, and then the preprocessing information can be input into the decision tree model.
[0183] At least some of the decision trees in the decision tree model can be used for classification prediction; wherein, some decision trees are selected in reverse order for classification prediction, that is, the last T-th decision tree must be selected (t=1,2,…,T), and a certain number of decision trees can be selected based on actual needs, for example, all decision trees in the decision tree model are used for classification prediction. For each decision tree, the classification prediction result corresponding to the preprocessing information can be output, and these data are weighted. The weighted processing result can represent the probability that the target data page is a cold data page or a hot data page, so that the data classification of the target data page can be determined.
[0184] Among them, when weighted processing, the weight of each decision tree is proportional to its contribution to the model performance. Generally speaking, the later the decision tree, the greater the weight. For example, the Tth decision tree has the largest weight.
[0185] The buffer management method provided in this embodiment uses continuous buffer mode and discrete buffer mode for management inside the solid-state drive, which can be easily applied to current solid-state drive products, can improve the utilization efficiency and turnover of the read and write buffer in the storage system, and reduce the waiting time of the read and write buffer in the storage system, improve the read and write speed of the solid-state drive, and can greatly improve the HRW / HRR performance and HSW / HSR performance. In the discrete buffer mode, fragmented hot and cold data pages can be extracted, and the classification of the fragmented data can be determined based on machine learning and other methods, so that the fragmented data can be collected on the entire data block, and the fragmented hot and cold data aggregation can be achieved, which can further improve the overall performance of the solid-state drive. Compared with the previous hot and cold data identification that only targets data blocks with large data volumes, this embodiment can identify fragmented hot and cold data pages based on the buffer management method, aggregate the hot and cold data pages to the data block, realize the separation of fragmented hot / cold data, and classify and aggregate the predicted fragmented hot and cold data pages to a larger data block, which is convenient for subsequent wear leveling and other related processing on the SSD.
[0186] The embodiment of the present invention also provides a solid state drive, comprising: a flash memory conversion layer and a flash memory chip; the flash memory conversion layer comprises a processor, a flash memory controller and a buffer manager; wherein the buffer manager is used to execute the buffer management method provided in any of the above embodiments. The architecture diagram of the solid state drive can be found in Figure 8 As shown, no further details are given here.
[0187] In the present embodiment, a buffer management device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0188] This embodiment provides a buffer management device, which is applied to a buffer manager, such as Fig.10 As shown, including:
[0189] A mode setting module 1001 is used to set the working mode of the buffer manager; the working mode includes a continuous buffer mode and a discrete buffer mode;
[0190] A continuous buffer module 1002, configured to determine, in response to a first read / write instruction, a plurality of continuous buffer spaces corresponding to the first read / write instruction, and perform continuous read / write operations according to the plurality of continuous buffer spaces when the buffer manager is in a continuous buffer mode;
[0191] The discrete buffer module 1003 is used to determine a single buffer space corresponding to a second read / write instruction in response to a second read / write instruction when the buffer manager is in discrete buffer mode, and perform discrete read / write operations according to the single buffer space.
[0192] In some optional implementations, the mode setting module 1001 is further used to:
[0193] At least one buffer identification list is configured according to the set working mode; the buffer identification list is used to record the buffer identification corresponding to the buffer space;
[0194] When the set working mode is the continuous buffer mode, each bit in the buffer identifier list corresponds to a corresponding buffer identifier;
[0195] When the set working mode is the discrete buffer mode, r bits in the buffer identifier list correspond to corresponding buffer identifiers; r>1.
[0196] In some optional implementations, 2^r is not less than the number of buffer identifiers in the buffer identifier list.
[0197] In some optional implementations, the mode setting module 1001 configures at least one buffer identification list according to the set working mode, including:
[0198] In the case where the set working mode is the continuous buffer mode, the depth of the buffer identifier list is set to a first depth, and the buffer identifier list is set to be full of buffer identifiers;
[0199] When the set working mode is the discrete buffer mode, the depth of the buffer identification list is set to a second depth, and the buffer identification list is set to be empty.
[0200] In some optional implementations, the buffer identification list is provided with a head pointer and a tail pointer;
[0201] The mode setting module 1001 configures at least one buffer identification list according to the set working mode, and further includes:
[0202] When the set working mode is the continuous buffer mode, the head pointer of the buffer identification list is pointed to the tail of the buffer identification list, and the tail pointer of the buffer identification list is pointed to the head of the buffer identification list;
[0203] When the set working mode is the discrete buffer mode, the head pointer and the tail pointer of the buffer identification list are pointed to the same position.
[0204] In some optional implementations, the mode setting module 1001 configures at least one buffer identification list according to the set working mode, and further includes:
[0205] When the set working mode is the continuous buffer mode, checking whether the highest bit of the buffer identification list is a preset value;
[0206] When the highest bit of the buffer identification list is a preset value, it is determined that the configuration of the buffer identification list is completed.
[0207] In some optional implementations, the continuous buffer module 1002 performs continuous read and write operations according to the plurality of continuous buffer spaces, including: determining a plurality of continuous buffer identifiers in the buffer identifier list corresponding to the plurality of buffer spaces currently required, and then releasing the plurality of continuous buffer identifiers in the buffer identifier list;
[0208] The discrete buffer module 1003 performs discrete read and write operations according to the single buffer space, including: after the flash memory controller or processor releases the target buffer identifier corresponding to the single buffer space, writing the target buffer identifier into the buffer identifier list.
[0209] In some optional implementations, the buffer identification list is provided with a head pointer and a tail pointer;
[0210] The device method pointer processing module is used to:
[0211] After the multiple consecutive buffer identifiers in the buffer identifier list are released, the tail pointer of the buffer identifier list is triggered to increase by one; after the target buffer identifier is written into the buffer identifier list, the head pointer of the buffer identifier list is triggered to increase by one.
[0212] In some optional implementations, the device further includes a mode switching module, configured to:
[0213] In response to a mode switching instruction, switch from the current first mode to another second mode; wherein the first mode is a continuous buffer mode and the second mode is a discrete buffer mode; or, the first mode is a discrete buffer mode and the second mode is a continuous buffer mode.
[0214] In some optional implementations, the mode switching module switches from the current first mode to another second mode, including:
[0215] When the first mode is the continuous buffer mode, releasing all buffer identifiers in the buffer identifier list, and then switching to the discrete buffer mode;
[0216] When the first mode is the continuous buffering mode, all released buffer identifiers are added to the buffer identifier list, and then the continuous buffering mode is switched.
[0217] In some optional implementations, the mode switching instruction is generated based on the following method:
[0218] In the case where the first mode is the continuous buffer mode, when the current amount of junk data exceeds a first preset threshold, generating a mode switching instruction for switching to the discrete buffer mode;
[0219] In the case where the first mode is the discrete buffer mode, when the current amount of junk data is less than a second preset threshold, a mode switching instruction for switching to the continuous buffer mode is generated.
[0220] In some optional implementations, the buffer identification list is configured to be used exclusively by one processor core; or, the buffer identification list is configured to be shared by multiple processor cores.
[0221] In some optional implementations, the buffer identification list is stored in a corresponding buffer register, and the bit width of the buffer register is consistent with the bit width of the buffer identification list.
[0222] In some optional implementations, the discrete buffer module 1003 is further configured to:
[0223] When the buffer manager is in discrete buffer mode, collecting target data pages in the single buffer space;
[0224] Determine a data classification of the target data page; the data classification includes a cold data page and a hot data page;
[0225] When the target data page is a cold data page, aggregating the cold data page into a cold data block;
[0226] When the target data page is a hot data page, the hot data page is aggregated into a hot data block.
[0227] In some optional implementations, the discrete buffer module 1003 determines the data classification of the target data page, including:
[0228] Extracting features of the target data page to determine feature information of the target data page; the feature information includes: at least one of attribute information, time information, a logical block address, and a physical block address of the target data page;
[0229] The characteristic information of the target data page is input into a pre-trained classification prediction model, and the data classification of the target data page is determined according to the classification prediction result of the classification prediction model.
[0230] In some optional implementations, the classification prediction model is a decision tree model, and the t-th decision tree is generated based on the t-1-th decision tree; ,in, Represents the t-th decision tree for the sample The predicted output is Represents the minimum square loss of the residual corresponding to the jth sample, where J is the number of samples;
[0231] The discrete buffer module 1003 inputs the characteristic information of the target data page into a pre-trained classification prediction model, and determines the data classification of the target data page according to the classification prediction result of the classification prediction model, including:
[0232] Preprocessing the characteristic information of the target data page to generate preprocessing information of the target data page; the preprocessing includes one or more of missing value processing, outlier processing, feature engineering processing, categorical feature processing, and standardization processing format conversion processing;
[0233] Inputting the preprocessed information into a pre-trained decision tree model to determine a predicted output of at least a portion of the decision tree;
[0234] The prediction outputs of at least part of the decision tree are weighted, and the data classification of the target data page is determined according to the weighted processing result.
[0235] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0236] The buffer management device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, including a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0237] The embodiment of the present invention also provides a computer device having the above Fig.10 The buffer management device shown.
[0238] See also Fig.11 , Fig.11 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.11 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.11 A processor 10 is taken as an example.
[0239] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0240] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0241] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0242] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0243] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.
[0244] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0245] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.
[0246] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A buffer management method, It is characterized in that Applied to a buffer manager, the method comprises: Setting the working mode of the buffer manager; the working mode includes a continuous buffer mode and a discrete buffer mode; When the buffer manager is in a continuous buffer mode, in response to a first read / write instruction, determining a plurality of continuous buffer spaces corresponding to the first read / write instruction, and performing continuous read / write operations according to the plurality of continuous buffer spaces; When the buffer manager is in discrete buffer mode, in response to a second read / write instruction, determining a single buffer space corresponding to the second read / write instruction, and performing discrete read / write operations according to the single buffer space; The method further comprises: At least one buffer identification list is configured according to the set working mode; the buffer identification list is used to record the buffer identification corresponding to the buffer space; Furthermore, configuring at least one buffer identification list according to the set working mode includes: When the set working mode is the continuous buffer mode, the buffer identifier list is set to be full of buffer identifiers; When the set working mode is the discrete buffer mode, the buffer identification list is set to be empty.
2. The method according to claim 1, It is characterized in that When the set working mode is the continuous buffer mode, each bit in the buffer identifier list corresponds to a corresponding buffer identifier; When the set working mode is the discrete buffer mode, r bits in the buffer identifier list correspond to corresponding buffer identifiers; r>1.
3. The method according to claim 2, It is characterized in that 2^r is not less than the number of buffer identifiers in the buffer identifier list.
4. The method according to claim 2, It is characterized in that The configuring at least one buffer identification list according to the set working mode also includes: When the set working mode is the continuous buffer mode, setting the depth of the buffer identification list to a first depth; When the set working mode is the discrete buffer mode, the depth of the buffer identification list is set to a second depth.
5. The method according to claim 4, It is characterized in that The buffer identification list is provided with a head pointer and a tail pointer; The configuring at least one buffer identification list according to the set working mode also includes: When the set working mode is the continuous buffer mode, the head pointer of the buffer identification list is pointed to the tail of the buffer identification list, and the tail pointer of the buffer identification list is pointed to the head of the buffer identification list; When the set working mode is the discrete buffer mode, the head pointer and the tail pointer of the buffer identification list are pointed to the same position.
6. The method according to claim 4, It is characterized in that The configuring at least one buffer identification list according to the set working mode also includes: When the set working mode is the continuous buffer mode, checking whether the highest bit of the buffer identification list is a preset value; When the highest bit of the buffer identification list is a preset value, it is determined that the configuration of the buffer identification list is completed.
7. The method according to claim 4, It is characterized in that The performing continuous read and write operations according to the plurality of continuous buffer spaces comprises: determining a plurality of continuous buffer identifiers in the buffer identifier list corresponding to the plurality of buffer spaces currently required, and then releasing the plurality of continuous buffer identifiers in the buffer identifier list; The performing discrete read and write operations according to the single buffer space includes: After the flash memory controller or the processor releases the target buffer identifier corresponding to the single buffer space, the target buffer identifier is written into the buffer identifier list.
8. The method according to claim 7, It is characterized in that The buffer identification list is provided with a head pointer and a tail pointer; The method further comprises: After releasing the plurality of consecutive buffer identifiers in the buffer identifier list, triggering the tail pointer of the buffer identifier list to be incremented by one; After the target buffer identifier is written into the buffer identifier list, the head pointer of the buffer identifier list is triggered to increase by one.
9. The method according to claim 4, It is characterized in that Also includes: In response to a mode switching instruction, switch from the current first mode to another second mode; wherein the first mode is a continuous buffer mode and the second mode is a discrete buffer mode; or, the first mode is a discrete buffer mode and the second mode is a continuous buffer mode.
10. The method according to claim 9, It is characterized in that The switching from the current first mode to another second mode includes: When the first mode is the continuous buffer mode, releasing all buffer identifiers in the buffer identifier list, and then switching to the discrete buffer mode; When the first mode is the continuous buffering mode, all released buffer identifiers are added to the buffer identifier list, and then the continuous buffering mode is switched.
11. The method according to claim 10, It is characterized in that The mode switching instruction is generated based on the following method: In the case where the first mode is the continuous buffer mode, when the current amount of junk data exceeds a first preset threshold, generating a mode switching instruction for switching to the discrete buffer mode; In the case where the first mode is the discrete buffer mode, when the current amount of junk data is less than a second preset threshold, a mode switching instruction for switching to the continuous buffer mode is generated.
12. The method according to claim 2, It is characterized in that The buffer identification list is configured to be used exclusively by one processor core; or, the buffer identification list is configured to be shared by multiple processor cores.
13. The method according to claim 2, It is characterized in that The buffer identification list is stored in a corresponding buffer register, and the bit width of the buffer register is consistent with the bit width of the buffer identification list.
14. The method according to claim 1, It is characterized in that Also includes: When the buffer manager is in discrete buffer mode, collecting target data pages in the single buffer space; Determine a data classification of the target data page; the data classification includes a cold data page and a hot data page; When the target data page is a cold data page, aggregating the cold data page into a cold data block; When the target data page is a hot data page, the hot data page is aggregated into a hot data block.
15. The method according to claim 14, It is characterized in that The determining the data classification of the target data page includes: Extracting features of the target data page to determine feature information of the target data page; the feature information includes: at least one of attribute information, time information, a logical block address, and a physical block address of the target data page; The characteristic information of the target data page is input into a pre-trained classification prediction model, and the data classification of the target data page is determined according to the classification prediction result of the classification prediction model.
16. The method according to claim 15, It is characterized in that The classification prediction model is a decision tree model, and the t-th decision tree is generated based on the t-1-th decision tree; ,in, Represents the t-th decision tree for the sample The predicted output is Represents the minimum square loss of the residual corresponding to the jth sample, where J is the number of samples; The step of inputting the characteristic information of the target data page into a pre-trained classification prediction model, and determining the data classification of the target data page according to the classification prediction result of the classification prediction model, comprises: Preprocessing the characteristic information of the target data page to generate preprocessing information of the target data page; the preprocessing includes one or more of missing value processing, outlier processing, feature engineering processing, categorical feature processing, and standardization processing format conversion processing; Inputting the preprocessed information into a pre-trained decision tree model to determine a predicted output of at least a portion of the decision tree; The prediction outputs of at least part of the decision tree are weighted, and the data classification of the target data page is determined according to the weighted processing result.
17. A computer device, It is characterized in that include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the buffer management method according to any one of claims 1 to 16 by executing the computer instructions.
18. A solid state hard disk, It is characterized in that include: Flash translation layer and flash chips; The flash translation layer includes a processor, a flash controller and a buffer manager; wherein the buffer manager is used to execute the buffer management method according to any one of claims 1 to 16.
19. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the buffer management method according to any one of claims 1 to 16.
20. A computer program product, It is characterized in that The method comprises computer instructions for causing a computer to execute the buffer management method according to any one of claims 1 to 16.
