File writing method and device, electronic equipment, computer program and storage medium

Abstract

The invention relates to a file writing method and device, electronic equipment, a computer program and a storage medium. The writing method comprises the steps of obtaining a plurality of target files to be written; for each target file in the plurality of target files, determining a characteristic value, associated with the file failure time, of each target file; based on the characteristic value of each target file, distributing a recovery circuit processor for each target file, so that the difference of failure time of each file contained in a recovery circuit pointed by the recovery circuit processor is smaller than a preset threshold value, each recovery circuit processor is one of a plurality of preset recovery circuit processors; and writing each target file into the recovery circuit pointed by the recovery circuit processor allocated for each target file.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a method, device, electronic device, computer program, and storage medium for writing a file. Background Art

[0002] Flexible Data Placement (FDP) is a newly approved Non-Volatile Memory Express (NVMe) specification. FDP Solid State Drive (SSD) is configured with several Reclaim Units (RUs), and multiple RUs can be organized into a Reclaim Group (RG). In addition, each RG can have one or more Reclaim Unit Handles (RUHs), and each RUH can point to a RU in the RG.

[0003] An important factor affecting the life and performance of SSD is write amplification (WA), which can also be called WAF. The main reason for write amplification is the fragmentation of data storage. "Fragmentation" can refer to the fact that various files with different degrees of hotness and coldness (such as the frequency of file access) are contained in the RU at the same time. At this time, when doing garbage collection (GC), if there is still valid data in the current RU, the valid data needs to be moved to the new RU before the invalid data in the RU can be erased. Moving valid data to a new RU will cause write amplification. For example, assuming that the write amplification factor is 5, for every 4KB of data written by the host, 20KB of data will eventually be written to the RU. Therefore, due to the fragmentation of data storage, the write overhead will increase, resulting in low memory usage efficiency. Summary of the invention

[0004] The present disclosure provides a file writing method, device, electronic device, computer program and storage medium to at least solve the problem in the above-mentioned related technologies that more writing overhead is generated due to the fragmentation of data storage.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for writing a file is provided, comprising: acquiring a plurality of target files to be written; for each target file among the plurality of target files, determining a characteristic value associated with a file expiration time of the each target file; based on the characteristic value of the each target file, allocating a recycling circuit processor to the each target file, so that a difference in expiration time of each file contained in a recycling circuit pointed to by the recycling circuit processor is less than a preset threshold, and each recycling circuit processor is one of a plurality of preset recycling circuit processors; and writing each target file into the recycling circuit pointed to by the recycling circuit processor allocated to the each target file.

[0006] Optionally, before allocating a recovery circuit processor to each target file based on the characteristic value of each target file, it also includes: determining the file type of each target file among the multiple target files, wherein the file type is one of a short-time open file type and a long-time open file type; wherein, allocating a recovery circuit processor to each target file based on the characteristic value of each target file includes: allocating a recovery circuit processor to each target file based on the characteristic value and file type of each target file.

[0007] Optionally, determining the file type of each target file among the multiple target files includes: setting the file type of each target file to the short-time open file type as a default file type; detecting whether each target file is in an open state at preset time intervals; and based on detecting that one or more target files are in the open state, converting the file type of the one or more target files from the short-time open file type to the long-time open file type.

[0008] Optionally, the allocating a recycling circuit processor to each target file based on the characteristic value and file type of each target file includes: based on the case where it is determined that the file type of the target file is the short-time open file type, inputting the characteristic value of each target file of the short-time open file type into a random forest model to obtain a predicted life cycle of each target file; calculating the expiration time of each target file based on the creation time of each target file and the predicted life cycle; and allocating a permanent isolation type recycling circuit processor to each target file based on the expiration time of each target file and the expiration time of the currently existing file in the recycling circuit pointed to by each recycling circuit processor among the preset multiple recycling circuit processors, wherein the permanent isolation type recycling circuit processor is configured to move the file from a first recycling circuit pointed to by the permanent isolation type recycling circuit processor to a second recycling circuit pointed to by the permanent isolation type recycling circuit processor.

[0009] Optionally, the method of allocating a permanent isolation type recovery circuit processor to each target file based on the expiration time of each target file and the expiration time of the currently existing file in the recovery circuit pointed to by each recovery circuit processor among the preset multiple recovery circuit processors includes: calculating, for each permanent isolation type recovery circuit processor among a first number of permanent isolation type recovery circuit processors included in the preset multiple recovery circuit processors, the mean square error between the expiration time of each target file and the expiration time of the currently existing file in the recovery circuit pointed to by each permanent isolation type recovery circuit processor; determining, among the first number of permanent isolation type recovery circuit processors, for each target file, a corresponding permanent isolation type recovery circuit processor with the smallest mean square error; and allocating the corresponding permanent isolation type recovery circuit processor with the smallest mean square error to each target file.

[0010] Optionally, based on determining that the file type of the target file is the short-time open file type, the characteristic value of the target file includes at least one of the following items: the creation time of the target file, the size of the target file, the number of files whose opening time is close to the opening time of the target file under each preset recycling circuit processor of the preset multiple recycling circuit processors, and the average life cycle of files whose file type is the same as the file type of the target file and have been deleted under each preset recycling circuit processor.

[0011] Optionally, based on determining that the file type of each target file is the long-open file type, the characteristic value is a unit size modification USM, and the USM is a ratio of the number of times the target file is modified to the size of the target file, and based on the characteristic value and file type of each target file, a recovery circuit processor is assigned to each target file, including: clustering the multiple USMs corresponding to the multiple target files using a K-means algorithm to divide the multiple target files into multiple groups; and assigning a recovery circuit processor from the preset multiple recovery circuit processors to each of the multiple groups.

[0012] Optionally, the clustering of the multiple USMs corresponding to the multiple target files using the K-means algorithm to divide the multiple target files into multiple groups includes: determining a second number of recovery circuit processors of the initial isolation type among the preset multiple recovery circuit processors, wherein the recovery circuit processor of the initial isolation type is configured to move the file from a third recovery circuit pointed to by the recovery circuit processor of the initial isolation type to a fourth recovery circuit pointed to by a recovery circuit processor of the same type as the initial isolation type; setting K of the K-means algorithm to the second number; clustering the multiple USMs corresponding to the multiple target files using the K-means algorithm with K set to the second number to divide the multiple target files into the K groups; and assigning a recovery circuit processor from the preset multiple recovery circuit processors to each of the multiple groups includes: for each of the multiple groups, selecting a recovery circuit processor of the initial isolation type from the second number of recovery circuit processors of the initial isolation type and assigning it to each group.

[0013] Optionally, the total number of the preset multiple recovery circuit processors is N; before obtaining the multiple target files to be written, it also includes: in response to the user's setting input, setting the number of recovery circuit processors whose processor type is a permanent isolation type to d, and setting the number of recovery circuit processors whose processor type is an initial isolation type to (Nd).

[0014] Optionally, the writing method further includes: detecting a first proportion of target files of a file type opened for a short time among the multiple target files at preset time intervals, or detecting a second proportion of target files of a file type opened for a long time among the multiple target files; dynamically adjusting the quantity d based on the first proportion or the second proportion; wherein the quantity d is proportional to the first proportion; and the quantity d is inversely proportional to the second proportion.

[0015] According to a second aspect of an embodiment of the present disclosure, a file writing device is provided, comprising: a file acquisition module, configured to acquire multiple target files to be written; a characteristic value determination module, configured to determine, for each target file among the multiple target files, a characteristic value associated with a file expiration time of the each target file; an allocation module, configured to allocate a recycling circuit processor to each target file based on the characteristic value of the each target file, so that the difference in expiration time of each file contained in the recycling circuit pointed to by the recycling circuit processor is less than a preset threshold, and each of the recycling circuit processors is one of a preset plurality of recycling circuit processors; and a writing module, configured to write each target file into the recycling circuit pointed to by the recycling circuit processor allocated to each target file.

[0016] Optionally, the writing device also includes: a file type determination module, configured to determine the file type of each target file among the multiple target files, wherein the file type is one of a short-time open file type and a long-time open file type; wherein the allocation module is configured to: allocate a recovery circuit processor to each target file based on the characteristic value and file type of each target file.

[0017] Optionally, the file type determination module is configured to: set the file type of each target file to the short-time open file type as a default file type; detect whether each target file is in an open state at preset time intervals; and based on detecting that one or more target files are in the open state, convert the file type of the one or more target files from the short-time open file type to the long-time open file type.

[0018] Optionally, the allocation module is configured to: based on determining that the file type of the target file is the short-time open file type, input the characteristic value of each target file of the short-time open file type into the random forest model to obtain the predicted life cycle of each target file; based on the creation time of each target file and the predicted life cycle, calculate the expiration time of each target file; based on the expiration time of each target file and the expiration time of the currently existing file in the recovery circuit pointed to by each recovery circuit processor in the preset multiple recovery circuit processors, allocate a permanent isolation type recovery circuit processor to each target file, wherein the permanent isolation type recovery circuit processor is configured to move the file from the first recovery circuit pointed to by the permanent isolation type recovery circuit processor to the second recovery circuit pointed to by the permanent isolation type recovery circuit processor.

[0019] Optionally, the allocation module is configured to: calculate, for each permanent isolation type recovery circuit processor among the first number of permanent isolation type recovery circuit processors included in the preset multiple recovery circuit processors, the mean square error between the expiration time of each target file and the expiration time of the currently existing file in the recovery circuit pointed to by each permanent isolation type recovery circuit processor; determine, among the first number of permanent isolation type recovery circuit processors, for each target file, the corresponding permanent isolation type recovery circuit processor with the smallest mean square error; and allocate the corresponding permanent isolation type recovery circuit processor with the smallest mean square error to each target file.

[0020] Optionally, based on determining that the file type of the target file is the short-time open file type, the characteristic value of the target file includes at least one of the following items: the creation time of the target file, the size of the target file, the number of files whose opening time is close to the opening time of the target file under each preset recycling circuit processor of the preset multiple recycling circuit processors, and the average life cycle of files whose file type is the same as the file type of the target file and have been deleted under each preset recycling circuit processor.

[0021] Optionally, based on determining that the file type of the target file is the long-open file type, the characteristic value is a unit size modification USM, and the USM is a ratio of the number of times the target file has been modified to the size of the target file, the allocation module is configured to: cluster the multiple USMs corresponding to the multiple target files using a K-means algorithm to divide the multiple target files into multiple groups; and allocate a recovery circuit processor from the preset multiple recovery circuit processors to each of the multiple groups.

[0022] Optionally, the allocation module is configured to: determine a second number of recovery circuit processors of the initial isolation type among the preset multiple recovery circuit processors, wherein the recovery circuit processor of the initial isolation type is configured to move files from a third recovery circuit pointed to by the recovery circuit processor of the initial isolation type to a fourth recovery circuit pointed to by a recovery circuit processor of the same type as the initial isolation type; set K of the K-means algorithm to the second number; cluster the multiple USMs corresponding to the multiple target files using the K-means algorithm with K set to the second number to divide the multiple target files into the K groups; for each of the multiple groups, select a recovery circuit processor of the initial isolation type from the second number of recovery circuit processors of the initial isolation type and assign it to each group.

[0023] Optionally, the total number of the preset multiple recovery circuit processors is N; the writing device also includes: a setting module, configured to respond to the user's setting input, set the number of recovery circuit processors whose processor type is a permanent isolation type to d, and set the number of recovery circuit processors whose processor type is an initial isolation type to (Nd).

[0024] Optionally, the writing device further includes: a ratio detection module, configured to detect a first ratio of target files of a file type that is opened for a short time in the multiple target files at a preset time interval, or to detect a second ratio of target files of a file type that is opened for a long time in the multiple target files; a dynamic adjustment module, configured to dynamically adjust the quantity d based on the first ratio or the second ratio; wherein the quantity d is proportional to the first ratio; and the quantity d is inversely proportional to the second ratio.

[0025] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a file writing method according to the present disclosure.

[0026] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute a file writing method according to the present disclosure.

[0027] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the file writing method according to the present disclosure is implemented.

[0028] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0029] In the present disclosure, since the difference in the expiration time of each file contained in the recycling circuit (recycling unit) pointed to by the same recycling circuit processor (recycling unit handle) can be made smaller than the preset threshold, that is, since the expiration time of each file written into the same recycling unit RU can be made roughly consistent as much as possible, therefore, when performing garbage collection, it is possible to achieve synchronous erasure of each file contained in the same recycling unit RU, avoiding the write amplification phenomenon caused by the simultaneous existence of invalid data and valid data in the same recycling unit RU. It can be seen that the file writing method provided by the present disclosure can reduce the fragmentation of the files contained in the recycling unit RU, and thus reduce the write overhead.

[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0032] Figure 1 is a schematic diagram showing a recycling group;

[0033] Figure 2 is a schematic diagram showing data migration by a PI type RUH and an II type RUH;

[0034] Figure 3 is a flowchart illustrating a method of writing a file according to an exemplary embodiment of the present disclosure;

[0035] Figure 4 is a diagram showing a distribution of error values ​​between a file life cycle predicted by a random forest model and a file real life cycle according to an exemplary embodiment of the present disclosure;

[0036] Figure 5 is a schematic diagram showing different degrees of hot and cold mixing of data caused by allocating different types of RUH according to an exemplary embodiment of the present disclosure;

[0037] Figure 6 is a schematic diagram showing various characteristic values ​​according to an exemplary embodiment of the present disclosure;

[0038] Figure 7 is a schematic diagram showing writing a target file E into RUs pointed to by different RUHs according to an exemplary embodiment of the present disclosure;

[0039] Figure 8 is a schematic diagram illustrating a compaction process of a RocksDB file according to an exemplary embodiment of the present disclosure;

[0040] Fig. 9 is a schematic diagram showing the life cycle of files at different layers according to an exemplary embodiment of the present disclosure;

[0041] Fig.10 is a schematic diagram showing the number of writes of a MySQL type file and a PostgreSQL type file according to an exemplary embodiment of the present disclosure;

[0042] Fig.11 is a schematic diagram showing different space utilizations caused by allocating different types of RUHs according to an exemplary embodiment of the present disclosure;

[0043] Fig.12 is a flowchart showing a specific implementation of a method for writing a file according to an exemplary embodiment of the present disclosure;

[0044] Fig.13 is a block diagram illustrating a file writing apparatus according to an exemplary embodiment of the present disclosure;

[0045] Fig.14 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure;

[0046] Fig.15 is a block diagram illustrating an example computer system according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0049] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0050] In one or more examples, the FDP solid state drive is configured with a plurality of recycling units (RUs), and a plurality of RUs may be organized into a recycling group (RG). Each RG may have one or more recycling unit handles (RUHs), and each RUH may point to a RU in the RG. In one or more examples, the RU may be referred to as a recycling circuit, and the RUH may be referred to as a recycling circuit processor. Figure 1 is a schematic diagram showing a recovery group. Figure 1, 4 recovery unit handles (RUH) are shown, namely RUH0, RUH1, RUH2 and RUH3. And 5 recovery units (RU) are also shown, namely RU0, RU1, RU2, RU3 and RU4. Among them, RUH0 can point to RU0, RUH1 can point to RU1, RUH2 can point to RU3, and RUH3 can point to RU4.

[0051] The maximum number of recovery unit handles (RUH) is 2 16 =65536, the host can use RUH to place data in RU. And when writing data to the pointed RU, if the current RU is not full, the data can be written to the RU; if the current RU is full, the controller can assign RUH to another empty RU; if the current RU write exceeds the capacity, the current RU can be filled first, and RUH can be assigned to another empty RU, and then the remaining data can be written to the new RU.

[0052] Furthermore, the recovery unit handle (RUH) can be divided into two types, namely, the Persistently Isolated (PI) type and the Initially Isolated (II) type. Among them, the PI type RUH only allows data to be moved (GC) to the RU pointed to by the same RUH. This move method can ensure a small write amplification, but the space utilization of the RU is low; the II type RUH allows data to be moved (GC) to the RU pointed to by the same type of RUH. At this time, the space utilization of the RU is higher, but the write amplification is also higher.

[0053] Figure 2 Schematic diagram showing data migration by RUH of type PI and RUH of type II. Figure 2 , shows a PI type RUH#0, which can point to RU1, RU2, RU3, and RU4. Also, three II type RUHs are shown, namely RUH#1, RUH#2, and RUH#3. Among them, RUH#1 can point to RU6, and RUH#2 can point to RU5.

[0054] When moving data in RU1, RU2, and RU3, the data needs to be moved to RU4 pointed to by the same RUH#0. When moving data in RU5 and RU6, the data can be moved to RU7 pointed to by the same type of RUH, such as type II. It can be seen that since type PI RUH only allows data to be moved (GC) to the RU pointed to by the same RUH, the degree of mixing of data with different hot and cold degrees in the same RU is low; since type II RUH allows data to be moved (GC) to the RU pointed to by the same type of RUH, the degree of mixing of data with different hot and cold degrees in the same RU is high.

[0055] The main reason for write amplification is the fragmentation of data storage. In one or more examples, "fragmentation" may refer to the fact that the RU contains various files with different degrees of hotness and coldness at the same time. At this time, when doing garbage collection (GC), if there is still valid data in the current RU, the valid data needs to be moved to the new RU first, and then the invalid data in the RU can be erased. Moving valid data to a new RU will adversely cause write amplification. It can be seen that due to the fragmentation of data storage, more write overhead will be generated.

[0056] In order to solve the above problems, the file writing method, device, electronic device, computer program and storage medium provided by the present disclosure can make the difference in the expiration time of each file contained in the recycling unit pointed to by the same recycling unit handle less than the preset threshold, that is, since the expiration time of each file written into the same recycling unit RU can be made roughly consistent as much as possible, therefore, when performing garbage collection, it is possible to achieve synchronous erasure of each file contained in the same recycling unit RU, avoiding the write amplification phenomenon caused by the simultaneous existence of invalid data and valid data in the same recycling unit RU. It can be seen that the file writing method provided by the present disclosure can reduce the fragmentation of the files contained in the recycling unit RU, and thus reduce the write overhead.

[0057] Figure 3 is a flowchart illustrating a method of writing a file according to an exemplary embodiment of the present disclosure.

[0058] Reference Figure 3 In step 301, multiple target files to be written may be obtained, wherein the target files may be files generated during the use of an application (Application, App). The file writing method provided by the present disclosure may be completed by four modules, which may be respectively: a monitor, a pre-allocator, a re-allocator, and a distributor.

[0059] In one or more examples, the monitor can be used to periodically detect all characteristics of the target file and record its information in a file record table; the pre-allocator can be used to predict the life cycle of the target file of the short-time open file type; the redistributor can be used to divide the target file into different groups based on the USM value of the target file; the distributor can be used to assign specific RUHs to the different divided target file groups. In one or more examples, the monitor module, the pre-allocator module, the redistributor module, and the distributor module can be implemented by separate circuits (e.g., ASICs configured to perform the functions of the respective modules) or one or more processors.

[0060] In step 302, a characteristic value for characterizing the file expiration time of each target file in the plurality of target files may be determined. In one or more examples, the file expiration time is the sum of the file creation time and the file life cycle.

[0061] In one or more examples, the "file life cycle" (e.g., the life cycle of a file) may refer to the length of time that a file goes through from the time it is created until it is deleted. The "life cycle" of a file can be used to characterize the hotness or coldness of a file, wherein the shorter the life cycle of a file, the higher the hotness of the file; the longer the life cycle of a file, the lower the hotness or colder the file. Furthermore, if two files have the same expiration time, it means that the two files will expire at the same time. In one or more examples, the hotness or coldness of data may refer to the frequency with which the data is accessed. For example, the hotness of first data that has a higher access frequency than second data may be higher than that of the second data, and vice versa, the coldness of the second data may be higher than that of the first data.

[0062] In step 303, a recycling unit handle can be allocated to each target file based on the characteristic value of each target file, so that the difference in expiration time of each file contained in the recycling unit pointed to by the recycling unit handle is less than a preset threshold, and each recycling unit handle can be one of the preset multiple recycling unit handles.

[0063] In this way, since the difference in the expiration time of each file contained in the recycling unit pointed to by the same recycling unit handle can be made smaller than the preset threshold, that is, since the expiration time of each file written into the same recycling unit RU can be made roughly consistent as much as possible, it is possible to synchronously erase each file contained in the same recycling unit RU during garbage collection, thus avoiding the write amplification phenomenon caused by the simultaneous existence of invalid data and valid data in the same recycling unit RU. The fragmentation of the files contained in the recycling unit RU is reduced, thereby reducing the write overhead.

[0064] According to an exemplary embodiment of the present disclosure, the total number of preset multiple recycling unit handles may be N. Before acquiring multiple target files to be written, the number of recycling unit handles whose handle type is permanent isolation (PI) type may be set to d in response to a setting input by a user, and the number of recycling unit handles whose handle type is initial isolation (II) type may be set to (Nd). Parameters N and d may be positive integers.

[0065] Exemplarily, assuming that the total number of recycling unit handles is 10, the user can decide the proportion of various types of RUH according to their own needs. For example, the user can set the number of PI type recycling unit handles to 5, and set the number of II type recycling unit handles to 5. For another example, the user can also set the number of PI type recycling unit handles to 6, and set the number of II type recycling unit handles to 4, and so on. In this way, since the user can set the proportion of various types of RUH according to actual conditions, the autonomy and flexibility of setting the proportion of various types of RUH are better.

[0066] According to an exemplary embodiment of the present disclosure, the file type of each target file in a plurality of target files can also be determined, wherein the file type can include a short-time open file type and a long-time open file type. Then, a recovery unit handle can be assigned to each target file based on the characteristic value of each target file and the file type. Exemplarily, the file of the "short-time open file type" can be a file of the Rocks db type, and the file of the "long-time open file type" can be a file of the MySQL type. According to the understanding of those of ordinary skill in the art, the file of the Rocks db type can refer to a file about key-value data stored in an embedded database. According to the understanding of those of ordinary skill in the art, the MySQL type can refer to an internal system file that stores metadata for forming a table definition (column name and other relationship metadata) of MySQL.

[0067] According to an exemplary embodiment of the present disclosure, the file type of each target file may be set to a short-time open file type by default, and whether each target file is in an open state (open) may be detected at a preset time interval T. When it is detected that the target file is in an open state, the file type of the target file may be converted from a short-time open file type to a long-time open file type.

[0068] According to an exemplary embodiment of the present disclosure, when the file type of the target file is a short-time open file type, the characteristic value of each target file of the short-time open file type can be input into the random forest model to obtain the predicted life cycle of each target file. For example, the pre-allocator can use the random forest model to predict the life cycle of each target file based on the characteristic value of the target file of the short-time open file type. Then, the expiration time of each target file can be calculated based on the creation time of each target file and the predicted life cycle. As mentioned above, the expiration time of the target file can be the sum of the creation time of the target file and the predicted life cycle of the target file.

[0069] Next, a permanent isolation type recycling unit handle can be allocated to each target file based on the expiration time of each target file and the expiration time of the currently existing file in the recycling unit pointed to by each recycling unit handle in the preset multiple recycling unit handles. Among them, the permanent isolation type recycling unit handle allows the file to be moved from the first recycling unit pointed to by the permanent isolation type recycling unit handle to the second recycling unit pointed to by the permanent isolation type recycling unit handle. That is, the permanent isolation (PI) type recycling unit handle only allows data to be moved between RUs pointed to by the same RUH.

[0070] It should be noted that as a common and effective machine learning algorithm, the random forest model has low computational complexity, which can reduce additional overhead and ensure the accuracy of prediction. In one or more examples, the random forest model can be a machine learning algorithm that combines the output results of multiple decision trees to obtain a result. For example, the random forest model can be an integrated learning method for classification, regression, and other tasks, which operates by building a large number of decision trees during training. For classification tasks, the output of the random forest model can be the category selected by the most decision trees. For regression tasks, the average prediction value of each decision tree is returned.

[0071] Figure 4 is a diagram showing the error value distribution between the file life cycle predicted by the random forest model and the actual file life cycle according to an exemplary embodiment of the present disclosure. Figure 4 , the number of files whose error between the predicted life cycle and the actual life cycle is within 20ms accounts for 70% of the total number of files; the number of files whose error is within 60ms accounts for 88% of the total number of files. Figure 4 As shown in the figure, the prediction accuracy of the random forest model has been effectively verified.

[0072] Thus, in the present disclosure, the random forest model can be used to accurately predict the life cycle of a file, and the expiration time of each target file can be calculated based on the creation time of each target file and the predicted life cycle, and then the RUH can be managed and allocated based on the expiration time of each target file. This can ensure that the degree of data mixing in the same RU is reduced, thereby reducing GC overhead, thereby reducing write amplification (WAF) and improving the write performance and life of the SSD.

[0073] Furthermore, when training the random forest model, the feature value of the file can be used as a sample input into the model, and the real life cycle of the file can be used as a label. In this way, after the random forest model outputs the predicted life cycle, the value of the loss function can be calculated based on the predicted life cycle and the real life cycle, and the parameters of the model can be adjusted based on the calculated value of the loss function to train the model.

[0074] According to an exemplary embodiment of the present disclosure, for each permanent isolation (PI) type recycling unit handle among a first number of permanent isolation (PI) type recycling unit handles included in a preset plurality of recycling unit handles, the mean square error between the expiration time of each target file and the expiration time of the currently existing file in the recycling unit pointed to by each permanent isolation (PI) type recycling unit handle can be calculated. Then, among the first number of permanent isolation (PI) type recycling unit handles, the corresponding permanent isolation (PI) type recycling unit handle with the smallest mean square error can be determined for each target file. Next, the corresponding permanent isolation (PI) type recycling unit handle with the smallest mean square error can be allocated to each target file.

[0075] For example, when calculating the above mean square error, it is assumed that the expiration time of the target file is t, and there are 3 RUHs of permanent isolation (PI) type, namely RUH1, RUH2 and RUH3. For RUH1, it is assumed that there are 3 existing files in the recycling unit pointed to by RUH1, and the expiration times of these 3 existing files are t1, t2, and t3 respectively. Then the mean square error corresponding to RUH1 can be: (t-t1) 2 +(t-t2) 2 +(t-t3) 2 Similarly, the mean square error corresponding to RUH2 and RUH3 can be calculated.

[0076] In this way, it can be ensured that the expiration times of the various files in the RU pointed to by each permanent isolation (PI) type recovery unit handle are close to each other, thereby reducing the degree of fragmentation of data storage.

[0077] It should be noted that, since the present disclosure allocates RUHs in units of files, and short-term open files are hardly changed, the life cycle of short-term open files is consistent with the life cycle of data contained in the short-term open files. Figure 5 2 is a schematic diagram showing different degrees of hot and cold mixing of data caused by allocating different types of RUHs according to an exemplary embodiment of the present disclosure.

[0078] Reference Figure 5 , assuming that there are currently 6 target files, namely target file A, target file B, target file C, target file D, target file E and target file F. Assume that the heat of target file A and target file B is the same, and the patterns corresponding to target file A and target file B are both "vertical bar" patterns; assume that the heat of target file C and target file D is the same, and the patterns corresponding to target file C and target file D are both "horizontal bar" patterns; assume that the heat of target file E and target file F is the same, and the patterns corresponding to target file E and target file F are both "black dot" patterns.

[0079] Since the target file A and the target file B have the same heat, the target file A and the target file B can be stored in the RU pointed to by the same RUH_0; similarly, the target file C and the target file D can be stored in the RU pointed to by the same RUH_1; the target file E and the target file F can be stored in the RU pointed to by the same RUH_2.

[0080] Furthermore, if the type of the allocated RUH is PI type, when performing data movement (GC), the target file A can only be moved from the current RU to another RU pointed to by the same recovery unit handle (PI_0); similarly, the target file C can only be moved from the current RU to another RU pointed to by the same recovery unit handle (PI_1); and the target file F can only be moved from the current RU to another RU pointed to by the same recovery unit handle (PI_2). In this way, after data erasure, target files A and B with the same or similar temperatures are stored in the same RU, target files C and D with the same or similar temperatures are stored in the same RU, and target files E and F with the same or similar temperatures are stored in the same RU. It can be seen that the feature of PI type RUH "only allowing data to be moved between RUs pointed to by the same RUH" can ensure that the degree of mixing of data of different temperatures is low. Therefore, PI type RUH is more advantageous in reducing write amplification. In one or more examples, if the difference in popularity between two files is within a threshold, the two files may be considered to have similar popularity.

[0081] Alternatively, if the type of the allocated RUH is type II, when performing data movement (GC), the target file A can be moved from the current RU to the RU pointed to by RUH (II_1) of the same type as RUH (II_0); similarly, the target file C can be moved from the current RU to the RU pointed to by RUH (II_2) of the same type as RUH (II_1); and the target file F can be moved from the current RU to the RU pointed to by RUH (II_0) of the same type as RUH (II_2). In this way, after data erasure, target files B and F with different temperatures are stored in the same RU, target files A and D with different temperatures are stored in the same RU, and target files C and E with different temperatures are stored in the same RU. It can be seen that the feature of type II RUH that "allows files to be moved between RUs pointed to by the same type of RUH" may lead to a high degree of mixing of data with different hotness. Therefore, compared with type PI RUH, type II RUH has a disadvantage in reducing write amplification.

[0082] According to an exemplary embodiment of the present disclosure, when the file type of the target file is a short-time open file type, the characteristic value of the target file may include at least one of the following items: the creation time of the target file; the size of the target file; the number of files whose opening time is close to the opening time of the target file under each preset recycling unit handle in a preset plurality of recycling unit handles; and the average life cycle of files whose file type is the same as the file type of the target file and have been deleted under each preset recycling unit handle.

[0083] The monitor can continuously extract information of all files and then update their feature values ​​and put them in the file record table. Due to the different properties of short-time open file types and long-time open file types, the monitor needs to extract corresponding feature values ​​based on the file types. In addition, most of the database files updated in situ are long-time open file types, and most of the appended database files are short-time open file types. In one or more examples, the monitor can periodically extract information of all files or a group of predetermined files.

[0084] Figure 6 is a schematic diagram showing various characteristic values ​​according to an exemplary embodiment of the present disclosure. Figure 6 , showing the characteristic values ​​corresponding to each file type in "short-time open file type" and "long-time open file type". In addition, when predicting the life cycle of the target file of "short-time open file type", the pre-allocator is mainly responsible; when grouping the USM corresponding to the target file of "long-time open file type", the redistributor is mainly responsible.

[0085] Figure 7 2 is a schematic diagram showing the writing of a target file E into RUs pointed to by different RUHs according to an exemplary embodiment of the present disclosure. Figure 7 , file A1 and file A2 can be two different components contained in the same file, so the heat of file A1 and file A2 is the same, and file A1 and file A2 are currently stored in the RU pointed to by RUH_0; file C1 and file C2 can be two different components contained in the same file, so the heat of file C1 and file C2 is the same, and file C1 and file C2 are currently stored in the RU pointed to by RUH_1; file B and file D are currently stored in the RU pointed to by RUH_1.

[0086] The target file E to be written into RU; file B; file D; file A1 and file A2; file C1 and file C2 have different hot and cold degrees. Figure 7 , different patterns can be used to represent files of different degrees of hotness and coldness.

[0087] Furthermore, Figure 7 The figure in the upper middle shows writing the target file E into the RU pointed to by RUH_0; Figure 7 The figure in the lower middle shows that the target file E is written to the RU pointed to by RUH_1. Comparing the two situations, it can be seen that the degree of hot and cold data mixing caused by writing the target file E to the RU pointed to by RUH_0 is low; the degree of hot and cold data mixing caused by writing the target file E to the RU pointed to by RUH_1 is high. It should be noted that the data of the same file may be written to different blocks at the same time, especially when the file is large. Therefore, reducing the degree of data mixing of files written at the same time can achieve better data separation, among which files with the same or similar heat are more likely to be stored together.

[0088] In one or more examples, for the feature value of "average life cycle of deleted files": files of the same type have similar life cycles, and the life cycles of deleted files are known. Therefore, the average life cycle of deleted files of the same type can be used to predict the life cycle of the target file.

[0089] In one or more examples, for the feature value of "size of target file": the file size can reflect the life cycle of the file to some extent. For example, media files are generally larger and colder, while directory files are smaller and hotter.

[0090] In one or more examples, for the feature value of "target file creation time": taking RocksDB file as an example, Figure 8is a schematic diagram showing a compaction process of a RocksDB file according to an exemplary embodiment of the present disclosure. Figure 8 , a total of two levels are shown, namely level(n) and level(n+1). After sst2, sst3, and sst5 are deleted, the newly generated files sst6 and sst7 in a group of compactions are written to the same level(n+1) at the same time. Fig. 9 is a schematic diagram showing the life cycle of files at different layers according to an exemplary embodiment of the present disclosure. Fig. 9 , shows the example lifecycles of files in the four layers L0, L1, L2, and L3. It can be seen that the lifecycles of files in the same layer are relatively similar, and the larger the layer, the longer the lifecycle of the file in that layer is generally. As mentioned earlier, the newly generated files in a group of compactions are written to the same layer at almost the same time. Therefore, the creation time of the target file can be used to reflect the lifecycle of the target file.

[0091] Regarding the characteristic value "USM": the write request for updating the file in place may be random, the data is updated inside the file, and the modification of the file is the modification of the logical block address (LBA). Moreover, more LBA writes often mean that the life cycle of the data written to the LBA is shorter. Fig.10 2 is a schematic diagram showing the number of writes to a MySQL type file and a PostgreSQL type file according to an exemplary embodiment of the present disclosure. Fig.10 ,The figure shows that USM is strongly correlated with the ,update frequency of data, therefore, USM can be used to reflect the life cycle of ,files, and the larger the USM, the shorter the life cycle of the ,file.

[0092] It should be noted that the "characteristic value" of the target file in the present disclosure is not limited to the contents listed above, but can also be other contents that can be used to characterize the file life cycle, which will not be repeated here. The above implementation is only an exemplary description.

[0093] According to an exemplary embodiment of the present disclosure, when the file type of the target file is a long-time open file type, the above-mentioned characteristic value may be a unit size modification (USM), and the USM may be a ratio of the number of times the target file is modified to the size of the target file. The multiple USMs corresponding to the multiple target files may be clustered using a K-means algorithm to divide the multiple target files into multiple groups. Then, a recycling unit handle may be allocated to each of the multiple groups.

[0094] It should be noted that the classification algorithm may be other types of classification algorithms besides the K-means algorithm, and the present disclosure does not impose any specific limitation on this, and the aforementioned implementation is merely an exemplary description.

[0095] According to an exemplary embodiment of the present disclosure, a second number of recycling unit handles of the initial isolation (II) type among a plurality of preset recycling unit handles can be determined. The recycling unit handle of the initial isolation (II) type allows a file to be moved from a third recycling unit pointed to by the recycling unit handle of the initial isolation (II) type to a fourth recycling unit pointed to by a recycling unit handle of the same type as the initial isolation (II) type. That is, an RUH of the initial isolation (II) type allows data to be moved between RUs pointed to by RUHs of the same type.

[0096] For the second number of recovery unit handles of type (II), K of the K-means algorithm can be set to the second number. Next, the multiple USMs corresponding to the multiple target files can be clustered using the K-means algorithm with K set to the second number to divide the multiple target files into a second number of groups. Then, for each of the multiple groups, an initial isolation (II) type recovery unit handle can be selected from the second number of initial isolation (II) type recovery unit handles and assigned to each group.

[0097] Exemplarily, as mentioned above, the user can set the number of recovery unit handles of the initial isolation (II) type to (Nd). Assuming that the numbers of these (Nd) II type RUHs are d+1 to N, the distributor can assign RUHs with IDs from d+1 to N and handle types of II type to the above second number of groups. Furthermore, a RUH with a larger ID can be assigned to a group with a larger USM value, and a RUH with a smaller ID can be assigned to a group with a smaller USM value.

[0098] Fig.11 FIG. 2 is a schematic diagram showing different space utilizations caused by allocating different types of RUHs according to an exemplary embodiment of the present disclosure. Fig.11, assuming that there are currently 5 target files, namely target file A1, target file A2, target file A3, target file B1, and target file B2. Moreover, target file A1, target file A2, and target file A3 can be different components of the same file, so the hotness and coldness of target file A1, target file A2, and target file A3 are the same or similar, and these three target files can all be represented by a "vertical bar" pattern. Similarly, target file B1 and target file B2 can be different components of the same file, so the hotness and coldness of target file B1 and target file B2 are the same, and these two target files can all be represented by a "horizontal bar" pattern. Target files A1, target file A2, and target file A3 can be stored in the RU pointed to by RUH_0, and target files B1 and target file B2 can be stored in the RU pointed to by RUH_1.

[0099] Furthermore, if the type of the allocated RUH is PI type, when performing data migration (GC), the target file B1 and the target file B2 can only be moved from the current RU to another RU pointed to by the same recycling unit handle (PI_1). In this way, after data erasure, the target file B1 and the target file B2 with the same heat are stored in the same RU, and the number of free RUs that can be recycled is currently two.

[0100] Alternatively, if the type of the allocated RUH is type II, when performing data migration (GC), the target file A1 can be moved from the current RU to the RU pointed to by the RUH (II_1) of the same type as the RU (II_0); the target files B1 and B2 can also be moved from the current RU to another RU pointed to by the same recycling unit handle (II_1). In this way, after data erasure, the target files A1, B1, and B2 with different heats are stored in the same RU, and the number of free RUs that can be recycled is three.

[0101] It can be seen that the characteristic of Type II RUH, which allows files to be moved between RUs pointed to by the same type of RUH, may lead to a high degree of mixing of data of different hotness. Therefore, compared with Type PI RUH, Type II RUH has a disadvantage in reducing write amplification. However, it is precisely because of the characteristic of Type II RUH, which allows files to be moved between RUs pointed to by the same type of RUH, that data can be moved in a timely manner, that is, data can be moved in a timely manner, and RU can be released in a timely manner. It can be seen that compared with Type PI RUH, since Type II RUH has more RUs available for copying when moving data, that is, more RUs can be selected, Type II RUH can accelerate the release of space, that is, it can accelerate the release of RU resources, and thus reduce the waste of space resources.

[0102] Thus, in the present disclosure, when writing a file, the characteristics of the file itself and the type of RUH can be comprehensively considered, and a better balance can be achieved between reducing write amplification and improving space utilization.

[0103] It should be noted that, when grouping target files, in addition to the above grouping method, other grouping methods may also be used. The present disclosure does not limit the specific method of grouping multiple target files, and the above implementation is only an exemplary description.

[0104] Reference, Figure 3 In step 304, each target file may be written into the recycling unit pointed to by the recycling unit handle allocated for each target file.

[0105] According to an exemplary embodiment of the present disclosure, a first proportion of target files of a file type opened for a short time in a plurality of target files may be detected at every preset time interval T, and / or a second proportion of target files of a file type opened for a long time in a plurality of target files may be detected. Then, the number d may be dynamically adjusted based on the first proportion and / or the second proportion, wherein the smaller the first proportion, the smaller the number d may be; and the larger the second proportion, the smaller the number d may be.

[0106] It should be noted that the total number of preset multiple recovery unit handles is generally fixed, but the load changes dynamically, that is, the proportion of target files of the file type that are opened for a short time and the proportion of target files of the file type that are opened for a long time may change dynamically over time. Therefore, the redistributor can detect the proportion of each type of file every T seconds, and then dynamically adjust d according to the proportion of various types of files, that is, dynamically adjust the number of PI type RUHs and the number of II type RUHs. In this way, by flexibly adjusting the number of each type of RUH based on the proportion of the number of various types of files, it can be ensured that various types of RUHs can meet the actual file writing needs, avoiding the phenomenon that a certain type of RUH is in short supply while another type of RUH is in excess, and ensuring the rational use of RUH resources.

[0107] Fig.12 FIG. 4 is a flowchart showing a specific implementation of a method for writing a file according to an exemplary embodiment of the present disclosure.

[0108] In step 1201, a plurality of target files to be written are obtained, wherein the target files may be files generated during the use of an application.

[0109] In step 1202, the monitor periodically monitors all target files.

[0110] In step 1203, the monitor records the characteristic values ​​of various types of target files in the file record table.

[0111] In one or more examples, for a target file of a file type that is opened for a short time, its characteristic value may include at least one of the following items: the creation time of the target file, the size of the target file, the number of files whose opening time is close to the opening time of the target file under each preset recycling unit handle in a preset plurality of recycling unit handles, and the average life cycle of files whose file type is the same as the target file and have been deleted under each preset recycling unit handle.

[0112] For a target file of a file type that is opened for a long time, the characteristic value may be a unit size modification USM, and the USM may be a ratio of the number of times the target file is modified to the size of the target file.

[0113] In step 1204, the pre-allocator predicts the life cycles of multiple target files based on the feature values ​​of the target files by using a random forest model, and calculates the expiration time of each target file based on the creation time of each target file and the predicted life cycle.

[0114] In step 1205, a permanent isolation (PI) type recycling unit handle is allocated to each target file based on the expiration time of each target file and the expiration time of the currently existing file in the recycling unit pointed to by each recycling unit handle in the preset multiple recycling unit handles.

[0115] In step 1206, the redistributor groups the multiple USMs corresponding to the multiple target files by using the K-means algorithm, so that the differences between the USMs of the target files in each group are relatively small, for example, the expiration time and the invalidation time of the target files in each group are relatively close.

[0116] In step 1207, for each USM group in the plurality of USM groups, an initial isolation (II) type recycling unit handle is allocated to each USM group.

[0117] In step 1208, the target file is written into the RU pointed to by the RUH allocated thereto.

[0118] In the present disclosure, the FDP is mainly adapted and optimized in the form of software, which can ensure the improvement of the overall service quality and reduce the cost. In addition, when writing files, the characteristics of the file itself and the type of RUH can be comprehensively considered to achieve a better balance between reducing write amplification and improving space utilization.

[0119] Fig.13 is a block diagram illustrating a file writing apparatus according to an exemplary embodiment of the present disclosure.

[0120] Reference Fig.13 The file writing device 1300 may include a file acquisition module 1301, a feature value determination module 1302, an allocation module 1303 and a writing module 1304. Each module may be implemented by a separate circuit or one or more processors.

[0121] The file acquisition module 1301 can acquire multiple target files to be written, wherein the target files can be files generated during the use of an application (App).

[0122] The feature value determination module 1302 may determine a feature value for characterizing the file expiration time of each target file among the plurality of target files, wherein the file expiration time is the sum of the file creation time and the file life cycle.

[0123] It should be noted that "file life cycle" can refer to the length of time from the creation of a file to its deletion. The "life cycle" of a file can be used to characterize the popularity of a file, where the shorter the life cycle of a file, the higher the popularity of the file; the longer the life cycle of a file, the lower the popularity of the file, that is, the colder it is. And if the expiration time of two files is the same, it means that the two files will expire at the same time.

[0124] The allocation module 1303 can allocate a recycling unit handle to each target file based on the characteristic value of each target file, so that the difference in expiration time of each file contained in the recycling unit pointed to by the recycling unit handle is less than a preset threshold, and each recycling unit handle can be one of the preset multiple recycling unit handles.

[0125] In this way, since the difference in the expiration time of each file contained in the recycling unit pointed to by the same recycling unit handle can be made smaller than the preset threshold, that is, since the expiration time of each file written into the same recycling unit RU can be made roughly consistent as much as possible, it is possible to synchronously erase each file contained in the same recycling unit RU during garbage collection, thus avoiding the write amplification phenomenon caused by the simultaneous existence of invalid data and valid data in the same recycling unit RU. The fragmentation of the files contained in the recycling unit RU is reduced, thereby reducing the write overhead.

[0126] According to an exemplary embodiment of the present disclosure, the file writing device 1300 may further include a setting module.

[0127] The total number of preset multiple recycling unit handles may be N. Before acquiring multiple target files to be written, the setting module may also, in response to a setting input from a user, set the number of recycling unit handles whose handle type is a permanent isolation (PI) type to d, and may set the number of recycling unit handles whose handle type is an initial isolation (II) type to (Nd).

[0128] For example, assuming that the total number of recycling unit handles is 10, the user can determine the proportion of various types of RUHs according to their own needs. For example, the user can set the number of PI type recycling unit handles to 5, and the number of II type recycling unit handles to 5; or, the user can also set the number of PI type recycling unit handles to 6, and the number of II type recycling unit handles to 4, and so on. In this way, since the user can set the proportion of various types of RUHs according to actual conditions, the autonomy and flexibility of setting the proportion of various types of RUHs are better.

[0129] According to an exemplary embodiment of the present disclosure, the file writing device 1300 may further include a file type determination module.

[0130] The file type determination module can also determine the file type of each target file in multiple target files, wherein the file type can include a short-time open file type and a long-time open file type. Then, the allocation module 1303 can allocate a recovery unit handle for each target file based on the characteristic value and file type of each target file. Exemplary, the file of "short-time open file type" can be a file of Rocks db type, and the file of "long-time open file type" can be a file of MySQL type.

[0131] According to an exemplary embodiment of the present disclosure, the above-mentioned file type determination module may default the file type of each target file to a short-time open file type, and may also detect whether each target file is in an open state (open) at a preset time interval T. In the case where it is detected that the target file is in an open state, the file type of the target file may be converted from a short-time open file type to a long-time open file type.

[0132] According to an exemplary embodiment of the present disclosure, when the file type of the target file is a short-time open file type, the allocation module 1303 can input the characteristic value of the target file into the random forest model to obtain the predicted life cycle of the target file. That is, the pre-allocator can use the random forest model to predict the life cycle of the target file based on the characteristic value of the target file of the short-time open file type.

[0133] Then, the allocation module 1303 can calculate the expiration time of each target file based on the creation time of each target file and the predicted life cycle. As mentioned above, the expiration time of the target file can be the sum of the creation time of the target file and the predicted life cycle of the target file.

[0134] Next, the allocation module 1303 can allocate a permanent isolation type recovery unit handle to each target file based on the expiration time of each target file and the expiration time of the currently existing file in the recovery unit pointed to by each recovery unit handle in the preset multiple recovery unit handles. Among them, the permanent isolation type recovery unit handle allows the file to be moved from the first recovery unit pointed to by the permanent isolation type recovery unit handle to the second recovery unit pointed to by the permanent isolation type recovery unit handle. That is, the permanent isolation (PI) type recovery unit handle only allows data to be moved between RUs pointed to by the same RUH.

[0135] It should be noted that as a common and effective machine learning algorithm, the random forest model has low computational complexity, which can reduce additional overhead and ensure prediction accuracy.

[0136] According to an exemplary embodiment of the present disclosure, for each permanent isolation (PI) type recovery unit handle in the first number of permanent isolation (PI) type recovery unit handles contained in the preset multiple recovery unit handles, the allocation module 1303 can calculate the mean square error between the expiration time of each target file and the expiration time of the currently existing file in the recovery unit pointed to by each permanent isolation (PI) type recovery unit handle. Then, the allocation module 1303 can determine the corresponding permanent isolation (PI) type recovery unit handle with the smallest mean square error for each target file in the first number of permanent isolation (PI) type recovery unit handles. Next, the allocation module 1303 can allocate the corresponding permanent isolation (PI) type recovery unit handle with the smallest mean square error to each target file. In this way, it can be ensured that the expiration times of each file in the RU pointed to by each permanent isolation (PI) type recovery unit handle are close to each other, thereby reducing the degree of fragmentation of data storage.

[0137] According to an exemplary embodiment of the present disclosure, when the file type of the target file is a short-time open file type, the characteristic value of the target file may include at least one of the following items: the creation time of the target file; the size of the target file; the number of files whose opening time is close to the opening time of the target file under each preset recycling unit handle in a preset plurality of recycling unit handles; and the average life cycle of files whose file type is the same as the file type of the target file and have been deleted under each preset recycling unit handle.

[0138] According to an exemplary embodiment of the present disclosure, when the file type of the target file is a long-time open file type, the above-mentioned characteristic value may be a unit size modification USM, and the USM may be a ratio of the number of times the target file is modified to the size of the target file. The allocation module 1303 may cluster the multiple USMs corresponding to the multiple target files using a K-means algorithm to divide the multiple target files into multiple groups. Then, the allocation module 1303 may allocate a recycling unit handle to each of the multiple groups.

[0139] It should be noted that the classification algorithm may be other types of classification algorithms besides the K-means algorithm, and the present disclosure does not impose any specific limitation on this, and the aforementioned implementation is merely an exemplary description.

[0140] According to an exemplary embodiment of the present disclosure, the allocation module 1303 can determine the second number of recovery unit handles of the initial isolation (II) type among the preset multiple recovery unit handles. The recovery unit handle of the initial isolation (II) type allows the file to be moved from the third recovery unit pointed to by the recovery unit handle of the initial isolation (II) type to the fourth recovery unit pointed to by the recovery unit handle of the same type as the initial isolation (II) type. That is, the RUH of the initial isolation (II) type allows data to be moved between RUs pointed to by RUHs of the same type.

[0141] Then, the allocation module 1303 may set K of the K-means algorithm to the above-mentioned second number. Next, the allocation module 1303 may cluster the multiple USMs corresponding to the multiple target files using the K-means algorithm with K set to the second number, so as to divide the multiple target files into a second number of groups. Then, for each of the multiple groups, the allocation module 1303 may select a recovery unit handle of the initial isolation (II) type from the second number of recovery unit handles of the initial isolation (II) type and allocate it to each group.

[0142] Thus, in the present disclosure, when writing a file, the characteristics of the file itself and the type of RUH can be comprehensively considered, and a better balance can be achieved between reducing write amplification and improving space utilization.

[0143] The writing module 1304 may write each target file into the recycling unit pointed to by the recycling unit handle allocated to each target file.

[0144] According to an exemplary embodiment of the present disclosure, the file writing device 1300 may further include a ratio detection module and a dynamic adjustment module.

[0145] The ratio detection module may also detect a first ratio of target files of a file type that is opened for a short time in the plurality of target files at every preset time interval T, and / or may detect a second ratio of target files of a file type that is opened for a long time in the plurality of target files. Then, the dynamic adjustment module may dynamically adjust the quantity d based on the first ratio and / or the second ratio, wherein the smaller the first ratio is, the smaller the quantity d may be; and the larger the second ratio is, the smaller the quantity d may be.

[0146] In this way, by flexibly adjusting the number of RUHs of each type based on the proportion of the number of files of various types, it can be ensured that all types of RUHs can meet the actual file writing needs, avoiding the phenomenon of shortage of one type of RUH and excess of another type of RUH, and ensuring the rational use of RUH resources.

[0147] Fig.14is a block diagram illustrating an electronic device 1400 according to an exemplary embodiment of the present disclosure.

[0148] Reference Fig.14 The electronic device 1400 includes at least one memory 1401 and at least one processor 1402. The at least one memory 1401 stores instructions. When the instructions are executed by the at least one processor 1402, a method for writing a file according to an exemplary embodiment of the present disclosure is executed.

[0149] As an example, the electronic device 1400 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instructions. Here, the electronic device 1400 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instructions (or instruction sets) individually or in combination. The electronic device 1400 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0150] In the electronic device 1400, the processor 1402 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0151] The processor 1402 may execute instructions or codes stored in the memory 1401, wherein the memory 1401 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0152] The memory 1401 may be integrated with the processor 1402, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1401 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1401 and the processor 1402 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1402 can read files stored in the memory.

[0153] In addition, the electronic device 1400 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 1400 may be connected to each other via a bus and / or a network.

[0154] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, and when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above-mentioned method of writing a file. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0155] Fig.15 is a block diagram illustrating an example computer system according to an exemplary embodiment of the present disclosure.

[0156] Fig.15 yes Figure 1 Device 1500 may correspond to writing device 1300 and electronic device 1400. Fig.15 As shown, device 1500 may include a bus 1510 , a processor 1520 , a memory 1530 , a storage component 1540 , an input component 1550 , an output component 1560 , and a communication interface 1570 .

[0157] Bus 1510 includes components that allow communication between components of device 1500. Processor 1520 is implemented by hardware, firmware, or a combination of hardware and software. Processor 1520 can be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other types of processing components. In some embodiments, processor 1520 includes one or more processors that can be programmed to perform functions. Memory 1530 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) for storing information and / or instructions for use by processor 1520.

[0158] Storage component 1540 stores information and / or software related to the operation and use of device 1500. For example, storage component 1540 may include a hard disk (such as a magnetic disk, optical disk, magneto-optical disk, and / or solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette tape, and / or other types of non-transitory computer-readable media and corresponding drives.

[0159] Input components 1550 include components that allow device 1500 to receive information, such as through user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). In addition, input components 1550 may also include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 1560 include components that provide output information from device 1500 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0160] The communication interface 1570 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 1500 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 1570 may allow the device 1500 to receive information from other devices and / or provide information to other devices. For example, the communication interface 1570 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, or the like.

[0161] Device 1500 may perform one or more processes / steps / operations described in the present disclosure. Device 1500 may perform these processes / steps / operations in response to processor 1520 executing software instructions stored by non-transitory computer-readable media such as memory 1530 and / or storage component 1540. Computer-readable media may be defined as non-transitory storage devices. Storage devices include storage space in a single physical storage device or storage space distributed in multiple physical storage devices.

[0162] The software instructions may be read into the memory 1530 and / or storage component 1540 from another computer-readable medium or from another device via the communication interface 1570. When executed, the software instructions stored in the memory 1530 and / or storage component 1540 may cause the processor 1520 to perform one or more processes described in the present disclosure. In addition, hard-wired circuits may be used in place of or in combination with software instructions to perform one or more processes described in the present disclosure. Therefore, the implementations described in the present disclosure are not limited to any specific combination of hardware circuitry and software.

[0163] Fig.15 The number and arrangement of components shown in the figure are provided as examples only. In practice, the device 1500 may include additional components, fewer components, different components, or different components. Fig.15 In addition, or as an alternative, one or more components of device 1500 (eg, one or more components) may perform one or more functions performed by another set of components of device 1500.

[0164] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including a computer program, and when the computer program is executed by a processor, the file writing method according to the present disclosure is implemented.

[0165] According to the file writing method, device, electronic device, computer program and storage medium disclosed in the present invention, since the difference in the expiration time of each file contained in the recycling unit pointed to by the same recycling unit handle can be made smaller than the preset threshold, that is, since the expiration time of each file written into the same recycling unit RU can be made roughly consistent as much as possible, therefore, when performing garbage collection, it is possible to achieve synchronous erasure of each file contained in the same recycling unit RU, avoiding the write amplification phenomenon caused by the simultaneous existence of invalid data and valid data in the same recycling unit RU. It can be seen that the file writing method provided by the present invention can reduce the fragmentation degree of the files contained in the recycling unit RU, and thus can reduce the write overhead.

[0166] According to an exemplary embodiment of the present disclosure, since the user can set the proportion of various types of RUHs according to actual conditions, the autonomy and flexibility of setting the proportion of various types of RUHs are good.

[0167] According to an exemplary embodiment of the present disclosure, when writing a file, the characteristics of the file itself and the type of RUH may be comprehensively considered, and a better balance may be achieved between reducing write amplification and improving space utilization.

[0168] According to an exemplary embodiment of the present disclosure, by flexibly adjusting the number of each type of RUH based on the proportion of the number of various types of files, it can be ensured that various types of RUHs can meet the actual file writing needs, avoiding the phenomenon of shortage of one type of RUH and excess of another type of RUH, and ensuring the rational use of RUH resources.

[0169] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0170] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for writing a file, characterized in that: include: Get multiple target files to be written; For each target file in the plurality of target files, determining a feature value associated with a file expiration time of each target file; Based on the characteristic value of each target file, a recycling circuit processor is allocated to each target file, so that the difference in the expiration time of each file contained in the recycling circuit pointed to by the recycling circuit processor is less than a preset threshold, and each recycling circuit processor is one of a plurality of preset recycling circuit processors; Each of the target files is written into a recycling circuit pointed to by a recycling circuit processor allocated for each of the target files.

2. The writing method according to claim 1, characterized in that: Before allocating a recycling circuit processor to each target file based on the feature value of each target file, the method further includes: Determine a file type of each target file among the plurality of target files, wherein the file type is one of a short-time open file type and a long-time open file type; The step of allocating a recycling circuit processor to each target file based on the feature value of each target file includes: Based on the feature value and file type of each target file, a recycling circuit processor is allocated to each target file.

3. The writing method according to claim 2, characterized in that: Determining the file type of each target file among the multiple target files includes: Setting the file type of each target file to the short-time open file type as a default file type; Detecting whether each target file is in an open state at a preset time interval; Based on detecting that one or more target files are in the open state, the file types of the one or more target files are converted from the short-time open file type to the long-time open file type.

4. The writing method according to claim 2, characterized in that: The allocating a recycling circuit processor to each target file based on the feature value and file type of each target file includes: Based on determining that the file type of the target file is the short-time open file type, inputting the feature value of each target file of the short-time open file type into a random forest model to obtain a predicted life cycle of each target file; Calculate the expiration time of each target file based on the creation time of each target file and the predicted life cycle; Based on the expiration time of each target file and the expiration time of the currently existing file in the recycling path pointed to by each recycling circuit processor in the preset multiple recycling circuit processors, a permanent isolation type recycling circuit processor is allocated to each target file, wherein the permanent isolation type recycling circuit processor is configured to move the file from the first recycling circuit pointed to by the permanent isolation type recycling circuit processor to the second recycling circuit pointed to by the permanent isolation type recycling circuit processor.

5. The writing method according to claim 4, characterized in that: The method of allocating a permanent isolation type recycling circuit processor to each target file based on the expiration time of each target file and the expiration time of the currently existing file in the recycling circuit pointed to by each recycling circuit processor in the preset plurality of recycling circuit processors comprises: For each permanent isolation type recycling circuit processor of a first number of permanent isolation type recycling circuit processors included in the preset plurality of recycling circuit processors, calculating a mean square error between the expiration time of each target file and the expiration time of a currently existing file in the recycling circuit pointed to by each permanent isolation type recycling circuit processor; Among the first number of permanent isolation type recovery circuit processors, determining, for each target file, a permanent isolation type recovery circuit processor having the smallest corresponding mean square error; Allocate the corresponding permanent isolation type recycling circuit processor with the minimum mean square error to each target file.

6. The writing method according to claim 4, characterized in that: Based on determining that the file type of the target file is the short-time open file type, the characteristic value of the target file includes at least one of the following items: The creation time of the target file, the size of the target file, the number of files whose opening time is close to the opening time of the target file under each of the preset recycling circuit processors, and the average life cycle of deleted files whose file type is the same as the file type of the target file under each preset recycling circuit processor.

7. The writing method according to claim 2, characterized in that: Based on determining that the file type of the target file is the long-time open file type, the characteristic value is a unit size modification USM, and the USM is a ratio of the number of times the target file is modified to the size of the target file, and based on the characteristic value and the file type of each target file, a recycling circuit processor is allocated to each target file, including: Clustering the multiple USMs corresponding to the multiple target files by using a K-means algorithm to divide the multiple target files into multiple groups; A recycling circuit processor from the preset plurality of recycling circuit processors is allocated to each of the plurality of groups.

8. The writing method according to claim 7, characterized in that: The clustering of the multiple USMs corresponding to the multiple target files by using a K-means algorithm to divide the multiple target files into multiple groups includes: Determining a second number of recycling circuit processors of the initial isolation type among the preset plurality of recycling circuit processors, wherein the recycling circuit processor of the initial isolation type is configured to move a file from a third recycling circuit pointed to by the recycling circuit processor of the initial isolation type to a fourth recycling circuit pointed to by a recycling circuit processor of the same type as the initial isolation type; Setting K of the K-means algorithm to the second number; clustering the multiple USMs corresponding to the multiple target files by using a K-means algorithm with K set to the second number, so as to divide the multiple target files into K groups; The allocating a recycling circuit processor from the preset plurality of recycling circuit processors to each of the plurality of groups comprises: For each of the plurality of groups, a recovery circuit processor of an initial isolation type is selected from the second number of recovery circuit processors of the initial isolation type and allocated to the each group.

9. The writing method according to claim 1, characterized in that: The total number of the preset multiple recycling circuit processors is N; Before obtaining multiple target files to be written, it also includes: In response to the user's setting input, the number of recycling circuit processors whose processor type is the permanent isolation type is set to d, and the number of recycling circuit processors whose processor type is the initial isolation type is set to (Nd).

10. The writing method according to claim 9, characterized in that: The writing method further comprises: At every preset time interval, detecting a first proportion of target files of a file type opened for a short time in the plurality of target files, or detecting a second proportion of target files of a file type opened for a long time in the plurality of target files; Dynamically adjusting the quantity d based on the first ratio or the second ratio; The number d is proportional to the first ratio; and the number d is inversely proportional to the second ratio.

11. A file writing device, characterized in that: include: A file acquisition module, configured to acquire multiple target files to be written; A feature value determination module is configured to determine, for each target file among the plurality of target files, a feature value associated with a file expiration time of each target file; an allocation module configured to allocate a recycling circuit processor to each target file based on the characteristic value of each target file, so that the difference in expiration time of each file contained in the recycling circuit pointed to by the recycling circuit processor is less than a preset threshold, and each recycling circuit processor is one of a plurality of preset recycling circuit processors; The writing module is configured to write each target file into the recycling circuit pointed to by the recycling circuit processor allocated to each target file.

12. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the file writing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the file writing method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the file writing method according to any one of claims 1 to 10 is implemented.

Images