Operation method for file, and electronic device and storage medium

By selecting the merge operation strategy based on the predicted decision value in the hard disk, using the log structure merge tree and Tiering/Leveling strategy, the merge strategy selection problem in hard disk file operations is solved, and query efficiency and throughput are improved.

WO2024235197A1PCT designated stage expired Publication Date: 2024-11-21DOUYIN VISION CO LTD +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/092869
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-17
Filing Date
2024-05-13
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Existing hard disk file operation methods are difficult to identify data access characteristics in different time and key-value spaces, resulting in the inability to effectively implement the merge strategy, which in turn affects the throughput and efficiency of the query.

Method used

By determining the file state within the first time interval in the hard disk, the merge operation strategy is selected based on these decision values, the log structure merge tree data structure and Tiering and Leveling strategies are used, and the log structure merge tree data structure and Tiering and Leveling strategies are used. Adaptively select the merge operation to improve query efficiency.

Benefits of technology

It significantly improves the throughput of file operations and hard disk bandwidth utilization, and improves user experience and operation efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024092869_21112024_PF_FP_ABST
    Figure CN2024092869_21112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the embodiments of the present disclosure are an operation method for a file, and an electronic device and a computer program medium. The method comprises: in response to receiving a request, determining, within a first time interval, a first prediction decision value for executing a first operation in a first state associated with a file to be operated in a hard disk, and a second prediction decision value for executing a second operation in the first state, wherein the first operation and the second operation are related to a merging operation which is performed on the file. The method may further comprise: determining a selection reference value on the basis of the first prediction decision value and the second prediction decision value; and selecting an operation from the first operation or the second operation on the basis of the selection reference value, so as to perform the merging operation on the file. By using the method in the embodiments of the present disclosure, an improved throughput can be realized when a file is operated, such that the operation efficiency is significantly improved, the bandwidth utilization rate of a hard disk can also be improved, and the user experience is further improved.
Need to check novelty before this filing date? Find Prior Art

Description

File operation method, electronic device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on May 17, 2023, with application number 202310558872.9 and invention name “Method for operating files, electronic device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates generally to the field of computers, and more particularly to an operating method for files in a hard disk, an electronic device, and a computer storage medium. Background Art

[0003] Hard drives are essential components in computing devices. They store a wide variety of data, often in the form of files. As computing devices process ever-increasing amounts of data, the number of files on disk is also increasing. When querying numerous files on a hard drive, achieving the desired high throughput becomes a key concern for users.

[0004] Summary of the Invention

[0005] According to an exemplary embodiment of the present disclosure, there are provided a method for operating a file in a hard disk, an electronic device, and a computer storage medium.

[0006] In a first aspect of the present disclosure, a method for operating a file is provided, comprising: in response to receiving a request, determining a first predicted decision value for performing a first operation in a first state associated with a file to be operated on a hard disk within a first time interval, and a second predicted decision value for performing a second operation in the first state, wherein the first operation and the second operation are related to a merge operation to be performed on the file. The method may also include determining a selection reference value based on the first predicted decision value and the second predicted decision value, and selecting an operation from the first operation or the second operation based on the selection reference value to be used for performing the merge operation on the file.

[0007] In a second aspect of the present disclosure, an electronic device is provided, comprising: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which instructions, when executed by the at least one processing unit, enable the electronic device to perform the method described in accordance with the first aspect of the present disclosure.

[0008] In a third aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has machine-executable instructions stored thereon. When the machine-executable instructions are executed by a device, the device may perform the method described according to the first aspect of the present disclosure.

[0009] According to a fourth aspect of the present disclosure, a computer program product is provided, comprising computer-executable instructions, wherein the computer-executable instructions implement the method described in the first aspect of the present disclosure when executed by a processor.

[0010] The purpose of providing the summary of the invention section is to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The summary of the invention section is not intended to identify the key features or essential features of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] FIG1 shows a schematic diagram of an example system in which embodiments of the present disclosure can be implemented;

[0013] FIG2 shows a flow chart of a method for operating a file according to an embodiment of the present disclosure;

[0014] FIG3 shows a calculation index associated with a file for determining a decision value according to an embodiment of the present disclosure;

[0015] FIG4 is a flowchart illustrating an exemplary method for a computing device to perform a query operation according to an embodiment of the present disclosure;

[0016] FIG5 illustrates a schematic diagram of an application multi-stage merging strategy for computing resources according to an embodiment of the present disclosure;

[0017] FIG6 shows a schematic block diagram of an example apparatus according to some embodiments of the present disclosure; and

[0018] FIG7 illustrates a block diagram of an example device that may be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0020] As the number of files on hard drives increases, merging files can reduce the number of queries required when performing operations (e.g., query operations) on these numerous files. However, existing technologies cannot identify different data access characteristics, making it difficult to implement optimal merging strategies at different times and in different key-value spaces. How to fully utilize data access characteristics and adaptively select merging strategies to achieve the desired high query throughput is an urgent problem to be solved.

[0021] In view of this, an embodiment of the present disclosure provides a method for operating a file, comprising: in response to receiving a request, determining a first predicted decision value for performing a first operation in a first state associated with a file to be operated on a hard disk within a first time interval and a second predicted decision value for performing a second operation in the first state, wherein the first operation and the second operation are related to a merge operation performed on the file. The method may also include determining a selection reference value based on the first predicted decision value and the second predicted decision value, and selecting an operation from the first operation or the second operation based on the selection reference value for performing a merge operation on the file. By adopting the method according to an embodiment of the present disclosure, it is possible to fully utilize the access characteristics of the data and adaptively select appropriate merge strategies in different time and key value spaces, thereby achieving improved throughput when operating on files, significantly improving operational efficiency, improving hard disk bandwidth utilization, and further improving user experience.

[0022] FIG1 illustrates a schematic diagram of an example system 100 in which embodiments of the present disclosure can be implemented. System 100 may include a computing device 120. Computing device 120 may include a memory 122 and a hard disk 124. Furthermore, although not shown, those skilled in the art will appreciate that computing device 120 may also include other devices such as a processor, input / output devices, and the like.

[0023] The data on the hard disk 124 can be stored in the memory 122 and the hard disk 124 according to a certain data structure. For example, the computing device 120 can use a log structure merge tree (LSM-tree) data structure to store data in the memory 122 and the hard disk 124. The log structure merge tree is a data structure optimized for writing and is widely used in database storage engines. It provides key-value pair (key-value) writing, deletion, modification and query functions. When using the log structure merge tree, the data can be first written and stored in the memory 122. When the amount of data reaches a threshold (for example, a threshold set by the user), the computing device 120 can sequentially persist the data to the hard disk 124 in the form of an ordered file sequence.

[0024] The log-structured merge tree continuously merges data on disk 124 to improve the order of the data on disk 124, thereby providing good query performance. The log-structured merge tree can store data on disk 124 in layers, with data being written to the next layer after each merge. For example, data just written to disk 124 can be stored in layer L0, merged once, written to layer L1, merged twice, written to layer L2, and so on.

[0025] When querying data on hard disk 124, the query begins at layer L0. Computing device 120 can perform a binary search on the ordered files at each layer until the corresponding key value is found and the data is returned. In practical applications, multiple instances of the log-structured merge tree are typically deployed on multiple computing devices to provide services to users simultaneously.

[0026] In some embodiments, the computing device 120 may include, but is not limited to, a personal computer, a server computer, a handheld or laptop device, a mobile device (such as a mobile phone, a personal digital assistant (PDA), a media player, etc.), a vehicle-mounted device, a multi-processor system, a consumer electronics product, a wearable electronic device, a smart home device, a minicomputer, a mainframe computer, an edge computing device, a distributed computing environment including any one of the above systems or devices, etc.

[0027] In some embodiments, the computing device 120 may, in response to receiving a request, determine a first predicted decision value for performing a first operation in a first state associated with a file to be operated in the hard disk 124 within a first time interval and a second predicted decision value for performing a second operation in the first state. The first operation and the second operation are related to a merge operation performed on the file. The computing device 120 may also determine a selection reference value based on the first predicted decision value and the second predicted decision value, and select an operation from the first operation or the second operation based on the selection reference value for performing a merge operation on the file. By adopting the operating method according to the embodiment of the present disclosure, it is possible to achieve improved throughput when operating on files, significantly improve operational efficiency, improve hard disk bandwidth utilization, and further enhance user experience.

[0028] A block diagram of an example system 100 in which embodiments of the present disclosure can be implemented is described above in conjunction with FIG1 . A flowchart of a method 200 for operating a file according to an embodiment of the present disclosure is described below in conjunction with FIG2 . The actions involved in method 200 are described below in conjunction with computing device 120 as shown in FIG1 . For example, in some embodiments, method 200 may be performed by computing device 120. It should be understood that method 200 may also include additional actions not shown and / or may omit actions shown, and the scope of the present disclosure is not limited in this respect.

[0029] In block 202, in response to receiving the request, computing device 120 may determine a first predicted decision value for performing a first operation and a second predicted decision value for performing a second operation in a first state associated with a file to be operated on in hard disk 124 within a first time interval. In some embodiments, the first operation and the second operation are related to a merge operation performed on the file.

[0030] In some embodiments, both the first operation and the second operation are related to a merge operation performed on files. Files can be stored in memory 122 and hard disk 124 according to a log-structured merge tree structure. When a merge strategy employing a log-structured merge tree is employed, the first operation may include skipping the merge operation, and the second operation may include executing the merge operation. For example, the first operation may be based on a Tiering strategy, and the second operation may be based on a Leveling strategy. Tiering-based operations involve skipping the merge operation, and accordingly, multiple ordered file sequences can be maintained at each layer. Leveling-based operations involve executing the merge operation, and accordingly, only one ordered file sequence can be maintained at each layer. Tiering-based operations require a smaller amount of file data to be merged, resulting in lower write overhead, but also higher query overhead. Leveling-based operations require a larger amount of file data to be merged, resulting in higher write overhead, but lower query overhead. In the following description, the first operation is based on a Tiering strategy, and the second operation is based on a Leveling strategy.

[0031] The computing device 110 may determine the state associated with the file within the first time interval, and determine the predicted decision value based on the determined state. In some embodiments, the computing device 120 may divide the time window according to the query volume and the number of merge operations, and determine the state associated with the file in each time window in ascending order of time. The state associated with the file may include but is not limited to the number of files to be operated on, the read penalty associated with the merge operation, and the amount of data in the file to be operated within the corresponding time interval. The read penalty means that under the same circumstances, the first operation (A) based on the Tiering strategy is adopted. tier ) Compared with the second operation based on the Leveling strategy (A level In some embodiments, the read penalty may be calculated based on the access granularity of data on the hard disk (eg, storage block).

[0032] The decision value can be identified as Q-value (Q value). Accordingly, the predicted decision value represents the predicted Q value. The Q value can be understood as a value that assists decision making, indicating the expected value of the "reward" that can be obtained by performing the corresponding operation or action in a given state. The larger the value of Q, the greater the "reward" that can be obtained by performing the corresponding action. Therefore, the Q value can be used as a decision value for determining an operation. In an embodiment of the present disclosure, the Q value can represent the expected value of taking an action or operation A in a given state (for example, the first state). iAfter that, how much read and write (IO) overhead is saved compared to another operation. For example, assuming the number of ordered file sequences is 2, in state S n Perform the second operation (A level ), the number of ordered file sequences can be reduced from 2 to 1, then, in the state S of the next time interval n+1 In other words, the decision value according to the embodiment of the present disclosure can represent the expected reward obtained when performing merge-related operations on files based on a given state. For example, the first predicted decision value can represent the first operation (A) on the file. tier ) when the prediction is obtained; the second prediction decision value can represent the second operation (A level ). Generally, the larger the predicted expected reward, the larger the predicted decision value, and the more inclined the computing device 120 is to adopt the corresponding operation.

[0033] As described above, the determination of the predicted decision value Q is associated with the state. The following will describe the calculation indicators associated with the file for determining (predicting) the decision value in conjunction with Figure 3. The Y-axis in Figure 3 corresponds to the calculation indicators associated with the file for calculating the decision value, including but not limited to: read and write load (which may include query, write, or a combination of query and write, etc.), the ordered sequence of files to be operated, the state number, and the time window number. The X-axis in Figure 3 corresponds to the time windows in increasing order of time, each time window has a corresponding time interval, and the computing device 110 can determine the state of the file for each time interval. For example, with the time window being T i+1 As an example, FIG3 shows that in this time window, the number of files in the ordered file sequence to be operated is 3, and the corresponding state number is S i+1 , and the query operation is performed on the file in this time window.

[0034] Assume that the first time interval is T i , and the state corresponding to this time interval is S i , the computing device 120 may determine that at the first time interval (eg, T i ) in a first state associated with a file in the hard disk 124 (e.g., S i ) to perform the first operation A tier The first predicted decision value Q(S i ,A tier ) and in the first state (eg, S i ) to perform the second operation A level The second prediction decision value Q(S i ,A levelIn some embodiments, the first prediction decision value Q(S) may be determined by a trained target network. i ,A tier ) and the second prediction decision value Q(S i ,A level ).

[0035] Continuing with the method 200 in FIG. 2 , the computing device 120 at block 204 determines the decision value Q(S) based on the first prediction. i ,A tier ) and the second prediction decision value Q(S i ,A level ), calculate the selection reference value. In some embodiments, the selection reference value may be a reference value used by the computing device 120 when selecting one of the first operation or the second operation.

[0036] If the first operation based on the Tiering strategy is currently adopted (A tier ), the computing device 120 can determine the selection reference value p by using formula (1):

[0037] After determining the selection reference value P, the computing device 120 may, at block 206, select the first operation (A) based on the selection reference value p. tier ) or the second operation (A level ) to merge the files.

[0038] In some embodiments, the selection reference value p represents the probability of selecting to perform the corresponding operation. When the selection reference value p is calculated using formula (1), p represents the probability of selecting to perform the first operation (A tier ), accordingly, 1-p represents the probability of choosing to perform the second operation (A level ) probability. The computing device 120 can use various appropriate methods to select an appropriate operation based on the selection reference value p. For example, the computing device 120 can set the operation threshold p th , and select the reference value p and the operation threshold p according to th For example, when the selected reference value p is greater than the operation threshold p th When the computing device 120 can choose to perform the first operation (A tier ) to implement the file operation; when the reference value p is not greater than the operation threshold p th When the computing device 120 can choose to perform the second operation (A level ) to implement the operation on the file. Other methods can also be used to select and perform appropriate operations according to the selection reference value p, which is not limited in this disclosure.

[0039] It is understandable that if the second operation (A level ), those skilled in the art can modify formula (1) (for example, the numerator in formula (1) will be the second prediction decision value Q(S i ,A level )), so that the calculated selection reference value p represents the selection of performing the second operation (A level ), 1-p represents the probability of choosing to perform the first operation (A tier The process of selecting an appropriate operation to be performed based on the selected reference value p may be similar to the process described above or adopt other suitable processes, which is not limited in the present disclosure.

[0040] By adopting the predicted decision value, the computing device 120 can more accurately determine whether to adopt the appropriate operation of the first operation or the second operation to merge the files to be operated on the hard disk, thereby achieving improved throughput when operating on the files, significantly improving operational efficiency, and improving hard disk bandwidth utilization, thereby further improving user experience.

[0041] In some embodiments, the request received by the computing device 120 at block 202 may include a merge request or a query request. When a merge request is received, the computing device 120 may default to the first operation (A tier ), and calculate the first prediction decision value Q(S i ,A tier ) and the second prediction decision value Q(S i ,A level ). In determining the first prediction decision value Q(S i ,A tier ) and the second prediction decision value Q(S i ,A level ), the computing device 120 may determine the selection reference value p according to the method described above, and select an appropriate operation to merge the files in the hard disk according to the selection reference value p.

[0042] In some embodiments, when the computing device 120 selects the first operation (A tier ), due to the first operation (A tier ) corresponds to skipping the merge operation, so the computing device 120 can skip the merge operation on the file and use the time interval T i The corresponding state S i To update the second time interval T i+1 The second state S associated with the file i+1 , and the second time interval Ti+1 With the first time interval T i In other words, when the first operation (A tier ), the computing device 110 can directly use the time interval T i The corresponding state S i To update the status of the file in the next time interval, that is, the first adjacent time interval T i and the second time interval T i+1 The status is the same.

[0043] In some embodiments, when the computing device selects the second operation (A level ), since the second operation corresponds to performing a merge operation on the files, the computing device 120 may perform a merge operation on the files in the hard disk, and obtain the data in the second time interval T based on the merge operation. i+1 The second state S associated with the file i+1 , In addition, the second time interval T i+1 With the first time interval T i adjacent.

[0044] As described above, the request received by the computing device 120 may also include a query request, that is, in the process of operating the files in the hard disk, in addition to the merge operation, there is also a query operation for the file. In some embodiments, when the request received by the computing device 120 in box 202 includes a query request, the computing device 120 may also perform a query operation. Figure 4 illustrates a flowchart of an illustrative method 400 for the computing device 120 to execute a query request in a first state according to an embodiment of the present disclosure. It should be understood that the method 400 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0045] As shown in FIG4 , the computing device 120 may perform a query operation and return data in a first state in response to a received query request in block 402 . In some implementations, the state in block 402 is a first time interval T i The corresponding first state S i It is understood that this is merely exemplary and method 400 can be used in any state within any time interval.

[0046] At block 404, computing device 120 may update the read penalty in the first state based on the query operation performed in block 402 to obtain an updated read penalty in the first state. In some embodiments, computing device 120 may determine the read penalty incurred by performing the query operation based on the number of storage blocks used when performing the query operation in block 402. Computing device 120 may further update the read penalty in the current state based on the determined read penalty.

[0047] Furthermore, at block 406, computing device 120 may determine whether the next query operation is the last query operation within the first time interval (i.e., the first time window). When computing device 120 determines that the next query operation is not the last query operation within the first time interval (i.e., the first time window), computing device 120 returns to continue executing the query operation in block 402 and obtains an updated read penalty. When computing device 120 determines that the next query operation is the last query operation within the first time interval (i.e., the first time window), computing device 120 may continue to block 408 in the flowchart of method 400 and update the decision value in the first state at block 408.

[0048] The following will describe in detail the schematic specific process of updating the decision value. In some embodiments, the computing device 120 may determine a first decision value and a second decision value in the first state. The first decision value corresponds to the first operation (A tier ), and the second decision value corresponds to the second operation (A level The first decision value may represent performing a first operation on the file in the first state (A tier ), and the second decision value can represent the expected reward of performing the second operation (A level ) is the expected reward obtained when the first decision value and the second decision value are used. In some embodiments, the first decision value and the second decision value may be values ​​pre-configured in a Q-table. The computing device 120 may determine the first decision value and the second decision value by querying the Q-table.

[0049] When configuring the Q table, for the first operation (A tier ) can be configured as follows. For example, the first operation (A tier ) represents the merging operation based on the Tiering strategy, which corresponds to skipping the merging operation on the file. For the case where the ordered file sequence is greater than 1, the first operation (A tier ) can be set to the size of the data amount (for example, the number of files) of the operation object, indicating that in the current case, relative to executing the second operation (A level) saves IO overhead. For the case where the ordered file sequence is equal to 1, perform the first operation (A tier )'s first decision value can be set to 0.

[0050] Since the second operation (A level ) corresponds to performing a merge operation on the files. Therefore, the second operation is also associated with the query load. This is because the greater the access load, the more the number of ordered file sequence searches can be reduced by executing the second operation, and the greater the overhead saved compared to the first operation. Therefore, the second decision value can be determined based on the query load. Various techniques known in the art or developed in the future can be used to determine the second decision value based on the query load, and this disclosure does not limit this.

[0051] After determining the first decision value and the second decision value, the computing device 120 may also determine the time interval T i At least one subsequent time interval T thereafter i+1 、T i+2 ...T i+x At least one corresponding metric value P i+1 、P i+2 ...P i+x Where x is a positive integer greater than or equal to 1. The metric value P i In some embodiments, due to the fact that in a certain state (S i+m (0≤i≤x)) to merge the files, then each merge can save IO overhead for the query of the file. Therefore, each time interval can have a corresponding metric value P that represents the benefit or reward of performing the second operation i+m (0≤i≤x).

[0052] In addition, depending on the first time interval T i The latest time interval T among the subsequent time intervals i+x Whether there is a merge operation between the first and second operations, the first benefit or missing benefit associated with performing the second operation is also considered. Specifically, when the first time interval T i The latest time interval T among the subsequent time intervals i+x When there is a merge operation between them, it can be considered that in the first state S i The merge operation performed under the latest time interval T i+x Status S i+x This is because the first time interval T i The latest time interval T among the subsequent time intervals i+x There are other merge operations between T i+x Status S i+xTherefore, it can be ignored in the first state S i Therefore, in this case, the first benefit (e.g., "missing benefit") for the second operation can be set to 0. In contrast, when the first time interval T i The latest time interval T among the subsequent time intervals i+x When there is no merge operation between them, it can be considered that in the first state S i The merge operation performed under the latest time interval T i+x Status S i+x In this case, the first benefit for the second operation (eg, “missing benefit”) may be set to the first state S i The number of files in N i , and the latest time interval T among the subsequent time intervals i+x Status S i+x The number of files in N i+x and the latest time interval T i+x Status S i+x The third decision value Q(S) corresponding to the second operation under i+x ,A level ). For example, the function can be expressed as the following formula (2), which is used to determine the first time interval T i The latest time interval T among the subsequent time intervals i+x When there is no merge operation between them, the first benefit value (or "missing benefit value") for the second operation is P m .

[0053] Accordingly, depending on the first time interval T i The latest time interval T among the subsequent time intervals i+x Is there a merge operation between them? The first benefit P related to the second operation m It can be equal to 0 or can be determined by formula (2).

[0054] In some embodiments, when the first time interval T i The latest time interval T among the subsequent time intervals i+x When there is a merge operation between the two, the computing device 120 may calculate the value of the at least one metric value P determined as described above. i+1 、P i+2 ...P i+x To update the first decision value and the second decision value, so as to obtain the updated first decision value and the updated second decision value respectively for performing the first operation and the second operation on the file in the first state. i The latest time interval T among the subsequent time intervals i+xWhen there is no merge operation between the two operations, the computing device 120 may determine the missing benefit value P associated with the second operation based on the third decision value according to the functional relationship determined in formula (2). m The computing device 120 may calculate the value of the at least one metric P based on the above-described determination. i+1 、P i+2 ...P i+x and the missing return value P m The first decision value and the second decision value are updated to obtain the updated first decision value and the updated second decision value for performing the first operation and the second operation on the file in the first state, respectively.

[0055] The process described above can be expressed by the following formula (3).

[0056] Among them, Q new (S i ,A i ) represents the updated first decision value or the updated second decision value, where A i Indicates the first operation (A tier ) or the second operation (A level ), P i+j Represents the first time interval T i The metric value of the subsequent time interval, and Q(S i+x ,A levle ) represents the latest time interval T i+x The third decision value corresponding to the second operation in the state of , j is a positive integer greater than 1. α is the learning rate, which can be configured by the user and can be any value between 0 and 1. The larger α is, the more inclined to follow the modified benefit G i:i+x To adjust the merging strategy, otherwise it tends to use the existing Q value to adjust the merging strategy.

[0057] By combining the above formulas (2) and (3), the updated first decision value Q can be calculated new (S i ,A tier ) and the updated second decision value Q new (S i ,A level ). The computing device 120 can convert the first state S i , the updated first decision value Q new (S i ,A tier ) and the updated second decision value Q new (S i ,A level) as training samples in a training sample set for training the update network. In some embodiments, the update network may be a neural network that obtains parameters through training, and the parameters obtained through training may be shared with the target network. The target network may also include a neural network, and the first predicted decision value and the second predicted decision value are generated at block 202 of method 200 by sharing the updated parameters of the update network. The training and prediction processes of the update network and the target network will be described in detail below.

[0058] In some embodiments, after obtaining the aforementioned training samples, the computing device 120 may first train an update network, share the parameters of the trained update network with the target network, and then use the target network with updated parameters to generate a first predicted decision value for performing a first operation in a first state and a second predicted decision value for performing a second operation in the first state. Thus, by using the updated Q values ​​as training samples to update the parameters of the target network, the obtained predicted decision values ​​may be made more accurate, thereby assisting the computing device 120 in determining a more appropriate operation, further contributing to improved throughput, significantly improving operational efficiency, and increasing hard disk bandwidth utilization, thereby further enhancing the user experience.

[0059] The following describes the parameter update process for the update network and the target network. The update network and the target network can be two separate neural networks. The parameters of the update network can be updated through a training process, and the updated parameters can be shared by the target network. The target network can then generate a first predicted decision value for a first operation and a second predicted decision value for a second operation based on the shared updated parameters. By using two separate neural networks, training and prediction can be performed concurrently, thereby improving execution efficiency.

[0060] In some embodiments, the input of any one of the update network and the target network may include a given state and an operation, for example, the state within a first time interval and a first operation or the state within a first time interval and a second operation. As described above, the state within any time interval may include the number of files to be operated, the read penalty associated with the merge operation, and the amount of data in the file. The output of any one of the update network and the target network may include a predicted decision value corresponding to the input operation. For example, when the input is the first operation, the output of the target network is the first predicted decision value, and when the input is the second operation, the output of the target network is the second predicted decision value.

[0061] In some embodiments, the computing device 120 may obtain a training set for training and updating the network by the method described above. The training set may be a first-in-first-out queue with a constant number of data. i ,A i)(For example, the updated decision value Q described above new (S i ,A i After the training set is written, computing device 120 may randomly select K training samples to train the updated network. In some embodiments, the K training samples may be part or all of the training samples in the training sample set. This disclosure does not limit the number of K. Computing device 120 may train the updated network using any suitable training method and loss function. This disclosure does not limit this.

[0062] In some embodiments, after training the update network, the computing device may share the parameters of the update network obtained through training with the target network (e.g., by copying them), so that the target network can complete the determination and output of the predicted decision value. In other words, the calculation of the predicted decision value Q is implemented by the target network, while the update of the target network parameters is completed by training the update network. This implementation method can enable the training process and the prediction and inference process to be executed in parallel, which can improve overall efficiency.

[0063] In the process of training, in order to avoid the training and updating of the network from taking up too many computing resources, and in order to avoid the negative impact of the poor merge strategy on system performance in the initial stage in the absence of user load characteristics, the method according to the embodiment of the present disclosure also separates the user operation of the log structure merge tree and the merge strategy training operation in terms of computing resources. That is, according to the embodiment of the present disclosure, a multi-stage merge strategy application method for computing resources is adopted. According to this application method, in the initial stage of training and updating the network, the computing device 120 only samples a portion (e.g., s%) of files in the hard disk operation area for training and guidance and application of the merge strategy, for example, selecting an operation from the first operation or the second operation to merge the randomly sampled files. Moreover, when the average return of one or more time windows in a time series is less than a threshold T, the computing device 120 only guides and applies the merge strategy for this portion (e.g., s%) of files in the hard disk operation area (i.e., selects the first operation or the second operation for this portion of files). When the average revenue of one or more time windows in the time series exceeds a threshold value T, computing device 120 can guide a merge strategy for all files in the hard disk operation area. That is, it can use the trained target network to select a merge operation for all files in the hard disk operation area. In some embodiments, the revenue can correspond to the metric value described above, representing the revenue or reward obtained by performing a certain type of operation under a given state.

[0064] In some embodiments, the computing device 120 may randomly sample files in the hard disk operation area according to a first ratio (e.g., s1%) to obtain sample files, and use the sample files as files to be operated. The computing device 120 may select an operation for merging for the files to be operated (i.e., sample files) according to the method 200 described above in conjunction with FIG. 2 , for example, the first operation (A) based on the Tiering strategy tier ) or the second operation based on the Leveling strategy (A level ). The computing device 120 may determine, for the file to be operated, the average revenue of at least one time window within a time interval. When the average revenue is less than the revenue threshold, the computing device 120 keeps the randomly sampled file as the file to be operated. The computing device 120 may continue to perform training and inference prediction for the files randomly sampled according to the first ratio until the average revenue of at least one time window within the time interval reaches or exceeds the revenue threshold. When the average revenue of at least one time window within the time interval reaches or exceeds the revenue threshold, the computing device 120 may randomly sample files in the hard disk operating area according to the second ratio, and use the sampled files as files to be operated to determine the merge operation for the files to be operated according to method 200 in Figure 2. In some embodiments, the second ratio may be greater than the first ratio. In some embodiments, the second ratio may be 100%. In other words, when the average revenue of at least one time window within the time interval reaches or exceeds the revenue threshold, the computing device may use the target network to perform a merge operation on all files in the hard disk operating area.

[0065] It can be understood that, according to actual needs and applications, multiple average revenue thresholds T1, T2….Tn can be set, where T1>T2>…..Tn. Accordingly, multiple random sampling ratios S1, S2….Sn can be set, and S1>S2>…..Sn. For example, when the initial sampling ratio is S1, when the average revenue of at least one time window within a time interval is greater than the revenue threshold T1, the computing device can sample files at the ratio of S2. When the average revenue of at least one time window within another time interval is greater than the revenue threshold T2, the computing device can continue to sample files at the ratio of S3, and so on. In this way, a multi-stage merging strategy guidance scheme for files can be implemented.

[0066] Figure 5 illustrates a schematic diagram 500 of a method for applying a multi-stage merge strategy to computing resources according to an embodiment of the present disclosure. The method in Figure 5 can be executed by a merge strategy training service module 510 in the computing device 120. The module 510 can implement the application of the multi-stage merge strategy through software and running via a processor. In other implementations, the module 510 can also be implemented by hardware. In other implementations, the module 510 can be implemented by a combination of software and hardware. The present disclosure does not limit the specific implementation method and form of the merge strategy training service module 510.

[0067] The merge strategy training service module 510 is used to guide and apply the merge strategy to the files in the operation area of ​​the hard disk 124 by training and updating the network, for example, selecting the first operation (A) based on the Tiering strategy. tier ) or the second operation based on the Leveling strategy (A level ) in the hard disk operation area. In the process of implementing the application method, the merge strategy training service module 510 can randomly select a portion (for example, s%) of files for training according to a first ratio in the initial stage of training and updating the network. The computing device 120 can determine the average benefit within a time series. When the computing device 120 determines that the average benefit of one or more time windows on the time series is less than a threshold value T, the computing device 120 only selects a merge strategy (that is, the first operation or the second operation) for the portion (for example, s%) of files in the hard disk operation area. When the computing device 120 determines that the average benefit of one or more time windows on the time series is greater than the threshold value T, the computing device 120 can sample all files in the hard disk operation area according to a second ratio (for example, 100%) and determine the corresponding merge operation.

[0068] For example, as shown in FIG5 , in the initial stage of training, the computing device 120 may select a portion of the files (i.e., operation objects) in the hard disk at a first ratio to guide the training and merging strategy selection. When the computing device 120 determines that the average revenue within a certain time interval is less than the revenue threshold T, the computing device 120 may use the target network only for the randomly sampled files for merging strategy selection. When the computing device 120 determines that the average revenue within a certain time interval is not less than T, the computing device 120 may use the target network for more (e.g., all) files in the operating area of ​​the hard disk 124 for merging strategy selection.

[0069] The values ​​of the average revenue threshold T (T1, T2....Tn) and the random sampling ratio S (S1, S2....Sn) can be set by the user. It is understandable that the larger the S value is set, the more training data there is, and the updated network can fit the Q value under the real load faster, so that the user can get the experience under the better merging strategy faster. It is understandable that in the early training stage, the performance jitter of the device will be relatively large. On the contrary, if the S value is set to a smaller value, the fitting speed will be relatively slow, but the performance jitter will be relatively small. Similarly, when T is set to a larger value, the neural network needs a longer fitting time to make the revenue reach the revenue threshold, but at this time, applying the merging strategy globally to all files to be operated will have a better user experience. Conversely, when T is set to a smaller value, the neural network fits the time period, but the application experience is not too high. Users can flexibly set the S value and T value according to their needs, and this disclosure does not limit this.

[0070] In some embodiments, the operating area in the hard disk as described above may include all storage areas in the hard disk. When files are stored in the hard disk 124 according to layers, the operating area in the hard disk may include each layer (e.g., the first layer, the second layer, or the third layer in FIG. 5 , etc.). In some embodiments, the operating area in the hard disk may also include each physical partition in a plurality of physical partitions in the hard disk that are respectively mapped to a plurality of virtual partitions in the internal memory. As shown in FIG. 5 , the first partition, the second partition, and the third partition, etc. In some embodiments, the operating area in the hard disk may also include each layer in each physical partition in the hard disk. Accordingly, the method according to an embodiment of the present disclosure may also include obtaining a plurality of virtual partitions in the internal memory 122, and in some implementations, the files stored in the hard disk 124 may be mapped to a plurality of virtual partitions.

[0071] In some embodiments, computing device 120 may store the received data in the form of a memory file on a file layer in memory 122. Computing device 120 may determine that the amount of data on the file layer exceeds a quantity threshold (e.g., a user-defined quantity threshold). When the amount of data exceeds the threshold, computing device 120 may partition the file layer to obtain multiple virtual partitions on the file layer.

[0072] In some embodiments, the computing device 120 may determine a partition key based on the key-value pairs corresponding to the files and partition the file layer into regions based on the partition keys. For example, for a given partition key, files with key values ​​greater than the partition key value and files with key values ​​less than the partition key value are respectively partitioned into two adjacent virtual partitions.

[0073] In some embodiments, computing device 120 may select a key as a partition key from multiple key-value pairs corresponding to a file. For example, computing device 120 may sample multiple key-value pairs corresponding to a file and obtain multiple sampled key-value pairs. Computing device 120 may sort the sampled key-value pairs and select multiple keys from the sorted key-value pairs as partition keys.

[0074] After determining the partition key and completing the partitioning, computing device 120 may mark the layer (e.g., L1 layer) where the partitioning is completed. This mark may indicate that the file layer now has multiple virtual partitions. After completing the virtual partitioning, computing device 120 may perform a merge operation on the files on that layer and implement physical partitioning of the hard disk. For example, when performing the merge operation, computing device 120 may obtain a sequence of key-value pairs corresponding to the files to be merged on that layer and sort the key-value pairs in the obtained sequence of key-value pairs. For example, computing device 120 may sort the key-value pairs according to the size of the key values ​​in the key-value pairs. Computing device 120 may sequentially read key-value pairs from the sequence of key-value pairs. When the value of the key in the key-value pair read equals the value of the partition key, computing device 120 terminates reading the key value and writes the file corresponding to the key read before the read operation was terminated to the first physical partition of hard disk 124. Computing device 120 may continue to read the keys in the sequence of key-value pairs and continue writing the files in the virtual partition to other corresponding physical partitions of hard disk 124 in the above manner. This achieves the mapping between virtual partitions and physical partitions.

[0075] The computing device 120 can use the determined physical partition (or each layer in the determined physical partition) as the operating area in the hard disk 124 described above, and execute the merge strategy selection method in method 200 on the files in the area to merge the files, thereby making full use of the access characteristics of the data and adaptively implementing a better merge strategy for the data under different time and key value spaces.

[0076] FIG6 shows a schematic block diagram of an example apparatus 600 according to some embodiments of the present disclosure. Apparatus 600 may be implemented in software, hardware, or a combination of both. As shown in FIG6 , apparatus 600 includes a first determination module 610 , a second determination module 620 , and a selection module 630 .

[0077] In some embodiments, the first determination module 610 is configured to, in response to receiving a request, determine a first predicted decision value for performing a first operation in a first state associated with a file to be operated on a hard disk within a first time interval, and a second predicted decision value for performing a second operation in the first state. In some embodiments, the first operation and the second operation are related to a merge operation performed on the file. The second determination module 620 is configured to determine a selection reference value based on the first predicted decision value and the second predicted decision value. The selection module 630 is configured to select an operation from the first operation or the second operation based on the selection reference value for merging the files.

[0078] The apparatus 600 in FIG. 6 can be used to implement the above-mentioned process described in conjunction with FIG. 1 to FIG. 5 , and for the sake of brevity, details thereof will not be repeated here.

[0079] The division of modules or units in the embodiments of the present disclosure is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the disclosed embodiments may be integrated into a single unit, exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0080] FIG7 illustrates a block diagram of an example device 700 that can be used to implement embodiments of the present disclosure. It should be understood that the device 700 shown in FIG7 is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. For example, the device 700 can correspond to the computing device 120 described herein in conjunction with FIG1 and can be used to perform the processes described above in FIG1 through FIG6 .

[0081] As shown in Figure 7, device 700 is in the form of a general-purpose computing device. Components of computing device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 720. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 700.

[0082] The computing device 700 typically includes a plurality of computer storage media. Such media can be any available media accessible to the computing device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 can be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the computing device 700.

[0083] The computing device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 7 , a disk drive for reading from or writing to a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules configured to perform various methods or actions of various implementations of the present disclosure.

[0084] The communication unit 740 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device 700 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or other network nodes.

[0085] Input device 750 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 760 may be one or more output devices, such as a display, a speaker, or a printer. Computing device 700 may also communicate with one or more external devices (not shown) via communication unit 740, as needed. External devices such as storage devices, display devices, and the like may also be used to communicate with one or more devices that allow a user to interact with computing device 700, or with any device that allows computing device 700 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0086] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0087] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0088] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0089] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0090] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0091] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for operating a file, the method comprising: In response to receiving the request, determining a first predicted decision value for performing a first operation in a first state associated with a file to be operated on a hard disk within a first time interval and a second predicted decision value for performing a second operation in the first state, wherein the first operation and the second operation are related to a merge operation performed on the file; Determining a selection reference value based on the first prediction decision value and the second prediction decision value; as well as Based on the selection reference value, an operation is selected from the first operation or the second operation to merge the files.

2. The method according to claim 1, wherein the files are stored in the hard disk according to the structure of a log structure merge tree.

3. The method of claim 1, wherein the first operation comprises skipping performing a merge operation and the second operation comprises performing a merge operation, and when the request comprises a merge request and the selected operation comprises the first operation, the method further comprises: Skip the merge operation on the file; as well as A second state associated with the file in a second time interval is updated with the first state, wherein the second time interval is adjacent to the first time interval.

4. The method of claim 1, wherein the first operation comprises skipping performing a merge operation, and the second operation comprises performing a merge operation, and when the request comprises a merge request and the one operation selected comprises the second operation, the method further comprises: performing a merge operation on the files; as well as Based on the merge operation, a second state associated with the file in a second time interval is obtained, wherein the second time interval is adjacent to the first time interval.

5. The method according to claim 1, wherein when the request comprises a query request, the method further comprises: determining a first decision value and a second decision value in the first state, The first decision value corresponds to the first operation, and the second decision value corresponds to the second operation.

6. The method according to claim 5, further comprising: determining at least one metric value corresponding to at least one subsequent time interval after the first time interval; as well as The first decision value and the second decision value are updated based on the at least one metric value to obtain an updated first decision value and an updated second decision value in the first state, respectively.

7. The method according to claim 6, wherein when there is no merging operation between the first time interval and the latest time interval of the at least one subsequent time interval, the method further comprises: determining a third decision value associated with the latest time interval of the at least one subsequent time interval, wherein the third decision value is associated with performing the second operation within the latest time interval; determining a missing benefit value for the second operation based on the third decision value; as well as The first decision value and the second decision value are updated based on the at least one metric value and the missing benefit value to obtain an updated first decision value and an updated second decision value in the first state, respectively.

8. The method according to claim 7, further comprising: The first state, the updated first decision value and the updated second decision value are used as training samples in a training sample set for training and updating the network.

9. The method according to claim 8, further comprising: Training the updated network using the training samples; as well as Using the trained parameters of the update network as the parameters of the target network, The target network is used to determine the first prediction decision value and the second prediction decision value.

10. The method according to claim 1, wherein the first status comprises the number of the files, the read penalty associated with the merge operation, and the amount of data in the files within the first time interval.

11. The method according to claim 1, wherein the file to be operated comprises a sample file obtained by sampling files in the hard disk according to a first ratio, and the method further comprises: For the file to be operated, determining an average revenue of at least one time window within a time interval; as well as In response to the average revenue being less than a revenue threshold, the sample file is kept as the file to be operated.

12. The method according to claim 11, further comprising: In response to the average revenue being greater than a revenue threshold, sampling the files in the hard disk according to a second ratio to obtain a sample file as the file to be operated, The second ratio is greater than the first ratio.

13. The method according to claim 1, further comprising: A plurality of virtual partitions are obtained in the memory, wherein the file to be operated is mapped to at least one virtual partition among the plurality of virtual partitions.

14. The method of claim 13, wherein obtaining a plurality of virtual partitions in memory comprises: storing the received data in the form of a memory file on a file layer of the memory; Determining that the data volume of the data on the file layer exceeds a data volume threshold; The file layer is partitioned to obtain the plurality of virtual partitions on the file layer.

15. An electronic device, comprising: at least one processing unit; At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to any one of claims 1 to 14.

16. A computer-readable storage medium having a computer program stored thereon, wherein the program implements the method according to any one of claims 1 to 14 when executed by a processor.

17. A computer program product having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Citation Information

Patent Citations

  • Dynamically adaptive LSM (Log-structured merge) tree combination method and system

    CN105159915A

  • Grading storage method and device based on LSM-tree, OLAP database system and medium

    CN108021702A

  • Self-adaptive data merging and storing method for data origin system

    CN110321349A

  • File merging method and device based on cassandra database

    CN111881092A

  • Operation method for file, electronic equipment and storage medium

    CN116610640A