Data storage method, storage medium, electronic device, and program product

Abstract

The application discloses a data storage method, a storage medium, an electronic device and a program product, relates to the technical field of storage, and the method of the application is characterized in that the first storage path refers to saving target data into the cache of a storage controller, returning a data storage response, and writing the target data in the cache into a storage medium; the second storage path refers to directly writing the target data into the storage medium and returning the data storage response without saving the target data into the cache. The data volume of the target data and the current cache residual amount of the storage controller are taken as references, and the shortest response time length is taken as a target, so that the path of writing the target data into the storage medium can be dynamically selected from the first storage path and the second storage path. In this way, the target data can be written into the storage medium according to the optimal storage path at all times, and therefore, the problem of unstable response speed of the storage controller in some technologies can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of storage, and in particular to a data storage method, a storage medium, an electronic device, and a program product. BACKGROUND

[0002] A storage controller is connected between a hard disk and a server, and is used to manage the hard disk and data on the hard disk, and provide a data access interface for the server. For example, the storage controller can abstract the hard disk as a logical volume, and present the logical volume to the server. Meanwhile, the storage controller can also receive read / write instructions of the server, and perform read / write operations on the hard disk according to the read / write instructions. In this way, the underlying details of the hard disk can be shielded from the server. For another example, in the read / write process, the storage controller can improve the IO (Input / Output) response speed of the hard disk through mechanisms such as caching, pre-reading, and write merging.

[0003] At present, in some technologies, when the storage controller writes data to the hard disk based on the caching mechanism, the response speed to the server is not stable enough, and needs to be improved. SUMMARY

[0004] The present application provides a data storage method, a data storage device, an electronic device, a computer readable storage medium, and a computer program product, to at least solve the problem of unstable response speed of the storage controller in the related art.

[0005] The present application provides a data storage method, which comprises:

[0006] receiving a data write request, wherein the data write request comprises target data to be stored;

[0007] selecting a target storage path with the shortest response time from among a first storage path and a second storage path according to a data amount of the target data and a current cache amount of a storage controller, wherein the first storage path refers to saving the target data into the cache of the storage controller, returning a data storage response, and writing the target data in the cache into a storage medium, and the second storage path refers to directly writing the target data into the storage medium and returning a data storage response without saving the target data into the cache;

[0008] writing the target data into the storage medium according to the target storage path.

[0009] The present application also provides a data storage device, which comprises:

[0010] a data receiving module configured to receive a data write request, wherein the data write request comprises target data to be stored;

[0011] determining a target storage path with the shortest response time from the first storage path and the second storage path according to the data amount of the target data and the current cache amount of the storage controller, wherein the first storage path refers to returning a data storage response when the target data is saved into the cache of the storage controller, and writing the target data in the cache into the storage medium, and the second storage path refers to directly writing the target data into the storage medium and returning a data storage response without saving the target data into the cache;

[0012] writing the target data into the storage medium according to the target storage path.

[0013] The application further provides an electronic device, comprising a memory for storing a computer program, and a processor for executing the computer program to implement the steps of the data storage method.

[0014] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the data storage method.

[0015] The application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of the data storage method.

[0016] In the technical scheme of some embodiments of the application, the first storage path refers to returning a data storage response when the target data is saved into the cache of the storage controller, and writing the target data in the cache into the storage medium, and the second storage path refers to directly writing the target data into the storage medium and returning a data storage response without saving the target data into the cache. With the data amount of the target data and the current cache amount of the storage controller as the reference, and with the shortest response time as the target, the path of writing the target data into the storage medium can be dynamically selected from the first storage path and the second storage path. In this way, the deficiencies of the first storage path and the second storage path can be overcome, and the target data can be always written into the storage medium according to the optimal storage path, so that the problem of unstable response speed of the storage controller in some technologies can be solved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort.

[0018] Figure 1An architecture schematic diagram of a storage system in some technologies;

[0019] Figure 2 An architecture schematic diagram of a storage system in some other technologies;

[0020] Figure 3 An architecture schematic diagram of a storage system provided by some embodiments of the present application;

[0021] Figure 4 A flow schematic diagram of a data storage method provided by some embodiments of the present application;

[0022] Figure 5 An architecture schematic diagram of a storage system provided by some other embodiments of the present application;

[0023] Figure 6 A module schematic diagram of a data storage device provided by some embodiments of the present application;

[0024] Figure 7 A module schematic diagram of an electronic device provided by some embodiments of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, any other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0026] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0027] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0028] For reference Figure 1 An architecture schematic diagram of a storage system in some technologies; Figure 1In this configuration, storage media can include, but are not limited to, SSDs (Solid State Drives) and HDDs (Hard Disk Drives). The server and storage controller are connected via an external interface card supporting the FC (Fibre Channel) protocol. The storage controller and storage media are connected via an external interface card supporting the SAS (Serial Attached SCSI) protocol. A cache is configured within the storage controller.

[0029] based on Figure 1 In the architecture shown, when an application on the server generates a data write request, the server can send the data write request to the storage controller via an external card. The data write request can include data generated by the application that needs to be written to the storage medium. After receiving the data write request via the external card, the storage controller does not immediately return a response to the server. Instead, it saves the data in the data write request to a cache via line 1, then returns a response to the server via line 2, and finally writes the data to the storage medium asynchronously via line 3. Asynchronous writing means that after returning a response to the server, the data in the cache is written to the storage medium at an appropriate time (such as when the storage system's workload is low).

[0030] After data in the cache is written to the storage medium, it continues to be stored in the cache. This way, when an application on the server needs to read data, it can directly retrieve it from the cache, thus improving data retrieval speed. If, as the amount of data in the cache increases, cache space becomes insufficient, data that has already been written to the storage medium can be gradually deleted from the cache according to the first-in, first-out (FIFO) principle.

[0031] In addition to lines 1-3 mentioned above, the memory can also write data to the storage medium via lines 4 and 5. In lines 4 and 5, the storage controller bypasses the cache, directly writes the data to the storage medium, and then returns a response to the server. Since saving data to the cache takes microseconds and writing data to the storage medium takes milliseconds, writing data to the storage medium in the manner shown in lines 1-3, compared to lines 4 and 5, not only eliminates the application's awareness of the underlying physical latency of the storage system but also significantly improves the throughput of the storage system.

[0032] However, the beneficial effect of the data write mode shown in lines 1-3 is limited to when the cache has sufficient space. When the cache has insufficient space, the data in the data write request cannot be written into the cache in time, and the time for the server to return a response is delayed, and can even exceed the delay of lines 4 and 5. For example, assuming that when the storage controller receives the data write request, the cache has been fully occupied, and the data in the cache has not been written into the storage medium (i.e., the data in the cache cannot be deleted), the data in the IO request needs to wait in the queue until at least part of the cache space is released. If the waiting time exceeds the time required by lines 4 and 5, the delay of the server to return a response in lines 1-3 will exceed the delay of lines 4 and 5.

[0033] With reference to the foregoing description, the architecture of a storage system in some other technologies is shown in FIG. 1. Figure 2 As shown in FIG. 1, the architecture of the storage system includes a server and a storage controller. The server and the storage controller are connected through a peripheral card. Figure 2 As shown in FIG. 1, the architecture of the storage system includes a server and a storage controller. The server and the storage controller are connected through a peripheral card. Figure 1 Basically similar, the main difference is that the storage controller includes a primary storage controller and a backup storage controller. The backup storage controller is used to back up the cache data in the primary storage controller. When the primary storage controller is down, the cache data in the primary storage controller can be found from the cache of the backup storage controller. The cache capacity of the backup storage controller can be greater than or equal to the cache capacity of the primary storage controller. The primary storage controller and the backup storage controller are connected through the peripheral card.

[0034] Based on the architecture shown in FIG. 1, when the application in the server generates a data write request, the server can send the data write request to the primary storage controller through the peripheral card. After receiving the data write request, the primary storage controller can write the data in the request into the local cache through line 1, and then write the data into the cache of the backup storage controller through line 2. After successfully writing the data into the cache of the primary and backup storage controllers, the primary storage controller returns a response to the server through line 3, and then asynchronously writes the data in the cache into the storage medium through line 4. Figure 2 After writing the data into the storage medium, the data can continue to be saved in the cache of the primary and backup storage controllers. When the cache space is insufficient, the data that has been written into the storage medium can be deleted from the cache of the primary and backup storage controllers according to the first-in, first-out principle.

[0035]

[0036] ​When the application in the server needs to read data, if the main storage controller is not down, the server can send a data read request to the main storage controller. The main storage controller first checks whether the data to be read by the application exists in the local cache, if exists, returns the data in the cache to the server, if not, reads the data from the storage medium and returns it to the server. Conversely, if the main storage controller is down, the server can send a data read request to the backup storage controller. The backup storage controller first checks whether the data to be read by the application exists in the local cache, if exists, returns the data in the cache to the server, if not, reads the data from the storage medium and returns it to the server.

[0037] Compared with the storage system in Figure 1 , the storage system in Figure 2 includes a main backup storage controller, and the data security is higher, but when the cache margin is insufficient or the data write request volume is large, the delay problem of the main storage controller returning a response to the server will be more serious, and the reasons are as follows:

[0038] 1) The communication resources between the main backup storage controller are limited. When the main storage controller writes data into the cache of the backup storage controller, if the data write request is large, the communication resources between the main backup storage controller are easy to be exhausted, and then data transmission congestion, long data transmission time, etc. occur, so the time of the main storage controller returning a response to the server will be more delayed.

[0039] 2) There is a limit to the single data transmission volume between the main backup storage controller. The single data transmission volume refers to the maximum data volume allowed to be transmitted between the main storage controller and the backup storage controller in each interaction process. Limited by the single data transmission volume, when the main storage controller writes data into the cache of the backup storage controller, if the data volume is large, the data needs to be divided into multiple small data blocks and transmitted multiple times, that is, multiple interactions between the main backup storage controller are needed. Thus, the time of the main storage controller returning a response to the server will also be more delayed. For example, assuming that the single data transmission volume between the main backup storage controller is 32K. If the main storage controller needs to send 128K data to the backup storage controller, the data needs to be divided into 4 small data blocks of 32K, and then 4 interactions with the backup storage controller are needed to complete the data transmission. Too many data interactions will undoubtedly make the time of the main storage controller returning a response to the server more delayed.

[0040] Therefore, the present application first provides a new architecture of a storage system. In combination with Figure 3 , the architecture schematic diagram of the storage system provided by some embodiments of the present application is provided. Compared with Figure 1 , the architecture of the storage system provided by the present application has the following advantages: Figure 3The storage system shown in the figure adds a decision module to the storage controller. The decision module can be an embedded processor integrated in the storage controller, a dedicated system on chip, etc. When the server sends a data write request to the storage controller, the data write request will be sent to the decision module. The decision module can determine the way of writing data to the storage medium by executing the data storage method of the present application.

[0041] For reference Figure 4 The flowchart of the data storage method provided for some embodiments of the present application is shown. The data storage method can be applied to the decision module in Figure 3 , and includes the following steps:

[0042] Step S401, receiving a data write request, the data write request including target data to be stored.

[0043] Specifically, Figure 3 After the decision module in receives the data write request sent by the server, it can parse the data write request to obtain the target data to be stored and the data volume of the target data.

[0044] Step S402, according to the data volume of the target data and the current cache capacity of the storage controller, screening the target storage path with the shortest response time from the first storage path and the second storage path, wherein the first storage path refers to saving the target data to the cache of the storage controller, returning a data storage response, and writing the target data in the cache to the storage medium, and the second storage path refers to directly writing the target data to the storage medium and returning a data storage response without saving the target data to the cache.

[0045] Wherein, the cache capacity refers to the unused cache capacity in the storage controller. The response time refers to the time interval from the time when the server sends the data write request to the time when the server receives the response to the data write request.

[0046] When storing the target data according to the first storage path, only the target data needs to be written into the cache to return a response to the server, and then the target data is written into the storage medium in an asynchronous manner. When storing the target data according to the second storage path, the target data is directly written into the storage medium to return a response to the server, bypassing the cache.

[0047] For reference Figure 1According to the related description, when the target data is written into the storage medium according to the first storage path, the response time is affected by the data amount of the target data and the current cache amount of the storage controller. Specifically, when the cache amount is greater than the data amount of the target data (i.e., when the cache amount is sufficient), the target data does not need to be queued when being written into the cache, and the time consumption can be in the order of microseconds, so the response time can also be in the order of microseconds. When the cache amount is less than the data amount of the target data, if the cache amount is greater than 0 and the gap between the data amount of the target data and the cache amount is small, the waiting time for writing the target data into the cache can be short, so the response time can also be relatively short, such as still in the order of microseconds. If the cache amount is greater than 0 and the gap between the data amount of the target data and the cache amount is large, such as the data amount is 1G and the cache amount is 16K, a long waiting time can be needed to obtain free cache space matching the data amount of the target data. In this case, the waiting time for writing the target data into the cache and the response time can be long, such as in the order of seconds or even minutes. If the cache amount is 0, the size of the data amount of the target data needs to be determined, and further determination of the length of the data queue waiting to be written into the cache in the cache queue is needed. If there are many data waiting to be written into the cache before the target data, the waiting time for writing the target data into the cache can be long, so the response time can also be long, such as in the order of milliseconds or even seconds. If only a small amount of data is waiting to be written into the cache before the target data, the data amount of the target data can be further determined. If the data amount of the target data is small, the waiting time for writing the target data into the cache can be short, so the response time can also be relatively short. Conversely, if the data amount of the target data is large, the waiting time for writing the target data into the cache can be long, so the response time can also be relatively long.

[0048] When the target data is written into the storage medium according to the second storage path, the time consumption can be in the order of milliseconds, but the specific response time is affected by the data amount of the target data. Specifically, the larger the data amount of the target data, the longer the time consumption for writing the target data into the storage medium, so the response time can also be longer.

[0049] Based on the above description, before the target data is written into the storage medium, the response times of the first storage path and the second storage path can be estimated, and then the target data is written into the storage medium according to the storage path with the shortest response time. The following describes how to estimate the response times of the first storage path and the second storage path.

[0050] In the embodiment, for any target data amount, data of the target data amount can be written into the storage medium according to the first storage path, and a first correspondence relationship between the target data amount, the cache surplus and the response time is recorded, and data of the target data amount can be written into the storage medium according to the second storage path, and a second correspondence relationship between the target data amount and the response time is recorded. Based on the first correspondence relationship and the second correspondence relationship, the correspondence relationship between the target data amount and the cache surplus is obtained.

[0051] Specifically, before the storage controller is put into use, a plurality of groups of data can be prepared, and each group of data can include a plurality of data. In the same group of data, the data amounts of the data are the same. Different groups of data correspond to different data amounts. For example, the first group of data includes a plurality of 32K data, the second group of data includes a plurality of 64K data, and the third group of data includes a plurality of 128K data. This is sequentially extended.

[0052] According to the first storage path, each data in each group can be sequentially stored into the storage medium, and the cache surplus and the response time corresponding to each data are counted. According to the second storage path, each data in each group can be sequentially stored into the storage medium, and the response time corresponding to each data is counted.

[0053] For example, it is assumed that the first group of data includes a plurality of 32K data. The cache in the storage controller is emptied, the data in the first group of data is sequentially written into the storage medium according to the first storage path, and the cache surplus and the response time corresponding to each data are counted, and a group of statistical data corresponding to the first group of data under the first storage path can be obtained. By repeatedly performing the above operation, a plurality of groups of statistical data corresponding to the first group of data under the first storage path can be obtained. The plurality of response times corresponding to the same cache surplus in the plurality of groups of statistical data are averaged to obtain the final response time corresponding to the corresponding cache surplus. For example, it is assumed that the above operation is repeated twice. When the data in the first group of data is written into the cache for the first time, the response time corresponding to the cache surplus of 2.5G is 3 microseconds. After the cache is emptied, when the data in the first group of data is written into the cache for the second time, the response time corresponding to the cache surplus of 2.5G is 3.2 microseconds. The average of 3 microseconds and 3.2 microseconds is 3.1 microseconds. In this way, the following correspondence relationship can be obtained: 32K, 2.5G, 3.1 microseconds. The correspondence relationship indicates that when data with a data amount of 32K is saved according to the first storage path, if the current cache surplus is 2.5G, the response time is 3.1 microseconds.

[0054] According to similar principles, the cache in the storage controller can be emptied, data in the first group of data can be written into the storage medium in sequence according to the second storage path, and the response time corresponding to each data can be counted, so that a group of statistical data corresponding to the first group of data under the second storage path can be obtained. By repeating the above operation, a plurality of groups of statistical data corresponding to the first group of data under the second storage path can be obtained. By averaging the plurality of response times in the plurality of groups of statistical data, the final response time corresponding to saving data with a data amount of 32K according to the second storage path can be determined.

[0055] In this way, the above operations can be sequentially repeated, and a plurality of groups of statistical results corresponding to each group of data under the first storage path and the second storage path can be obtained. Based on the plurality of groups of statistical results, a first correspondence relationship between the data amount, the cache amount and the response time under the first storage path can be established, and a second correspondence relationship between the data amount and the response time under the second storage path can be established.

[0056] Based on the first correspondence relationship, the data amount of the target data and the current cache amount of the storage controller, the first response time when the target data is stored according to the first storage path can be determined, and based on the second correspondence relationship and the data amount of the target data, the second response time when the target data is stored according to the second storage path can be determined. By comparing the first response time and the second response time, the target storage path with the shortest response time can be screened from the first storage path and the second storage path.

[0057] Step S403: writing the target data into the storage medium according to the target storage path.

[0058] Specifically, the path for writing the target data into the storage medium can be dynamically selected from the first storage path and the second storage path. That is, when the cache amount is sufficient, the target data is preferentially written into the storage medium according to the first storage path. When the cache amount is insufficient or the data amount is large, the target data is preferentially written into the storage medium according to the second storage path. In this way, the target data can be written into the storage medium according to the optimal storage path at all times, and the response speed of the storage controller can be improved.

[0059] In summary, in the technical solutions of some embodiments of the present application, the first storage path refers to saving the target data into the cache of the storage controller, returning a data storage response, and writing the target data in the cache into the storage medium. The second storage path refers to directly writing the target data into the storage medium and returning a data storage response without saving the target data into the cache. With the data volume of the target data and the current cache amount of the storage controller as the reference, and with the shortest response time as the target, the path for writing the target data into the storage medium can be dynamically selected from the first storage path and the second storage path. In this way, the deficiencies of the first storage path and the second storage path can be overcome, and the target data can always be written into the storage medium according to the optimal storage path, so that the problem of unstable response speed of the storage controller in some technologies can be solved.

[0060] In some embodiments, the step S402 of screening the target storage path with the shortest response time from the first storage path and the second storage path according to the data volume of the target data and the current cache amount of the storage controller can include:

[0061] estimating the minimum cache amount required by the storage controller when the response time of the first storage path is less than the response time of the second storage path according to the data volume of the target data;

[0062] determining the target storage path based on the current cache amount of the storage controller and the minimum cache amount.

[0063] Specifically, in general cases, the response time of the first storage path should be less than the response time of the second storage path. However, as described above, in cases such as insufficient cache amount, the response time of the first storage path will gradually increase and even be greater than the response time of the second storage path. Therefore, the minimum cache amount can also be referred to as the turning point when the response time of the first storage path is greater than the response time of the second storage path.

[0064] Based on the first correspondence and the second correspondence obtained in step S402, the minimum cache amount corresponding to each data volume can be obtained. For example, assuming that in the second storage path, the response time corresponding to the data volume of 32K is 3 milliseconds. In the first storage path, the correspondence among the data volume of 32K, the cache amount, and the response time is as follows:

[0065] 32K, 500K, 3 microseconds;

[0066] 32K, 100K, 3 microseconds;

[0067] 32K, 30K, 3.5 microseconds;

[0068] 32K, 25K, 1 millisecond;

[0069] 32K, 20K, 3.1 ms;

[0070] 32K, 15K, 5 ms;

[0071] 32K, 10K, 8 ms;

[0072] It can be seen from the above correspondence that, in the case of a data amount of 32K, if the response time of the first storage path is to be less than that of the second storage path, the current cache surplus of the storage controller must be no less than 25K. If the current cache surplus of the storage controller is less than 25K, the response time of the first storage path will be greater than that of the second storage path. Therefore, the minimum cache surplus corresponding to a data amount of 32K can be 25K.

[0073] When the current cache surplus of the storage controller is not less than the minimum cache surplus, the response time of the first storage path is always less than or equal to that of the second storage path, in which case the first storage path can be used as the target storage path. When the current cache surplus of the storage controller is less than the minimum cache surplus, the response time of the first storage path is greater than that of the second storage path, in which case the second storage path can be used as the target storage path.

[0074] In this embodiment, the minimum cache surplus corresponding to each data amount can be obtained in advance, so that, when the target storage path with the shortest response time is selected, the target storage path can be selected based on the correspondence among the current cache surplus of the storage controller, the data amount of the target data, and the minimum cache surplus, and the amount of calculation is small.

[0075] For reference Figure 5 The architecture of the storage system provided by some embodiments of the present application is shown in the following figure. Figure 5 In the figure, the storage controller includes a primary storage controller and a backup storage controller. The backup storage controller is used to back up the cache data in the primary storage controller. The decision module is arranged in the primary storage controller. Figure 5The lines 1, 2, 3, 4 in the figure correspond to the first storage path, and the lines 5, 6 correspond to the second storage path. After receiving the data write request sent by the server, the decision module can find the minimum cache margin corresponding to the data volume of the target data, and can check the current cache margins of the main storage controller and the backup storage controller. In the main storage controller and the backup storage controller, if the current cache margin of any storage controller is less than the minimum cache margin corresponding to the data volume of the target data, the second storage path is taken as the target storage path. If the current cache margins of the main storage controller and the backup storage controller are both not less than the minimum cache margin, it indicates that according to the performance of the main storage controller and the backup storage controller before being put into use, the response time of the first storage path can be less than that of the second storage path. However, in the process of using the main storage controller and the backup storage controller, with the aging, wear and tear and other problems of the main storage controller and the backup storage controller, the maximum communication rate between the main storage controller and the backup storage controller may be affected, thereby causing the response time of the first storage path to be possibly greater than that of the second storage path.

[0076] In view of this, when the current cache margins of the main storage controller and the backup storage controller are both not less than the minimum cache margin, the target time required for transmitting the target data between the main storage controller and the backup storage controller can be determined according to the data volume of the target data. If the target time is less than a preset first theoretical landing time, the first storage path is taken as the target storage path, and if the target time is not less than the first theoretical landing time, the second storage path is taken as the target storage path, where the first theoretical landing time refers to the theoretical response time when data is written to the storage medium according to the second storage path. In specific implementation, based on the data volume of the target data, the response time corresponding to the data volume of the target data can be found in the statistical result corresponding to the second storage path in step S402, and the found response time is taken as the first theoretical landing time.

[0077] In some embodiments, in the case of taking the second storage path as the target storage path, the actual landing time when the target data is written to the storage medium can be recorded. Based on the actual landing time, the first theoretical landing time can be updated, that is, the statistical result corresponding to the second storage path in step S402 is updated. In this way, in the process of using the main storage controller and the backup storage controller, the above statistical result can be dynamically updated to ensure the accuracy of the statistical result, and thus the accuracy of the target storage path can be improved when the target storage path is determined based on the statistical result subsequently.

[0078] In some embodiments, at least one actual landing time can be recorded based on at least one data write request in a recent specified time period (such as the last 10 minutes), and the first theoretical landing time can be cumulatively updated based on the at least one actual landing time. The calculation formula when updating is shown in expression (1).

[0079]

[0080] wherein, is the updated first theoretical landing time, is the first theoretical landing time before updating (i.e. the first theoretical landing time in the statistical result), is the sum of at least one actual landing time recorded within the specified time, is the number of actual landing times recorded within the specified time, represents the average of landing times in the most recent specified time, represents the reference weight of the first theoretical landing time before updating on the updated first theoretical landing time in the process of updating the first theoretical landing time, represents the reference weight of the average of landing times in the most recent specified time on the updated first theoretical landing time in the process of updating the first theoretical landing time.

[0081] In some embodiments, the above-mentioned determination of the target time required for the main storage controller and the backup storage controller to transmit the target data can include:

[0082] determining the number of interactions required for the main storage controller and the backup storage controller in the process of transmitting the target data based on the data amount of the target data and a preset single data transmission amount, wherein the single data transmission amount refers to the maximum data amount allowed to be transmitted between the main storage controller and the backup storage controller in each interaction process;

[0083] obtaining a single theoretical interaction time between the main storage controller and the backup storage controller;

[0084] determining the target time based on the number of interactions and the single theoretical interaction time.

[0085] Specifically, the single data transmission amount is determined by the physical line between the main storage controller and the backup storage controller, the transmission protocol, etc., and this value can be a fixed value, so it can be preset in the decision module of the main storage controller. The single theoretical interaction time refers to the interaction time of the main storage controller and the backup storage controller, i.e. the transmission time required for a single data transmission between the main storage controller and the backup storage controller according to the single data transmission amount.

[0086] In some embodiments, the target time can be calculated based on expression (2).

[0087]

[0088] T1 = size / minsize + t_comm * ceil(size % minsize / minsize), size % minsize > 0? 1 : 0

[0089] Further, in some embodiments, the initial value of the single theoretical interaction time length can be obtained based on the following method:

[0090] According to the single data transmission amount, a plurality of data are transmitted between the primary storage controller and the backup storage controller, and the transmission time length of each data is counted;

[0091] The transmission time length counted is averaged to obtain the initial value of the single theoretical interaction time length.

[0092] Specifically, before the primary and backup storage controllers are put into use, a plurality of data can be transmitted between the primary storage controller and the backup storage controller according to the single data transmission amount, and the transmission time length of each data is averaged to obtain the initial value of the single theoretical interaction time length. For example, assuming that the single data transmission amount is 32K, a plurality of 32K data can be transmitted between the primary storage controller and the backup storage controller in turn, and then the transmission time length of each data is averaged to obtain the initial value of the single theoretical interaction time length.

[0093] With the aging of the primary and backup storage controllers and other problems, the single theoretical interaction time length can be dynamically changed. Therefore, in the case of taking the first storage path as the target storage path, the actual interaction time length of the primary storage controller and the backup storage controller in each interaction process can be recorded, and then the single theoretical interaction time length can be updated based on the actual interaction time length. In this way, the accuracy of the single theoretical interaction time length can be ensured, and then when the target storage path is determined according to the single theoretical interaction time length in the subsequent process, the accuracy of the target storage path can be ensured.

[0094] In some embodiments, at least one actual interaction time length can be recorded based on at least one data write request in a specified time length (such as the last 10 minutes), and the single theoretical interaction time length can be cumulatively updated based on the at least one actual interaction time length, and the calculation formula when updating is shown in expression (3).

[0095]

[0096] wherein, denotes the updated single theoretical interaction time length, denotes the single theoretical interaction time length before updating, denotes the sum of the actual interaction time lengths in the specified time length, an actual interaction time length average value in a specified time length, an actual interaction time length average value in a specified time length, a reference weight of the single theoretical interaction time length before the update to the single theoretical interaction time length after the update in the process of updating the single theoretical interaction time length, a reference weight of the actual interaction time length average value of the most recent specified time length to the single theoretical interaction time length after the update in the process of updating the single theoretical interaction time length.

[0097] In some embodiments, before the storage controller is shipped, the storage controller can be tested to obtain the correspondence between the data amount and the cache surplus, and then the correspondence between the data amount and the cache surplus can be taken as the performance parameter list of the storage controller, and the performance parameter list can be burned in the storage controller as the factory parameter of the storage controller. In this way, during the running process of the storage controller, the minimum cache surplus corresponding to the data amount of the target data can be directly determined based on the performance parameter list.

[0098] Further, considering that when writing data with a larger data amount (such as data larger than 1024K) in the storage medium, the response time of the first storage path is usually greater than the response time of the second storage path, that is, the target storage path corresponding to these data with a larger data amount can be fixed as the second storage path, therefore, the larger data amount can be excluded in the performance parameter list of the storage controller, so as to reduce the data amount of the performance parameter list. Similarly, when writing data with a smaller data amount (such as data smaller than 32K) in the storage medium, the response time of the first storage path is usually less than the response time of the second storage path, therefore, the smaller data amount can also be excluded in the performance parameter list of the storage controller.

[0099] Based on the above description, the minimum cache surplus required by the storage controller when the response time of the first storage path is less than the response time of the second storage path according to the data amount of the target data includes:

[0100] Obtaining the performance parameter list of the storage controller, the performance parameter list including a plurality of correspondences between different data amounts and cache surpluses;

[0101] If the data amount of the target data is in the performance parameter list, the cache surplus corresponding to the data amount of the target data is taken as the minimum cache surplus required by the storage controller.

[0102] If the data amount of the target data is not in the performance parameter list, the target storage path is determined according to the following logic:

[0103] If the data amount of the target data is greater than a first data amount threshold (e.g., 1024K), the second storage path is taken as the target storage path, and the first data amount threshold is greater than or equal to the maximum data amount in the performance parameter list;

[0104] If the data amount of the target data is less than a second data amount threshold (e.g., 32K), it is determined whether a preset second theoretical landing time length is greater than a preset third theoretical landing time length, the second theoretical landing time length is a theoretical response time length when the target data with the data amount less than the second data amount threshold is written into the storage medium according to the second storage path, the third theoretical landing time length is a theoretical response time length when the target data with the data amount less than the second data amount threshold is written into the storage medium according to the first storage path, and the second data amount threshold is less than or equal to the minimum data amount in the performance parameter list;

[0105] If the second theoretical landing time length is greater than the third theoretical landing time length, the first storage path is taken as the target storage path, and if the second theoretical landing time length is not greater than the third theoretical landing time length, the second storage path is taken as the target storage path.

[0106] Specifically, an initial value of the second theoretical landing time length can be greater than an initial value of the third theoretical landing time length. For example, the second theoretical landing time length can be 10 milliseconds, and the initial value of the third theoretical landing time length can be 1 millisecond. That is, the above-described condition that the response time length of the first storage path is generally less than the response time length of the second storage path when writing data with a small data amount (e.g., data less than 32K) into the storage medium is met. However, with the aging of the hard disk storage controller and other problems, the response time length of the first storage path (i.e., the third theoretical landing time length) when writing data with a small data amount (e.g., data less than 32K) into the storage medium can also be greater than the response time length of the second storage path (i.e., the second theoretical landing time length). In view of this, in the case where the data amount of the target data is less than the second data amount threshold, the actual landing time length when the target data is written into the storage medium can be recorded in the process of writing the target data into the storage medium. When the first storage path is taken as the target storage path, the third theoretical landing time length can be updated based on the actual landing time length, and when the second storage path is taken as the target storage path, the second theoretical landing time length is updated based on the actual landing time length.

[0107] Specifically, a plurality of first actual landing time lengths corresponding to the second theoretical landing time length and a plurality of second actual landing time lengths corresponding to the third theoretical landing time length can be recorded within a recent specified time length (e.g., the last 20 minutes). Based on the plurality of first actual landing time lengths, the second theoretical landing time length can be updated according to expression (4). Based on the plurality of second actual landing time lengths, the third theoretical landing time length can be updated according to expression (5).

[0108]

[0109] wherein, is the updated second theoretical landing time, is the second theoretical landing time before being updated, is the sum of at least one first actual landing time recorded within the specified time period, is the number of first actual landing times recorded within the specified time period, represents the average of the first landing times of the most recent specified time period, represents the reference weight of the second theoretical landing time before being updated to the updated second theoretical landing time in the process of updating the second theoretical landing time, represents the reference weight of the average of the first landing times of the most recent specified time period to the updated second theoretical landing time in the process of updating the second theoretical landing time.

[0110]

[0111] wherein, 1 is the updated third theoretical landing time, is the third theoretical landing time before being updated, is the sum of at least one second actual landing time recorded within the specified time period, is the number of second actual landing times recorded within the specified time period, represents the average of the second landing times of the most recent specified time period, represents the reference weight of the third theoretical landing time before being updated to the updated third theoretical landing time in the process of updating the third theoretical landing time, represents the reference weight of the average of the second landing times of the most recent specified time period to the updated third theoretical landing time in the process of updating the third theoretical landing time.

[0112] By updating the second theoretical landing time and the third theoretical landing time in real time, the data accuracy can be ensured, and thus the target storage path can be determined more accurately in the subsequent determination.

[0113] From the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware, but in many cases the former is a better embodiment.

[0114] The embodiments of the present application also provide a data storage device for a model training task, which is described in combination with Figure 6 , a module schematic diagram of the data storage device provided by some embodiments of the present application. Figure 6In some embodiments, the data storage device comprises:

[0115] The data receiving module 601 receives a data write request, the data write request comprising target data to be stored;

[0116] The path determining module 602 filters a target storage path with the shortest response time from among the first storage path and the second storage path according to the data volume of the target data and the current cache volume of the storage controller, wherein the first storage path refers to saving the target data in the cache of the storage controller, returning a data storage response, and writing the target data in the cache to the storage medium, and the second storage path refers to directly writing the target data to the storage medium and returning a data storage response without saving the target data in the cache.

[0117] The data writing module 603 writes the target data to the storage medium according to the target storage path.

[0118] In some embodiments, the path determining module 602 is configured to:

[0119] estimate the minimum cache volume required by the storage controller when the response time of the first storage path is less than that of the second storage path according to the data volume of the target data; and

[0120] determine the target storage path based on the current cache volume and the minimum cache volume of the storage controller.

[0121] In some embodiments, the storage controller comprises a primary storage controller and a backup storage controller, and the path determining module 602 is configured to:

[0122] if the current cache volume of any of the primary storage controller and the backup storage controller is less than the minimum cache volume, then the second storage path is selected as the target storage path.

[0123] In some embodiments, the storage controller comprises a primary storage controller and a backup storage controller, and the path determining module 602 is configured to:

[0124] if the current cache volume of any of the primary storage controller and the backup storage controller is less than the minimum cache volume, then the second storage path is selected as the target storage path.

[0125] if the target time is less than a first theoretical disk-falling time, then the first storage path is selected as the target storage path, and if the target time is not less than the first theoretical disk-falling time, then the second storage path is selected as the target storage path, wherein the first theoretical disk-falling time refers to the theoretical response time when writing data to the storage medium according to the second storage path.

[0126] In some embodiments, the path determination module 602 is configured to:

[0127] In the case of taking the second storage path as the target storage path, record the actual landing time length when the target data is written to the storage medium;

[0128] Update the first theoretical landing time length based on the actual landing time length.

[0129] In some embodiments, the path determination module 602 is configured to:

[0130] Determine the number of interactions required between the primary storage controller and the backup storage controller during the transmission of the target data based on the data volume of the target data and a preset single data transmission volume, wherein the single data transmission volume refers to the maximum data volume allowed to be transmitted between the primary storage controller and the backup storage controller during each interaction;

[0131] Obtain the single theoretical interaction time length between the primary storage controller and the backup storage controller;

[0132] Determine the target time length based on the number of interactions and the single theoretical interaction time length.

[0133] In some embodiments, the path determination module 602 is configured to:

[0134] In the case of taking the first storage path as the target storage path, record the actual interaction time length of the primary storage controller and the backup storage controller during each interaction;

[0135] Update the single theoretical interaction time length based on the actual interaction time length.

[0136] In some embodiments, the path determination module 602 obtains the single theoretical interaction time length based on the following method:

[0137] According to the single data transmission volume, transmit a plurality of data between the primary storage controller and the backup storage controller and count the transmission time length of each data;

[0138] Take the average of the counted transmission time lengths to obtain an initial value of the single theoretical interaction time length.

[0139] In some embodiments, the path determination module 602 is configured to:

[0140] Obtain a performance parameter list of the storage controller, the performance parameter list including a plurality of corresponding relationships between different data volumes and cache reserves;

[0141] If the data volume of the target data is in the performance parameter list, take the cache reserve corresponding to the data volume of the target data as the minimum cache reserve required by the storage controller.

[0142] In some embodiments, if the data amount of the target data is not in the performance parameter list, the path determination module 602 is configured to:

[0143] If the data amount of the target data is greater than a first data amount threshold, the second storage path is taken as the target storage path, the first data amount threshold is greater than or equal to the maximum data amount in the performance parameter list;

[0144] If the data amount of the target data is less than a second data amount threshold, it is determined whether a preset second theoretical landing time is greater than a preset third theoretical landing time, the second theoretical landing time refers to a theoretical response time when the target data with the data amount less than the second data amount threshold is written into the storage medium according to the second storage path, the third theoretical landing time refers to a theoretical response time when the target data with the data amount less than the second data amount threshold is written into the storage medium according to the first storage path, the second data amount threshold is less than or equal to the minimum data amount in the performance parameter list;

[0145] If the second theoretical landing time is greater than the third theoretical landing time, the first storage path is taken as the target storage path, if the second theoretical landing time is not greater than the third theoretical landing time, the second storage path is taken as the target storage path.

[0146] In some embodiments, the path determination module 602 is configured to:

[0147] In the case that the data amount of the target data is less than the second data amount threshold, an actual landing time when the target data is written into the storage medium is recorded;

[0148] When the first storage path is taken as the target storage path, the third theoretical landing time is updated based on the actual landing time, when the second storage path is taken as the target storage path, the second theoretical landing time is updated based on the actual landing time.

[0149] In some embodiments, for any target data amount in the performance parameter list, the path determination module 602 obtains the corresponding relationship between the target data amount and the cache surplus based on the following method:

[0150] According to the first storage path, data with a plurality of target data amounts is written into the storage medium, and a first corresponding relationship among the target data amount, the cache surplus and the response time is recorded;

[0151] According to the second storage path, data with a plurality of target data amounts is written into the storage medium, and a second corresponding relationship between the target data amount and the response time is recorded;

[0152] Based on the first corresponding relationship and the second corresponding relationship, the corresponding relationship between the target data amount and the cache surplus is obtained.

[0153] For reference Figure 7Embodiments of the present application also provide an electronic device, comprising a memory 10 and a processor 20, the memory 10 storing a computer program, and the processor 20 being configured to execute the computer program to perform the steps in any of the above state data saving method embodiments.

[0154] Embodiments of the present application also provide a computer readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above state data saving method embodiments when executed.

[0155] In an example embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing computer programs.

[0156] Embodiments of the present application also provide a computer program product comprising a computer program, wherein the computer program is executed by a processor to perform the steps in any of the above state data saving method embodiments.

[0157] Embodiments of the present application also provide another computer program product comprising a non-volatile computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to perform the steps in any of the above state data saving method embodiments.

[0158] Those skilled in the art will further appreciate that the functions implemented by each of the examples described herein can be implemented in hardware, software, or a combination of both, and that the disclosure encompasses software, firmware, and hardware implementations. The specification has been written with a focus on functionality to be implemented and the manner in which the described examples can be implemented, rather than specific steps taken to implement the described examples. Those skilled in the art will appreciate that many design choices and implementation approaches exist concerning specific implementation details including hardware and software implementations, and that the described examples can be implemented in a variety of ways.

[0159] The above introduces in detail a data storage method, a storage medium, an electronic device and a program product provided by the present application. The principles and implementation manners of the present application are described by applying specific examples, and the above example is only used to help understand the method of the present application and its core idea. It should be pointed out that, for ordinary skilled persons in the technical field, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A data storage method, characterized in that, The method includes: Receive a data write request, the data write request including the target data to be stored; Based on the amount of the target data and the current cache balance of the storage controller, the target storage path with the shortest response time is selected from the first storage path and the second storage path. The first storage path refers to returning a data storage response and writing the target data in the cache to the storage medium when the target data is saved to the cache of the storage controller. The second storage path refers to directly writing the target data to the storage medium and returning a data storage response when the target data is not saved to the cache. Write the target data into the storage medium according to the target storage path; The step of selecting the target storage path with the shortest response time from the first and second storage paths based on the data volume of the target data and the current cache availability of the storage controller includes: Based on the amount of the target data, the minimum cache space required by the storage controller when the response time of the first storage path is less than the response time of the second storage path is estimated. Based on the current cache balance and the minimum cache balance of the storage controller, the target storage path is determined. The storage controller includes a primary storage controller and a backup storage controller. And, determining the target storage path based on the current cache balance and the minimum cache balance of the storage controller includes: If the current cache balance of either the primary storage controller or the backup storage controller is less than the minimum cache balance, then the second storage path is used as the target storage path.

2. The method according to claim 1, characterized in that, The storage controller includes a primary storage controller and a backup storage controller; Determining the target storage path based on the current cache balance and the minimum cache balance of the storage controller includes: If the current cache balance of both the primary storage controller and the backup storage controller is not less than the minimum cache balance, then the target duration required to transmit the target data between the primary storage controller and the backup storage controller is determined based on the data volume of the target data. If the target duration is less than the preset first theoretical disk write duration, then the first storage path is used as the target storage path; if the target duration is not less than the first theoretical disk write duration, then the second storage path is used as the target storage path. The first theoretical disk write duration refers to the theoretical response time when writing data to the storage medium according to the second storage path.

3. The method according to claim 2, characterized in that, The method further includes: When the second storage path is used as the target storage path, the actual disk write time when the target data is written to the storage medium is recorded. Based on the actual drop time, update the first theoretical drop time.

4. The method according to claim 2, characterized in that, The determination of the target duration required for transmitting the target data between the primary storage controller and the backup storage controller includes: Based on the data volume of the target data and the preset single data transmission volume, the number of interactions required between the primary storage controller and the backup storage controller during the transmission of the target data is determined, wherein the single data transmission volume refers to the maximum amount of data that can be transmitted between the primary storage controller and the backup storage controller during each interaction. Obtain the theoretical duration of a single interaction between the primary storage controller and the backup storage controller; The target duration is determined based on the number of interactions and the theoretical duration of a single interaction.

5. The method according to claim 4, characterized in that, The method further includes: When the first storage path is used as the target storage path, the actual interaction time between the primary storage controller and the backup storage controller is recorded in each interaction process. The theoretical duration of a single interaction is updated based on the actual interaction duration.

6. The method according to claim 4, characterized in that, The initial value of the theoretical duration of a single interaction was obtained based on the following method: According to the single data transmission volume, multiple data are transmitted between the primary storage controller and the backup storage controller, and the transmission time of each data is counted. The average of the statistically obtained transmission durations is used to obtain the initial value of the theoretical duration of a single interaction.

7. The method according to claim 1, characterized in that, The minimum cache space required by the storage controller when the estimated response time of the first storage path is less than the response time of the second storage path based on the data volume of the target data includes: Obtain the performance parameter list of the storage controller, which includes multiple different correspondences between data volume and cache availability; If the amount of the target data is in the performance parameter list, then the cache space corresponding to the amount of the target data is used as the minimum cache space required by the storage controller.

8. The method according to claim 7, characterized in that, If the target data volume is not in the performance parameter list, the method further includes: If the amount of the target data is greater than the first data amount threshold, then the second storage path is used as the target storage path, and the first data amount threshold is greater than or equal to the maximum data amount in the performance parameter list. If the amount of the target data is less than the second data amount threshold, then it is determined whether the preset second theoretical disk persistence time is greater than the preset third theoretical disk persistence time. The second theoretical disk persistence time refers to the theoretical response time when the target data with a amount less than the second data amount threshold is written to the storage medium according to the second storage path. The third theoretical disk persistence time refers to the theoretical response time when the target data with a amount less than the second data amount threshold is written to the storage medium according to the first storage path. The second data amount threshold is less than or equal to the minimum data amount in the performance parameter list. If the second theoretical disk write time is greater than the third theoretical disk write time, then the first storage path is used as the target storage path; if the second theoretical disk write time is not greater than the third theoretical disk write time, then the second storage path is used as the target storage path.

9. The method according to claim 8, characterized in that, The method further includes: If the amount of the target data is less than the second data amount threshold, record the actual disk write time when writing the target data to the storage medium; When the first storage path is used as the target storage path, the third theoretical disk persistence time is updated based on the actual disk persistence time. When the second storage path is used as the target storage path, the second theoretical disk persistence time is updated based on the actual disk persistence time.

10. The method according to claim 7, characterized in that, For any target data volume in the performance parameter list, the correspondence between the target data volume and the cache remaining space is obtained based on the following method: According to the first storage path, write multiple target data amounts into the storage medium, and record the first correspondence between the target data amount, cache balance, and response time; According to the second storage path, write multiple target data volumes into the storage medium and record the second correspondence between the target data volume and the response time; Based on the first correspondence and the second correspondence, the correspondence between the target data volume and the cache balance is obtained.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 10.

12. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory being used to store a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 10.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 10.

Images