An object storage method, apparatus, device, and medium

Abstract

This invention relates to the field of object storage technology and discloses an object storage method, apparatus, device, and medium applied to an object storage gateway. The method includes: when a data processing request is received from a client, obtaining the client's historical load information and the storage platform's historical health information; processing the historical load information and historical health information to determine the target asynchronous I / O load corresponding to the current data processing request; if the data processing request is a data upload request, determining the number of asynchronous I / O channels and the data block size corresponding to the data upload request based on the target asynchronous I / O load; and segmenting and uploading the upload object corresponding to the data upload request based on the data block size and the number of asynchronous I / O channels. This invention determines the current asynchronous transmission load by combining historical data when a data upload request is received, thereby determining an appropriate data block size and the number of asynchronous I / O channels, ensuring transmission performance during data upload.

Description

Technical Field

[0001] This invention relates to the field of object storage technology, and more specifically to an object storage method, apparatus, device, and medium. Background Technology

[0002] The application scenarios involving large amounts of real-time data storage are becoming increasingly common, such as video surveillance and artificial intelligence. In these scenarios, the amount of data stored in video surveillance is experiencing explosive growth, leading to increasing challenges for video storage. How to adaptively scale video storage and how to meet the low latency and high load requirements of video surveillance and AI training are becoming increasingly urgent problems to solve. Therefore, the scheduling of data storage or download tasks by storage gateways is becoming increasingly important.

[0003] Currently, in object storage scenarios, storage gateways often determine the scheduling scheme and execute data upload or download based solely on pre-written scheduling rules, failing to fully utilize load capacity and resulting in poor data storage processing efficiency. Summary of the Invention

[0004] In view of this, the present invention provides an object storage method, apparatus, device and medium to solve the problem of poor processing efficiency during data storage due to the inability to fully utilize load capacity.

[0005] In a first aspect, the present invention provides an object storage method applied to an object storage gateway, the method comprising:

[0006] When a data processing request is received from a client, the client's historical load information and the storage platform's historical health information are obtained.

[0007] The historical load information and the historical health information are processed to determine the target asynchronous I / O load corresponding to the current data processing request;

[0008] If the data processing request is a data upload request, then the number of asynchronous I / O channels and the data block size corresponding to the data upload request are determined based on the target asynchronous I / O load;

[0009] Based on the data block size and the number of asynchronous I / O channels, the upload object corresponding to the data upload request is segmented and uploaded in blocks.

[0010] The method provided in this aspect determines the asynchronous I / O load corresponding to the current data processing request based on the client's historical load information and the storage platform's historical health information when receiving a data processing request from the client. Then, when the data processing request is an upload request, the method combines the asynchronous I / O load determined based on the historical situation to determine the data block size and the number of channels corresponding to the upload request in order to perform data upload. This can improve the performance of the storage gateway when uploading data and ensure data processing efficiency.

[0011] In one optional implementation, if the data processing request is a data download request, then the data block size of the download object corresponding to the data download request within the storage platform is determined;

[0012] Based on the target asynchronous I / O load and the data block size of the download object, determine the number of asynchronous I / O channels corresponding to the data download request;

[0013] Based on the number of asynchronous I / O channels, the data block of the download object corresponding to the data download request is sent to the client.

[0014] In this implementation, when the data processing request is a data download request, the number of channels for asynchronous transmission is determined based on the size of the data block corresponding to the download request in the storage platform and the determined asynchronous I / O load. Then, the data block of the download object is sent to the client through the determined number of channels, thus ensuring the transmission performance during the data download process.

[0015] In one optional implementation, processing the historical load information and the historical health information to determine the target asynchronous I / O load allocated to the current data processing request includes:

[0016] The historical load information is processed using a first neural network model to obtain the first load information corresponding to the current time period of the client.

[0017] The historical health information is processed using a second neural network model to obtain the first health information corresponding to the current time period on the storage platform;

[0018] Based on the first load information and the first health information, the target asynchronous I / O load corresponding to this data processing request is determined.

[0019] This implementation uses a neural network model to predict the load information of the client and the health information of the storage platform at the current time period, thereby determining the asynchronous I / O load of the current data processing request. This ensures the accuracy of determining the asynchronous I / O load, and in turn, ensures that the number of channels determined subsequently matches the current actual situation, thus guaranteeing the data processing effect.

[0020] In one optional implementation, the first health information includes: average I / O information and disk status information corresponding to each disk group in the storage platform;

[0021] The step of determining the target asynchronous I / O load corresponding to this data processing request based on the first load information and the first health information includes:

[0022] Obtain the preset weights corresponding to each disk group, and determine the second load information corresponding to the storage platform based on the preset weights, average I / O information, and disk status information corresponding to each disk group.

[0023] The target asynchronous I / O load is determined by substituting the first load information and the second load information into a preset limit formula.

[0024] In this implementation, the load information of the storage platform is determined by the status information and average I / O information of each disk group in the storage platform. Then, by combining the load information of the client and the preset limit formula, the asynchronous I / O load that conforms to the actual situation can be accurately obtained, ensuring the accuracy of load calculation.

[0025] In one optional implementation, the first neural network model is a first long short-term memory network model trained on a first training dataset, and the second neural network model is a second long short-term memory network model trained on a second training dataset.

[0026] The step of processing the historical load information using a first neural network model to obtain the first load information corresponding to the current time period of the client includes:

[0027] The historical load information is divided into time periods to obtain load information corresponding to multiple preset time periods, and then input into the first long short-term memory network model to obtain the first load information corresponding to the current time period;

[0028] The process of using a second neural network model to process the historical health information to obtain the first health information corresponding to the current time period on the storage platform includes:

[0029] The historical health information is divided into time periods to obtain health information corresponding to multiple preset time periods. This information is then input into the second long short-term memory network model to obtain the first health information corresponding to the current time period.

[0030] In this implementation, a long short-term memory network is trained using a dedicated training dataset to obtain neural network models corresponding to the client's load information and the storage platform's health information. Then, historical health information and historical load information are input into the corresponding long short-term memory network models to ensure the accuracy of the first load information and first health information corresponding to the current time period.

[0031] In an optional implementation, the method further includes:

[0032] Determine the metadata corresponding to the starting and ending data blocks of the uploaded object during the chunked upload process;

[0033] Based on the metadata corresponding to the starting data block and the ending data block, as well as the data block size, the metadata corresponding to the uploaded object is determined and recorded in the metadata management module of the storage gateway.

[0034] In this implementation, after data is uploaded, the metadata of the uploaded object is determined and recorded in the metadata management module based only on the metadata of the starting and ending data blocks and the size of the data block division. This ensures the simplicity of the metadata and avoids excessive metadata volume.

[0035] In one optional implementation, sending the data block of the download object corresponding to the data download request to the client based on the number of asynchronous I / O channels includes:

[0036] Obtain the metadata of the downloaded object, and determine the metadata of each data block of the downloaded object based on the metadata of the downloaded object;

[0037] Based on the metadata of each data block, the location and sending order of each data block of the download object in the storage platform are determined;

[0038] Based on the number of asynchronous I / O channels and the position and sending order of each data block, the data blocks of the download object in the storage platform are sent to the client.

[0039] In this implementation, the location and sending order of each data block of the download object are determined by the metadata corresponding to the download object, thus ensuring the accuracy of the data download process.

[0040] In a second aspect, the present invention provides an object storage device applied to an object storage gateway, the device comprising:

[0041] The request receiving module is used to obtain the client's historical load information and the storage platform's historical health information when it receives a data processing request from the client.

[0042] The load determination module is used to process the historical load information and the historical health information to determine the target asynchronous I / O load corresponding to the current data processing request;

[0043] The request discrimination module is used to determine the number of asynchronous I / O channels and the data block size corresponding to the data upload request based on the target asynchronous I / O load if the data processing request is a data upload request.

[0044] The data processing module is used to slice and upload the upload object corresponding to the data upload request based on the data block size and the number of asynchronous I / O channels.

[0045] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the object storage method described in the first aspect or any corresponding embodiment thereof.

[0046] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions that cause a computer to perform the object storage method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0047] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating an object storage method according to an embodiment of the present invention;

[0049] Figure 2 This is a flowchart illustrating another object storage method according to an embodiment of the present invention;

[0050] Figure 3 This is an example diagram of an object storage system architecture according to an embodiment of the present invention;

[0051] Figure 4 This is a flowchart illustrating a client-side object upload process according to an embodiment of the present invention.

[0052] Figure 5 This is a flowchart illustrating a client downloading an object according to an embodiment of the present invention;

[0053] Figure 6 This is an example diagram of the data processing flow of an adaptive scheduling module according to an embodiment of the present invention;

[0054] Figure 7 This is an example diagram of the architecture of an LSTM model according to an embodiment of the present invention;

[0055] Figure 8 This is an example diagram of a data reading process for a storage platform according to an embodiment of the present invention;

[0056] Figure 9 This is an example diagram of a data writing process for a storage platform according to an embodiment of the present invention;

[0057] Figure 10 This is a structural block diagram of an object storage device according to an embodiment of the present invention;

[0058] Figure 11 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The application scenarios involving large amounts of real-time data storage are becoming increasingly common, such as video surveillance and artificial intelligence. In these scenarios, the amount of data stored in video surveillance is experiencing explosive growth, leading to increasing challenges for video storage. How to adaptively scale video storage and how to meet the low latency and high load requirements of video surveillance and AI training are becoming increasingly urgent problems to solve. Therefore, the scheduling of data storage or download tasks by storage gateways is becoming increasingly important.

[0061] Currently, in object storage scenarios, storage gateways often determine the scheduling scheme and execute data upload or download based solely on pre-written scheduling rules, failing to fully utilize load capacity and resulting in poor data storage processing efficiency.

[0062] To address this, this invention provides an object storage method. Upon receiving a data processing request from a client, the method determines the asynchronous I / O load corresponding to the current data processing request based on the client's historical load information and the storage platform's historical health information. Then, when the data processing request is an upload request, the method combines the asynchronous I / O load determined based on historical data to determine the data block size and the number of channels corresponding to the upload request for data upload. This improves the performance of the storage gateway during data upload and ensures data processing efficiency.

[0063] According to an embodiment of the present invention, an object storage method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0064] This embodiment provides an object storage method applied to an object storage gateway. Figure 1 This is a flowchart of an object storage method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0065] Step S101: When a data processing request is received from the client, obtain the client's historical load information and the storage platform's historical health information.

[0066] For object storage procedures, there are three main components: the client initiating the data upload / download request, the storage gateway executing the data transmission, and the storage platform storing the actual data. In the storage gateway involved in this method embodiment, it receives a data processing request from the client, indicating that the client requests to upload or download a specific object. After receiving the data processing request, the storage gateway needs to determine the asynchronous I / O load corresponding to this data processing for data transmission. To determine the most reasonable load allocation, historical information from both the client and the storage platform can be used.

[0067] Therefore, it is necessary to obtain historical load information for the client, which refers to data records related to the workload undertaken by the client during operation over a past period. This includes historical information that characterizes the client's load capacity, such as disk I / O (input / output) operation frequency and data volume, memory usage, and network bandwidth usage. For the storage platform's historical health information, this can include historical status information for each disk group within the storage platform, historical average I / O information, and other historical information that characterizes the storage platform's load capacity.

[0068] Step S102: Process the historical load information and historical health information to determine the target asynchronous I / O load corresponding to the current data processing request.

[0069] For storage gateways, there is a dedicated scheduling module that analyzes the historical load information of clients and the historical health information of storage platforms to predict the expected load of clients and the expected health status of storage platforms during the time period corresponding to the current data processing request. By combining the limiting relationship between the asynchronous I / O load of storage gateways, the load of clients, and the load of storage platforms, the expected range of asynchronous I / O load during the time period corresponding to the current data processing request can be obtained.

[0070] The optimal target asynchronous I / O load is determined within the expected range based on specific rules. For example, the median value of the expected range of asynchronous I / O load can be selected as the target asynchronous I / O load for this data processing request, or the load value corresponding to a specific quantile value within the expected range can be selected as the target asynchronous I / O load. There are no restrictions on how to determine the load value within the expected range, and it can be set according to the actual situation.

[0071] Step S103: If the data processing request is a data upload request, then determine the number of asynchronous I / O channels and the data block size corresponding to the data upload request based on the target asynchronous I / O load.

[0072] For object storage procedures, the storage gateway needs to divide the objects uploaded by the client into blocks, and then upload the divided data blocks one by one to the storage platform through asynchronous I / O channels. Therefore, it is necessary to determine the specific number of asynchronous I / O channels and the size of the data blocks in this data upload process.

[0073] During data upload, multiple I / O channels can be understood as each channel being responsible for uploading its assigned data block to the storage platform one by one, with these I / O channels operating concurrently. Determining the target asynchronous I / O load in the above steps can be understood as the load capacity allocated by the storage gateway for this data processing request, capable of concurrent uploads. Simply put, the product of the number of asynchronous I / O channels the storage gateway can ultimately allocate and the data block size corresponds to the target asynchronous I / O load. Therefore, after determining the asynchronous I / O load, the specific number of I / O channels and the data block size can be determined according to preset rules, as long as their product corresponds to the target asynchronous I / O load.

[0074] For example, the number of channels corresponding to different object size ranges can be pre-defined. The number of asynchronous I / O channels is determined based on the specific size of the uploaded object. Then, the data block size is determined by dividing the target asynchronous I / O load by the corresponding number of channels. Alternatively, referring to the example above, the data block size for different object size ranges can be pre-defined. Then, the number of asynchronous I / O channels is obtained by dividing the target asynchronous I / O load by the corresponding data block size. The above are just example implementations. The specific method for determining the number of channels and the data block size is not limited here, as long as their product corresponds to the target asynchronous I / O load.

[0075] Step S104: Based on the data block size and the number of asynchronous I / O channels, the upload object corresponding to the data upload request is segmented and uploaded in blocks.

[0076] After determining the data block size and the number of asynchronous I / O channels, the uploaded object can be divided into blocks according to the data block size. For example, if the data block size is 5MB and the uploaded object is 100MB, it can be divided into 20 data blocks. At the same time, if the number of asynchronous I / O channels is determined to be 5, these 20 data blocks can be sent to each channel in the corresponding order, so that each channel transmits the allocated data blocks to the storage platform one by one.

[0077] The object storage method provided in this embodiment determines the asynchronous I / O load corresponding to the current data processing request based on the client's historical load information and the storage platform's historical health information when receiving a data processing request from the client. Then, when the data processing request is an upload request, the method combines the asynchronous I / O load determined based on the historical situation to determine the data block size and the number of channels corresponding to the upload request for data upload, thereby improving the performance of the storage gateway when uploading data and ensuring data processing efficiency.

[0078] According to an embodiment of the present invention, another object storage method embodiment is provided, applied to an object storage gateway. Figure 2 This is a flowchart of another object storage method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0079] Step S201: When a data processing request is received from a client, obtain the client's historical load information and the storage platform's historical health information. For detailed implementation details, refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0080] Step S202: Process the historical load information and historical health information to determine the target asynchronous I / O load corresponding to the current data processing request.

[0081] Specifically, step S202 includes:

[0082] Step S202-1: Process the historical load information using the first neural network model to obtain the first load information corresponding to the current time period of the client.

[0083] This can be understood as follows: a pre-trained first neural network model is pre-configured in the storage gateway's scheduling module, specifically designed to predict client load information. After obtaining historical client load information, this historical information can be input into the neural network model to predict the client load information for the current time period.

[0084] Specifically, the first neural network model is a first long short-term memory network model trained on the first training dataset.

[0085] Long Short-Term Memory (LSTM) network models are relatively good at time series prediction because they can effectively process long-term data and remember long-term information. Therefore, they can be trained using training data related to client load information to obtain a first LSTM network model for predicting client load information.

[0086] Step S202-1 includes:

[0087] The historical load information is divided into time periods to obtain the load information corresponding to each of the multiple preset time periods. The load information is then input into the first long short-term memory network model to obtain the first load information corresponding to the current time period.

[0088] Based on the above explanation of Long Short-Term Memory (LSTM) networks, it can be understood that they are mainly used for analyzing and predicting time series data. Therefore, it is necessary to divide the client's historical load information into time periods to obtain the client's load information under multiple historical time periods. For example, the client's historical load information can be divided by hour to obtain the historical load information corresponding to each of the past several hours. After completing the time period division, the data is input into the first LTM network model to obtain the load information corresponding to the current time period.

[0089] Step S202-2: The historical health information is processed using the second neural network model to obtain the first health information corresponding to the current time period on the storage platform.

[0090] As mentioned above, a pre-trained second neural network model is also pre-configured in the storage gateway's scheduling module, specifically for predicting the load information of the storage platform. After obtaining the historical health information of the storage platform, this historical health information can be input into the trained second neural network model to predict the health information of the storage platform for the current time period.

[0091] Specifically, the second neural network model is a second long short-term memory network model based on the second training dataset.

[0092] Similarly, a first long short-term memory network model for predicting client load information can be obtained by training it with training data related to the health information of the storage platform, i.e., a second training dataset.

[0093] Step S202-2 includes:

[0094] Historical health information is divided into time periods to obtain health information corresponding to multiple preset time periods. This information is then input into a second long short-term memory network model to obtain the first health information corresponding to the current time period.

[0095] Referring to step S202-1 above, the historical health information of the storage platform can also be divided into multiple time periods, yielding health information for each period. It's important to note that due to differences between the storage platform and the client, the length of each time period may differ. For example, historical information on the client can be divided by hour or minutes, while health information on the storage platform can be divided by day or hours. After dividing the historical health information, it can be input into the second Long Short-Term Memory (LSTM) network model to obtain the first health information corresponding to the current time period.

[0096] For example, the first and second long short-term memory network models mentioned above can be integrated into one model. By directly inputting the historical information corresponding to the client and the storage platform, the prediction information for the current time period can be obtained.

[0097] Step S202-3: Based on the first load information and the first health information, determine the target asynchronous I / O load corresponding to this data processing request.

[0098] Specifically, the first health information includes: average I / O information and disk status information for each disk group in the storage platform.

[0099] This can be understood as follows: for a storage platform, which has multiple disk groups, the predicted health information of the storage platform in the current time period, i.e., the first health information, can include the average I / O information and disk status information of each disk group in the current time period.

[0100] Step S202-3 above includes:

[0101] Step S202-3-1: Obtain the preset weights corresponding to each disk group, and determine the second load information corresponding to the storage platform based on the preset weights, average I / O information, and disk status information corresponding to each disk group.

[0102] For a storage platform, its current load capacity is mainly related to the disk status information and average I / O information. Furthermore, the weights of each disk can be pre-set based on the actual conditions of the storage platform. By multiplying and summing the status information, average I / O information, and their respective weights, the load of the storage platform in the current time period can be obtained, i.e., the second load information.

[0103] For example, suppose the storage platform has n hard disk groups, and the weight ratio of these n hard disk groups is: (W0, W1, W2, ..., W...). n-1 The predicted health status parameters for these n hard disk groups are: (H0, H1, H2..., H...). n-1 ), and the average I / O of these n disk groups: (I0, I1, I2..., I n-1 The relationship between the total load of each hard disk group, i.e., the second load information, Total_disk_load, and the above information is as follows:

[0104]

[0105] Step S202-3-2: Substitute the first load information and the second load information into the preset limit formula to determine the target asynchronous I / O load.

[0106] Assuming the asynchronous I / O load is Total_aio_load, and the client load, i.e., the first load information, is Total_user_load, then the constraint relationship among Total_aio_load, Total_user_load, and Total_disk_load is as follows:

[0107] Total_user_load<Total_aio_load*weight<Total_disk_load

[0108] The weight parameter is set based on experience and can be adjusted according to the actual situation.

[0109] Based on the above formula, the range of asynchronous I / O load Total_aio_load can be obtained. Within this range, the corresponding target asynchronous I / O load can be determined. For example, a specific quantile, such as 1 / 2 or 1 / 4, can be selected as the target asynchronous I / O load. The specific determination method can be configured according to the actual situation.

[0110] Step S203: If the data processing request is a data upload request, then determine the number of asynchronous I / O channels and the data block size corresponding to the data upload request based on the target asynchronous I / O load. For detailed implementation details, refer to [link to implementation details]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0111] Step S204: Based on the data block size and the number of asynchronous I / O channels, the upload object corresponding to the data upload request is segmented and uploaded in blocks. For detailed implementation details, please refer to [link / reference]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0112] Furthermore, after segmenting and uploading the upload object corresponding to the data upload request, the method also includes:

[0113] Determine the metadata corresponding to the starting and ending data blocks of the uploaded object during the chunked upload process;

[0114] Based on the metadata corresponding to the starting and ending data blocks and the size of the data block division, the metadata corresponding to the uploaded object is determined and recorded in the metadata management module of the storage gateway.

[0115] This can be understood as follows: for object storage, after data blocks are partitioned and uploaded, it is necessary to record the metadata corresponding to each data block during the data block upload process, such as offset and data block name, so as to facilitate subsequent reading of the corresponding data blocks. Since the size of each data block is the same after data block partitioning in the above process, the metadata of the first starting data block, that is, the data block name and offset information corresponding to the first uploaded data block, and the metadata of the last data block, that is, the data block name and offset information corresponding to the last uploaded data block, can be recorded. At the same time, the metadata corresponding to these two data blocks and the data block partition size are used as the metadata corresponding to the uploaded object.

[0116] When the uploaded object needs to be downloaded later, the name and offset of each data block can be calculated based on the metadata of the starting and ending data blocks and the size of each data block. This allows the metadata of each data block to be determined, making it easier to find and download the corresponding data block in the storage platform.

[0117] Step S205: If the data processing request is a data download request, then determine the size of the data block of the download object corresponding to the data download request within the storage platform.

[0118] This can be understood as follows: when a data processing request is a data download request, the downloaded object is a pre-divided data block in the storage platform, meaning the size of the data block is already determined. Therefore, when a data processing request is a download request, the size of the corresponding data block in the storage platform can be determined through the metadata of the downloaded object in the storage gateway.

[0119] Step S206: Determine the number of asynchronous I / O channels corresponding to the data download request based on the target asynchronous I / O load and the data block size of the download object.

[0120] Since the target asynchronous I / O load has been determined in the steps above, the number of asynchronous I / O channels allocated for this data download request can be determined based on the specific data block size. For example, assuming the data block size is 5MB and the target asynchronous I / O load is 30MB, then 6 asynchronous I / O channels can be allocated for data download.

[0121] Step S207: Based on the number of asynchronous I / O channels, send the data block of the download object corresponding to the data download request to the client.

[0122] Once the number of asynchronous I / O channels is determined, the data blocks of the downloadable object can be retrieved from the storage platform and sent to the client through the asynchronous I / O channels.

[0123] Specifically, in step S207, based on the number of asynchronous I / O channels, the data block of the download object corresponding to the data download request is sent to the client, including:

[0124] Step S207-1: Obtain the metadata of the download object, and determine the metadata of each data block of the download object based on the metadata of the download object.

[0125] This can be understood as follows: when downloading data, it is necessary to locate the position and transmission order of each data block in the storage platform in order to transmit data based on the I / O channel. Therefore, it is necessary to determine the metadata of each data block of the download object in order to locate and send the subsequent data blocks.

[0126] Step S207-2: Based on the metadata of each data block, determine the location and sending order of each data block of the downloaded object in the storage platform.

[0127] The metadata records the offsets corresponding to each data block. These offsets include the offset relative to the storage location and the offset of each data block relative to the starting object of the download object, i.e., which data block it belongs to within the download object. These specific offsets determine the position of each data block in the storage platform and the subsequent sending order. For details on how to determine the position and sending order, please refer to the relevant background knowledge on metadata in object storage; it will not be elaborated upon here.

[0128] It should be noted that steps S207-1 and S207-2 above can be executed immediately after receiving the data download request, and their execution order does not need to be restricted to after the target asynchronous I / O load is determined.

[0129] Step S207-3: Based on the number of asynchronous I / O channels and the location and sending order of each data block, send the data blocks of the downloaded object in the storage platform to the client.

[0130] Several asynchronous I / O channels are created based on the number of asynchronous I / O channels. After determining the location of each data block, each data block can be located. At the same time, combined with the corresponding sending order, the data blocks are sent to each asynchronous I / O channel in sequence to send the data blocks to the client, thereby realizing the download of the download object.

[0131] The object storage method provided in this embodiment of the invention determines the asynchronous I / O load corresponding to the current data processing request based on historical information when receiving a data processing request from a client. When the data processing request is an upload request, the method determines the data block size and the number of channels corresponding to the upload request based on the asynchronous I / O load determined based on the historical information. When the data processing request is a data download request, the method determines the number of channels for asynchronous transmission based on the size of the data block corresponding to the download request within the storage platform and the determined asynchronous I / O load. This ensures the performance of the storage gateway during the data processing process.

[0132] To facilitate understanding of the above method embodiments, an example diagram of an object storage system architecture is provided according to an embodiment of the present invention, such as... Figure 3 As shown, the system mainly consists of an object storage gateway and a new storage platform. The object storage gateway includes a data segmentation and asynchronous I / O module, a metadata management module, and an adaptive scheduling module.

[0133] In large object upload scenarios, issues such as high latency and low resource utilization arise. The data segmentation and asynchronous I / O modules divide large objects into smaller chunks according to the size specified by the adaptive scheduling module, and then use the given number of I / O channels in the adaptive scheduling to asynchronously send the data chunks to the new storage platform. In large object download scenarios, the given number of I / O channels in the adaptive scheduling module asynchronously downloads the data chunks of large objects from the storage platform to the client.

[0134] like Figure 4 The diagram shown is a flowchart illustrating a client-side object upload process according to an embodiment of the present invention. It mainly includes:

[0135] (a) When a client uploads a large object, the adaptive scheduling module in the object storage gateway determines the size of the smaller data blocks and the number of concurrent asynchronous I / O operations based on the client's historical data and the historical status of the new storage platform.

[0136] (b) Then the data slicing and asynchronous I / O module cuts large blocks of data into smaller blocks, and sends the smaller blocks of data to the storage platform for writing via asynchronous I / O.

[0137] (c) After all the data has been successfully written, write the names and offset information of these small data blocks into the metadata.

[0138] After the object upload process involves segmenting the data into chunks, it's necessary to record the names of these small chunks and their offsets within the overall large chunk. Recording the metadata for all these small chunks would lead to excessively large metadata when there are too many chunks, potentially affecting the writing of very large objects. The metadata management module in the aforementioned new storage gateway avoids recording the metadata for all small chunks. Instead, it records only the specified small chunks of data for this large object upload, generally of a fixed size, and then sequentially records the metadata for the first and last chunks. This allows for the regular calculation of the name, offset, and size of each chunk.

[0139] like Figure 5 The diagram shown is a flowchart illustrating a client downloading an object according to an embodiment of the present invention. It mainly includes:

[0140] (a) After receiving a request to download an object, the object storage gateway first reads the metadata from the database to obtain information such as the name and offset of the small block of data corresponding to the object.

[0141] (b) The adaptive scheduling module determines the asynchronous I / O concurrency size based on the predicted client load and the load of the new storage platform.

[0142] (c) The asynchronous I / O module sends these small data blocks, along with their names and offsets, to the new storage platform in sequence according to the asynchronous I / O concurrency size.

[0143] The adaptive scheduling module in the storage gateway is primarily used to predict the load at the next moment based on historical client load and the historical load of the new storage platform, using new machine learning algorithms. For the specific prediction process, please refer to [link / reference needed]. Figure 6 This is an example diagram of the data processing flow of an adaptive scheduling module according to an embodiment of the present invention.

[0144] First, acquire relevant data, including client load at different times, such as recording load at 10-minute intervals. Then, acquire the health status of each disk group on the storage platform, such as recording the health status of each disk at a 1-day interval.

[0145] Next, preprocessing is performed. After obtaining the client's historical load and the historical health status of each SSD, invalid data is cleaned up, such as data cleaning and homogenization.

[0146] Finally, prediction is performed using an LSTM (Long Short-Term Memory) network. The LSTM model is subjected to multi-parameter fitting and adaptive training so that it can perform specific prediction functions.

[0147] For specific details on the LSTM processing flow, please refer to [link / reference]. Figure 7 The diagram shown is an example architecture of an LSTM model according to an embodiment of the present invention. LSTM is a type of RNN. Compared to RNN, LSTM filters historical data, ensuring that key information is continuously passed on. It mainly consists of an input layer, an LSTM layer, and an output layer. The input layer takes historical monitoring data and feeds it into the LSTM. The LSTM filters this information and outputs it to the next LSTM. The output result is the prediction result.

[0148] Finally, based on the prediction results, the data block size and the number of asynchronous I / O channels are determined. For example, assume the storage platform has n hard disk groups, and the preset weight ratio of these n hard disk groups is: (W0, W1, W2, ..., W...). n-1 ).

[0149] The predicted storage platform health information includes: health status parameters (H0, H1, H2..., H...) for these n hard disk groups. n-1 The average I / O of these n disk groups: (I0, I1, I2..., I...) n-1 The predicted client load is: Total_user_load.

[0150] The size of the blocks: block_size, the number of AIO channels: aio_nums. The corresponding preset weight parameters for both are weight1 and weight2, respectively.

[0151] The range of the product of block_size and aio_nums can be determined by the following constraints.

[0152]

[0153] Then, the specific data block size and number of channels are determined based on the product range of block_size and aio_nums in order to upload the data.

[0154] Regarding the above Figure 3 The novel storage platform in the system includes components such as a proxy, a router, and various host SSDs and related processes. The proxy component communicates with the object storage gateway, handles protocol conversion, and manages data read / write operations. The router component determines routing information based on the disk group ratio determined by the object storage gateway's adaptive scheduling module when writing data to the novel storage platform. Specifically, it determines which pool and disk group the small data block should be written to, and updates the SSD group information after writing. Each host SSD starts its own related process. After each SSD writes data of the same block size, the proxy module returns the pool ID, SSD ID, and code ID to the object storage gateway. The next time data is read, the object storage gateway can retrieve the data based on this information.

[0155] For example, such as Figure 8 The diagram shown is an example of a data reading process for a storage platform according to an embodiment of the present invention, which mainly includes:

[0156] (a) The proxy receives an HTTP read request from the object storage gateway, including pool_id and blob_id.

[0157] (b) Then check the router module to see which hard drive group to read the data from.

[0158] (c) The proxy reads data from the corresponding hard disk group and returns it to the client.

[0159] For example, such as Figure 9 The diagram shown is an example of a data writing process for a storage platform according to an embodiment of the present invention, which mainly includes:

[0160] (a) The proxy receives an HTTP write request from the object storage gateway.

[0161] (b) In the router module, query which hard disk group the small block of data is written to.

[0162] (c) The proxy writes small blocks of data to the corresponding disk group, and at the same time, the proxy returns the disk group ID and blob ID to the object storage gateway.

[0163] By introducing the LSTM (Long Short-Term Memory) algorithm, historical client load and the historical health status of each hard drive on the new storage platform are collected as inputs to obtain prediction results, namely the client load and the health status parameters of each hard drive at the next moment. Then, mathematical modeling is performed on the performance of the object storage system to calculate the optimal chunk size and AIO size at the next moment, thereby ensuring that the storage gateway achieves the highest possible performance during data upload while meeting performance requirements.

[0164] This embodiment also provides an object storage device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0165] This embodiment provides an object storage device, such as Figure 10 As shown, it includes:

[0166] The request receiving module 401 is used to obtain the client's historical load information and the storage platform's historical health information when it receives a data processing request from the client.

[0167] The load determination module 402 is used to process historical load information and historical health information to determine the target asynchronous I / O load corresponding to the current data processing request.

[0168] The request discrimination module 403 is used to determine the number of asynchronous I / O channels and the data block size corresponding to the data upload request based on the target asynchronous I / O load if the data processing request is a data upload request.

[0169] The data processing module 404 is used to slice and upload the upload object corresponding to the data upload request based on the data block size and the number of asynchronous I / O channels.

[0170] In some optional implementations, when the data processing request is a data download request, the request discrimination module 403 is also used to determine the data block size of the download object corresponding to the data download request within the storage platform;

[0171] Based on the target asynchronous I / O load and the data block size of the downloaded object, determine the number of asynchronous I / O channels corresponding to the data download request;

[0172] The data processing module 404 is also used to send the data block of the download object corresponding to the data download request to the client based on the number of asynchronous I / O channels.

[0173] In some optional implementations, the load determination module 402, when processing historical load information and historical health information to determine the target asynchronous I / O load allocated to the current data processing request, includes:

[0174] The historical load information is processed using the first neural network model to obtain the first load information corresponding to the current time period of the client.

[0175] The historical health information is processed using a second neural network model to obtain the first health information corresponding to the current time period on the storage platform;

[0176] Based on the first load information and the first health information, the target asynchronous I / O load corresponding to this data processing request is determined.

[0177] In some optional implementations, the first health information includes: average I / O information and disk status information corresponding to each disk group in the storage platform;

[0178] The load determination module 402, when determining the target asynchronous I / O load corresponding to the current data processing request based on the first load information and the first health information, includes:

[0179] Obtain the preset weights corresponding to each disk group, and determine the second load information corresponding to the storage platform based on the preset weights, average I / O information, and disk status information corresponding to each disk group.

[0180] The first and second load information are substituted into a preset limit formula to determine the target asynchronous I / O load.

[0181] In some optional implementations, the first neural network model is a first long short-term memory network model trained on a first training dataset, and the second neural network model is a second long short-term memory network model trained on a second training dataset.

[0182] The load determination module 402, when processing historical load information using the first neural network model to obtain the first load information corresponding to the current time period of the client, includes:

[0183] The historical load information is divided into time periods to obtain the load information corresponding to each of the multiple preset time periods. The load information is then input into the first long short-term memory network model to obtain the first load information corresponding to the current time period.

[0184] The load determination module 402, when processing historical health information using the second neural network model to obtain the first health information corresponding to the current time period of the storage platform, includes:

[0185] Historical health information is divided into time periods to obtain health information corresponding to multiple preset time periods. This information is then input into a second long short-term memory network model to obtain the first health information corresponding to the current time period.

[0186] In some optional implementations, the apparatus is further configured to determine the metadata corresponding to the start and end data blocks of the uploaded object during the chunked upload process;

[0187] Based on the metadata corresponding to the starting and ending data blocks and the size of the data block division, the metadata corresponding to the uploaded object is determined and recorded in the metadata management module of the storage gateway.

[0188] In some optional implementations, when the data processing module 404 sends the data block of the download object corresponding to the data download request to the client based on the number of asynchronous I / O channels, it includes:

[0189] Obtain the metadata of the downloaded object, and determine the metadata of each data block of the downloaded object based on the metadata of the downloaded object;

[0190] Based on the metadata of each data block, determine the location and sending order of each data block of the downloaded object in the storage platform;

[0191] Based on the number of asynchronous I / O channels and the location and sending order of each data block, the data blocks of the downloaded object in the storage platform are sent to the client.

[0192] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0193] In this embodiment, the object storage device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0194] This invention also provides a computer device having the above-described features. Figure 10 The object storage device shown.

[0195] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 11As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 11 Take a processor 10 as an example.

[0196] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0197] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0198] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0199] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0200] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 11 Taking the example of a connection between China and Israel via a bus.

[0201] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0202] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0203] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. An object storage method, applied to an object storage gateway, characterized in that, The method includes: When a data processing request is received from a client, the client's historical load information and the storage platform's historical health information are obtained. Processing the historical load information and historical health information to determine the target asynchronous I / O load corresponding to the current data processing request includes: processing the historical load information using a first neural network model to obtain first load information corresponding to the current time period of the client; processing the historical health information using a second neural network model to obtain first health information corresponding to the current time period of the storage platform; and determining the target asynchronous I / O load corresponding to the current data processing request based on the first load information and the first health information, including: obtaining preset weights corresponding to each disk group; determining second load information corresponding to the storage platform based on the preset weights, average I / O information, and disk status information of each disk group; and substituting the first load information and the second load information into a preset limiting formula to determine the target asynchronous I / O load, wherein the preset limiting formula is: Total_user_load < Total_aio_load weight < Total_disk_load, where Total_user_load represents the first load information, Total_aio_load represents the asynchronous I / O load, weight represents the weight parameter set based on experience, and Total_disk_load represents the second load information; If the data processing request is a data upload request, then the number of asynchronous I / O channels and the data block size corresponding to the data upload request are determined based on the target asynchronous I / O load; Based on the data block size and the number of asynchronous I / O channels, the upload object corresponding to the data upload request is segmented and uploaded in blocks.

2. The method according to claim 1, characterized in that, The method further includes: If the data processing request is a data download request, then determine the data block size of the download object corresponding to the data download request within the storage platform; Based on the target asynchronous I / O load and the data block size of the download object, determine the number of asynchronous I / O channels corresponding to the data download request; Based on the number of asynchronous I / O channels, the data block of the download object corresponding to the data download request is sent to the client.

3. The method according to claim 1, characterized in that, The first neural network model is a first long short-term memory network model trained on the first training dataset, and the second neural network model is a second long short-term memory network model trained on the second training dataset. The step of processing the historical load information using a first neural network model to obtain the first load information corresponding to the current time period of the client includes: The historical load information is divided into time periods to obtain load information corresponding to multiple preset time periods, and then input into the first long short-term memory network model to obtain the first load information corresponding to the current time period; The process of using a second neural network model to process the historical health information to obtain the first health information corresponding to the current time period on the storage platform includes: The historical health information is divided into time periods to obtain health information corresponding to multiple preset time periods. This information is then input into the second long short-term memory network model to obtain the first health information corresponding to the current time period.

4. The method according to claim 1, characterized in that, The method further includes: Determine the metadata corresponding to the starting and ending data blocks of the uploaded object during the chunked upload process; Based on the metadata corresponding to the starting data block and the ending data block, as well as the data block size, the metadata corresponding to the uploaded object is determined and recorded in the metadata management module of the storage gateway.

5. The method according to claim 2, characterized in that, The step of sending the data block of the download object corresponding to the data download request to the client based on the number of asynchronous I / O channels includes: Obtain the metadata of the downloaded object, and determine the metadata of each data block of the downloaded object based on the metadata of the downloaded object; Based on the metadata of each data block, the location and sending order of each data block of the download object in the storage platform are determined; Based on the number of asynchronous I / O channels and the position and sending order of each data block, the data blocks of the download object in the storage platform are sent to the client.

6. An object storage device, applied to an object storage gateway, characterized in that, The device includes: The request receiving module is used to obtain the client's historical load information and the storage platform's historical health information when it receives a data processing request from the client. A load determination module is used to process the historical load information and the historical health information to determine the target asynchronous I / O load corresponding to the current data processing request. This includes: processing the historical load information using a first neural network model to obtain first load information corresponding to the current time period of the client; processing the historical health information using a second neural network model to obtain first health information corresponding to the current time period of the storage platform; and determining the target asynchronous I / O load corresponding to the current data processing request based on the first load information and the first health information. This includes: obtaining preset weights corresponding to each disk group; determining second load information corresponding to the storage platform based on the preset weights, average I / O information, and disk status information of each disk group; and substituting the first load information and the second load information into a preset limiting formula to determine the target asynchronous I / O load. The preset limiting formula is: Total_user_load < Total_aio_load. weight < Total_disk_load, where Total_user_load represents the first load information, Total_aio_load represents the asynchronous I / O load, weight represents the weight parameter set based on experience, and Total_disk_load represents the second load information; The request discrimination module is used to determine the number of asynchronous I / O channels and the data block size corresponding to the data upload request based on the target asynchronous I / O load if the data processing request is a data upload request. The data processing module is used to slice and upload the upload object corresponding to the data upload request based on the data block size and the number of asynchronous I / O channels.

7. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the object storage method of any one of claims 1 to 5 by executing the computer instructions.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the object storage method according to any one of claims 1 to 5.

Images