Object storage method and device, equipment and medium

Abstract

The invention relates to the technical field of object storage, and discloses an object storage method, device and equipment and a medium, which are applied to an object storage gateway, the method comprises the following steps: when a data processing request of a client is received, obtaining historical load information of the client and historical health information of a storage platform; processing the historical load information and the historical health information, and determining a target asynchronous I / O load corresponding to the data processing request; if the data processing request is a data uploading request, determining the number of asynchronous I / O channels and the data block division size corresponding to the data uploading request based on the target asynchronous I / O load; the uploading object corresponding to the data uploading request is subjected to block cutting uploading on the basis of the data block division size and the asynchronous I / O channel number, when the data uploading request is received, the current asynchronous transmission load is determined in combination with the historical situation, and then the appropriate data block division size and the appropriate asynchronous I / O channel number are determined; and the transmission performance during data uploading is ensured.

Description

Technical Field

[0001] The present invention relates to the field of object storage technology, and in particular to an object storage method, device, equipment and medium. Background Art

[0002] Currently, there are more and more application scenarios with large amounts of real-time data storage, such as video surveillance and artificial intelligence. In these application scenarios, the amount of data stored in video surveillance has exploded, resulting in more and more challenges for video storage. How to adaptively expand video storage and how to meet the low latency and high load requirements of video surveillance and artificial intelligence training are becoming more and more urgent problems to be solved. For storage gateways, scheduling when performing data storage or data download tasks is becoming more and more important.

[0003] Currently, in object storage scenarios, when performing scheduling, the storage gateway often only determines the scheduling plan and executes data upload or download based on pre-written scheduling rules, which fails to fully utilize the load capacity, resulting in poor processing efficiency during data storage. Summary of the invention

[0004] In view of this, the present invention provides an object storage method, apparatus, device and medium to solve the problem that the load capacity cannot be fully utilized, resulting in poor processing efficiency during data storage.

[0005] In a first aspect, the present invention provides an object storage method, which is applied to an object storage gateway, and the method includes:

[0006] When receiving a data processing request from a client, obtain the client's historical load information and the storage platform's historical health information;

[0007] Processing the historical load information and the historical health information to determine a target asynchronous I / O load corresponding to the current data processing request;

[0008] If the data processing request is a data upload request, determining the number of asynchronous I / O channels and the data block division size corresponding to the data upload request based on the target asynchronous I / O load;

[0009] Based on the data block division size and the number of asynchronous I / O channels, the upload object corresponding to the data upload request is divided into blocks and uploaded.

[0010] The method provided in this aspect determines the asynchronous I / O load corresponding to the data processing request according to the historical load information of the client and the historical health information of the storage platform when receiving the data processing request from the client, and then when the data processing request is an upload request, the data block division size and the number of channels corresponding to the upload request are determined in combination with the asynchronous I / O load determined according to the historical situation to perform data upload, thereby improving the performance of the storage gateway when uploading data and ensuring data processing efficiency.

[0011] In an optional implementation, if the data processing request is a data download request, determining a data block size of a download object corresponding to the data download request in the storage platform;

[0012] Determining the number of asynchronous I / O channels corresponding to the data download request based on the target asynchronous I / O load and the data block size of the download object;

[0013] Based on the number of asynchronous I / O channels, a data block of the download object corresponding to the data download request is sent to the client.

[0014] This embodiment, when the data processing request is a data download request, determines the number of channels for asynchronous transmission based on the size of the data block corresponding to the download request in the storage platform and the determined asynchronous I / O load, and then sends the data block of the download object to the client through the determined number of channels, thereby ensuring the transmission performance during the data download process.

[0015] In an optional implementation, the processing of the historical load information and the historical health information to determine the target asynchronous I / O load allocated to the current data processing request includes:

[0016] Processing the historical load information using a first neural network model to obtain first load information corresponding to a current period of the client;

[0017] Processing the historical health information using a second neural network model to obtain first health information corresponding to a current period of the storage platform;

[0018] Based on the first load information and the first health information, a target asynchronous I / O load corresponding to the current data processing request is determined.

[0019] In this embodiment, a neural network model is used to predict the load information corresponding to the client in the current period and the health information corresponding to the storage platform, so as to determine the asynchronous I / O load of this data processing request, ensure the accuracy when determining the asynchronous I / O load, and further ensure that the number of channels determined subsequently conforms to the current actual situation, thereby ensuring the data processing effect.

[0020] In an optional implementation, the first health information includes: average I / O information and disk status information corresponding to each disk group in the storage platform;

[0021] The determining, based on the first load information and the first health information, a target asynchronous I / O load corresponding to the current data processing request includes:

[0022] Obtaining a preset weight corresponding to each disk group, and determining second load information corresponding to the storage platform according to the preset weight corresponding to each disk group and average I / O information and disk status information;

[0023] The first load information and the second load information are substituted into a preset restriction formula to determine the target asynchronous I / O load.

[0024] This implementation method determines the load information of the storage platform through the status information and average I / O information of each disk group in the storage platform, and then combines the load information of the client and the preset restriction formula to accurately obtain the asynchronous I / O load that conforms to the actual situation, thereby ensuring the accuracy of the load calculation.

[0025] In an optional implementation, the first neural network model is a first long short-term memory network model trained based on a first training data set, and the second neural network model is a second long short-term memory network model based on a second training data set;

[0026] The using the first neural network model to process the historical load information to obtain the first load information corresponding to the current period of the client includes:

[0027] Dividing the historical load information into time periods to obtain load information corresponding to each of a plurality of preset time periods, and inputting the load information into the first long short-term memory network model to obtain first load information corresponding to the current time period;

[0028] The using the second neural network model to process the historical health information to obtain the first health information corresponding to the current period of the storage platform includes:

[0029] The historical health information is divided into time periods to obtain health information corresponding to each of a plurality of preset time periods, and the health information is input into the second long short-term memory network model to obtain first health information corresponding to the current time period.

[0030] In this embodiment, the long short-term memory network is trained through a special training data set to obtain the neural network models corresponding to the client's load information and the storage platform's health information, and then the historical health information and historical load information are input into the corresponding long short-term memory network model to ensure the accuracy of the first load information and the first health information corresponding to the current time period that are finally determined.

[0031] In an optional embodiment, the method further includes:

[0032] Determine the metadata corresponding to the starting data block and the ending data block of the uploaded object during the block uploading process;

[0033] Based on the metadata corresponding to the start data block and the end data block respectively and the size of the data block division, the metadata corresponding to the upload object is determined and recorded in the metadata management module of the storage gateway.

[0034] In this embodiment, after the data is uploaded, the metadata of the uploaded object is determined only based on the metadata of the starting data block and the ending data block and the size of the data block divisions and recorded in the metadata management module, thereby ensuring the simplicity of the metadata and avoiding excessive metadata data volume.

[0035] In an optional implementation, sending the data block of the download object corresponding to the data download request to the client based on the number of asynchronous I / O channels includes:

[0036] Acquire metadata of the download object, and determine metadata of each data block of the download object based on the metadata of the download object;

[0037] Based on the metadata of each data block, determine the location and sending order of each data block of the download object in the storage platform;

[0038] Based on the number of the asynchronous I / O channels and the positions and sending order of the respective data blocks, the data blocks of the download object in the storage platform are sent to the client.

[0039] In this implementation manner, the location and sending order of each data block of the download object are determined by metadata corresponding to the download object, thereby ensuring the accuracy of the data download process.

[0040] In a second aspect, the present invention provides an object storage device, applied to an object storage gateway, the device comprising:

[0041] A request receiving module, used to obtain the historical load information of the client and the historical health information of the storage platform when receiving a data processing request from the client;

[0042] A load determination module, used to process the historical load information and the historical health information to determine a target asynchronous I / O load corresponding to the current data processing request;

[0043] A request identification module, configured to determine, if the data processing request is a data upload request, the number of asynchronous I / O channels and the data block division size corresponding to the data upload request based on the target asynchronous I / O load;

[0044] The data processing module is used to perform block uploading on the upload object corresponding to the data upload request based on the data block division size and the number of the asynchronous I / O channels.

[0045] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the object storage method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0046] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the object storage method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 is a flowchart of an object storage method according to an embodiment of the present invention;

[0049] Figure 2 is a flow chart of another object storage method according to an embodiment of the present invention;

[0050] Figure 3 is an example diagram of an object storage system architecture according to an embodiment of the present invention;

[0051] Figure 4 This is an example diagram of a process when a client uploads an object according to an embodiment of the present invention;

[0052] Figure 5 This is an example diagram of a process when a client downloads an object according to an embodiment of the present invention;

[0053] Figure 6 is an example diagram of a data processing flow of an adaptive scheduling module according to an embodiment of the present invention;

[0054] Figure 7 is an example diagram of the architecture of an LSTM model according to an embodiment of the present invention;

[0055] Figure 8 This is an example diagram of a storage platform data reading process according to an embodiment of the present invention;

[0056] Fig. 9 This is an example diagram of a storage platform data writing process according to an embodiment of the present invention;

[0057] Fig.10 is a structural block diagram of an object storage device according to an embodiment of the present invention;

[0058] Fig.11 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0060] Currently, there are more and more application scenarios with large amounts of real-time data storage, such as video surveillance and artificial intelligence. In these application scenarios, the amount of data stored in video surveillance has exploded, resulting in more and more challenges for video storage. How to adaptively expand video storage and how to meet the low latency and high load requirements of video surveillance and artificial intelligence training are becoming more and more urgent problems to be solved. For storage gateways, scheduling when performing data storage or data download tasks is becoming more and more important.

[0061] Currently, in object storage scenarios, when performing scheduling, the storage gateway often only determines the scheduling plan and executes data upload or download based on pre-written scheduling rules, which fails to fully utilize the load capacity, resulting in poor processing efficiency during data storage.

[0062] To this end, an embodiment of the present invention provides an object storage method. When receiving a data processing request from a client, the asynchronous I / O load corresponding to the data processing request is determined based on the historical load information of the client and the historical health information of the storage platform. Then, when the data processing request is an upload request, the data block size and the number of channels corresponding to the upload request are determined in combination with the asynchronous I / O load determined based on historical conditions to perform data upload. This can improve the performance of the storage gateway when uploading data and ensure data processing efficiency.

[0063] According to an embodiment of the present invention, an object storage method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0064] In this embodiment, an object storage method is provided, which is applied to an object storage gateway. Figure 1 is a flow chart of an object storage method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0065] Step S101, when receiving a data processing request from a client, obtaining historical load information of the client and historical health information of the storage platform.

[0066] For the object storage process, it mainly includes three subjects, namely the client that initiates the data upload / download request, the storage gateway used to execute the data transmission, and the storage platform used to store specific data. For the storage gateway involved in the embodiment of the present method, it will receive a data processing request from the client, which indicates that the client requests to upload or download a certain object. After receiving the data processing request, the storage gateway needs to determine the asynchronous I / O load corresponding to this data processing for data transmission. In order to determine the most reasonable load distribution, it can be determined in combination with the historical information of the client and the storage platform.

[0067] Therefore, it is necessary to obtain the client's historical load information, which refers to the data records related to the workload that the client has borne during operation in the past period of time, such as disk I / O (input / output) operation frequency and data volume, memory usage, network bandwidth usage, and other historical information that can characterize the client's load capacity. For the historical health information of the storage platform, it can include historical status information corresponding to each disk group within the storage platform, historical average I / O information, and other historical information that can characterize the storage platform's load capacity.

[0068] Step S102: Process the historical load information and the historical health information to determine the target asynchronous I / O load corresponding to the current data processing request.

[0069] For the storage gateway, it has a special scheduling module inside, which is used to analyze the historical load information of the client and the historical health information of the storage platform to predict the expected load of the client and the expected health status of the storage platform in the time period corresponding to this data processing request, thereby combining the restriction relationship between the asynchronous I / O load of the storage gateway and the load of the client and the load of the storage platform to obtain the expected range of the asynchronous I / O load in the time period corresponding to this data processing request.

[0070] The optimal target asynchronous I / O load is determined within the expected range according to specific determination rules. For example, the middle value of the expected range of the asynchronous I / O load is selected as the target asynchronous I / O load corresponding to this data processing request, or the load value corresponding to a specific percentile value within the expected range is selected as the target asynchronous I / O load. There is no restriction on how to determine the load value within the expected range, and it can be set according to actual conditions.

[0071] Step S103: If the data processing request is a data upload request, the number of asynchronous I / O channels and the data block division size corresponding to the data upload request are determined based on the target asynchronous I / O load.

[0072] For the object storage process, the storage gateway needs to divide the objects uploaded by the client into blocks, and upload the divided data blocks one by one to the storage platform through asynchronous I / O channels. Therefore, it is necessary to determine the specific number of asynchronous I / O channels and the size of the data blocks in this data upload process.

[0073] When uploading data, it can be understood that multiple I / O channels are responsible for uploading the data blocks assigned to them one by one to the storage platform, and these I / O channels are in a concurrent operation state. The target asynchronous I / O load determined in the above steps can be understood as the load capacity that the storage gateway allocates for concurrent uploads for this data processing request. It can be simply understood that the product of the number of asynchronous I / O channels that the storage gateway can finally divide and the size of the data block division is equivalent to the target asynchronous I / O load determined above. Therefore, after determining the asynchronous I / O load, the specific number of I / O channels and the size of the data block division can be determined according to the preset rules, as long as the product of the two can correspond to the target asynchronous I / O load.

[0074] For example, the number of channels corresponding to different object size intervals can be pre-set, and the number of asynchronous I / O channels can be determined according to the specific size of the uploaded object, and then the target asynchronous I / O load is divided by the corresponding number of channels to determine the data block division size. Or, referring to the above example, the data block division size of different object size intervals can be pre-set, and then the target asynchronous I / O load is divided by the corresponding data block division size to obtain the number of asynchronous I / O channels. The above is only an example of the implementation of the example row, and the specific method of determining the number of channels and the data block division size is not limited here, as long as the product of the two corresponds to the target asynchronous I / O load.

[0075] Step S104: based on the data block size and the number of asynchronous I / O channels, the upload object corresponding to the data upload request is divided into blocks and uploaded.

[0076] After determining the data block size and the number of asynchronous I / O channels, the upload object can be segmented according to the data block size. For example, if the data block size is 5mb and the upload object is 100mb, 20 data blocks can be obtained. If the number of asynchronous I / O channels is 5, the 20 data blocks can be sent to each channel in turn according to the corresponding arrangement order, so that each channel can transmit the allocated data blocks to the storage platform one by one.

[0077] The object storage method provided in this embodiment determines the asynchronous I / O load corresponding to the data processing request based on the historical load information of the client and the historical health information of the storage platform when receiving the data processing request from the client, and then when the data processing request is an upload request, the asynchronous I / O load determined according to the historical situation is combined to determine the data block division size and the number of channels corresponding to the upload request to perform data upload, thereby improving the performance of the storage gateway when uploading data and ensuring data processing efficiency.

[0078] According to an embodiment of the present invention, another object storage method embodiment is provided, which is applied to an object storage gateway. Figure 2 is a flowchart of another object storage method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0079] Step S201: When receiving a data processing request from a client, obtain the client's historical load information and the storage platform's historical health information. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0080] Step S202: Process the historical load information and the historical health information to determine the target asynchronous I / O load corresponding to the current data processing request.

[0081] Specifically, in step S202, it includes:

[0082] Step S202-1, use the first neural network model to process the historical load information to obtain the first load information corresponding to the current time period of the client.

[0083] It can be understood that a trained first neural network model is pre-configured in the scheduling module of the storage gateway, which is specifically used to predict the load information of the client. After obtaining the historical load information of the client, the historical information can be input into the neural network model to predict the load information of the client corresponding to the current period.

[0084] Specifically, the first neural network model is a first long short-term memory network model trained based on a first training data set.

[0085] The long short-term memory network model, namely the LSTM model, can effectively process long sequence data and remember long-term information, which leads to its relatively good ability to predict time series. Therefore, it can be trained with training data related to client load information to obtain the first long short-term memory network model for predicting client load information.

[0086] In step S202-1, it includes:

[0087] The historical load information is divided into time periods to obtain load information corresponding to each of a plurality of preset time periods, and the load information is input into a first long short-term memory network model to obtain first load information corresponding to the current time period.

[0088] Based on the above explanation of the long short-term memory network, it can be understood that it is mainly used to analyze and predict time series data. Therefore, it is necessary to divide the client's historical load information into time periods to obtain the client's load information in multiple historical time periods. For example, the client's historical load information can be divided by hours to obtain the corresponding historical load information for the past several hours. After the time period division is completed, the data is input into the first long short-term memory network model to obtain the load information corresponding to the current time period.

[0089] Step S202-2, use the second neural network model to process the historical health information to obtain the first health information corresponding to the current period of the storage platform.

[0090] As mentioned above, a trained second neural network model is also pre-configured in the scheduling module of the storage gateway, which is specifically used to predict the load information of the storage platform. After obtaining the historical health information of the storage platform, the historical health information can be input into the trained second neural network model to predict the health information of the storage platform corresponding to the current period.

[0091] Specifically, the second neural network model is a second long short-term memory network model based on the second training data set.

[0092] Similarly, the first long short-term memory network model for predicting client load information can be obtained by training it with training data related to the storage platform health information, that is, the second training data set.

[0093] In step S202-2, it includes:

[0094] The historical health information is divided into time periods to obtain health information corresponding to each of a plurality of preset time periods, and the health information is input into a second long short-term memory network model to obtain first health information corresponding to the current time period.

[0095] Referring to the content in the above step S202-1, the historical health information of the storage platform can also be divided into multiple time periods to obtain the health information corresponding to each of the multiple time periods. It should be noted that due to the differences between the storage platform and the client, when dividing the time periods, the time period lengths between the two may be different. For example, when dividing the historical information of the client, it can be divided according to each hour or several minutes. For the health information of the storage platform, it can be divided according to each day or several hours. After the historical health information is divided, it can be input into the second long short-term memory network model to obtain the first health information corresponding to the current time period.

[0096] Exemplarily, the first long short-term memory network model and the second long short-term memory network model mentioned above can be integrated into one model, and the historical information corresponding to the client and the storage platform can be directly input into it to obtain the corresponding prediction information for the current time period.

[0097] Step S202-3: Determine the target asynchronous I / O load corresponding to the current data processing request based on the first load information and the first health information.

[0098] Specifically, the first health information includes: average I / O information and disk status information corresponding to each disk group in the storage platform.

[0099] It can be understood that for a storage platform, it has multiple disk groups inside. The predicted health information of the storage platform in the current period, that is, the first health information, can include the average I / O information and disk status information corresponding to each disk group in the current period.

[0100] In the above step S202-3, it includes:

[0101] Step S202-3-1, obtain the preset weight corresponding to each disk group, and determine the second load information corresponding to the storage platform according to the preset weight corresponding to each disk group and the average I / O information and disk status information.

[0102] For the storage platform, its load capacity in the current period is mainly related to the disk status information and average I / O information. At the same time, the weight corresponding to each disk can be set in advance according to the actual situation of the storage platform. By multiplying and adding the status information and the average I / O information and their corresponding weights, the corresponding load of the storage platform in the current period can be obtained, that is, the second load information.

[0103] For example, assuming that the number of hard disk groups in the storage platform is n, the weight ratio of these n hard disk groups is: (W 0 ,W 1 ,W 2 ...,W n-1 ); The predicted health status parameters of the n hard disk groups: (H 0 ,H 1 ,H 2 ...,H n-1 ), and the average IO status of these n hard disk groups: (I 0 ,I 1 ,I 2 ...,I n-1 ); The total load of each hard disk group, that is, the second load information: Total_disk_load and the relationship between the above information is:

[0104]

[0105] Step S202-3-2, the first load information and the second load information are substituted into a preset restriction formula to determine a target asynchronous I / O load.

[0106] Assume that the asynchronous I / O load is Total_aio_load, and the client load, that is, the first load information, is Total_user_load. Then the limiting relationship among Total_aio_load, Total_user_load and Total_disk_load is:

[0107] Total_user_load<Total_aio_load*weight<Total_disk_load

[0108] Among them, weight is a weight parameter set based on experience and can be set based on actual conditions.

[0109] Based on the above formula, the range of the asynchronous I / O load Total_aio_load can be obtained. The corresponding target asynchronous I / O load is determined within the range. For example, a specific quantile, such as 1 / 2 or 1 / 4, is selected and the corresponding value is used as the target asynchronous I / O load. The specific determination method can be configured based on the actual situation.

[0110] Step S203: If the data processing request is a data upload request, the number of asynchronous I / O channels and the data block size corresponding to the data upload request are determined based on the target asynchronous I / O load. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0111] Step S204: based on the data block size and the number of asynchronous I / O channels, the upload object corresponding to the data upload request is divided into blocks and uploaded. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0112] Furthermore, after uploading the upload object corresponding to the data upload request in pieces, the method further includes:

[0113] Determine the metadata corresponding to the starting data block and the ending data block of the uploaded object during the block uploading process;

[0114] Based on the metadata corresponding to the start data block and the end data block and the data block division size, the metadata corresponding to the uploaded object is determined and recorded in the metadata management module of the storage gateway.

[0115] It can be understood that, for object storage, after data blocks are divided and uploaded, it is necessary to record the metadata corresponding to each data block in the data block upload process, such as offset, data block name and other information, in order to facilitate the subsequent reading of the corresponding data blocks. Since the sizes of each data block are the same after the data blocks are divided in the above process, it is possible to record the metadata of the first starting data block, that is, the data block name and offset information corresponding to the first uploaded data block; and the metadata of the last data block, that is, the data block name and offset information corresponding to the last uploaded data block, and at the same time, the metadata corresponding to these two data blocks and the data block division size are used as the metadata corresponding to the uploaded object.

[0116] When the uploaded object needs to be downloaded later, the data block name and offset corresponding to each data block can be inferred based on the metadata of the starting data block and the ending data block, as well as the size of each data block, so as to determine the metadata corresponding to each data block, so as to find the corresponding data block in the storage platform and download it.

[0117] Step S205: If the data processing request is a data download request, determine the data block size of the download object corresponding to the data download request in the storage platform.

[0118] It can be understood that when the data processing request is a data download request, the download object is a data block that has been cut in the storage platform, that is, the size of the data block has been determined. Therefore, when the data processing request is a download request, the size of the data block corresponding to the download object in the storage platform can be determined through the metadata of the download object in the storage gateway.

[0119] Step S206: determining the number of asynchronous I / O channels corresponding to the data download request based on the target asynchronous I / O load and the data block size of the download object.

[0120] Since the current target asynchronous I / O load has been determined in the above steps, the number of asynchronous I / O channels allocated for this data download request can be determined in combination with the specific data block size. For example, assuming that the data block size is 5mb and the target asynchronous I / O load is 30mb, 6 asynchronous I / O channels can be allocated for data download.

[0121] Step S207: Based on the number of asynchronous I / O channels, the data block of the download object corresponding to the data download request is sent to the client.

[0122] After the number of asynchronous I / O channels is determined, the data blocks of the download object can be searched from the storage platform and sent to the client through the asynchronous I / O channel.

[0123] Specifically, in step S207, based on the number of asynchronous I / O channels, the data block of the download object corresponding to the data download request is sent to the client, including:

[0124] Step S207 - 1 : acquiring metadata of the download object, and determining metadata of each data block of the download object based on the metadata of the download object.

[0125] It can be understood that when downloading data, it is necessary to find the location and transmission order of each data block in the storage platform in order to transmit data based on the I / O channel. Therefore, it is necessary to determine the metadata of each data block of the download object in order to search and send subsequent data blocks.

[0126] Step S207-2: Based on the metadata of each data block, determine the location and sending order of each data block of the download object in the storage platform.

[0127] The metadata records the offsets corresponding to each data block. The offsets include the offset relative to the storage location and the offset of each data block relative to the starting object of the download object, that is, the number of data blocks belonging to the download object. The specific offsets can be used to determine the location of each data block in the storage platform and the order in which it is sent later. For specific information about how to determine the location and the order in which it is sent, please refer to the relevant background knowledge of metadata in object storage, which will not be repeated here.

[0128] It should be noted that the above steps S207 - 1 and S207 - 2 can be executed after receiving the data download request, and the execution order thereof does not need to be limited to after determining the target asynchronous I / O load.

[0129] Step S207-3: based on the number of asynchronous I / O channels and the position and sending order of each data block, the data blocks of the download object in the storage platform are sent to the client.

[0130] Several asynchronous I / O channels are created based on the number of asynchronous I / O channels. After the location of each data block is determined, each data block can be found and sent to each asynchronous I / O channel in turn in combination with the corresponding sending order to send the data block to the client to realize the download of the download object.

[0131] The object storage method provided by the embodiment of the present invention determines the asynchronous I / O load corresponding to the data processing request according to historical information when receiving the data processing request from the client; when the data processing request is an upload request, the data block division size and the number of channels corresponding to the upload request are determined in combination with the asynchronous I / O load determined according to historical conditions; when the data processing request is a data download request, the number of channels for asynchronous transmission is determined according to the size of the data block corresponding to the download request in the storage platform and the determined asynchronous I / O load, thereby ensuring the performance effect of the storage gateway during the data processing process.

[0132] In order to facilitate understanding of the above method embodiment, according to an embodiment of the present invention, an example diagram of an object storage system architecture is provided, as shown in FIG. Figure 3 As shown, the system mainly includes: an object storage gateway and a new storage platform. The object storage gateway includes data segmentation and asynchronous IO module, metadata management module, and adaptive scheduling module.

[0133] In the large object upload scenario, there will be problems such as high latency and low resource utilization. The data block and asynchronous IO module will cut the large object into small blocks according to the size given by the adaptive scheduling module, and then use the adaptive scheduling to send the data blocks asynchronously to the new storage platform with the given number of IO channels. In the large object download scenario, the given number of IO channels will be adaptively scheduled to asynchronously download the data blocks of the large object in the storage platform to the client.

[0134] like Figure 4 FIG. 1 is a flowchart showing an example of a process when a client uploads an object according to an embodiment of the present invention. It mainly includes:

[0135] (a) When a client uploads a large object, the adaptive scheduling module in the object storage gateway determines the size of the small data blocks and the number of concurrent asynchronous IOs based on the client's historical status and the historical status of the new storage platform.

[0136] (b) Then the data block cutting and asynchronous IO module cuts the large block data into small blocks, and sends the small block data to the storage platform for writing through asynchronous IO.

[0137] (c) When all data are written successfully, the names and offset information of these small blocks of data are written into the metadata.

[0138] After the above object uploading process is completed, it is necessary to record the names of these small data blocks and the offset of the small data blocks in the total large data blocks. If the metadata of all these small data blocks is recorded, when there are too many small data blocks, the problem of metadata being too large will arise, which will affect the writing of very large objects. The main function of the metadata management module in the above-mentioned new storage gateway is that it does not need to record the metadata of all small data blocks, but records the generally fixed size of the small data blocks specified for this large object upload, and then records the metadata of the first small block and the last block in turn, so that the name, offset and size of each small block can be calculated according to the rules.

[0139] like Figure 5 FIG. 1 is a flow chart showing an example of a process when a client downloads an object according to an embodiment of the present invention. It mainly includes:

[0140] (a) After receiving a request to download an object, the object storage gateway first reads the metadata from the database to obtain the name and offset of the small block of data corresponding to the object.

[0141] (b) The adaptive scheduling module determines the asynchronous IO concurrency size based on the predicted client load and the new storage platform load.

[0142] (c) The asynchronous IO module sends these small data blocks to the new storage platform in sequence by name, offset, and then according to the asynchronous IO concurrency size.

[0143] The adaptive scheduling module in the storage gateway is mainly used to predict the load situation at the next moment based on the historical load of the client and the historical load of the new storage platform using a new machine learning algorithm. For the specific prediction process, please refer to Figure 6 , is an example diagram of a data processing flow of an adaptive scheduling module according to an embodiment of the present invention.

[0144] First, obtain relevant data, including the client's load status at each time period, such as recording the load status at a 10-minute granularity. Then obtain the health status of each hard disk group on the storage platform, such as the health status of each hard disk at a 1-day granularity.

[0145] Then, preprocessing is performed to obtain the historical load of the client and the historical health status of each SSD, and then invalid data is cleared, such as data cleaning and homogenization.

[0146] Finally, prediction is performed through LSTM (Long Short-Term Memory Network), and multi-parameter fitting and adaptive training are performed on the LSTM model so that it can perform specific prediction functions.

[0147] For details, please refer to the LSTM processing flow. Figure 7 The figure shows an example of the architecture of an LSTM model according to an embodiment of the present invention. LSTM is a type of RNN. Compared with RNN, LSTM can filter the history and pass on the key information. It is mainly divided into input layer, LSTM layer and output layer. The input layer inputs the historical monitoring data into LSTM, and LSTM filters the information and outputs it to the next LSTM. The output result is the prediction result.

[0148] Finally, the data block partition size and the number of asynchronous I / O channels are determined according to the prediction results. For example, assuming that the storage platform has n hard disk groups, the preset weight ratio of these n hard disk groups is: (W 0 ,W 1 ,W 2 ...,W n-1 ).

[0149] The predicted storage platform health information includes: the health status parameters of the n hard disk groups (H 0 ,H 1 ,H 2 ...,H n-1 ); The average IO status of these n hard disk groups: (I 0 ,I 1 ,I 2 ...,I n-1 ); The predicted client load is: Total_user_load.

[0150] The size of the blocks is block_size, the number of AIO channels is aio_nums. The corresponding preset weight parameters are weight1 and weight2.

[0151] The product range of block_size and aio_nums can be determined by the following restriction relationship.

[0152]

[0153] Then, the specific data block size and number of channels are determined according to the product range of block_size and aio_nums for data upload.

[0154] For the above Figure 3 The new storage platform in the system shown includes components such as proxy, router, and each host SSD and related processes. The proxy component is the component that communicates with the object storage gateway, converts protocols, and reads and writes data. The router component is the object storage gateway that determines the routing information based on the proportion of each hard disk group determined by the object storage gateway adaptive scheduling module when writing data to the new storage platform, that is, determines which pool and which hard disk groups the small block of data is written to, and updates the SSD group related information after writing. Each host SSD will each pull up a related process. After each SSD writes data of the same block size, the proxy module will return the pool id, ssd id, and recode id to the object storage gateway. The next time the data is read, the object storage gateway can read the data based on the above information.

[0155] For example, Figure 8 FIG. 2 is an example diagram of a storage platform data reading process according to an embodiment of the present invention, which mainly includes:

[0156] (a) The proxy receives the object storage gateway HTTP read request, including pool_id and blob_id.

[0157] (b) Go to the router module to query which hard disk group to read data from.

[0158] (c) The proxy reads data from the corresponding hard disk group and returns it to the client.

[0159] For example, Fig. 9 FIG. 1 is an example diagram of a storage platform data writing process according to an embodiment of the present invention, which mainly includes:

[0160] (a) The proxy receives an HTTP write request from the object storage gateway.

[0161] (b) Query in the router module which hard disk group the small block of data is written to.

[0162] (c) The proxy writes the small block of data to the corresponding hard disk group, and returns the hard disk group ID and blob ID to the object storage gateway.

[0163] By introducing the LSTM (Long Short-Term Memory Network) algorithm, the historical client load and the historical health status of each hard disk of the new storage platform are collected and input, and the prediction result, that is, the client load and health status parameters of each hard disk at the next moment, is obtained. Then, the performance of the above object storage system is mathematically modeled, and the optimal block size and AIO size at the next moment are calculated, so that the storage gateway can have the highest possible performance during data upload under the performance conditions.

[0164] In this embodiment, an object storage device is also provided, which is used to implement the above embodiments and preferred implementations, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0165] This embodiment provides an object storage device, such as Fig.10 As shown, including:

[0166] The request receiving module 401 is used to obtain the historical load information of the client and the historical health information of the storage platform when receiving a data processing request from the client.

[0167] The load determination module 402 is used to process the historical load information and the historical health information to determine the target asynchronous I / O load corresponding to the current data processing request.

[0168] The request identification module 403 is used to determine the number of asynchronous I / O channels and the data block division size corresponding to the data upload request based on the target asynchronous I / O load if the data processing request is a data upload request.

[0169] The data processing module 404 is used to perform block uploading on the upload object corresponding to the data upload request based on the data block division size and the number of asynchronous I / O channels.

[0170] In some optional implementations, when the data processing request is a data download request, the request identification module 403 is further used to determine the data block size of the download object corresponding to the data download request in the storage platform;

[0171] Determine the number of asynchronous I / O channels corresponding to the data download request based on the target asynchronous I / O load and the data block size of the download object;

[0172] The data processing module 404 is further configured to send the data block of the download object corresponding to the data download request to the client based on the number of asynchronous I / O channels.

[0173] In some optional implementations, the load determination module 402, when processing the historical load information and the historical health information to determine the target asynchronous I / O load allocated to the current data processing request, includes:

[0174] The historical load information is processed using the first neural network model to obtain first load information corresponding to the current period of the client;

[0175] Processing the historical health information using the second neural network model to obtain first health information corresponding to the current period of the storage platform;

[0176] Based on the first load information and the first health information, a target asynchronous I / O load corresponding to the current data processing request is determined.

[0177] In some optional implementations, the first health information includes: average I / O information and disk status information corresponding to each disk group in the storage platform;

[0178] The load determination module 402, when determining the target asynchronous I / O load corresponding to the current data processing request based on the first load information and the first health information, includes:

[0179] Obtaining a preset weight corresponding to each disk group, and determining second load information corresponding to the storage platform according to the preset weight corresponding to each disk group and the average I / O information and disk status information;

[0180] The first load information and the second load information are put into a preset restriction formula to determine a target asynchronous I / O load.

[0181] In some optional implementations, the first neural network model is a first long short-term memory network model trained based on a first training data set, and the second neural network model is a second long short-term memory network model based on a second training data set;

[0182] The load determination module 402, when processing the historical load information using the first neural network model to obtain the first load information corresponding to the current period of the client, includes:

[0183] Divide the historical load information into time periods to obtain load information corresponding to each of a plurality of preset time periods, and input the load information into a first long short-term memory network model to obtain first load information corresponding to a current time period;

[0184] The load determination module 402, when processing the historical health information using the second neural network model to obtain the first health information corresponding to the current period of the storage platform, includes:

[0185] The historical health information is divided into time periods to obtain health information corresponding to each of a plurality of preset time periods, and the health information is input into a second long short-term memory network model to obtain first health information corresponding to the current time period.

[0186] In some optional implementations, the device is further used to determine metadata corresponding to the starting data block and the ending data block of the uploaded object during the block uploading process;

[0187] Based on the metadata corresponding to the start data block and the end data block and the data block division size, the metadata corresponding to the uploaded object is determined and recorded in the metadata management module of the storage gateway.

[0188] In some optional implementations, the data processing module 404, when sending the data block of the download object corresponding to the data download request to the client based on the number of asynchronous I / O channels, includes:

[0189] Acquire metadata of the download object, and determine metadata of each data block of the download object based on the metadata of the download object;

[0190] Based on the metadata of each data block, determine the location and sending order of each data block of the download object in the storage platform;

[0191] Based on the number of asynchronous I / O channels and the location and sending order of each data block, the data blocks of the download object in the storage platform are sent to the client.

[0192] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0193] The object storage device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0194] The embodiment of the present invention also provides a computer device having the above Fig.10 The object storage device shown.

[0195] See also Fig.11 , Fig.11 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.11 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.11 A processor 10 is taken as an example.

[0196] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0197] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0198] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0199] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0200] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig.11 The example of connecting through bus is taken in the following.

[0201] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0202] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0203] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. An object storage method, applied to an object storage gateway, characterized in that: The method comprises: When receiving a data processing request from a client, obtain the client's historical load information and the storage platform's historical health information; Processing the historical load information and the historical health information to determine a target asynchronous I / O load corresponding to the current data processing request; If the data processing request is a data upload request, determining the number of asynchronous I / O channels and the data block division size corresponding to the data upload request based on the target asynchronous I / O load; Based on the data block division size and the number of asynchronous I / O channels, the upload object corresponding to the data upload request is divided into blocks and uploaded.

2. The method according to claim 1, characterized in that The method further comprises: If the data processing request is a data download request, determining a data block size of a download object corresponding to the data download request in the storage platform; Determining the number of asynchronous I / O channels corresponding to the data download request based on the target asynchronous I / O load and the data block size of the download object; Based on the number of asynchronous I / O channels, a data block of the download object corresponding to the data download request is sent to the client.

3. The method according to claim 1, characterized in that The processing of the historical load information and the historical health information to determine the target asynchronous I / O load allocated to the current data processing request includes: Processing the historical load information using a first neural network model to obtain first load information corresponding to a current period of the client; Processing the historical health information using a second neural network model to obtain first health information corresponding to a current period of the storage platform; Based on the first load information and the first health information, a target asynchronous I / O load corresponding to the current data processing request is determined.

4. The method according to claim 3, characterized in that: The first health information includes: average I / O information and disk status information corresponding to each disk group in the storage platform; The determining, based on the first load information and the first health information, a target asynchronous I / O load corresponding to the current data processing request includes: Obtaining a preset weight corresponding to each disk group, and determining second load information corresponding to the storage platform according to the preset weight corresponding to each disk group and average I / O information and disk status information; The first load information and the second load information are substituted into a preset restriction formula to determine the target asynchronous I / O load.

5. The method according to claim 3, characterized in that: The first neural network model is a first long short-term memory network model trained based on a first training data set, and the second neural network model is a second long short-term memory network model based on a second training data set; The using the first neural network model to process the historical load information to obtain the first load information corresponding to the current period of the client includes: Dividing the historical load information into time periods to obtain load information corresponding to each of a plurality of preset time periods, and inputting the load information into the first long short-term memory network model to obtain first load information corresponding to the current time period; The using the second neural network model to process the historical health information to obtain the first health information corresponding to the current period of the storage platform includes: The historical health information is divided into time periods to obtain health information corresponding to each of a plurality of preset time periods, and the health information is input into the second long short-term memory network model to obtain first health information corresponding to the current time period.

6. The method according to claim 1, characterized in that The method further comprises: Determine the metadata corresponding to the starting data block and the ending data block of the uploaded object during the block uploading process; Based on the metadata corresponding to the start data block and the end data block respectively and the size of the data block division, the metadata corresponding to the upload object is determined and recorded in the metadata management module of the storage gateway.

7. The method according to claim 2, characterized in that The step of sending the data block of the download object corresponding to the data download request to the client based on the number of the asynchronous I / O channels includes: Acquire metadata of the download object, and determine metadata of each data block of the download object based on the metadata of the download object; Based on the metadata of each data block, determine the location and sending order of each data block of the download object in the storage platform; Based on the number of the asynchronous I / O channels and the positions and sending order of the respective data blocks, the data blocks of the download object in the storage platform are sent to the client.

8. An object storage device, applied to an object storage gateway, characterized in that: The device comprises: A request receiving module, used to obtain the historical load information of the client and the historical health information of the storage platform when receiving a data processing request from the client; A load determination module, used to process the historical load information and the historical health information to determine a target asynchronous I / O load corresponding to the current data processing request; A request identification module, configured to determine, if the data processing request is a data upload request, the number of asynchronous I / O channels and the data block division size corresponding to the data upload request based on the target asynchronous I / O load; The data processing module is used to perform block uploading on the upload object corresponding to the data upload request based on the data block division size and the number of the asynchronous I / O channels.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the object storage method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the object storage method according to any one of claims 1 to 7.

Images