Cloud computer integrated storage method and device, electronic equipment, chip and storage medium

Abstract

The invention provides a cloud computer integrated storage method and device, electronic equipment, a chip and a medium, and the method comprises the steps: determining the prediction time when the usage amount of a local storage space exceeds a preset threshold value based on local storage data, and enabling the prediction time to be obtained through the prediction of a trained target prediction model; according to the storage information configuration file, a target cloud storage space is determined from at least one cloud storage space contained in the storage information configuration file, and the cloud storage space is used for the cloud application to store data; before the prediction time, mounting the target cloud storage space to a local mounting point; and uploading the file written into the local mounting point to the target cloud storage space. According to the invention, the cloud storage space of other cloud applications can be integrated to the local cloud computer client, the integrated storage of the cloud computer client and the cloud storage space is realized, the utilization rate of the storage space is improved, elastic capacity expansion can be realized according to requirements through automatic mounting, the user experience is improved, and the capacity expansion efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of cloud computing, and in particular to a cloud computer integrated storage method, device, electronic device, chip and storage medium. Background Art

[0002] With the continuous development of cloud computing technology, cloud computers are increasingly widely used. However, in the process of using cloud computers, users' storage needs are often dynamically changing. Complex and changeable application scenarios are prone to data surges. When the existing system disk and data disk of the cloud computer cannot meet the capacity requirements, capacity expansion is required. The current capacity expansion process is inefficient and wastes resources. Summary of the invention

[0003] The present invention provides a cloud computer integrated storage method, device, electronic device, chip and storage medium.

[0004] A first aspect embodiment of the present disclosure proposes a cloud computer integrated storage method, the method comprising, based on local storage data, determining a predicted time when the usage of local storage space exceeds a preset threshold, the predicted time being predicted by a trained target prediction model; according to a storage information configuration file, determining a target cloud storage space from at least one cloud storage space included in the storage information configuration file, the cloud storage space being used for cloud application storage data; before the predicted time, mounting the target cloud storage space to a local mount point; and uploading files written to the local mount point to the target cloud storage space.

[0005] In some embodiments of the present disclosure, before determining the predicted time when the local storage space usage exceeds a preset threshold based on the local storage data, the method also includes: obtaining the local storage data for a preset time period, and generating a training set and a test set based on the local storage data for the preset time period; training a vector machine prediction model based on the training set and the test set; training a time series model based on the training set and the test set; and performing integrated fitting processing on the trained vector machine prediction model and the time series model to obtain a target prediction model.

[0006] In some embodiments of the present disclosure, before mounting the target cloud storage space to the local mount point, the method also includes: receiving a storage information configuration file, the storage information configuration file including configuration information of at least one cloud storage space; and authenticating and connecting the at least one cloud storage space according to the configuration information of the at least one cloud storage space contained in the storage information configuration file.

[0007] In some embodiments of the present disclosure, authenticating and connecting to at least one cloud storage space based on the configuration information of at least one cloud storage space included in a storage information configuration file includes: determining an access key identifier and a private key identifier of at least one cloud storage space based on the configuration information of at least one cloud storage space; and authenticating and connecting to at least one cloud storage space based on the access key identifier and the private key identifier of the at least one cloud storage space.

[0008] In some embodiments of the present disclosure, determining a target cloud storage space from at least one cloud storage space included in a storage information configuration file includes: determining a sorting result of at least one cloud storage space according to the storage information configuration file; and determining a cloud storage space whose sorting result meets a preset condition as a target storage space.

[0009] In some embodiments of the present disclosure, uploading files written to a local mount point to a target cloud storage space includes: using a sliding window to traverse and compare the files written to the local mount point and the target files stored in the target cloud storage space to determine whether there are differences between the files written to the local mount point and the target files stored in the target cloud storage space; when there are differences between the files written to the local mount point and the target files stored in the target cloud storage space, using a sliding window to traverse and compare the files written to the local mount point and the target files stored in the target cloud storage space to determine difference data blocks; generating difference files based on the difference data blocks; and uploading the difference files to the target cloud storage space.

[0010] A second aspect of the present disclosure provides a cloud computer integrated storage device, comprising a first processing unit, for determining, based on local storage data, a predicted time when the usage of local storage space exceeds a preset threshold, the predicted time being predicted by a trained target prediction model; a second processing unit, for determining, based on a storage information configuration file, a target cloud storage space from at least one cloud storage space included in the storage information configuration file, the cloud storage space being used for cloud application storage data; a third processing unit, for mounting the target cloud storage space to a local mount point before the predicted time; and a fourth processing unit, for uploading files written to the local mount point to the target cloud storage space.

[0011] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect embodiment of the present disclosure.

[0012] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0013] A fifth aspect embodiment of the present disclosure provides a computer program product, including a computer program, which executes the method described in the first aspect embodiment of the present disclosure when a processor is used to execute the computer program.

[0014] The sixth aspect embodiment of the present disclosure proposes a chip, which includes one or more interfaces and one or more processors; the interface is used to receive signals from a memory of an electronic device and send signals to the processor, the signals include computer instructions stored in the memory, and when the processor executes the computer instructions, the electronic device executes the method described in the first aspect embodiment of the present disclosure.

[0015] In summary, the cloud computer integrated storage method proposed in the present invention can predict the time when the local storage space usage exceeds a preset threshold through a target prediction model. By mounting the storage space of other cloud applications to the local before the predicted time, the local storage space can be automatically expanded, data surge can be avoided, and the expansion efficiency can be improved. After mounting the storage space of other cloud applications to the local, integrated storage of local and cloud applications can be achieved, the utilization rate of storage space can be improved, storage resources can be balanced, and resource waste can be avoided.

[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0018] Figure 1 A flowchart of a cloud computer integrated storage method provided in an embodiment of the present disclosure;

[0019] Figure 2 A flowchart of a cloud computer integrated storage method provided in an embodiment of the present disclosure;

[0020] Figure 3 A flowchart of a cloud computer integrated storage method provided in an embodiment of the present disclosure;

[0021] Figure 4 An architecture diagram of a cloud computer elastic expansion system provided by an embodiment of the present disclosure;

[0022] Figure 5A schematic diagram of the structure of a cloud computer integrated storage device provided in an embodiment of the present disclosure;

[0023] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0024] Figure 7 A schematic diagram of the chip structure provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0026] With the continuous development of cloud computing technology, cloud computers are being used more and more widely. However, in the process of using cloud computers, users' storage needs are often dynamically changing, and complex and changeable application scenarios are prone to data surges. When the existing system disk and data disk of the cloud computer cannot meet the capacity requirements, capacity expansion is required. At present, traditional storage capacity expansion methods include: manually creating an expansion order on the cloud service platform, expanding capacity by ordering new disks, or calling the factory method encapsulated by the underlying disk expansion API to expand capacity.

[0027] Traditional storage expansion methods usually require manual intervention, which is not only inefficient, but also easily leads to service interruptions, affecting user experience. In addition, traditional storage expansion methods cannot shrink after expansion, which is prone to unreasonable expansion, resulting in waste of resources and increased costs.

[0028] Therefore, in order to solve the above technical problems, the present invention proposes a cloud computer integrated storage method, which can be executed by an electronic device, for example, it can be executed by a server. For example, the method can be executed by a server (or electronic device) deployed with a cloud storage gateway, or a server (or electronic device) with a cloud storage gateway function, which is described as a cloud storage gateway in the following text. The method realizes efficient, secure and flexible data storage and management through virtualization technology and hybrid cloud storage services, and aims to form integrated storage between local cloud computers and other cloud application instances to achieve the purpose of sharing storage space across cloud applications. Provide cloud computer users with a convenient and reliable cloud storage environment without changing the existing business architecture. In addition, the method can enable users to fully utilize the cloud storage space of other cloud application instances in the cloud computer, so that when the disk is insufficient, the storage space can be flexibly expanded without the need for underlying subscription, which can save costs and resources.

[0029] The specific contents of this method are as follows.

[0030] Figure 1 A flowchart of a cloud computer integrated storage method provided by an embodiment of the present disclosure. Figure 1 As shown, the method may include the following steps.

[0031] Step 101, based on local storage data, determine the predicted time when the local storage space usage exceeds a preset threshold.

[0032] In some embodiments, for example, the local storage data may be data related to the local storage space of the cloud computer client. The local storage space is the storage space used by the cloud computer client to store data. The local storage space supports unique cloud computer client storage data, that is, different cloud computer clients each have different local storage spaces. For example, when the local storage space is the local disk of the cloud computer client, the local storage data may be, for example, disk capacity, disk usage, disk usage rate, disk read / write times per second (Input / Output Operations Per Second, IOPS) information, etc.

[0033] In some embodiments, the predicted time is the predicted time when the local storage space usage exceeds a preset threshold value, obtained by using a trained target prediction model. In other words, the trained target prediction model can be used to determine the time when the local storage space usage exceeds the preset threshold value. The target prediction model can be used to predict the time when the local storage space usage rate exceeds the preset threshold value.

[0034] In some embodiments, local storage space data changes dynamically over time. For example, the utilization rate of local storage space increases over time. When the local storage space is used more, it may cause the efficiency of the cloud computer client to decrease, or even cause problems such as downtime. Therefore, it is necessary to expand the local storage space in time. The target prediction model can predict the changes in the local storage space data and determine the time when the local storage space usage exceeds a preset threshold as the prediction time.

[0035] Optionally, the preset threshold may be set according to actual application scenarios, which is not limited in the present disclosure.

[0036] In some embodiments, optionally, a trained target prediction model can be used to periodically determine the prediction time. For example, the target prediction model can determine that the usage of the local storage space will not exceed a preset threshold over a long period of time. The target prediction model can then reacquire the data of the local storage space after a period of time, and make predictions again based on the newly acquired data of the local storage space to determine the prediction time.

[0037] Step 102: According to the storage information configuration file, determine a target cloud storage space from at least one cloud storage space included in the storage information configuration file.

[0038] In some embodiments, the cloud storage space is used for cloud applications to store data, that is, cloud applications can store data in the cloud storage space.

[0039] In some embodiments, for example, a storage information configuration file input by a user can be obtained from a cloud computer client. Optionally, the storage information configuration file includes configuration parameters of at least one cloud storage space. The cloud computer client can be authenticated and connected through the configuration information of the cloud storage space, so that the cloud computer client can access the cloud storage space according to the configuration parameters of the cloud storage space.

[0040] In some embodiments, the configuration parameters of the storage space can be, for example, configuration parameters, which may include the storage type (type) of the cloud storage space, whether to enable user authentication (env_auth), the access key identifier (access_key_id), the private key identifier (secret_access_keys), the endpoint (endpoint), the region (region), etc.

[0041] Among them, the storage type can be the storage type of data in the cloud storage space. For example, when the storage type field is S3, it can indicate that the storage type of the cloud storage space is object storage; whether to enable user verification can be used to indicate whether to enable the use of access keys and private keys to verify the cloud computer client accessing the cloud storage space. For example, when user verification is enabled, the access request of the cloud computer client needs to include the access key and the private key. The cloud storage space will only accept the access request of the cloud computer client when it determines that the above access key and private key are correct; optionally, the access key can be a unique identifier that identifies the user, or identifies the cloud computer client, similar to a user name, which can be used to determine the identity of the visitor; the private key can be a key used to encrypt the cloud storage space, similar to a password. When the private key corresponds to the cloud storage space, the cloud computer client can access the cloud storage space; the endpoint can be the Universal Resource Locator (URL) address of the cloud storage space, that is, the endpoint can be the interface path of the cloud computer client when requesting access to the cloud storage space; the region field can be used to indicate the resource pool where the cloud storage space is located.

[0042] In the above embodiment, each user can optionally have a corresponding access key and private key to access the cloud storage space, and each user can have different operation permissions for the specified cloud storage space, such as obtaining, creating, deleting, etc. For example, in order to achieve data isolation between different applications, the cloud storage service console can create a new account for the user and specify the operation permissions for the specific cloud storage space for the account. For example, the cloud storage service console can create a new account for the cloud computer user through the access control management component, and specify the permissions of the account as read and write. After the creation is completed, the user can access the account list component to obtain and record the values ​​of the access key and the private key.

[0043] In some embodiments, for the above-mentioned endpoint parameters, when requesting access to a cloud storage space, the user needs to specify the request address of the cloud storage space, that is, the URL address of the above-mentioned cloud storage space, so that the cloud storage gateway can find the corresponding cloud storage space according to the URL address for authentication and connection. By way of example, the user can determine the URL address corresponding to the cloud storage space through the specific information of the cloud storage space stored in the cloud application console. For example, the user can trigger a jump to the bucket details page providing cloud services through the bucket list page provided by the cloud storage service console (cloud application console) corresponding to the cloud application, and view and record the domain name address URL provided to the outside world, which is composed of the bucket name and the resource pool domain name. In other words, the user can determine the resource pool where the cloud storage space is located and the name of the specific storage bucket from the cloud application console, and compose the URL address of the cloud storage space according to the resource pool and bucket name.

[0044] In some embodiments, optionally, the cloud computer client can create a configuration file through the remote storage information configuration box. The user can enter the drive letter and the corresponding storage domain name, access key and other information in the configuration box to create a storage information configuration file. The cloud storage gateway can receive the storage information configuration file created by the cloud computer client.

[0045] In other words, the configuration file may also include a drive letter. For example, the drive letter may be a drive letter that maps the cloud storage space to the local drive. The cloud computer client may search and manage the cloud storage space based on the drive letter. The cloud computer client may assign a drive letter to the cloud storage space based on the configuration file.

[0046] In some embodiments, determining a target cloud storage space from at least one cloud storage space included in a storage information configuration file includes: determining a sorting result of at least one cloud storage space according to the storage information configuration file; and determining a cloud storage space whose sorting result meets preset conditions as a target storage space.

[0047] In other words, after receiving the configuration file, the distance from the cloud storage space to the cloud computer client can be determined based on the URL address of the cloud storage space in the configuration file, and at least one cloud storage space included in the configuration file can be sorted according to the distance. For example, the cloud storage space can be sorted in ascending order according to the distance from the cloud storage space to the cloud computer client, and the sorting result can be stored in the local storage space. Optionally, the configuration file can include a dictionary data structure, and the sorting result can be saved in the dictionary data structure. Optionally, the dictionary data structure can also be used to store the correspondence between the cloud storage space and the drive character.

[0048] In the above embodiment, when sorting at least one cloud storage space, a comprehensive sorting can also be performed based on parameters such as the remaining capacity of the cloud storage space, the usage rate, and the distance to the cloud computer client. For example, a weighted calculation can be performed based on the remaining capacity of the cloud storage space and the distance from the cloud storage space to the cloud computer client to determine the score of at least one cloud storage space, and the cloud storage space can be sorted based on the score of the cloud storage space, or the cloud storage space can be sorted based on other parameters, which is not limited by the present disclosure.

[0049] In some embodiments, the preset conditions can be determined according to the actual application scenario, which is not limited by the present disclosure. For example, when the cloud storage space can be sorted in ascending order according to the distance from the cloud computer client, the preset condition can be determined that the cloud storage space ranked first is the target cloud storage space, or the top 3 cloud storage spaces are the target cloud storage spaces, etc., that is, the cloud storage space with a closer distance can be preferentially selected as the target cloud storage space to reduce latency; for example, the number of determined target cloud storage spaces can be greater than 1, that is, multiple target cloud storage spaces can be mounted to meet local storage needs.

[0050] For another example, when the cloud storage spaces are sorted based on their scores, the preset condition may be to determine that a cloud storage space with a score greater than or equal to the preset score is the target cloud storage space, or the preset condition may be to determine that a cloud storage space with a higher ranking is the target cloud storage space. At this time, when calculating the score, it may be determined that the closer the storage space is to the cloud computer client, the higher the score, and the larger the remaining capacity, the higher the score, and so on. In other words, a cloud storage space with a large capacity and a short distance may be preferentially selected as the target cloud storage space.

[0051] In some embodiments, the cloud storage gateway can determine a target cloud storage space from at least one cloud storage space included in the configuration file. For example, based on the above-mentioned sorting results, the cloud storage space closest to the cloud computer client can be determined as the target cloud storage space. For example, the target cloud storage space can be a cloud storage space for mounting, that is, the target cloud storage space can be mounted to the local storage space of the cloud computer client. After the mounting is completed, the cloud computer client can perform operations such as storing data (such as getting data GetObject, creating new data PutObject, deleting data DeleteObject, etc.) in the target cloud storage space. When the nearest cloud storage space is determined to be the target cloud storage space, due to the shorter distance, the delay time of the operation can be reduced and the operation efficiency can be improved.

[0052] In some embodiments, before mounting the target cloud storage space to the local mount point, the method also includes: receiving a storage information configuration file, the storage information configuration file including configuration information of at least one cloud storage space; authenticating and connecting at least one cloud storage space according to the configuration information of at least one cloud storage space contained in the storage information configuration file.

[0053] In some embodiments, authenticating and connecting at least one cloud storage space based on the configuration information of at least one cloud storage space included in the storage information configuration file includes: determining the access key identifier and the private key identifier of at least one cloud storage space based on the configuration information of at least one cloud storage space; and authenticating and connecting at least one cloud storage space based on the access key identifier and the private key identifier of the at least one cloud storage space.

[0054] In other words, after receiving the storage information configuration file, the cloud storage gateway can authenticate and connect to at least one cloud storage space according to the configuration information of the cloud storage space, for example, it can locate the cloud storage space according to the URL address of the above cloud storage gateway, and determine the access key and private key according to the above configuration parameters to perform identity authentication. For example, the cloud computer client can obtain the command line application from the cloud computer client, execute the corresponding executable file, and set its environment variables in the system properties, so as to realize the authentication and connection of the cloud storage space through command line operation in the cloud computer terminal.

[0055] Step 103: before the predicted time, mount the target cloud storage space to the local mount point.

[0056] In some embodiments, after determining the target cloud storage space, the target cloud storage space can be mounted to a local mount point. For example, the mounting can be performed before the predicted time, or the mounting can be performed at the predicted time. In other words, the predicted time can be the latest time for mounting. When the predicted time is exceeded, a data surge may occur. If the mounting is not completed at this time, data loss may occur due to failure to expand capacity in time.

[0057] In some embodiments, mounting the target cloud storage space to a local mount point may refer to mounting the target cloud storage space to a cloud storage client. At this time, a mount directory may be created on the cloud computer client according to the drive letter of the target cloud storage space, and the target cloud storage space may be mounted to the cloud computer client through a mount command based on the Common Internet File System (CIFS) protocol. This operation allows the file system to be implemented in the user space without modifying or mutating the kernel and mapping it to a local drive letter. In other words, after the mounting is completed, the user can operate the target cloud storage space on the cloud computer client, and the user can operate the target cloud storage space like operating the local storage space, such as storing files in the target cloud storage space, etc.

[0058] In some embodiments, when using the mount command to mount, the mount command needs to be followed by the configuration name, bucket name, and mount drive letter, where the configuration name is the name of the storage information configuration file, in other words, it can be the name of the configuration file that needs to be executed, the bucket name is the bucket name of the target cloud storage space determined from the cloud application console, and the mount drive letter is the drive letter corresponding to the target cloud storage space, that is, the drive letter of the target cloud storage space that the user wants to display on the cloud computer. In addition to the above mount command, the cache timeout and cache mode of the file attributes can also be set to improve the efficiency of file reading and writing.

[0059] Through the above mount command, a virtual file system can be created on the cloud computer client, and the object storage space of other cloud application instances (i.e. the target cloud storage space) can be mapped to the local cloud computer client. After the mount is completed, the cloud computer client can check whether it is successfully mounted through the file manager or terminal. If a new drive letter appears, the mount is successful. At this time, the user can store and use data on the cloud computer client like operating a local disk.

[0060] In some embodiments, after the mounting is completed, the target prediction model can periodically predict the usage of the local storage space. When the target prediction model predicts that the usage of the local storage space is low over a long period of time and will not exceed a preset threshold, that is, when the target prediction model determines that the local storage space can meet the local storage requirements of the cloud computer client for a long time in the future, there is no need to expand the local storage space, and the previously mounted target cloud storage space can be released for use by other users.

[0061] Alternatively, the usage of the local storage space can be manually observed, and when it is confirmed that the local storage space does not need to be expanded, the mounted target cloud storage space can be uninstalled, etc., which is not limited by the present disclosure. In other words, the solution of the present invention can flexibly mount and uninstall the local storage space to meet the needs of different scenarios.

[0062] Step 104: Upload the file written to the local mount point to the target cloud storage space.

[0063] In some embodiments, after the mounting is completed, the user can store and use data on the cloud computer client just like operating a local disk. For example, the user can store files in the target cloud storage space. At this time, the user can write the files to be stored to the local mount point through the cloud computer client, and the cloud storage gateway can scan the files written to the local mount point. Optionally, the cloud storage gateway can periodically scan the local mount point to determine whether there are newly written files, which can avoid frequently starting scanning tasks, occupying server bandwidth and resources, and affecting the data processing capabilities of the cloud storage gateway.

[0064] For example, a scheduled task can be set to periodically scan local mount points, where the scheduled task information can include, for example, task name, description, time interval, and other information. In addition, the scheduled task can specify a script path to facilitate the use of scripts to implement the scheduled task. The functions of the script include scanning files written to the mount point, comparing the differences between new and old files, and uploading files to the cloud storage space, etc.

[0065] In other words, the cloud storage gateway can periodically scan the files written to the local mount point. Preferably, the cloud storage gateway can upload the files only when it determines that there is a difference between the files written to the mount point and the files stored in the cloud storage space; or, optionally, the cloud storage gateway may not compare the files written to the mount point with the files stored in the target cloud storage space, and directly upload the files written to the mount point when it scans that the files exist in the local mount point. The present disclosure does not limit this.

[0066] In summary, the above-mentioned embodiments of the present application, by using a trained target prediction model to determine the prediction time, can realize the prediction of the time when capacity expansion is required, so as to automatically expand the capacity when it is needed, and avoid data loss problems caused by the failure to expand the capacity in time due to a surge in data; by determining the cloud storage space with the shortest distance from multiple cloud storage spaces as the target cloud storage space, the delay of the cloud computer client when operating the cloud storage space can be reduced; by mounting the target cloud storage space to the local mount point, the local storage space can be automatically expanded, and the integrated storage of cloud applications and cloud computer clients can be realized, the utilization rate of storage resources can be improved, and the resource utilization rate of each storage space can be balanced; by uploading the files written to the local mount point to the target cloud storage space, the files can be stored in the cloud storage space of other cloud applications, and the file synchronization can be realized.

[0067] Figure 2 A flowchart of a cloud computer integrated storage method provided by an embodiment of the present disclosure. Figure 2 As shown, the method may include the following steps.

[0068] Step 201, obtaining the local storage data of a preset time period, and generating a training set and a test set according to the local storage data of the preset time period.

[0069] In some embodiments, local storage data of the local storage space within a preset time period can be collected, such as historical local storage data within a period of time, etc. For example, when the local storage space is a local disk, the local storage data may include, for example, the local historical data disk capacity, the local historical system disk capacity, and the IOPS information corresponding to the data disk and the system disk, etc. Optionally, data from different disks can be collected for training the model, which can enhance the generalization ability of the model and its adaptability to different application scenarios.

[0070] In some embodiments, the collected locally stored data can be processed so that it can be used as training data for model training, such as using median filtering and sliding average methods to remove noise, smoothing, cleaning and preprocessing the data so that the data can be used for model training, and the data processing of the locally stored data can also include dividing the locally stored data into training sets and test sets.

[0071] Step 202: train a vector machine prediction model based on the training set and the test set.

[0072] In some embodiments, for example, a vector machine prediction model may be used as a first prediction model, and the vector machine prediction model may be trained using training set data and test set data. Optionally, a training process is as follows.

[0073] In some embodiments, for example, the training data in the training set and the test set may include two parts: features and labels. For example, disk I / O performance data can be used as a feature of a prediction model to describe sample information, where one sample can be a locally stored data.

[0074] The initial vector machine model is iteratively trained using the training set and the test set. The model of the vector machine can be expressed as:

[0075]

[0076] Among them, K(x i ,x)=exp(-‖xx i ‖ 2 / 2λ 2 ) represents the kernel function, which is the support vector x i The inner product between the new input sample x, x represents the new input sample, x i is the support vector (also called eigenvector), y i is the category label corresponding to the training data, b is the hyperplane parameter, is the Lagrangian factor, L is the support vector x i The number of

[0077] In some embodiments, the support vector may be, optionally, a data point in the training data that is closest to the classification decision surface and is the most difficult to classify, and the new input sample is the training data that needs to be classified.

[0078] In some embodiments, when the vector machine model is iteratively trained, model parameters can be adjusted based on the prediction results and the test set to improve the prediction performance of the model. When the iterative training meets the target conditions, the trained model is used as the final vector machine model to predict whether the disk capacity exceeds the threshold.

[0079] Step 203: train the time series model based on the training set and the test set.

[0080] In some embodiments, for example, a time series model can be used as the second prediction model. For example, an autoregressive moving average model (ARMA) can be used as the second prediction model. The time series model can be trained using training set data and test set data. Optionally, a training process is as follows.

[0081] In some embodiments, a time series prediction model can be established based on the time series prediction model and the training set and the test set, including an encoder module and a decoder module. The encoder module extracts features and labels from the input data and passes these features to the decoder. The label data is input into the decoder module for processing to obtain the predicted data, the mean absolute error function is used as the loss function, the sliding window feature is added, the gap between the disk utilization prediction data and the label data is calculated, and the gradient descent method is continuously fitted to obtain a trained time series model.

[0082] Step 204, performing integrated fitting processing on the trained vector machine prediction model and the time series model to obtain a target prediction model.

[0083] In some embodiments, the trained vector machine prediction model and time series model may be integrated and fitted, that is, multiple prediction models may be integrated to obtain a target prediction model. Optionally, the predicted value of the capacity at a certain moment may be determined as the weighted predicted value of the above two models. In other words, the obtained target prediction model may be expressed as:

[0084]

[0085] Among them, R t is the predicted value at time t, ε i is the model weight coefficient, r jt is the predicted value of the jth model at time t, where the weighted coefficient satisfies:

[0086]

[0087] Among them, E i is the absolute error of a single model. Optionally, the target prediction model uses the mean absolute error as the loss function, expressed as The final target prediction model is obtained by continuous fitting through the gradient descent method.

[0088] Optionally, when training a vector machine model and a time series model, the training data in the training set and the test set can be divided into a training set and a test set of a first preset time period, and a test set and a training set of a second preset time period. The models can be trained using training sets and test sets of different preset time periods respectively. This can improve the generalization ability of the models when the models are integrated. Alternatively, the first preset time period and the second preset time period can be ignored, and all the training set and test set data can be used to train the two models, and the present disclosure is not limited to this.

[0089] In summary, the above embodiments of the present application, by using historical local storage data to train the vector machine prediction model and the time series model, and integrating and fitting the trained vector machine prediction model and the time series model to obtain the target prediction model, can improve the prediction accuracy of the target prediction model, and its adaptability and generalization ability to different application scenarios.

[0090] Figure 3 A flowchart of a cloud computer integrated storage method provided by an embodiment of the present disclosure. Figure 3 As shown, the method may include the following steps.

[0091] Step 301, using a sliding window to traverse and compare the files written to the local mount point and the target files stored in the target cloud storage space, to determine whether there are differences between the files written to the local mount point and the target files stored in the target cloud storage space.

[0092] In some embodiments, when the cloud storage gateway scans and finds that there are files written to the local mount point, it can determine whether there are differences between the files written to the local mount point and the target files stored in the target cloud storage space, so that the files can be uploaded when differences are determined to achieve file synchronization.

[0093] In some embodiments, determining whether there is a difference between a file written to a local mount point and a target file stored in a target cloud storage space includes: when the target file stored in the target cloud storage space is empty, determining that there is a difference between the file written to the local mount point and the target file stored in the target cloud storage space; when the size or modification timestamp of the file written to the local mount point and the target file stored in the target cloud storage space are different, determining that there is a difference between the file written to the local mount point and the target file stored in the target cloud storage space; when the size and modification timestamp of the file written to the local mount point and the target file stored in the target cloud storage space are the same, using a sliding window to traverse and compare the file written to the local mount point and the target file stored in the target cloud storage space, and when there are difference data blocks, determining that there is a difference between the file written to the local mount point and the target file stored in the target cloud storage space.

[0094] Specifically, when the target file saved in the target cloud storage space is empty, it means that the file written to the local mount point does not exist in the target cloud storage space. For example, when the file written to the mount point is fs, it will be determined whether there is a file fs in the target cloud storage space before transferring it to the target cloud storage space. When the file fs does not exist in the target cloud storage space, there is a difference between the file written to the local mount point and the target file saved in the target cloud storage space, and the file written to the mount point needs to be uploaded to the target cloud storage space.

[0095] Specifically, when the target file saved in the target cloud storage space is not empty, it is necessary to determine whether there is a difference between the file written to the local mount point and the target file saved in the target cloud storage space based on the parameters of the file. For example, the size and modification timestamp of the file written to the local mount point and the size and modification timestamp of the target file saved in the target cloud storage space can be determined separately. When the size or modification timestamp of the file written to the local mount point is different from that of the target file saved in the target cloud storage space, it means that the file written to the mount point has been modified relative to the target file saved in the target cloud storage space. Therefore, it can be determined that there is a difference between the file written to the local mount point and the target file saved in the target cloud storage space, and the file written to the mount point needs to be uploaded to the target cloud storage space.

[0096] Specifically, when the size and modification timestamp of the file written to the local mount point are the same as the target file saved in the target cloud storage space, it is necessary to further compare the contents of the two files to determine whether there are any differences. At this time, since the cloud storage gateway scans the mount point files and uploads files as a periodic task, it is possible that when the file written to the mount point is updated, the cloud storage gateway completes the upload of the previous file at the same time, and there is a delay in the file upload synchronization. When the modification timestamp of the target file saved in the cloud storage space is the same as the modification timestamp of the file written to the local mount point, due to the delay in uploading files, there may be differences between the two files. Therefore, it is necessary to traverse and compare the two files to determine whether there are any differences.

[0097] Optionally, a sliding window can be used to traverse and compare the files written to the local mount point and the target files stored in the target cloud storage space. When there are difference data blocks, it is determined that there are differences between the files written to the local mount point and the target files stored in the target cloud storage space. The specific comparison and traversal process is as follows.

[0098] In some embodiments, a window may be created and the size of the sliding window may be set. The size of the window may affect the computational efficiency. If the sliding window is too small, more data blocks need to be processed, resulting in increased computation and matching time. If the window is too large, the transmission efficiency may be reduced. In other words, the data in a window may be referred to as a data block. The file traversal comparison may be achieved through the sliding window. The size of the sliding window may be set based on the actual scenario, and the present disclosure does not limit this.

[0099] Optionally, on the target cloud storage space side, a window can be used to slide from the leftmost side of the target file fd until the entire file is traversed. Each time the window slides, the number of the data block in the window is recorded, and the starting offset address and length of the data block are recorded, and the window data is encoded to generate weak digests and strong digests respectively. When encoding the window data, this solution proposes to use two verification algorithms in combination, for example, Adler-32 and SHA-256 verification algorithms can be used in combination to improve the calculation speed while ensuring data security.

[0100] In some embodiments, the target cloud storage space side can use an object storage structure to store the above information, where the weak summary information is used as the key value, and the strong summary and the data block number are combined in a specific format as the value. After completing the traversal of the entire target file, the checksum set corresponding to the target file fd is sent to the cloud storage gateway.

[0101] In some embodiments, the cloud storage gateway can verify the files written to the local mount point based on the received verification code set. Similarly, the cloud storage gateway can use a sliding window to slide from the leftmost side of the file written to the local mount point, and calculate the weak summary and strong summary information of the data block with the same verification algorithm, and search from the verification code set. If the same key value exists, the strong summary and data block number information are unlocked in a specific format, and the value of the strong summary is compared. If the values ​​are the same, the sliding window continues to slide backward until the entire file written to the local mount point is traversed. If they are different, it is necessary to store the data block number and the data block content in the sliding window, and use the data block number as the key value and the data block content as the value to generate a difference file.

[0102] In other words, when at least one of the weak summary information, strong summary information and data block content of the file written to the mount point is different, it is determined that there is a difference between the file written to the local mount point and the target file stored in the target cloud storage space, and the file written to the mount point needs to be uploaded to the target cloud storage space.

[0103] Step 302: When there is a difference between the file written to the local mount point and the target file stored in the target cloud storage space, a sliding window is used to traverse and compare the file written to the local mount point and the target file stored in the target cloud storage space to determine the difference data blocks.

[0104] In some embodiments, based on the above comparison results, when there is a difference between the file written to the local mount point and the target file stored in the target cloud storage space, a difference file can be generated, wherein the difference file contains the difference part between the file written to the local mount point and the target file stored in the target cloud storage space, and the difference part is the difference data block.

[0105] In some embodiments, when there is a difference between a file written to a local mount point and a target file stored in a target cloud storage space, generating a difference file includes: when the size or modification timestamp of the file written to the local mount point and the target file stored in the target cloud storage space is different, using a sliding window to traverse and compare the file written to the local mount point and the target file stored in the target cloud storage space to determine the difference data blocks; generating a difference file based on the difference data blocks.

[0106] Specifically, when the size or modification timestamp of the file written to the local mount point is different from that of the target file stored in the target cloud storage space, it means that the file at the local mount point has been updated. Similarly, the sliding window method of step 301 can be used to compare the file written to the local mount point with the target file stored in the target cloud storage space to determine the different parts of the two files, that is, the difference data blocks.

[0107] Specifically, when the file written to the local mount point has the same size or modification timestamp as the target file stored in the target cloud storage space, but it is determined through sliding window verification that there are difference data blocks, a difference file can be generated based on the difference data blocks.

[0108] Step 303: Generate a difference file based on the difference data block.

[0109] In some embodiments, after determining the difference data block according to step 302, a difference file can be generated according to the difference data block. Specifically, the data block number of the difference data block can be used as the key value and the data block content can be used as the value to form the difference file.

[0110] In some embodiments, when the target file saved in the target cloud storage space is empty, it can be determined that the file written to the local mount point is a differential file. Specifically, when the target file saved in the target cloud storage space is empty, it means that the target cloud storage space has not saved the target file corresponding to the file written to the local mount point. At this time, the entire file written to the local mount point needs to be uploaded, so that the file written to the local mount point can be determined as a differential file.

[0111] Step 304: Upload the difference file to the target cloud storage space.

[0112] In some embodiments, after determining the difference file, the cloud storage gateway can upload the difference file to the target cloud storage space to upload the file, so that the target cloud storage space can update the stored target file based on the difference file to achieve synchronization with the locally written file.

[0113] Specifically, when the target file corresponding to the file written to the mount point is not saved in the target cloud storage space, the difference file can be directly saved as the target file to implement the update of the target file.

[0114] Specifically, when the uploaded difference file contains only part of the difference data blocks, the target cloud storage space can determine a part based on the difference data blocks, and update the difference part to update the target file. The specific process is as follows.

[0115] In some embodiments, when the target cloud storage space receives a difference file, it first creates a temporary file, and reorganizes the temporary file through the received difference file. Specifically, for data block numbers that do not exist in the difference file, the content of the corresponding data block number in the target file is directly copied to the location of the temporary file data block number. For existing data block number values, the data block content corresponding to the data block number in the difference file is stored in the temporary file, thereby ensuring that the temporary file is the same as the file written to the local mount point. After the temporary file is reorganized, the name, access rights, modification timestamp, owner and other information of the temporary file are modified so that the final file and the file written to the local mount point remain synchronized. After the temporary file is reorganized, the temporary file can be used as a new target file to update the target file. After the update is completed, the integrity check of the updated target file continues to ensure that there is no data damage or loss during the synchronization process.

[0116] In summary, the above-mentioned embodiments of the present application can scan the files written to the mount point, and when there is a difference between the files written to the mount point and the files stored in the target cloud storage space, a difference file can be generated and uploaded, so as to update the files stored in the target cloud storage space based on the difference file. By uploading only when differences occur, and only uploading the difference parts, it is possible to reduce the pressure on the server, reduce the use of service bandwidth, and save transmission resources.

[0117] For the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the described order of actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously.

[0118] Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0119] The technical solution of the present disclosure is further described in detail below in conjunction with specific application examples.

[0120] The disclosed embodiment provides a cloud computer elastic expansion method based on a cloud storage gateway.

[0121] like Figure 4As shown, the system architecture of this example includes: cloud computer client, cloud storage gateway, other cloud application instances, and other cloud application instance remote storage service end. Among them, the cloud computer client is used to receive the cloud storage information of other cloud application instances input by the user, build a disk capacity prediction model, and serve as the data source end of the storage data.

[0122] In some embodiments, a cloud storage gateway is installed on a cloud computer client, and based on the Common Internet File System (CIFS) protocol, the remote storage services of other cloud application instances are mapped to a network drive on the cloud computer client to achieve integrated storage between the cloud computer client and other cloud application instances; other cloud application instances refer to software applications that provide services over the Internet, which are usually hosted and managed by a third-party service provider. Users can access them over the network, and they have high accessibility, scalability, and easy maintenance, such as mobile cloud space; other cloud application instance remote storage service clients are servers used for remote storage by other cloud application instance clients, and are used to provide powerful data storage and management functions for cloud applications. In this solution, capacity expansion services can be provided for cloud computer clients.

[0123] The application scenario of this example is: the user uses the remote storage information configuration box provided by the cloud computer client. The user can enter the drive letter and the corresponding storage domain name and access key information in the configuration box, and sort according to the distance between the cloud computer user's geographical location and the storage domain name, and save the sorting results in the dictionary data structure. When the predicted value of the cloud computer disk capacity exceeds the preset threshold, the network mapper is mounted through the CIFS protocol to integrate other cloud application instances into the cloud computer client, and other cloud application instances, such as cloud space, can achieve flexible expansion.

[0124] The detailed steps of this program are as follows:

[0125] Step 1: The cloud computer client receives the cloud storage information of other cloud application instances input by the user, including the access domain name and access key information.

[0126] Considering the scenario requirements of this proposal, this proposal specially designs a remote storage information configuration box. Users can enter the drive letter and the corresponding storage domain name and access key information in the configuration box, and sort them according to the distance between the cloud computer user's geographical location and the storage domain name, and save the sorting results in the dictionary data structure.

[0127] When this solution expands storage space, the storage used by other cloud application instances at the bottom layer needs to be an object storage service. Object storage provides services to the outside world in the form of RESTful APIs, and implements data interaction between clients and servers through the HTTP protocol. When requesting to store data, a corresponding request address, i.e., a request URL, is required. In object storage, each resource pool corresponds to a different domain name. To obtain the domain name, users can trigger a jump to the bucket details page that provides cloud services through the bucket list page provided by the cloud storage service console corresponding to other cloud application instances, and view and record the domain name address URL that provides services to the outside world, which consists of the bucket name and the resource pool domain name.

[0128] Since object storage implements permission allocation based on users, each account has an Access key and a secret key, which are keys for accessing the cloud object storage API and have account access permission control. Each account can set different operation permissions for specified resources (such as storage buckets, objects, etc.), including operations such as GetObject, PutObject, and DeleteObject. In order to achieve data isolation between different applications, it is necessary to create a new account for cloud computer users through the access control management component in the cloud storage service console corresponding to other cloud application instances, and specify the account's permissions as read and write. After the creation is complete, you can access the account list component to obtain and record the values ​​of the Access key and the secret key.

[0129] Step 2: The cloud computer client obtains the cloud storage gateway command line application to configure the authentication and connection information for connecting to the remote cloud storage service.

[0130] In order to establish integrated storage between the object storage services of the local cloud computer and other cloud application instances, it is necessary to obtain the cloud storage gateway command line application on the cloud computer client, execute the corresponding executable file, and set its environment variables in the system properties so that it can be operated through the command line in the cloud computer terminal. At the same time, the configuration file is generated through the cloud storage gateway instructions.

[0131] The configuration file includes the following fields: type, which is used to specify the storage type. In this example, the type value can be set to S3. env_auth is used to indicate whether verification is enabled. In this example, it can be set to false. The value of the access_key_id field is the Access key recorded in step 1, which is used to complete user authentication and ensure user data isolation in the cloud storage service. The value of the secret_access_keys field is the secret key in step 1. The endpoint is the endpoint, and the value is the URL of the object storage bucket obtained in step 1, that is, the interface path when accessing data requests. The region field is used to indicate the resource pool where the object storage bucket is located. In order to achieve lower access latency, the specific value should be combined with the geographical location of the cloud computer user.

[0132] In the design of this step, the configuration file ensures that all necessary authentication and connection information for accessing the remote cloud storage service is configured in the cloud computer client.

[0133] Step 3: Obtain the historical data disk and system disk capacity and IOPS information of the cloud computer in the preset time period as training data, and build a cloud computer disk capacity prediction model;

[0134] When considering disk capacity prediction, a single prediction model may fail to capture the complex relationships in disk capacity data, resulting in poor performance of the model on both the training set and the test set. In addition, due to the different access frequencies of cloud computer users to system disks and data disks and the different disk I / O, it cannot be generalized to different data sets or different application scenarios. Therefore, this solution designs an integrated model to build a cloud computer disk capacity prediction model. The specific steps of the model construction process include:

[0135] Sub-step 3-1: Obtain the historical data disk, system disk capacity and IOPS information of the cloud computer in a preset time period, and divide it into a training set and a label set.

[0136] Specifically, first obtain the historical data disk, system disk disk capacity and IOPS information of the cloud computer in a preset time period, use the median filter and sliding average method to remove noise, smooth and clean the data and pre-process it, and divide the historical capacity data set into a training set and a label set.

[0137] Sub-step 3-2: Select a first prediction model and iteratively train the model using a disk I / O performance training data set.

[0138] Specifically, this solution selects the vector machine prediction model as the first prediction model. The disk I / O performance data can be used as a feature of the prediction model. The training data includes two parts: features and labels. The disk I / O performance data can be used as a feature of the prediction model to describe sample information.

[0139] The initial vector machine model is iteratively trained using the training data set. The model of the vector machine can be expressed as:

[0140]

[0141] Among them, K(x i ,x)=exp(-‖xx i ‖ 2 / 2λ 2 ) represents the kernel function, which is the support vector x i The inner product between the new input sample x, x represents the new input sample, x i is the support vector (also called eigenvector), y i is the corresponding category label, b is the hyperplane parameter, is the Lagrangian factor, L is the support vector x i The number of

[0142] Based on the prediction results and the test set, the model parameters are adjusted to improve the prediction performance of the model. When the iterative training meets the target conditions, the trained model is used as the final vector machine model to predict whether the disk capacity will exceed the threshold.

[0143] Sub-step 3-3: Select the second prediction model, and continuously fit it through the gradient descent method to obtain a disk usage prediction model.

[0144] Specifically, this solution selects the ARMA time series model. Data within the second preset time period is selected from the training data set as the first time series data, and a time series prediction model is established according to the time series prediction model and the first time series data set, including an encoder module and a decoder module. The encoder module extracts features and labels from the input data and passes these features to the decoder. The label data is input into the decoder module for processing to obtain the predicted data, the mean absolute error function is used as the loss function, the sliding window feature is added, the gap between the disk utilization prediction data and the label data is calculated, and the disk utilization prediction model is obtained by continuous fitting through the gradient descent method.

[0145] The predicted value of the capacity of the integrated model at a certain moment is the weighted predicted value of the above two models, and its value is:

[0146]

[0147] Among them, Rt is the predicted value at time t, ε i is the model weight coefficient, r jt is the predicted value of the jth model at time t, where the weighted coefficient satisfies:

[0148]

[0149] Among them, E i is the absolute error of a single model. This scheme uses the mean absolute error as the loss function, expressed as The final disk capacity prediction model is obtained by continuous fitting through the gradient descent method.

[0150] The collection model proposed in this example can obtain a more accurate disk capacity prediction model and reduce the impact of frequent mounting and unmounting of network mappers on the performance of cloud computer clients.

[0151] Step 4: Based on the disk usage prediction model, the time node when the disk capacity exceeds the threshold is estimated, and the expansion time of the cloud computer client is calculated based on the preset capacity threshold. The remote cloud storage services of other cloud application instances are mounted to the cloud computer client based on the CIFS protocol.

[0152] Based on the above prediction model and combined with the preset capacity threshold, the expansion time of the cloud computer client is calculated. When expansion is needed, the remote cloud storage services of other cloud application instances are mounted to the cloud computer client based on the CIFS protocol to avoid data loss caused by failure to expand capacity in time when data surges.

[0153] According to the dictionary structure in step 1, when the storage capacity is sufficient, the first object in the dictionary structure is preferentially selected as the service providing remote cloud storage, and a mount directory is created on the cloud computer client according to the drive letter of the first object, and the object storage of other cloud application instances is mounted to the cloud computer client through the mount command based on the CIFS protocol. This operation allows the file system to be implemented in the user space without modifying or mutating the kernel, and mapping it to a local drive letter.

[0154] When using the mount command to mount, you need to add the configuration name, bucket name, and mount drive letter after the mount command, where the configuration name is the configuration file name created in step 2, the bucket name is the bucket name of the object storage corresponding to the application instance confirmed by the first object in step 1, and the mount drive letter is the drive letter corresponding to the first object, that is, the drive letter of the remote object storage that the user wants to display on the cloud computer. In addition to the above mount commands, you can also set the cache timeout and cache mode of the file attributes to improve the efficiency of file reading and writing.

[0155] With the above command, you can create a virtual file system on the cloud computer client and map the object storage space of other cloud application instances to the local cloud computer. After the mount is completed, you can check whether it is successfully mounted through the file manager or terminal. If a new drive letter appears, the mount is successful, and the cloud computer user can store and use data like a local disk.

[0156] Step 5: After the mount is completed, the cloud storage gateway scans the files written to the mount point in real time and periodically uploads them to the object storage service corresponding to other cloud application instances.

[0157] Considering that frequently starting synchronization tasks can easily cause the server bandwidth and resources to be occupied, affecting the data processing capacity of the cloud storage gateway. After the mounting is completed, this solution is designed to set a scheduled task in the cloud computer task scheduler. The information of the scheduled task includes task name, description, time interval, etc., and the script path is specified in the startup program in operation to achieve delayed upload of files. The functions of the script file include scanning files written to the mount point, comparing the differences between new and old files, and uploading files to the object storage bucket of other cloud application instances.

[0158] When comparing the differences between new and old files, the overall process of this script is to slide the sliding window from the leftmost side of the file to be uploaded in turn. Each time it slides, the checksum of the data in the sliding window is calculated and matched with the checksum in the checksum set corresponding to the target file. If a data block entry in the checksum set can be matched, it means that the data block is the same as a data block in the object storage bucket and does not need to be transmitted. If they are different, it means that the data block in the sliding window has been updated and needs to be uploaded. After the match is completed, the sliding window continues to slide to the end offset address of the data block, and continues to take data blocks from this offset for matching. Repeat the above steps until the end of the file. The specific steps are as follows:

[0159] (1) Preliminary inspection

[0160] The script periodically scans the files written to the mount point. When the file written to the mount point is fs, it will first determine whether there is a file fs in the bucket before transferring it to the object storage bucket. If not, it will be uploaded directly. If so, it will compare the size and modification timestamp of the file fs and the target file fd in the object storage bucket. If they are different, the file must have been updated and continue to compare the difference.

[0161] (2) Set the sliding window size

[0162] Create a window and set the size of the sliding window. The size of the window affects the calculation efficiency. If the sliding window is too small, more data blocks need to be processed, resulting in increased calculation and matching time; if the window is too large, it may lead to reduced transmission efficiency. For example, the size of the sliding window can be set to 700 bytes.

[0163] (3) Obtain the verification code set of the target file

[0164] The window starts to slide from the leftmost side of the target file fd until the entire file is traversed. Each time the window slides, the number of the data block in the window is recorded, and the starting offset address and length of the data block are recorded. The window data is encoded using the Adler-32 and SHA-256 verification algorithms respectively, and weak and strong summaries are generated respectively. The Adler-32 calculation speed is relatively fast and can quickly detect whether the data has changed, but the probability of conflict is relatively high. The SHA-256 verification algorithm has a high calculation complexity, but the probability of conflict is small and the security is higher. This scheme proposes to use the two verification algorithms in combination to improve the calculation speed while ensuring the security of the data.

[0165] To further reduce the complexity of subsequent searches, this proposal uses an object storage structure to store the above information, where weak summary information is used as the key value, and strong summary and data block number are combined in a specific format as the value. After completing the traversal of the entire target file, the checksum set corresponding to the target file fd is sent to the cloud computer.

[0166] After the cloud computer client receives the verification code set, the sliding window starts to slide from the leftmost side of the file to be uploaded, and calculates the weak summary and strong summary information of the data block with the same verification algorithm, and searches from the verification code set. If the same key value exists, the strong summary and data block number information are unlocked in a specific format, and the value of the strong summary is compared. If the values ​​are the same, the sliding window continues to slide backward until the entire file to be uploaded is traversed. If they are different, the data block number and the data block content in the sliding window need to be stored, and the data block number is used as the key value and the data block content as the value to form a difference file and send it to the remote object storage service.

[0167] When the object storage service receives a differential file, it first creates a temporary file and reorganizes the temporary file using the received differential file. Specifically, for keys that do not exist in the differential file, the data block content of the corresponding key in the target file fd is directly copied to the location of the temporary file key. For existing key values, the corresponding data block in the differential file is stored in the temporary file, thereby ensuring that the temporary file is the same as the fs file. After the temporary file is reorganized, the name, access rights, modification timestamp, owner and other information of the temporary file are modified so that the final file and the fs file are synchronized. After the reorganization is completed, continue to perform integrity checks on the updated file to ensure that there is no data damage or loss during the synchronization process.

[0168] After the above steps, the purpose of elastic expansion of cloud computer storage space can be achieved, and because of differential upload, the pressure on the server and network bandwidth are saved. While making full use of personal cloud application storage space, the storage space of cloud computers is elastically expanded, allowing users to store data like operating local disks.

[0169] In summary, the above examples of the present disclosure achieve efficient, secure and flexible data storage and management through virtualization technology and distributed architecture, realize the purpose of sharing storage space across cloud applications, and provide users with a convenient and reliable cloud storage environment.

[0170] Other advantages of this example include:

[0171] (1) Through intelligent analysis and cloud storage gateway hybrid cloud storage services, in the face of a surge in disk storage capacity data and disk I / O, cloud PC clients can be elastically expanded without manual operation, thereby improving the stability of cloud PC client services and data security at a lower cost.

[0172] (2) When determining whether it is necessary to expand the capacity of the cloud computer client, the disk capacity prediction model and capacity threshold are combined to accurately predict the future automatic expansion time, thereby reducing the impact of frequent mounting and unmounting of the network mapper on the performance of the cloud computer client.

[0173] (3) Capacity can be elastically expanded or reduced based on usage needs. When the demand for capacity storage is high, the remote cloud storage service can be mounted to the cloud computer client. When the demand is low, the relevant mount points can be unmounted to achieve the purpose of elastic expansion and reduction and maximize resource utilization.

[0174] (4) When uploading files written to the mount point of the cloud computer client to the remote object storage service corresponding to other cloud application instances, this solution proposes to upload only new files and updated parts of existing files, thereby reducing the pressure on the server and reducing the use of service bandwidth.

[0175] Figure 51 is a block diagram of a cloud computer integrated storage device 500 provided in an embodiment of the present disclosure. Figure 5 As shown, the device 500 includes: a first processing unit 510, used to determine, based on local storage data, a predicted time when the usage of the local storage space exceeds a preset threshold, the predicted time being predicted by using a trained target prediction model; a second processing unit 520, used to determine, based on a storage information configuration file, a target cloud storage space from at least one cloud storage space included in the storage information configuration file, the cloud storage space being used for cloud application storage data; a third processing unit 530, used to mount the target cloud storage space to a local mount point before the predicted time; and a fourth processing unit 540, used to upload files written to the local mount point to the target cloud storage space.

[0176] In summary, the above-mentioned device of the present invention can predict the time when the local storage space usage exceeds a preset threshold through a target prediction model. By mounting the storage space of other cloud applications to the local before the predicted time, the local storage space can be automatically expanded, data surge can be avoided, and the expansion efficiency can be improved. After mounting the storage space of other cloud applications to the local, integrated storage of local and cloud applications can be achieved, the utilization rate of storage space can be improved, storage resources can be balanced, and resource waste can be avoided.

[0177] In some embodiments, the cloud computer integrated storage device also includes a fifth processing unit, which is used to obtain local storage data for a preset time period, generate a training set and a test set based on the local storage data for the preset time period; train a vector machine prediction model based on the training set and the test set; train a time series model based on the training set and the test set; and perform integrated fitting processing on the trained vector machine prediction model and the time series model to obtain a target prediction model.

[0178] In some embodiments, the cloud computer integrated storage device also includes a sixth processing unit for receiving a storage information configuration file, the storage information configuration file including configuration information of at least one cloud storage space; authenticating and connecting at least one cloud storage space according to the configuration information of at least one cloud storage space contained in the storage information configuration file.

[0179] In some embodiments, the sixth processing unit is also used to determine the access key identifier and private key identifier of at least one cloud storage space based on the configuration information of at least one cloud storage space; and authenticate and connect at least one cloud storage space based on the access key identifier and private key identifier of at least one cloud storage space.

[0180] In some embodiments, the second processing unit is further used to determine a sorting result of at least one cloud storage space according to the storage information configuration file; and determine the cloud storage space whose sorting result meets a preset condition as the target storage space.

[0181] In some embodiments, the fourth processing unit is also used to use a sliding window to traverse and compare the files written to the local mount point and the target files stored in the target cloud storage space to determine whether there are differences between the files written to the local mount point and the target files stored in the target cloud storage space; when there are differences between the files written to the local mount point and the target files stored in the target cloud storage space, use a sliding window to traverse and compare the files written to the local mount point and the target files stored in the target cloud storage space to determine difference data blocks; generate difference files based on the difference data blocks; and upload the difference files to the target cloud storage space.

[0182] Figure 6 A block diagram of an electronic device 600 for implementing the above method is provided in an embodiment of the present disclosure.

[0183] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present disclosure, the embodiment of the present disclosure also provides an electronic device, such as Figure 6 As shown, the electronic device 600 includes:

[0184] Communication interface 601, capable of exchanging information with other devices;

[0185] The processor 602 is connected to the communication interface 601 to implement information interaction with other devices, and is used to execute the method provided by one or more of the above technical solutions when running the computer program;

[0186] A memory 603 on which the computer program is stored.

[0187] Specifically, the processor 602 may be configured to determine a first family to which the first object belongs based on the communication data of the first object; determine the first family as a target family when the first family satisfies a first preset condition; and push target information to the target family.

[0188] It should be noted that the specific processing process of the processor 602 can be understood by referring to the above method.

[0189] Of course, in actual application, the various components in the electronic device 600 are coupled together through the bus system 604. It can be understood that the bus system 604 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 6 Various buses are labeled as bus system 604 .

[0190] The memory 603 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 600. Examples of such data include: any computer program used to operate on the electronic device 600.

[0191] The method disclosed in the above embodiment of the present application can be applied to the processor 602, or implemented by the processor 602. The processor 602 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 602. The above-mentioned first processor 602 can be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 602 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 603, and the processor 602 reads the information in the memory 603 and completes the steps of the above method in combination with its hardware.

[0192] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0193] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the above instructions can be executed by the processor 620 of the electronic device 600 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0194] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the above embodiments of the present disclosure.

[0195] The embodiment of the present disclosure also provides a chip, such as Figure 7As shown, the chip includes a processor 710 and an interface 720. The number of the processor 710 may be one or more, the number of the interface 720 may be multiple, and the interface circuit is used to receive a signal from the memory of the electronic device and send a signal to the processor, the signal includes a computer instruction stored in the memory, and when the processor executes the computer instruction, the electronic device executes the method described in the above embodiment of the present disclosure.

[0196] It should be noted that, for the convenience of description, only the parts related to the relevant invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0197] It should be understood that the "system", "device", "unit" and / or "module" used in this application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0198] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0199] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0200] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0201] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0202] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0203] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0204] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0205] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0206] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0207] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent commodity, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0208] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A cloud computer integrated storage method, characterized in that: The method comprises: Determine, based on the local storage data, a predicted time at which the usage of the local storage space exceeds a preset threshold, wherein the predicted time is predicted using a trained target prediction model; According to the storage information configuration file, determining a target cloud storage space from at least one cloud storage space included in the storage information configuration file, the cloud storage space being used for cloud application storage data; Before the predicted time, mounting the target cloud storage space to a local mount point; Upload the files written to the local mount point to the target cloud storage space.

2. The method according to claim 1, characterized in that After determining, based on the local storage data, a predicted time when the local storage space usage exceeds a preset threshold, the method further includes: Acquire local storage data for a preset time period, and generate a training set and a test set based on the local storage data for the preset time period; According to the training set and the test set, training a vector machine prediction model; Training a time series model based on the training set and the test set; The trained vector machine prediction model and the time series model are integrated and fitted to obtain the target prediction model.

3. The method according to claim 1, characterized in that Before mounting the target cloud storage space to the local mount point, the method further includes: receiving a storage information configuration file, wherein the storage information configuration file includes configuration information of the at least one cloud storage space; According to the configuration information of at least one cloud storage space included in the storage information configuration file, the at least one cloud storage space is authenticated and connected.

4. The method according to claim 3, characterized in that The authenticating and connecting the at least one cloud storage space according to the configuration information of the at least one cloud storage space included in the storage information configuration file comprises: Determining an access key identifier and a private key identifier of the at least one cloud storage space according to configuration information of the at least one cloud storage space; Based on the access key identifier and the private key identifier of the at least one cloud storage space, the at least one cloud storage space is authenticated and connected.

5. The method according to claim 1, characterized in that The step of determining a target cloud storage space from at least one cloud storage space included in the storage information configuration file comprises: Determining a sorting result of the at least one cloud storage space according to the storage information configuration file; The cloud storage space whose sorting result meets the preset condition is determined as the target storage space.

6. The method according to claim 1, characterized in that The uploading of the file written to the local mount point to the target cloud storage space includes: Using a sliding window to traverse and compare the file written to the local mount point and the target file stored in the target cloud storage space to determine whether there is a difference between the file written to the local mount point and the target file stored in the target cloud storage space; When there is a difference between the file written to the local mount point and the target file stored in the target cloud storage space, a sliding window is used to traverse and compare the file written to the local mount point and the target file stored in the target cloud storage space to determine a difference data block; generating a difference file based on the difference data block; The difference file is uploaded to the target cloud storage space.

7. A cloud computer integrated storage device, characterized in that: The device includes: A first processing unit is used to determine, based on the local storage data, a predicted time when the usage of the local storage space exceeds a preset threshold, where the predicted time is predicted by using a trained target prediction model; a second processing unit, configured to determine, according to the storage information configuration file, a target cloud storage space from at least one cloud storage space included in the storage information configuration file, the cloud storage space being used for cloud application storage data; A third processing unit is used to mount the target cloud storage space to a local mount point before the predicted time; The fourth processing unit is used to upload the file written into the local mount point to the target cloud storage space.

8. An electronic device, characterized in that: include: one or more processors; A storage device communicatively connected to the one or more processors, on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images