Distributed data optimization storage method

Abstract

The invention belongs to the technical field of distributed storage, and provides a distributed data optimization storage method, which comprises the steps of basic data acquisition, basic feature extraction, k proximity activeness prediction and storage space allocation scheme matching. According to the method, the access frequency, the access time, the access IP region, the file size and the file label of the to-be-distributed storage file are collected, so that the basic attribute information of the to-be-distributed storage file is obtained, and data input is provided for activity prediction; through basic feature extraction, form conversion of basic data is realized, and an input vector is provided for the subsequent process; through the k proximity activeness prediction model, the activeness of the to-be-allocated storage file is predicted, and an allocation basis is provided for a storage scheme; through the storage space allocation scheme library, the pre-allocation of the to-be-allocated storage file is realized, the file access efficiency is improved, and the system response time is reduced.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a distributed data optimization storage method. Background Art

[0002] Distributed storage is a technology that stores data in a dispersed manner across multiple physical or virtual nodes. This storage method aims to improve the availability, reliability, and scalability of data. Compared with traditional centralized storage, distributed storage has the advantages of more secure data, more flexible storage, and stronger read and write performance. However, general distributed storage systems generally use the same storage solution for different data, resulting in low utilization of storage resources and slower system response time as the amount of stored data increases. Patent CN 109669636 B provides a distributed intelligent storage system that alleviates the problems of distributed storage to a certain extent through active value analysis.

[0003] However, existing distributed storage technologies do not have the ability to predict active data, adjust it in advance, and allocate it specifically to different storage media. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a distributed data optimization storage method, which obtains basic attribute information of the storage file to be allocated by collecting the number of accesses, access time, access IP region, file size and file label of the storage file to be allocated, and provides data input for activity prediction; through basic feature extraction, the form conversion of basic data is realized to provide input vectors for subsequent use; through the k-nearest activity prediction model, the activity of the storage file to be allocated is predicted to provide an allocation basis for the storage plan; through the storage space allocation plan library, the pre-allocation of the storage file to be allocated is realized, the access efficiency of the file is improved, and the system response time is reduced.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A distributed data optimization storage method, comprising:

[0007] Obtain basic data of the file to be allocated for storage; the basic data includes: number of accesses, access time, access IP region, file size, and file tag;

[0008] Extracting basic features of the storage file to be allocated according to the basic data;

[0009] Inputting the basic features into a pre-trained k-nearest activity prediction model to obtain a target activity prediction result;

[0010] The target activity prediction result is matched with a preset storage space allocation solution library to obtain a target migration solution.

[0011] Preferably, the training process of the k-nearest activity prediction model includes:

[0012] Use the preset server to build a data access database;

[0013] The data access database is divided according to hours to obtain a data access hour set; the data access hour set includes: the basic data of all storage files in the same time period arranged in date order;

[0014] Performing data cleaning on the data access hour set to obtain a data hour set to be trained;

[0015] Extracting features from the hourly data set to be trained to obtain a feature set to be trained;

[0016] Dividing the feature set to be trained into a training set and a test set;

[0017] Construct KNN model, loss function and optimization algorithm;

[0018] Training the KNN model according to the training set, and using the KNN model to perform prediction to obtain a prediction set;

[0019] According to the prediction set and the test set, the KNN model is iteratively optimized using the loss function and the optimization algorithm to obtain an optimized k-nearest activity prediction model.

[0020] Preferably, extracting basic features of the to-be-allocated storage file according to the basic data includes:

[0021] Determine the access time of the storage file to be allocated to obtain a time vector;

[0022] Determine the level of the access count of the storage file to be allocated in a preset access lookup table to obtain a count vector;

[0023] Determine the server serial number closest to the access IP region of the storage file to be allocated, and obtain a region vector;

[0024] Determine the level of the file size of the to-be-allocated storage file in a preset capacity lookup table to obtain a size vector;

[0025] Determine the sequence number of the file label of the to-be-allocated storage file in a preset label lookup table to obtain a label vector;

[0026] The time vector, the number vector, the region vector, the size vector and the label vector are sequentially combined to obtain the basic features.

[0027] Preferably, the loss function is: Wherein, loss is the loss function; h is the time period; n h is the total number of training data in the hth time period; α h is the loss weight of the hth time period; x hi is the actual number of visits to the i-th training data in the h-th time period; is the predicted number of visits to the i-th training data in the h-th time period.

[0028] Preferably, the target activity prediction result is matched with a preset storage space allocation solution library to obtain a target migration solution, including:

[0029] When the target activity prediction result is low activity, storing the to-be-allocated storage file in a low-speed large-capacity storage device;

[0030] When the target activity prediction result is medium activity, determining the data size of the to-be-allocated storage file to obtain a first determination result;

[0031] Allocate the server serial number for the storage file to be allocated according to the access IP region;

[0032] When the first judgment result is a small-capacity file, storing the to-be-allocated storage file in a medium-speed large-capacity device corresponding to the server serial number;

[0033] When the first judgment result is a large-capacity file, splitting the to-be-allocated storage file into a header file and a remaining file in a reading order, storing the header file in the medium-speed large-capacity device corresponding to the server serial number, and storing the remaining file in the low-speed large-capacity storage device;

[0034] When the target activity prediction result is high activity, determining the data size of the to-be-allocated storage file to obtain a second determination result;

[0035] Allocate the server serial number for the storage file to be allocated according to the access IP region;

[0036] When the second judgment result is a small-capacity file, storing the to-be-allocated storage file in a high-speed, low-capacity device corresponding to the server serial number;

[0037] When the second judgment result is a large-capacity file, the to-be-allocated storage file is split into a plurality of block files according to a reading order, and all the block files are dispersedly stored in a plurality of the high-speed and low-capacity devices that are closest to the server serial number.

[0038] Preferably, it also includes:

[0039] When the target user downloads the header file, the remaining file is evenly split into a plurality of data segments, and all the data segments are synchronously transmitted to the terminal device of the target user;

[0040] After the header file and all the data fragments are completely transmitted, the header file and all the data fragments are sequentially spliced.

[0041] Preferably, it also includes:

[0042] Collect the upload request of the target user, authenticate the upload request of the target user, and obtain the authentication result;

[0043] When the authentication result is passed, parsing the address information in the upload request, and allocating a temporary storage server serial number according to the address information;

[0044] Receiving the uploaded data of the target user, obtaining a temporary uploaded file, and storing the uploaded file in a medium-speed and large-capacity device corresponding to the serial number of the temporary storage server;

[0045] Extracting temporary summary information of the temporarily uploaded file;

[0046] According to a preset summary information database, similarity calculation is performed on the temporary summary information using cosine similarity to obtain repetition rate information;

[0047] When the repetition rate information is 1, the address pointer of the temporary upload file is updated, and the temporary upload file in the medium-speed large-capacity device corresponding to the temporary storage server serial number is cleared.

[0048] Preferably, extracting temporary summary information of the temporary uploaded file includes:

[0049] Identify the data type of the temporarily uploaded file and obtain a type identification result; the type identification result includes: text data and video data;

[0050] When the type identification result is the video data, the key frame images of the temporary uploaded file are extracted, and all the key frame images are combined to obtain the temporary summary information.

[0051] The present invention discloses the following technical effects:

[0052] The present invention provides a distributed data optimization storage method. By collecting the number of accesses, access time, access IP region, file size and file label of storage files to be allocated, the problem of the source of input data for a prediction model is solved, and the acquisition of basic data is realized; by extracting basic features, the problem of vectorization of basic data is solved, and the form conversion of basic data is realized; by using a k-nearest neighbor activity prediction model, the problem of predicting the activity of storage files to be allocated is solved, and the acquisition of allocation basis is realized; by using a storage space allocation solution library, the defects of low file access efficiency and long system response time are solved, and the pre-allocation of storage files to be allocated is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 A schematic diagram of a distributed data optimization storage process provided by an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of the training process of the k-neighboring activity prediction model provided in an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of the basic feature extraction process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The purpose of the present invention is to provide a distributed data optimization storage method, which obtains basic attribute information of the storage file to be allocated by collecting the number of accesses, access time, access IP region, file size and file label of the storage file to be allocated, and provides data input for activity prediction; through basic feature extraction, the form conversion of basic data is realized to provide input vectors for subsequent use; through a k-nearest activity prediction model, the activity of the storage file to be allocated is predicted to provide an allocation basis for the storage plan; through a storage space allocation plan library, the pre-allocation of the storage file to be allocated is realized, the access efficiency of the file is improved, and the system response time is reduced.

[0059] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] Figure 1 A schematic diagram of a distributed data optimization storage process provided by an embodiment of the present invention, such as Figure 1 As shown, the present invention provides a distributed data optimization storage method, comprising:

[0061] Obtain basic data of the file to be allocated for storage; the basic data includes: number of accesses, access time, access IP region, file size, and file tag;

[0062] Extracting basic features of the storage file to be allocated according to the basic data;

[0063] Inputting the basic features into a pre-trained k-nearest activity prediction model to obtain a target activity prediction result;

[0064] The target activity prediction result is matched with a preset storage space allocation solution library to obtain a target migration solution.

[0065] refer to Figure 2 The training process of the k-nearest activity prediction model includes:

[0066] Step 301: constructing a data access database using a preset server;

[0067] Step 302: Divide the data access database by hour to obtain a data access hour set; the data access hour set includes: the basic data of all storage files in the same time period arranged in date order;

[0068] Step 303: performing data cleaning on the data access hour set to obtain a data hour set to be trained;

[0069] Step 304: extracting features from the hourly data set to be trained to obtain a feature set to be trained;

[0070] Step 305: Divide the feature set to be trained into a training set and a test set;

[0071] Step 306: construct a KNN model, a loss function, and an optimization algorithm;

[0072] Step 307: training the KNN model according to the training set, and using the KNN model to perform prediction to obtain a prediction set;

[0073] Step 308: According to the prediction set and the test set, the KNN model is iteratively optimized using the loss function and the optimization algorithm to obtain an optimized k-nearest activity prediction model.

[0074] refer to Figure 3 , extracting basic features of the storage file to be allocated according to the basic data, including:

[0075] Step 201: Determine the access time of the storage file to be allocated and obtain a time vector;

[0076] Step 202: Determine the level of the access count of the storage file to be allocated in a preset access lookup table, and obtain a count vector;

[0077] Step 203: Determine the server serial number closest to the access IP region of the to-be-allocated storage file, and obtain a region vector;

[0078] Step 204: Determine the level of the file size of the to-be-allocated storage file in the preset capacity lookup table to obtain a size vector;

[0079] Step 205: Determine the sequence number of the file tag of the to-be-allocated storage file in a preset tag lookup table to obtain a tag vector;

[0080] Step 206: sequentially combine the time vector, the frequency vector, the region vector, the size vector and the label vector to obtain the basic features.

[0081] Specifically, the implementation principle of the KNN nearest neighbor classification algorithm is: in order to determine the category of an unknown sample, all samples of known categories are used as references, the distance between the unknown sample and all known samples is calculated, and the K known samples closest to the unknown sample are selected. According to the majority-voting rule, the unknown sample and the K nearest neighbor samples are classified into the same category. The key to the KNN algorithm is that all features of the sample must be comparable and quantified. If there are non-numerical types in the sample features, they must be quantified into numerical values. For example, if the sample features contain color, the distance calculation can be achieved by converting the color into grayscale values. The sample features must be normalized. The sample has multiple parameters, each of which has its own domain and value range. They have different effects on the distance calculation. For example, the influence of a larger value will overshadow the parameter with a smaller value. Therefore, the sample parameters must be scaled. The simplest way is to normalize the values ​​of all features. A distance function is needed to calculate the distance between two samples. Commonly used distance functions include: Euclidean distance, cosine distance, Hamming distance, Manhattan distance, etc. Euclidean distance is generally selected as the distance metric, but this is only applicable to continuous variables. In the case of non-continuous variables such as text classification, the Hamming distance can be used as a metric. Under normal circumstances, if some special algorithms are used to calculate the metric, the accuracy of K-nearest neighbor classification can be significantly improved, such as using the large margin nearest neighbor method or the nearest neighbor component analysis method. Based on the implementation principle of knn, this embodiment uses the regression analysis capability of knn as the core to achieve activity prediction.

[0082] Preferably, the time vector, the frequency vector, the region vector, the size vector and the label vector are all represented by 01; wherein, the time vector, the frequency vector and the size vector are all two-dimensional vectors; the region vector is a six-dimensional vector (if the implementer has more servers, the dimension of the region vector can be increased accordingly); the label vector is dimensioned according to the valuable tags in the database (generally, tags ranked high in the order of the number of citations are used).

[0083] Preferably, the loss function is: Wherein, loss is the loss function; h is the time period; n h is the total number of training data in the hth time period; α h is the loss weight of the hth time period; x hi is the actual number of visits to the i-th training data in the h-th time period; is the predicted number of visits to the i-th training data in the h-th time period.

[0084] Furthermore, the target activity prediction result is matched with a preset storage space allocation solution library (see Table 1) to obtain a target migration solution, including:

[0085] When the target activity prediction result is low activity, storing the to-be-allocated storage file in a low-speed large-capacity storage device;

[0086] When the target activity prediction result is medium activity, determining the data size of the to-be-allocated storage file to obtain a first determination result;

[0087] Allocate the server serial number for the storage file to be allocated according to the access IP region;

[0088] When the first judgment result is a small-capacity file, storing the to-be-allocated storage file in a medium-speed large-capacity device corresponding to the server serial number;

[0089] When the first judgment result is a large-capacity file, splitting the to-be-allocated storage file into a header file and a remaining file in a reading order, storing the header file in the medium-speed large-capacity device corresponding to the server serial number, and storing the remaining file in the low-speed large-capacity storage device;

[0090] When the target activity prediction result is high activity, determining the data size of the to-be-allocated storage file to obtain a second determination result;

[0091] Allocate the server serial number for the storage file to be allocated according to the access IP region;

[0092] When the second judgment result is a small-capacity file, storing the to-be-allocated storage file in a high-speed, low-capacity device corresponding to the server serial number;

[0093] When the second judgment result is a large-capacity file, the to-be-allocated storage file is split into a plurality of block files according to a reading order, and all the block files are dispersedly stored in a plurality of the high-speed and low-capacity devices that are closest to the server serial number.

[0094] Table 1

[0095]

[0096]

[0097] Preferably, it also includes:

[0098] When the target user downloads the header file, the remaining file is evenly split into a plurality of data segments, and all the data segments are synchronously transmitted to the terminal device of the target user;

[0099] After the header file and all the data fragments are completely transmitted, the header file and all the data fragments are sequentially spliced.

[0100] Furthermore, it also includes:

[0101] Collect the upload request of the target user, authenticate the upload request of the target user, and obtain the authentication result;

[0102] When the authentication result is passed, parsing the address information in the upload request, and allocating a temporary storage server serial number according to the address information;

[0103] Receiving the uploaded data of the target user, obtaining a temporary uploaded file, and storing the uploaded file in a medium-speed and large-capacity device corresponding to the serial number of the temporary storage server;

[0104] Extracting temporary summary information of the temporarily uploaded file;

[0105] According to a preset summary information database, similarity calculation is performed on the temporary summary information using cosine similarity to obtain repetition rate information;

[0106] When the repetition rate information is 1, the address pointer of the temporary upload file is updated, and the temporary upload file in the medium-speed large-capacity device corresponding to the temporary storage server serial number is cleared.

[0107] Specifically, in the field of data analysis, cosine similarity is used to measure the similarity between two non-zero vectors in the inner product space. It is equal to the cosine of the angle between the two vectors, that is, the dot product of the vectors divided by the product of their lengths. Therefore, cosine similarity has nothing to do with the size of the vectors, but only with their angle. Cosine similarity always belongs to an interval, for example, the cosine similarity of two proportional vectors is 1, the similarity of two orthogonal vectors is 0, and the similarity of two opposite vectors is -1. In some cases, the component values ​​of the vectors cannot be negative, in which case the cosine similarity is bounded. For example, in information retrieval and text mining, each word corresponds to a unique coordinate. Documents are represented by vectors of the frequency of word occurrences in them. Therefore, we can use cosine similarity to measure the similarity of two documents in terms of topics, which is independent of the length of the documents. In the field of data mining, this technique is also used to measure the cohesion within clusters. One advantage of cosine similarity is that it has low computational complexity, especially for sparse vectors, because it only considers non-zero coordinates. Cosine similarity is the cosine of the angle between two n-dimensional vectors in n-dimensional space. It is equal to the dot product (vector product) of the two vectors divided by the product of the lengths (or sizes) of the two vectors. Its calculation formula is:

[0108]

[0109] Wherein, S is the repetition rate information; A and B are two pieces of temporary summary information for similarity comparison.

[0110] Preferably, extracting temporary summary information of the temporary uploaded file includes:

[0111] Identify the data type of the temporarily uploaded file and obtain a type identification result; the type identification result includes: text data and video data;

[0112] When the type identification result is the video data, the key frame images of the temporary uploaded file are extracted, and all the key frame images are combined to obtain the temporary summary information.

[0113] The beneficial effects of the present invention are as follows:

[0114] The present invention obtains basic attribute information of the storage files to be allocated by collecting the number of accesses, access time, access IP region, file size and file tag of the storage files to be allocated, and provides data input for activity prediction; realizes the form conversion of basic data through basic feature extraction, and provides input vectors for subsequent use; realizes the prediction of the activity of the storage files to be allocated through the k-nearest activity prediction model, and provides an allocation basis for the storage plan; realizes the pre-allocation of the storage files to be allocated through the storage space allocation plan library, improves the access efficiency of the files, and reduces the system response time

[0115] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0116] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A distributed data optimization storage method, characterized in that: include: Get the basic data of the storage file to be allocated; The basic data includes: number of visits, visit time, visit IP region, file size and file tag; Extracting basic features of the storage file to be allocated according to the basic data; Inputting the basic features into a pre-trained k-nearest activity prediction model to obtain a target activity prediction result; The target activity prediction result is matched with a preset storage space allocation solution library to obtain a target migration solution.

2. The distributed data optimization storage method according to claim 1, characterized in that: The training process of the k-nearest activity prediction model includes: Use the preset server to build a data access database; The data access database is divided according to hours to obtain a data access hour set; the data access hour set includes: the basic data of all storage files in the same time period arranged in date order; Performing data cleaning on the data access hour set to obtain a data hour set to be trained; Extracting features from the hourly data set to be trained to obtain a feature set to be trained; Dividing the feature set to be trained into a training set and a test set; Construct KNN model, loss function and optimization algorithm; Training the KNN model according to the training set, and using the KNN model to perform prediction to obtain a prediction set; According to the prediction set and the test set, the KNN model is iteratively optimized using the loss function and the optimization algorithm to obtain an optimized k-nearest activity prediction model.

3. The distributed data optimization storage method according to claim 1, characterized in that: Extracting basic features of the storage file to be allocated according to the basic data includes: Determine the access time of the storage file to be allocated to obtain a time vector; Determine the level of the access count of the storage file to be allocated in a preset access lookup table to obtain a count vector; Determine the server serial number closest to the access IP region of the storage file to be allocated, and obtain a region vector; Determine the level of the file size of the to-be-allocated storage file in a preset capacity lookup table to obtain a size vector; Determine the sequence number of the file label of the to-be-allocated storage file in a preset label lookup table to obtain a label vector; The time vector, the number vector, the region vector, the size vector and the label vector are sequentially combined to obtain the basic features.

4. The distributed data optimization storage method according to claim 2, characterized in that: The loss function is: Wherein, loss is the loss function; h is the time period; n h is the total number of training data in the hth time period; α h is the loss weight of the hth time period; x hi is the actual number of visits to the i-th training data in the h-th time period; is the predicted number of visits to the i-th training data in the h-th time period.

5. The distributed data optimization storage method according to claim 3, characterized in that: The target activity prediction result is matched with a preset storage space allocation solution library to obtain a target migration solution, including: When the target activity prediction result is low activity, storing the to-be-allocated storage file in a low-speed large-capacity storage device; When the target activity prediction result is medium activity, determining the data size of the to-be-allocated storage file to obtain a first determination result; Allocate the server serial number for the storage file to be allocated according to the access IP region; When the first judgment result is a small-capacity file, storing the to-be-allocated storage file in a medium-speed large-capacity device corresponding to the server serial number; When the first judgment result is a large-capacity file, splitting the to-be-allocated storage file into a header file and a remaining file in a reading order, storing the header file in the medium-speed large-capacity device corresponding to the server serial number, and storing the remaining file in the low-speed large-capacity storage device; When the target activity prediction result is high activity, determining the data size of the to-be-allocated storage file to obtain a second determination result; Allocate the server serial number for the storage file to be allocated according to the access IP region; When the second judgment result is a small-capacity file, storing the to-be-allocated storage file in a high-speed, low-capacity device corresponding to the server serial number; When the second judgment result is a large-capacity file, the to-be-allocated storage file is split into a plurality of block files according to a reading order, and all the block files are dispersedly stored in a plurality of the high-speed and low-capacity devices that are closest to the server serial number.

6. The distributed data optimization storage method according to claim 5, characterized in that: Also includes: When the target user downloads the header file, the remaining file is evenly split into a plurality of data segments, and all the data segments are synchronously transmitted to the terminal device of the target user; After the header file and all the data fragments are completely transmitted, the header file and all the data fragments are sequentially spliced.

7. The distributed data optimization storage method according to claim 5, characterized in that: Also includes: Collect the upload request of the target user, authenticate the upload request of the target user, and obtain the authentication result; When the authentication result is passed, parsing the address information in the upload request, and allocating a temporary storage server serial number according to the address information; Receiving the uploaded data of the target user, obtaining a temporary uploaded file, and storing the uploaded file in a medium-speed and large-capacity device corresponding to the serial number of the temporary storage server; Extracting temporary summary information of the temporarily uploaded file; According to a preset summary information database, similarity calculation is performed on the temporary summary information using cosine similarity to obtain repetition rate information; When the repetition rate information is 1, the address pointer of the temporary upload file is updated, and the temporary upload file in the medium-speed large-capacity device corresponding to the temporary storage server serial number is cleared.

8. The distributed data optimization storage method according to claim 7, characterized in that: Extracting temporary summary information of the temporary uploaded file includes: Identify the data type of the temporarily uploaded file and obtain a type identification result; the type identification result includes: text data and video data; When the type identification result is the video data, the key frame images of the temporary uploaded file are extracted, and all the key frame images are combined to obtain the temporary summary information.

Images