A data distributed storage method and computer device

Abstract

The present invention relates to the field of data storage technology, and in particular to a data distributed storage method and computer equipment. The method uses the historical time period before the real-time time period to predict data for the real-time time period, obtains the predicted access volume ratio, evaluates the degree of change of hotspot information in the real-time time period by the difference between the predicted access volume ratio and the actual access volume ratio, obtains the intercepted data length of the historical data by using the degree of change of the hotspot information, and obtains reference data. The predicted access data for the future time period can be obtained based on the reference data. The bandwidth of the node server is adaptively and effectively adjusted based on the difference in access volume between the predicted access data and the access data in the real-time time period. The embodiment of the present invention intercepts the historical data used for prediction, obtains accurate predicted access data, and can make timely and accurate adaptive adjustments to the node server bandwidth based on the predicted access data.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a data distributed storage method and computer equipment. Background Art

[0002] In a distributed storage system with a central meta-server, the functions of the meta-server include data location, access control, unified namespace management, high availability guarantee, load balancing scheduling, etc. The communication bandwidth between the node server and the central meta-server affects the operating efficiency of the entire storage system.

[0003] The staff can set the bandwidth according to the experience based on the actual operation of the system to meet the scheduling operation of the system, but this method is difficult to ensure the rationality and timeliness of bandwidth adjustment. Therefore, the existing technology can use machine learning methods combined with historical information to predict the access data at future times, and then determine the bandwidth of the node server in time to ensure the real-time adjustment of bandwidth. However, this method does not take into account the impact of hotspot information during data access, because the data information in the distributed system is dispersed and stored in multiple node servers, so the node server storing hotspot information data will receive a higher amount of access. If the hotspot information changes during the actual data access and storage process, it will cause the historical data to contain a large amount of useless information, affecting the prediction results of the node server, and then causing inaccurate bandwidth settings, affecting data access and storage. Summary of the invention

[0004] In order to solve the technical problem that the historical data contains a lot of useless information, the prediction results of the node server are inaccurate, and the bandwidth setting is inaccurate because the influence of hot information data is not taken into account in the prior art, the purpose of the present invention is to provide a data distributed storage method and a computer device, and the technical solution adopted is as follows:

[0005] The present invention proposes a data distributed storage method, the method comprising:

[0006] Obtain several historical time periods of the real-time time period; obtain storage information of each node server in the distributed storage cluster; the storage information includes the number of visits in each time period and the access feature set of each user;

[0007] Under each node server, obtain the visit volume share of the node server in each time period; obtain the user distribution similarity between each historical time period and the real-time time period; the user distribution similarity is obtained based on the similarity of the access feature sets between users; according to the user distribution similarity and visit volume share in each historical time period, obtain the predicted visit volume share in the real-time time period; according to the difference between the actual visit volume share and the predicted visit volume share in the real-time time period, obtain the degree of change of the hotspot information;

[0008] Obtaining the length of intercepted data according to the degree of change of the hotspot information, intercepting data in a historical time period as reference data according to the length of intercepted data, and predicting the predicted access data in a future time period according to the reference data;

[0009] The bandwidth of the node server is adjusted according to the access volume difference between the predicted access data and the access data in the real time period.

[0010] Furthermore, the visit volume proportion is the proportion of the visit volume of a node server in the visit volume of all node servers in a time period.

[0011] Furthermore, the method for obtaining the user distribution similarity includes:

[0012] For any two users, the intersection-and-union ratio between the access feature sets of the two users is obtained as the similarity, and if the similarity is greater than a preset threshold, the two users are judged to be similar;

[0013] The user distribution similarity is obtained according to the proportion of similar users between the historical time period and the real-time time period.

[0014] Furthermore, the method for obtaining the predicted visit volume ratio includes:

[0015] The user distribution similarity in each historical time period is used as the weight of the visit volume share, and the visit volume share in all historical time periods is weighted and averaged to obtain the predicted visit volume share.

[0016] Furthermore, the method for obtaining the degree of change of the hotspot information includes:

[0017] The absolute value of the difference between the actual visit volume ratio and the predicted visit volume ratio in the real-time time period is normalized to obtain the degree of change of the hot spot information.

[0018] Furthermore, the method for obtaining the length of the intercepted data in the future time period includes:

[0019] The change degree of the hot information is negatively correlated and normalized to obtain a length adjustment coefficient, and the length adjustment coefficient is multiplied by the length of the access data combined in all historical time periods to obtain the intercepted data length.

[0020] Furthermore, the method for adjusting the bandwidth of the server includes:

[0021] The proportion of the access volume difference in the access volume in the real-time time period is used as the bandwidth adjustment coefficient, the bandwidth adjustment coefficient is multiplied by the bandwidth in the real-time time period to obtain the bandwidth adjustment amount, and the bandwidth adjustment amount is added to the bandwidth in the real-time time period to obtain the adjusted bandwidth of the node server.

[0022] Furthermore, the access feature set includes at least: access time, device type, user identity and authority, and network information.

[0023] The present invention also proposes a data distributed storage computer device, the computer device comprising:

[0024] The distributed storage information acquisition module is used to acquire several historical time periods of the real-time time period; acquire the storage information of each node server in the distributed storage cluster; the storage information includes the access volume of each time period and the access feature set of each user;

[0025] The first node server analysis module is used to obtain the visit volume ratio of the node server in each time period under each node server; obtain the user distribution similarity between each historical time period and the real-time time period; the user distribution similarity is obtained based on the similarity of the access feature sets between users;

[0026] The second node server analysis module is used to obtain the predicted visit volume share in the real-time time period according to the user distribution similarity and visit volume share in each historical time period under each node server; obtain the degree of change of hotspot information according to the difference between the actual visit volume share and the predicted visit volume share in the real-time time period; obtain the access data length of the future time period according to the degree of change of the hotspot information, and obtain the predicted access data in the future time period;

[0027] The node server bandwidth adjustment module is used to adjust the bandwidth of the node server according to the difference between the predicted access data and the access data in the real time period.

[0028] The present invention also proposes a data distributed storage program product, which is used to implement the steps of any one of the data distributed storage methods.

[0029] The present invention has the following beneficial effects:

[0030] Before making predictions for future moments, the present invention first uses the historical time periods before the real-time time period to make data predictions for the real-time time period, obtains the predicted visit volume percentage, and then evaluates the degree of change of hotspot information in the real-time time period by the difference between the predicted visit volume percentage and the actual visit volume percentage, that is, the greater the difference, the greater the degree of change of the hotspot information, the less useful information the historical information contains, and the more data from the closer historical moments are needed as reference data. By effectively intercepting the reference data, the accuracy of the predicted access data in the future time period can be increased, and then the bandwidth of the node server can be adaptively and effectively adjusted according to the difference in the visits between the predicted access data and the access data in the real-time time period. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 A flow chart of a data distributed storage method provided by an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of a distributed storage system cluster provided by an embodiment of the present invention;

[0034] Figure 3 A block diagram of a data distributed storage computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of a data distributed storage method and a computer device proposed according to the present invention, its specific implementation, structure, features and effects, in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0036] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0037] The scenario targeted by the embodiments of the present invention is the bandwidth adjustment and allocation problem of node servers in a distributed storage system. For a single device in a distributed storage system, a dual-control hybrid storage method using the VPX architecture is used, which consists of two control units and a switching unit, a backplane unit, and a power supply unit. Each control unit includes a front-end interface module, a processing module, and a storage interface. The front-end interface is responsible for network communication with other devices, and the dual-control units can back up each other. In the entire distributed cluster, a high-performance device acts as the central node to store brief information such as the directory structure of the data. Other devices communicate with the central node as sub-server nodes. Please refer to Figure 2 , which shows a schematic diagram of a distributed storage system cluster provided by an embodiment of the present invention. When a client needs to access and write stored data, it first communicates with the central node, obtains the node server that needs to be accessed according to the processing module of the central node, and then passes the corresponding information data to the relevant child nodes, so as to complete the access storage between the client and the distributed cluster. The central node has a control relationship with other node servers, so it can call the bandwidth control log in the node server, the access data of the node server, and the relevant data of the access user. Such data can be collectively referred to as storage information in the embodiment of the present invention. The central node generally stores the directory data structure and hierarchical relationship of the data file, as well as the node server location of the corresponding data block storage. When the user obtains access data, it needs to connect to the central node to parse the required data, and then obtain the location of the data stored in the sub-server, and then connect to the node server for data interaction.

[0038] A specific scheme of a data distributed storage method and a computer device provided by the present invention is described in detail below with reference to the accompanying drawings.

[0039] See also Figure 1 , which shows a flow chart of a data distributed storage method provided by an embodiment of the present invention, the method comprising:

[0040] Step S1: obtaining several historical time periods of the real-time time period; obtaining storage information of each node server in the distributed storage cluster; the storage information includes the access volume of each time period and the access feature set of each user.

[0041] The embodiment of the present invention aims to determine the change of hot information in the current real-time time period through historical information, and to predict the data in the future time period through historical information, so it is necessary to first divide the historical time period. In one embodiment of the present invention, the length of a time period is divided into 24 hours, and the time period within one month before the real-time time period is selected as the historical time period.

[0042] Further obtain the storage information of each node server in the distributed storage cluster, that is, the distributed storage system. The storage information includes the number of visits in each time period and the access feature set of each user. It should be noted that the storage information can be directly called through the central node as described in the above scenario introduction.

[0043] Preferably, in the embodiment of the present invention, the access feature set used includes at least: access time, device type, user identity and authority, and network information, wherein the network information includes at least IP address, bandwidth limitation, and delay.

[0044] Step S2: Under each node server, obtain the visit volume share of the node server in each time period; obtain the user distribution similarity between each historical time period and the real-time time period; the user distribution similarity is obtained based on the similarity of the access feature sets between users; according to the user distribution similarity and visit volume share in each historical time period, obtain the predicted visit volume share in the real-time time period; according to the difference between the actual visit volume share and the predicted visit volume share in the real-time time period, obtain the degree of change of the hot spot information.

[0045] It should be noted that the processing method for each node server in the embodiment of the present invention is the same, so only one node server is used as an example for explanation in the subsequent description.

[0046] The percentage of visits to a node server in each time period reflects the importance of visits to the node server in that time period. The percentage of visits in the historical time period is used to predict the percentage of visits in the real-time time period. Because visits have already occurred in the real-time time period, there is also an actual percentage of visits in the real-time time period. In the subsequent process, the degree of change of hot information can be evaluated by comparing the difference between the two.

[0047] Because different historical time periods have different references to real-time time periods, the more similar the historical time period is to the real-time time period, the more similar the access data is, and the more similar the access volume distribution is, that is, it should have a higher weight when predicting. Therefore, the embodiment of the present invention further analyzes the user distribution similarity between each historical time period and the real-time time period. The user distribution similarity can be used as a weight in the prediction process. The predicted access volume share in the real-time time period can be obtained based on the user distribution similarity and access volume share in each historical time period. The user distribution similarity is obtained based on the similarity of the access feature sets between users, that is, the more similar the access feature sets between users are, the more similar the two users are, and the more similar users there are between the two time periods, the greater the user distribution similarity.

[0048] Preferably, in one embodiment of the present invention, the method for obtaining user distribution similarity includes:

[0049] For any two users, the intersection-and-union ratio between the access feature sets of the two users is obtained as the similarity. If the similarity is greater than a preset threshold, the two users are judged to be similar. The intersection-and-union ratio is used as the similarity, which means that the more completely similar element types there are between the two access feature sets, the greater the similarity between the two sets. The user distribution similarity is obtained based on the proportion of similar users between the historical time period and the real-time time period. In an embodiment of the present invention, the proportion of the number of users in the historical time period that are similar to the real-time time period is selected as the user distribution similarity.

[0050] In the embodiment of the present invention, because the intersection-over-union ratio is data with a value range between 0 and 1, the threshold is set to 0.7.

[0051] Preferably, in one embodiment of the present invention, the user distribution similarity in each historical time period is used as the weight of the visit volume share, and the visit volume share in all historical time periods is weighted and averaged to obtain the predicted visit volume share. It should be noted that the weighted average is to multiply the user distribution similarity by the visit volume share and then find the average of all products.

[0052] If the difference between the actual visit volume ratio and the predicted visit volume ratio in the real-time time period is larger, it means that the hotspot data in the real-time time period has changed dramatically relative to the historical time period, and the historical data that can provide access information has become less. If the historical information is selected blindly without intercepting it, the predicted access data of the future time period will be inaccurate. Therefore, the embodiment of the present invention first uses the difference between the actual visit volume ratio and the predicted visit volume ratio in the real-time time period to obtain the degree of change of the hotspot information, and in the subsequent steps, the length of the historical information to be intercepted is determined by the degree of change of the hotspot information.

[0053] Preferably, in an embodiment of the present invention, the method for obtaining the degree of change of hotspot information includes:

[0054] The absolute value of the difference between the actual visit volume ratio and the predicted visit volume ratio in the real-time time period is normalized to obtain the degree of change of the hotspot information. The normalization method in the embodiment of the present invention uses the sigmoid function mapping method. Those skilled in the art may also use other basic mathematical means such as hyperbolic tangent function and range normalization for normalization, which will not be described in detail here.

[0055] Step S3: Obtain the length of intercepted data according to the degree of change of hot spot information, intercept data in a historical time period as reference data according to the length of intercepted data, and predict access data in a future time period according to the reference data.

[0056] According to step S2, after obtaining the degree of change of the hotspot information, the length of the historical data can be intercepted, and the intercepted data length can be further obtained. The data in the historical time period can be intercepted according to the intercepted data length as reference data. That is, the access data in all historical time periods are regarded as a data sequence, and the data segment closest to the real-time time period is selected as reference data according to the intercepted data length. The access data in the future time period can be predicted based on the reference data to obtain predicted access data.

[0057] It should be noted that the prediction method for predicting access data can use the neural network prediction method proposed in the background technology, which is a technical means well known to those skilled in the art. In the embodiment of the present invention, the prediction model corresponds to one for each node server, the basic framework is an LSTM model, the input is reference data, and the output is the predicted access data in the future time period. When the model is trained, the loss function used is the cross entropy loss function, and the training set is a data set consisting of access data of different lengths of history.

[0058] Preferably, in an embodiment of the present invention, the method for obtaining the length of intercepted data in a future time period includes:

[0059] The length adjustment coefficient is obtained by negatively mapping and normalizing the degree of change of the hot information. That is, the greater the degree of change of the hot information, the less available historical information is, and the smaller the length adjustment coefficient is, and the shorter the length of the intercepted data is. Because the length adjustment coefficient is a value between 0 and 1, it can be regarded as a scaling factor. The length adjustment coefficient is multiplied by the length of the access data merged in all historical time periods to obtain the length of the intercepted data. It should be noted that the merging of the access data is to connect the first heads of the sequence composed of the access data according to the chronological order to obtain the access data merged in all historical time periods.

[0060] In the embodiment of the present invention, the negative correlation mapping and normalization method uses the reciprocal method. In order to avoid the denominator being 0, the reciprocal of the data added with the positive integer 1 is used as the result of negative correlation mapping and normalization.

[0061] Step S4: adjusting the bandwidth of the node server according to the access volume difference between the predicted access data and the access data in the real time period.

[0062] After obtaining accurate prediction data, the bandwidth of the node server can be adjusted according to the difference in access volume between the predicted access data and the access data in the real time period.

[0063] Preferably, in an embodiment of the present invention, the method for adjusting the bandwidth of the server includes:

[0064] The proportion of the access volume difference in the real-time time period is used as the bandwidth adjustment coefficient, and the bandwidth adjustment coefficient is multiplied by the bandwidth in the real-time time period to obtain the bandwidth adjustment amount. The bandwidth adjustment amount is added to the bandwidth in the real-time time period to obtain the adjusted bandwidth of the node server.

[0065] In summary, before making predictions for future moments, the embodiment of the present invention first uses the historical time period before the real-time time period to make data predictions for the real-time time period to obtain the predicted visit volume percentage, and then evaluates the degree of change of hotspot information in the real-time time period by the difference between the predicted visit volume percentage and the actual visit volume percentage, and obtains the intercepted data length of the historical data using the degree of change of the hotspot information to obtain reference data. The predicted access data for the future time period can be obtained based on the reference data. The bandwidth of the node server is adaptively and effectively adjusted based on the difference in the visits between the predicted access data and the access data in the real-time time period. The embodiment of the present invention performs targeted analysis on each node server separately, determines the changes in hotspot information in the real-time state based on the characteristics of the historical information, and then intercepts the historical data used for prediction to obtain accurate predicted access data, and can make timely and accurate adaptive adjustments to the node server bandwidth based on the predicted access data.

[0066] Based on the same inventive concept, please refer to Figure 3 , which shows a block diagram of a data distributed storage computer device provided by an embodiment of the present invention. The embodiment of the present invention also proposes a data distributed storage computer device, the computer device comprising:

[0067] The distributed storage information acquisition module 101 is used to acquire several historical time periods of the real-time time period; acquire the storage information of each node server in the distributed storage cluster; the storage information includes the access volume of each time period and the access feature set of each user;

[0068] The first node server analysis module 102 is used to obtain the visit volume ratio of the node server in each time period under each node server; obtain the user distribution similarity between each historical time period and the real-time time period; the user distribution similarity is obtained based on the similarity of the access feature sets between users;

[0069] The second node server analysis module 103 is used to obtain the predicted visit volume ratio in the real-time time period according to the user distribution similarity and visit volume ratio in each historical time period under each node server; obtain the degree of change of hotspot information according to the difference between the actual visit volume ratio and the predicted visit volume ratio in the real-time time period; obtain the access data length of the future time period according to the degree of change of the hotspot information, and obtain the predicted access data in the future time period;

[0070] The node server bandwidth adjustment module 104 is used to adjust the bandwidth of the node server according to the difference between the predicted access data and the access data in the real time period.

[0071] The present invention also proposes a data distributed storage program product, which is used to implement any step of a data distributed storage method.

[0072] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0073] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A data distributed storage method, characterized in that: The method comprises: Obtain several historical time periods of the real-time time period; obtain storage information of each node server in the distributed storage cluster; the storage information includes the number of visits in each time period and the access feature set of each user; Under each node server, obtain the visit volume share of the node server in each time period; obtain the user distribution similarity between each historical time period and the real-time time period; the user distribution similarity is obtained based on the similarity of the access feature sets between users; according to the user distribution similarity and visit volume share in each historical time period, obtain the predicted visit volume share in the real-time time period; according to the difference between the actual visit volume share and the predicted visit volume share in the real-time time period, obtain the degree of change of the hotspot information; Obtaining the length of intercepted data according to the degree of change of the hotspot information, intercepting data in a historical time period as reference data according to the length of intercepted data, and predicting the predicted access data in a future time period according to the reference data; The bandwidth of the node server is adjusted according to the access volume difference between the predicted access data and the access data in the real time period.

2. A data distributed storage method according to claim 1, characterized in that: The visit volume ratio is the ratio of the visit volume of a node server to the visit volume of all node servers in a time period.

3. A data distributed storage method according to claim 1, characterized in that: The method for obtaining the user distribution similarity includes: For any two users, the intersection-and-union ratio between the access feature sets of the two users is obtained as the similarity, and if the similarity is greater than a preset threshold, the two users are judged to be similar; The user distribution similarity is obtained according to the proportion of similar users between the historical time period and the real-time time period.

4. A data distributed storage method according to claim 1, characterized in that: The method for obtaining the predicted visit volume ratio includes: The user distribution similarity in each historical time period is used as the weight of the visit volume share, and the visit volume share in all historical time periods is weighted and averaged to obtain the predicted visit volume share.

5. A data distributed storage method according to claim 1, characterized in that: The method for obtaining the degree of change of the hotspot information includes: The absolute value of the difference between the actual visit volume ratio and the predicted visit volume ratio in the real-time time period is normalized to obtain the degree of change of the hot spot information.

6. A data distributed storage method according to claim 1, characterized in that: The method for obtaining the intercepted data length in the future time period includes: The change degree of the hot information is negatively correlated and normalized to obtain a length adjustment coefficient, and the length adjustment coefficient is multiplied by the length of the access data combined in all historical time periods to obtain the intercepted data length.

7. A data distributed storage method according to claim 1, characterized in that: The method for adjusting the bandwidth of the server includes: The proportion of the access volume difference in the access volume in the real-time time period is used as the bandwidth adjustment coefficient, the bandwidth adjustment coefficient is multiplied by the bandwidth in the real-time time period to obtain the bandwidth adjustment amount, and the bandwidth adjustment amount is added to the bandwidth in the real-time time period to obtain the adjusted bandwidth of the node server.

8. A data distributed storage method according to claim 1, characterized in that: The access feature set includes at least: access time, device type, user identity and authority, and network information.

9. A data distributed storage computer device, characterized in that: The computer device comprises: The distributed storage information acquisition module is used to acquire several historical time periods of the real-time time period; acquire the storage information of each node server in the distributed storage cluster; the storage information includes the access volume of each time period and the access feature set of each user; The first node server analysis module is used to obtain the visit volume ratio of the node server in each time period under each node server; obtain the user distribution similarity between each historical time period and the real-time time period; the user distribution similarity is obtained based on the similarity of the access feature sets between users; The second node server analysis module is used to obtain the predicted visit volume share in the real-time time period according to the user distribution similarity and visit volume share in each historical time period under each node server; obtain the degree of change of hotspot information according to the difference between the actual visit volume share and the predicted visit volume share in the real-time time period; obtain the access data length of the future time period according to the degree of change of the hotspot information, and obtain the predicted access data in the future time period; The node server bandwidth adjustment module is used to adjust the bandwidth of the node server according to the difference between the predicted access data and the access data in the real time period.

Images