A distributed data storage management system and method based on big data

Abstract

The present invention discloses a distributed data storage management system and method based on big data, and relates to the technical field of distributed data storage. The method comprises the following steps: obtaining historical data of user nodes and storage nodes, determining a target node from a first storage node, and determining a first storage mode of user data in the first storage node according to a user's processing requirement for user data; determining a second storage mode of storing user data in the first storage node according to a user's download requirement for user data based on the historical data of the user node and the first storage node, and providing download and processing services for the user; the user processes the data directly on the target node, thereby reducing unnecessary forwarding and download times and improving efficiency; data fragments are stored on multiple nodes, with high availability and fault tolerance, and little impact from failures of storage nodes and target nodes; the target node does not store user data, and user operations on the target node have little impact on the security of the entire distributed system.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed data storage, and in particular to a distributed data storage management system and method based on big data. Background Art

[0002] Distributed data storage is a technology that stores data in multiple independent nodes. This technology improves data reliability, availability and scalability by dividing the data into multiple data slices and storing them in different physical locations. Distributed data storage usually consists of multiple data storage nodes, each of which can independently process data storage and access requests. These nodes are interconnected through the network to form a distributed network. In a distributed data storage system, sometimes users only need to process local information and content of the data, but when actually transmitting, users need to download the entire data and upload the entire data after processing. This is mainly due to the data management and transmission mechanism. In order to ensure the integrity and consistency of the data, the system usually transmits the entire data object, which affects both the user experience and the transmission efficiency. Therefore, how to improve the data storage effect and enhance the user experience has become an urgent problem to be solved. Summary of the invention

[0003] The purpose of the present invention is to provide a distributed data storage management system and method based on big data to solve the problems raised in the above background technology.

[0004] In one aspect of the present invention, a distributed data storage management method based on big data is provided, comprising:

[0005] S11, acquiring historical data of user nodes and storage nodes, wherein the storage nodes include a first storage node and a second storage node, wherein the first storage node performs distributed storage of user data and directly interacts with the user node, allowing the user to process the user data on the first storage node; and the second storage node is used to store all user data and does not directly interact with the user node;

[0006] S12, based on the historical data of the user node and the first storage node, determining a target node from the first storage node, and determining a first storage mode of the user data in the first storage node according to the user's processing requirements for the user data; the user directly processes the user data through the target node;

[0007] S13, based on the historical data of the user node and the first storage node, determine the second storage mode of storing the user data in the first storage node according to the user's downloading demand for the user data, and provide downloading and processing services for the user.

[0008] In step S12, the determining of the target node from the first storage node further includes the following steps:

[0009] S21, for i=1, 2, ..., n, perform the following steps S22, where n represents the number of first storage nodes, and i is used to distinguish different first storage nodes;

[0010] S22, taking the i-th first storage node as a candidate target node, and calculating the response time F when the i-th first storage node is taken as a candidate target node i , Where m is the number of active time periods of user nodes, k is used to distinguish different active time periods of user nodes, and U k is the probability that the user node has data processing requirements in the kth active time period, P k (i, j) is the probability that the candidate target node and the jth first storage node have not established a connection during the kth active time period, TS(i, j) is the time it takes for the candidate target node to establish a connection with the jth first storage node, T j is the time it takes for the jth first storage node to transmit user data to the candidate target node, Z i The time required for the candidate target node to integrate the user data of other first storage nodes; j and i have different values, which are also used to distinguish different first storage nodes;

[0011] S23, determining a candidate target node with the shortest response time from all candidate target nodes as the target node.

[0012] When the candidate target node and the jth first storage node are not connected, the time to establish a connection will be generated, so the coefficient P needs to be multiplied. k (i, j); The probability that users need to process data at different times is different, so the coefficient U is set for each time period k , all coefficients U k The sum of is 1. User data can be documents, audio, video, etc. When a user has a data processing request, such as the need to modify or delete a characteristic part of a document, the document can be modified directly on the target node without downloading the document to the local computer and then uploading it. It represents the expected connection time of the jth first storage node. For the entire user data, the overall expected connection time is the maximum value of the expected connection time of all first storage nodes; similarly, the time for transmitting data is also the maximum value.

[0013] In step S22, the following steps are also included:

[0014] S31, based on the historical data of the user node, determine the time interval in which the user node has activity records, and divide the time interval into m active time periods on average, where m is set by the user node;

[0015] S32, counting the number of activities n1 of the user node in the kth active time period k And the total number of activities of the user node N1 k , we get the probability U that the user node has data processing requirements in the kth active time period k ,

[0016] S33, determining the total duration n2 of the connection between the jth first storage node and the candidate target node in the kth active time period according to the historical data of the jth first storage node k , determine the total duration n3 during which there is no connection between the jth first storage node and the candidate target node in the kth active time period k , according to n2 k and n3 k Determine the probability P that the candidate target node and the jth first storage node are not connected during the kth active time k (i,j),

[0017] S34, use network tools to test in the actual network environment to determine the time TS(i,j) for establishing the connection between the j-th first storage node and the candidate target node; perform data integration test on the candidate target node and record the time Z required to complete the data integration i .

[0018] The connection between nodes is established and disconnected on demand. When data needs to be sent or received, the node will establish a connection and disconnect after the task is completed to save resources. The network connection between nodes takes a certain amount of time. For example, when using the TCP protocol, the TCP connection needs to be established through a three-way handshake process. This process requires sending and receiving at least three data packets. By using network tools such as ping or traceroute to measure the time it takes for the data packet to go from the sender to the receiver and then return, taking the average of multiple measurements can reduce the impact of randomness.

[0019] In step S22, the following steps are also included:

[0020] S41, determine the data transmission speed V(i,j) between the jth first storage node and the candidate target node, Arrange them in ascending order, and subtract the previous item from the next item in the sequence to get the difference sequence E1, E2, ..., E n-2, each difference in the difference sequence corresponds to a first storage node, which will be associated with E1, E2, ..., E n-2 The corresponding data transmission speeds between the first storage node and the candidate target node are denoted as V1, V2, ..., V n-2 ;

[0021] S42, judge E1×(V1+V2+…+V n-2 )+E2×(V2+...+V n-2 )+…+E n-2 ×V n-2 Is it greater than dv, dv is the amount of user data, if not, first convert the data volume E1×(V1+V2+…+V n-2 )、E2×(V2+…+V n-2 ),…,E n-2 ×V n-2 Mark it on the corresponding first storage node, and then distribute the remaining data in dv to other first storage nodes except the candidate target node in proportion. The amount of data allocated to the jth first storage node is The amount of data allocated to each first storage node and the amount of marked data are added to obtain the amount of user data that each first storage node needs to store, which is the first storage mode of user data; the time T spent by the jth first storage node to transmit user data to the alternative target node is determined according to the amount of user data that the jth first storage node needs to store and the data transmission speed between the jth first storage node and the alternative target node j , Where dv j The amount of data allocated to the jth first storage node; if greater than, mark dv in the order of the difference sequence.

[0022] In a high-latency network environment, the time overhead of establishing a connection will account for a large proportion. There are differences. In order to reduce the data transmission time, the node that establishes the connection first starts data transmission first. In the sequence, during the time period from the previous data A1 to the next data A2, the first storage node corresponding to A1 can perform data transmission; after reaching A2, the first storage nodes corresponding to A1 and A2 have completed the connection establishment, and at this time A1 and A2 can both perform data transmission. For the next data A3 of data A2 in the sequence, during the time period from A2 to A3, A1 and A2 perform data transmission together. At this time, A1 has a total of two data transmissions, and A2 has a total of one data transmission. By analogy, when dv is fully allocated or reaches the end of the sequence, the connection time is balanced.

[0023] When the volume of user data is large enough, all first storage nodes except the target node participate in data transmission, and the corresponding user data is allocated for transmission according to the transmission speed of the first storage node;

[0024] In step S42, the step of marking dv in the order of the difference sequence further includes the following steps:

[0025] Let the j(1), j(2), ...j(n-2)th first storage nodes be connected to E1, E2, ..., E n-2 Correspondingly, the data transmission speeds between the j(1), j(2), ...j(n-2)th first storage node and the candidate target node are V j(1) 、V j(2) ,…,V j(n-2) ;

[0026] S51, E1×V j(1) The data volume of the j(1)th first storage node is temporarily marked. If the temporarily marked data volume is not less than dv, the marking is completed and the dv data volume is marked on the j(1)th first storage node; if the temporarily marked data volume is less than dv, the variable a=2 is set, and the marked data volume is set to E1×V j(1) , enter S52;

[0027] S52, based on the generated temporary marking, E a ×[V j(1) +…+V j(a) ] is temporarily annotated to the j(1), ..., j(a)th first storage nodes, and the data volumes temporarily annotated to the j(1), ..., j(a)th first storage nodes are E a ×V j(1) ,…,E a ×V j(a) , increase the amount of labeled data by E a ×[V j(1) +…+V j(a) ], if the amount of marked data is not less than dv or a=n-1, the marking is completed, and the amount of data marked by the first storage node in all times is added up to obtain the amount of data marked by each first storage node. Where A a for The ath data in the sequence, A max for If the amount of marked data is less than dv, add 1 to the value of a and return to step S52.

[0028] In step S13, the step of distributing and storing the user data in the first storage node further includes the following steps:

[0029] S61, from the historical data of the user node, determine the proportions R1 and R2 of the user's processing demand and download demand for user data, where R1+R2=1; take the user node as a candidate target node, remove the determined target node, and perform steps S41 and S42 on other first storage nodes to obtain the first storage mode of the user data when the user node is a candidate target node. In this case, the first storage mode of the user data is the second storage mode;

[0030] S62, let the amount of data stored in the j-th first storage node of the first storage mode be DV1 j In the second storage mode, the amount of data stored in the jth first storage node is DV2 j , and finally the amount of data stored in the jth first storage node is R1×DV1 j +R2×DV2 j ; Traverse the value of j to obtain the amount of data that all first storage nodes need to store, slice the user data according to the amount of data that each first storage node needs to store, and store the data slices in the first storage nodes; wherein, the target node does not store data slices, and the amount of data that needs to be stored is 0.

[0031] When a user needs to download data, it is similar to processing data on the target node, with the only difference being that the target node is not generated from the first type of storage node, but is directly determined as a user node. A weighted sum is taken of the user's data download requirements and data processing requirements, and both are taken into consideration to ultimately determine the size of the data slice and the data storage method.

[0032] After completing the user's data processing request, the target node periodically deletes the user data. When the target node fails, the target node is removed from the first type of storage node. The target node itself does not store user data, and has little impact on the security of user data. When a first type of storage node that is not a target node fails, since the first type of storage node only stores part of the data slices, only part of the user data is affected. At this time, the missing data slices can be obtained from the second storage node.

[0033] In another aspect of the present invention, a distributed data storage management system based on big data is provided, including: storage nodes, user nodes and management nodes; the storage nodes are used for distributed storage of user data generated by user nodes, and when the user has a data processing demand, the user data is concentrated in the target node for integration, and the user is provided with a service of directly processing data at the target node; when the user has a data download request, the user data in the storage node is sent to the user node; the user node downloads the user data from the storage node when the user has a data download demand, and sends a request to the management node when the user has a data processing demand, and obtains the authority to directly process data at the target node; the management node is used to determine the target node from the storage node, and update the user data stored in the storage node.

[0034] The storage node also includes a first storage node, a second storage node and a communication unit, wherein the first storage node is used to store user data in slices, and the second storage node is used to store complete user data; the communication unit is used to establish and disconnect data transmission connections within the first storage node, within the second storage node, and between the first storage node and the second storage node. The management node also includes a first storage mode determination unit, a second storage mode determination unit, a data update unit and a distributed storage unit; the first storage mode determination unit determines the first storage mode of the target node and user data from the first storage node based on the connection between the first storage nodes; the second storage mode determination unit determines the second storage mode of the user data based on the connection between the first storage node and the user node; the data update unit determines the data slice processed by the user when the user directly processes the user data at the target node, and updates the information of the data slice on the first storage node corresponding to the processed data slice, and the information of the unprocessed data slice remains unchanged; the distributed storage unit slices the user data based on the first storage mode and the second storage mode of the user data, and distributes the user data in the first storage node except the target node. The first storage method determination unit determines the time interval in which the user node has activity records based on the historical data of the user node, divides the time interval into active time periods on average, and determines the target node based on the probability that the user node has data processing requirements in the active time period and the probability that no connection is established between the alternative target node and the first storage node during the active time period.

[0035] Compared with the prior art, the beneficial effects achieved by the present invention are: users process data directly on the target node, reducing unnecessary forwarding and downloading times and improving efficiency; data shards are stored on multiple nodes, with high availability and fault tolerance, and are less affected by failures of storage nodes and target nodes; the target node does not store user data, and user operations on the target node have little impact on the security of the entire distributed system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0037] Figure 1 It is a structural diagram of a distributed data storage management system based on big data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] In the embodiments of the present invention, please refer to Figure 1 , provides a distributed data storage management system based on big data, including: storage nodes, user nodes and management nodes; the storage nodes are used for distributed storage of user data generated by user nodes, and when users have data processing needs, the user data is concentrated in the target node for integration, and the service of directly processing data at the target node is provided for the user; when the user has a data download request, the user data in the storage node is sent to the user node; the user node downloads the user data from the storage node when the user has a data download demand, and sends a request to the management node when the user has a data processing demand, and obtains the authority to directly process data at the target node; the management node is used to determine the target node from the storage node, and update the user data stored in the storage node.

[0040] The storage node also includes a first storage node, a second storage node and a communication unit, the first storage node is used to store user data in slices, and the second storage node is used to store complete user data; the communication unit is used to establish and disconnect data transmission connections within the first storage node, within the second storage node, and between the first storage node and the second storage node.

[0041] The management node also includes a first storage method determination unit, a second storage method determination unit, a data update unit and a distributed storage unit; the first storage method determination unit determines the first storage method of the target node and the user data from the first storage node based on the connection status between the first storage nodes; the second storage method determination unit determines the second storage method of the user data based on the connection status between the first storage node and the user node; the data update unit, when the user directly processes the user data at the target node, determines the data slice processed by the user, and updates the information of the data slice on the first storage node corresponding to the processed data slice, and the information of the unprocessed data slice remains unchanged; the distributed storage unit, based on the first storage method and the second storage method of the user data, slices the user data, and distributes the user data on the first storage node except the target node.

[0042] The first storage method determination unit determines the time interval in which the user node has activity records based on the historical data of the user node, divides the time interval into active time periods on average, and determines the target node based on the probability that the user node has data processing requirements in the active time period and the probability that no connection is established between the alternative target node and the first storage node during the active time period.

[0043] In an embodiment of the present invention, a distributed data storage management method based on big data is provided, comprising:

[0044] S11, obtaining historical data of user nodes and storage nodes, the storage nodes including a first storage node and a second storage node, the first storage node performs distributed storage on user data, interacts directly with the user node, and allows the user to process the user data on the first storage node; the second storage node is used to store all user data and does not interact directly with the user node.

[0045] S12, based on the historical data of the user node and the first storage node, determining a target node from the first storage node, and determining a first storage mode of the user data in the first storage node according to the user's processing requirements for the user data; the user directly processes the user data through the target node;

[0046] Determining the target node from the first storage node also includes the following steps:

[0047] S21, for i=1, 2, ..., n, perform the following steps S22, where n represents the number of first storage nodes, and i is used to distinguish different first storage nodes;

[0048] S22, based on the historical data of the user node, determine the time interval in which the user node has activity records, and divide the time interval into m active time periods on average, where m is set by the user node;

[0049] Count the number of activities n1 of the user node in the kth active time period k And the total number of activities of the user node N1 k , we get the probability U that the user node has data processing requirements in the kth active time period k ,

[0050] According to the historical data of the jth first storage node, determine the total duration n2 of the connection between the jth first storage node and the candidate target node in the kth active time period k , determine the total duration n3 during which there is no connection between the jth first storage node and the candidate target node in the kth active time period k , according to n2 k and n3 k Determine the probability P that the candidate target node and the jth first storage node are not connected during the kth active time k (i,j),

[0051] Use network tools to test in the actual network environment to determine the time TS(i,j) required to establish a connection between the jth first storage node and the candidate target node; perform data integration tests on the candidate target nodes and record the time Z required to complete data integration i ;

[0052] Take the i-th first storage node as the candidate target node and calculate the response time F when the i-th first storage node is used as the candidate target node i , Where m is the number of active time periods of user nodes, k is used to distinguish different active time periods of user nodes, and U k is the probability that the user node has data processing requirements in the kth active time period, P k (i, j) is the probability that the candidate target node and the jth first storage node have not established a connection during the kth active time period, TS(i, j) is the time it takes for the candidate target node to establish a connection with the jth first storage node, T j is the time it takes for the jth first storage node to transmit user data to the candidate target node, Z i The time required for the candidate target node to integrate the user data of other first storage nodes; j and i have different values, which are also used to distinguish different first storage nodes;

[0053] Determine the data transmission speed V(i,j) between the jth first storage node and the candidate target node, and Arrange them in ascending order, and subtract the previous item from the next item in the sequence to get the difference sequence E1, E2, ..., E n-2 , each difference in the difference sequence corresponds to a first storage node, which will be associated with E1, E2, ..., E n-2 The corresponding data transmission speeds between the first storage node and the candidate target node are denoted as V1, V2, ..., V n-2 ;

[0054] S42, judge E1×(V1+V2+…+V n-2 )+E2×(V2+…+V n-2 )+…+E n-2 ×V n-2 Is it greater than dv, dv is the amount of user data, if not, first convert the data volume E1×(V1+V2+…+V n-2 )、E2×(V2+…+V n-2 ),…,E n-2 ×V n-2 Mark it on the corresponding first storage node, and then distribute the remaining data in dv to other first storage nodes except the candidate target node in proportion. The amount of data allocated to the jth first storage node is The amount of data allocated to each first storage node and the amount of marked data are added to obtain the amount of user data that each first storage node needs to store, which is the first storage mode of user data; the time T spent by the jth first storage node to transmit user data to the alternative target node is determined according to the amount of user data that the jth first storage node needs to store and the data transmission speed between the jth first storage node and the alternative target node j , Where dv j The amount of data allocated to the jth first storage node; if it is greater than, mark dv in the order of the difference sequence; let the j(1), j(2), ...j(n-2)th first storage nodes be respectively associated with E1, E2, ..., E n-2 Correspondingly, the data transmission speeds between the j(1), j(2), ...j(n-2)th first storage node and the candidate target node are V j(1) 、V j(2) ,…,V j(n-2) ; Marking dv in the order of the difference sequence also includes the following steps S51 and S52:

[0055] S51, E1×V j(1)The data volume of the j(1)th first storage node is temporarily marked. If the temporarily marked data volume is not less than dv, the marking is completed and the dv data volume is marked on the j(1)th first storage node; if the temporarily marked data volume is less than dv, the variable a=2 is set, and the marked data volume is set to E1×V j(1) , enter S52;

[0056] S52, based on the generated temporary marking, E a ×[V j(1) +…+V j(a) ] is temporarily annotated to the j(1), ..., j(a)th first storage node, and the data volumes temporarily annotated to the j(1), ..., j(a)th first storage node are E a ×V j(1) ,…,E a ×V j(a) , increase the amount of labeled data by E a ×[V j(1) +…+V j(a) ], if the amount of marked data is not less than dv or a=n-1, the marking is completed, and the amount of data marked by the first storage node in all times is added up to obtain the amount of data marked by each first storage node. Where A a for The ath data in the sequence, A max for If the amount of marked data is less than dv, add 1 to the value of a and return to step S52.

[0057] S23, determining a candidate target node with the shortest response time from all candidate target nodes as the target node.

[0058] When there are four first storage nodes B1, B2, B3 and B4, first take B1 as the candidate target node, calculate the expected connection time pie between B2, B3, B4 and B1 and record them as lj2, lj3, lj4,

[0059]

[0060] When lj2, lj3, and lj4 are 1, 5, and 10 seconds respectively, and the data transmission speeds between B2, B3, B4 and B1 are 3, 2, and 1 mbs, then the difference sequence is 4 and 5 seconds, where 4 corresponds to B2 and 5 corresponds to B3;

[0061] If dv is 40mb, since 3×(4+5)+2×5=37mb, which is less than 40mb, B2, B3, and B4 all need to participate in data transmission, and the remaining 3mb of user data is allocated according to 3:2:1 based on the data transmission speed between B2, B3, B4 and B1, that is, B2, B3, and B4 are allocated 1.5, 1, and 0.5mb of user data respectively; when B2 is connected to B1, B2 first performs data transmission for 4s, then B2 and B3 perform data transmission for 5s, and finally B2, B3, and B4 perform data transmission together; for the node B2 that starts transmission first, the allocated data volume is 1.5mb, and the marked data volume is 3×(4+5)=27mb, so the time T1 spent by B2 to transmit user data to the alternative target node is 1.5 / 3=0.5s; F1 is obtained by calculation.

[0062] Calculate B1, B2, B3 and B4 respectively, find the minimum value among F1, F2, F3 and F4, and the corresponding node is the target node.

[0063] If B1 eventually becomes the target node, the amount of user data that B2 needs to store under the first storage mode is 28.5 MB, and the amount of user data that B2 needs to store under the second storage mode is weighted summed to finally obtain the amount of data stored by B2, and the data slices of the corresponding amount are stored on B2;

[0064] If dv is 10mb, since 30 is less than 12, B3 and B4 do not participate in data transmission. At this time, a is 1, corresponding to the first number 4 in the difference sequence, corresponding to node B2, and the data volume marked by B2 is 10mb. Still 10s, through To compensate for it, Similarly, if B1 eventually becomes the target node, the amount of user data that B2 needs to store under the first storage mode is 10mb, and the weighted sum of the amount of user data that B2 needs to store under the second storage mode is performed to finally obtain the amount of data stored by B2, and the data slices of the corresponding amount of data are stored on B2. When B1 fails, the target node is generated from B2, B3 and B4.

[0065] S13, based on the historical data of the user node and the first storage node, determine the second storage mode of storing the user data in the first storage node according to the user's download demand for the user data, and provide download and processing services for the user; determine the proportions R1 and R2 of the user's processing demand and download demand for the user data from the historical data of the user node, R1+R2=1; take the user node as an alternative target node, remove the determined target node, and perform steps S41 and S42 on other first storage nodes to obtain the first storage mode of the user data when the user node is an alternative target node. In this case, the first storage mode of the user data is the second storage mode; let the amount of data stored in the jth first storage node of the first storage mode be DV1 j In the second storage mode, the amount of data stored in the jth first storage node is DV2 j , and finally the amount of data stored in the jth first storage node is R1×DV1 j +R2×DV2 j ; Traverse the value of j to obtain the amount of data that all first storage nodes need to store, slice the user data according to the amount of data that each first storage node needs to store, and store the data slices in the first storage nodes; wherein, the target node does not store data slices, and the amount of data that needs to be stored is 0.

[0066] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0067] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A distributed data storage management method based on big data, characterized in that: The following steps are involved: S11, acquiring historical data of a user node and a storage node, wherein the storage node includes a first storage node and a second storage node, wherein the first storage node performs distributed storage on the user data, directly interacts with the user node, and allows the user to process the user data on the first storage node; The second storage node is used to store all user data and does not interact directly with the user node; S12, based on the historical data of the user node and the first storage node, determining a target node from the first storage node, and determining a first storage mode of the user data in the first storage node according to the user's processing requirements for the user data; Users process user data directly through the target node; The determining of the target node from the first storage node also The method comprises the following steps S21, S22 and S23; S21, for i=1, 2, ..., n, perform the following steps S22, where n represents the number of first storage nodes, and i is used to distinguish different first storage nodes; S22, taking the i-th first storage node as a candidate target node, and calculating the response time F when the i-th first storage node is taken as a candidate target node i , Where m is the number of active time periods of user nodes, k is used to distinguish different active time periods of user nodes, and U k is the probability that the user node has data processing requirements in the kth active time period, P k (i, j) is the probability that the candidate target node and the jth first storage node have not established a connection during the kth active time period, TS(i, j) is the time it takes for the candidate target node to establish a connection with the jth first storage node, T j is the time it takes for the jth first storage node to transmit user data to the candidate target node, Z i The time required to integrate user data of other first storage nodes for the candidate target node; The values ​​of j and i are different, and are also used to distinguish different first storage nodes; S23, determining a candidate target node with the shortest response time from all candidate target nodes as the target node; In step S22, the following steps S31 to S34 are also included: S31, based on the historical data of the user node, determine the time interval in which the user node has activity records, and divide the time interval into m active time periods on average, where m is set by the user node; S32, counting the number of activities n1 of the user node in the kth active time period k And the total number of activities of the user node N1 k , we get the probability U that the user node has data processing requirements in the kth active time period k , S33, determining the total duration n2 of the connection between the jth first storage node and the candidate target node in the kth active time period according to the historical data of the jth first storage node k , determine the total duration n3 during which there is no connection between the jth first storage node and the candidate target node in the kth active time period k , according to n2 k and n3 k Determine the probability P that the candidate target node and the jth first storage node are not connected during the kth active time k (i,j), S34, using a network tool to perform a test in an actual network environment to determine the time TS(i,j) for establishing a connection between the j-th first storage node and the candidate target node; Perform data integration test on candidate target nodes and record the time required to complete data integration. i ; S13, based on the historical data of the user node and the first storage node, and according to the user's downloading demand for the user data, determine a second storage mode in which the user data is distributed and stored in the first storage node, and provide download and processing services for the user.

2. A distributed data storage management method based on big data according to claim 1, characterized in that: In step S22, the following steps are also included: S41, determine the data transmission speed V(i,j) between the jth first storage node and the candidate target node, Arrange them in ascending order, and subtract the previous item from the next item in the sequence to get the difference sequence E1, E2, ..., E n-2 , each difference in the difference sequence corresponds to a first storage node, which will be associated with E1, E2, ..., E n-2 The corresponding data transmission speeds between the first storage node and the candidate target node are denoted as V1, V2, ..., V n-2 ; S42, judge E1×(V1+V2+…+V n-2 )+E2×(V2+…+V n-2 )+…+E n-2 ×V n-2 Is it greater than dv, dv is the amount of user data, if not, first convert the data volume E1×(V1+V2+…+V n-2 )、E2×(V2+…+V n-2 ),…,E n-2 ×V n-2 Mark it on the corresponding first storage node, and then distribute the remaining data in dv to other first storage nodes except the candidate target node in proportion. The amount of data allocated to the jth first storage node is The amount of data allocated to each first storage node and the amount of marked data are added to obtain the amount of user data that each first storage node needs to store, which is the first storage mode of user data; the time T spent by the jth first storage node to transmit user data to the alternative target node is determined according to the amount of user data that the jth first storage node needs to store and the data transmission speed between the jth first storage node and the alternative target node j , Where dv j The amount of data allocated to the jth first storage node; if greater than, mark dv in the order of the difference sequence.

3. A distributed data storage management method based on big data according to claim 2, characterized in that: In step S42, the step of marking dv in the order of the difference sequence further includes the following steps: Let the j(1), j(2), ...j(n-2)th first storage nodes be connected to E1, E2, ..., E n-2 Correspondingly, the data transmission speeds between the j(1), j(2), ...j(n-2)th first storage node and the candidate target node are V j(1) 、V j(2) ,…,V j(n-2) ; S51, E1×V j(1) The data volume of the j(1)th first storage node is temporarily marked. If the temporarily marked data volume is not less than dv, the marking is completed and the dv data volume is marked on the j(1)th first storage node; if the temporarily marked data volume is less than dv, the variable a=2 is set, and the marked data volume is set to E1×V j(1) , enter S52; S52, based on the generated temporary marking, E a ×[V j(1) +…+V j(a) ] is temporarily annotated to the j(1), ..., j(a)th first storage nodes, and the data volumes temporarily annotated to the j(1), ..., j(a)th first storage nodes are E a ×V j(1) ,…,E a ×V j(a) , increase the amount of labeled data by E a ×[V j(1) +…+V j(a) ], if the amount of marked data is not less than dv or a=n-1, the marking is completed, and the amount of data marked by the first storage node in all times is added up to obtain the amount of data marked by each first storage node. Where A a for The ath data in the sequence, A max for If the amount of marked data is less than dv, add 1 to the value of a and return to step S52.

4. The distributed data storage management method based on big data according to claim 3 is characterized in that: In step S13, distributing and storing the user data in the first storage node further includes the following steps: S61, from the historical data of the user node, determine the proportions R1 and R2 of the user's processing demand and download demand for user data, where R1+R2=1; take the user node as a candidate target node, remove the determined target node, and perform steps S41 and S42 on other first storage nodes to obtain the first storage mode of the user data when the user node is a candidate target node. In this case, the first storage mode of the user data is the second storage mode; S62, let the amount of data stored in the j-th first storage node of the first storage mode be DV1 j In the second storage mode, the amount of data stored in the jth first storage node is DV2 j , and finally the amount of data stored in the jth first storage node is R1×DV1 j +R2×DV2 j ; Traverse the value of j to obtain the amount of data that all first storage nodes need to store, slice the user data according to the amount of data that each first storage node needs to store, and store the data slices in the first storage nodes; wherein, the target node does not store data slices, and the amount of data that needs to be stored is 0.

5. A distributed data storage management system based on big data, characterized in that: include: Storage nodes, user nodes, and management nodes; The storage node is used to distribute the user data generated by the user node, and when the user has a data processing requirement, the user data is centralized to the target node for integration, providing the user with a service of directly processing the data at the target node; when the user has a data download request, the user data in the storage node is sent to the user node; The user node downloads user data from the storage node when the user has a data download demand, and sends a request to the management node to obtain the authority to directly process data at the target node when the user has a data processing demand; the management node is used to determine the target node from the storage node and update the user data stored in the storage node; The storage node further includes a first storage node, a second storage node and a communication unit, wherein the first storage node is used to store user data in slices, and the second storage node is used to store complete user data; the communication unit is used to establish and disconnect data transmission connections within the first storage node, within the second storage node, and between the first storage node and the second storage node; The management node further includes a first storage mode determination unit, a second storage mode determination unit, a data update unit and a distributed storage unit; the first storage mode determination unit determines the first storage mode of the target node and the user data from the first storage node based on the connection status between the first storage nodes; The second storage mode determination unit determines the second storage mode of the user data based on the connection between the first storage node and the user node; the data update unit determines the data slice processed by the user when the user directly processes the user data at the target node, and updates the information of the data slice on the first storage node corresponding to the processed data slice, while the information of the unprocessed data slice remains unchanged; the distributed storage unit divides the user data into slices based on the first storage mode and the second storage mode of the user data, and distributes the user data in the first storage node except the target node; Determining the target node from the first storage node further includes the following steps S21, S22 and S23: S21, for i=1, 2, ..., n, perform the following steps S22, where n represents the number of first storage nodes, and i is used to distinguish different first storage nodes; S22, taking the i-th first storage node as a candidate target node, and calculating the response time F when the i-th first storage node is taken as a candidate target node i , Where m is the number of active time periods of user nodes, k is used to distinguish different active time periods of user nodes, and U k is the probability that the user node has data processing requirements in the kth active time period, P k (i, j) is the probability that the candidate target node and the jth first storage node have not established a connection during the kth active time period, TS(i, j) is the time it takes for the candidate target node to establish a connection with the jth first storage node, T j is the time it takes for the jth first storage node to transmit user data to the candidate target node, Z i The time required to integrate user data of other first storage nodes for the candidate target node; The values ​​of j and i are different, and are also used to distinguish different first storage nodes; S23, determining a candidate target node with the shortest response time from all candidate target nodes as the target node; In step S22, the following steps S31 to S34 are also included: S31, based on the historical data of the user node, determine the time interval in which the user node has activity records, and divide the time interval into m active time periods on average, where m is set by the user node; S32, counting the number of activities n1 of the user node in the kth active time period k And the total number of activities of the user node N1 k , we get the probability U that the user node has data processing requirements in the kth active time period k , S33, determining the total duration n2 of the connection between the jth first storage node and the candidate target node in the kth active time period according to the historical data of the jth first storage node k , determine the total duration n3 during which there is no connection between the jth first storage node and the candidate target node in the kth active time period k , according to n2 k and n3 k Determine the probability P that the candidate target node and the jth first storage node are not connected during the kth active time k (i,j), S34, using a network tool to perform a test in an actual network environment to determine the time TS(i,j) for establishing a connection between the j-th first storage node and the candidate target node; Perform data integration test on candidate target nodes and record the time required to complete data integration. i .

6. A distributed data storage management system based on big data according to claim 5, characterized in that: The first storage method determination unit determines the time interval in which the user node has activity records based on the historical data of the user node, divides the time interval into active time periods on average, and determines the target node based on the probability that the user node has data processing requirements in the active time period and the probability that no connection is established between the alternative target node and the first storage node during the active time period.

Images