A visual big data operation and maintenance management platform
By obtaining the storage proportion of digital nodes and moving data blocks through the preset interval, the data block movement delay problem caused by uneven storage of DataNode is solved, and the storage equalization and proportion balance of data blocks is realized, which improves the Processing speed and user experience of big data systems.
Patent Information
- Application Number
- CN202311494027.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-10
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2043-11-10
AI Technical Summary
In a distributed file system, when DataNode storage is uneven, the data block movement delay causes the storage capacity to break through the security threshold, affecting the processing speed and user experience of the big data system.
The storage proportion of digital nodes is obtained by a preset interval, a connection between high-storage-based nodes and low-storage-based nodes is established, and data block movement simulation is carried out to ensure that the storage of data blocks in digital nodes is evenly divided and storage is balanced. The proportion is completed without affecting the use of the node.
It realizes that the storage equalization of data blocks is completed without affecting the use of digital nodes, balances the storage proportion of digital nodes, and improves the processing speed and user experience of the big data system.
Smart Images

Figure CN117459589B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data management, and in particular to a visualized big data operation and maintenance management platform. Background Art
[0002] Online shopping has become an inseparable part of our lives. When using online shopping platforms, the platforms can recommend similar products based on the content we search for and the types of products we browse, as well as other operations that stimulate our shopping desire. All of these are based on the analysis of big data systems. However, ensuring the normal operation of big data systems or optimizing work efficiency is critical, especially for data storage and indexing.
[0003] For the distributed file system (HDFS), during operation and maintenance management, the data blocks in the DataNode can only be moved within the specified time to address the uneven storage of the DataNode. If this time is not met, the DataNode storage capacity may exceed the safety threshold, which will affect the data processing speed of the entire big data system and the interaction with users, thereby affecting the user experience. Summary of the invention
[0004] In view of the above problems, the present invention provides a visual big data operation and maintenance management platform. The present invention obtains the storage ratio of digital nodes at preset time intervals, and matches the digital nodes according to the obtained data, so that the digital nodes with high digital node storage ratio are connected with the digital nodes with low digital node storage ratio. The simulation process of data block movement is completed first, and then according to the usage of the digital nodes, the storage of data blocks in the digital nodes can be evenly distributed without affecting the use of the digital nodes, thereby balancing the storage ratio of the digital nodes.
[0005] The technical solution of the present invention is: a visual big data operation and maintenance management platform, comprising:
[0006] The access data table update module is used to update the access data table, the access data table includes a number of data storage information, the data storage information includes the data block name, data block storage location and data block size;
[0007] The digital node monitoring module is used to obtain the name of the data block stored in the digital node and the digital node storage ratio, and determine whether to perform the digital node capacity balancing operation according to the digital node storage ratio;
[0008] A digital node capacity balancing module is used to perform a digital node capacity balancing operation, that is, to balance the digital node capacity according to the data block names stored in the digital nodes and the digital node storage ratio;
[0009] The digital node capacity balancing module includes a digital node address splitting unit, which is used to split the digital node address into a transmission set and a receiving set according to the digital node storage ratio;
[0010] An evenly distributed set generation unit, used to establish an evenly distributed set according to the data block names corresponding to the digital node addresses in the transmission set;
[0011] A block transmission pair generation unit: used to form a block transmission pair according to corresponding digital node addresses in a transmission set and a receiving set;
[0012] A first data block transmission strategy establishing unit, configured to establish a first data block transmission strategy according to a matching result of the block transmission pair and a first data block transmission set composed of digital block names stored in an evenly divided set, and update an evenly divided set and a digital node storage ratio in the transmission set;
[0013] A second data block transmission strategy establishing unit, configured to establish a second data block transmission strategy according to the evenly divided set and the digital node storage ratio updated by the first data block transmission strategy establishing unit;
[0014] A digital node capacity balancing execution unit is used to execute a balancing operation according to a first data block transmission strategy and a second data block transmission strategy.
[0015] As a preferred aspect, updating the access data table by the access data table update module specifically includes the following steps:
[0016] Obtain each data storage information G in the access data table within the first preset period n Number of visits H n , n=1, 2, 3······N, N is the total amount of data storage information; according to the number of accesses H n In descending order, store the corresponding data in the access data table G n Rearrange and update the access data table.
[0017] As a preferred aspect, obtaining the storage ratio of the digital node through the digital node monitoring module and determining whether to perform the digital node capacity balancing operation specifically includes the following steps:
[0018] Get the digital node storage ratio X m , m=1, 2, 3······M, M is the total number of digital nodes; calculate the storage ratio difference δ between each pair of digital nodes ij , and satisfy i<j, i, j∈{1, 2, 3······M}, count all the ij|>A”, where A is the first threshold, and whether “ψ / M>B” is established is judged. B is the second threshold. If “ψ / M>B” is established, the digital node capacity balancing operation is performed. Otherwise, the digital node storage ratio is obtained again after a second preset period.
[0019] As a preferred aspect, the digital node capacity balancing operation is performed by the digital node capacity balancing module, specifically including the following steps:
[0020] Get all digital node addresses Z through digital node address splitting unit m And the corresponding digital node storage ratio X m The corresponding digital node storage capacity Q m , based on the digital node storage ratio X m Split digital node address Z m , establish a transmission set Sd and a receiving set Rp;
[0021] The digital node address Z in the transmission set m The corresponding data block name P n The equal distribution set E is established by the equal distribution set generation unit m ;
[0022] The block transmission pair generation unit generates the transmission set Sd and the receiving set Rp according to the transmission time T hy Match and generate block transfer pairs (Sd h , Rp y ), h is the number corresponding to the element in the transmission set Sd, h=1, 2, 3······H, H is the total number of elements in the transmission set Sd, y is the number corresponding to the element in the reception set Rp, y=1, 2, 3······Y, Y is the total number of elements in the reception set Rp;
[0023] Get block transfer pair (Sd h , Rp y ) and the evenly divided set E m and establishes a first data block transmission strategy (Sd h , Rp y , S1 h ), and update the equal distribution set E in the transmission set m And the digital node storage ratio X m ;
[0024] Obtain the updated evenly distributed set E of the first data block transmission strategy establishment unit m And the digital node storage ratio X m, establish a second data block transmission strategy (Sd h ,γ2 k , S2 h );
[0025] Based on the first data block transmission strategy (Sd h , Rp y , S1 h ) and the second data block transmission strategy (Sd h ,γ2 k , S2 h ), the data blocks are moved through the digital node capacity balancing execution unit.
[0026] As a preferred aspect, the digital node address is split into a transmission set and a reception set by a digital node address splitting unit, and the specific steps are as follows:
[0027] Get all digital node addresses Z m And the corresponding digital node storage ratio X m The corresponding digital node storage capacity Q m , calculate the average threshold , we get the average interval (J-ε, J+ε), where ε is the error proportion;
[0028] Traverse all digital nodes storage ratio X m , judge "X m ≥J" is true, if "X m ≥J” holds true, then the corresponding digital node address Z m Stored in the transmission set Sd, if "X m ≥J” is not true, then the corresponding digital node address Z m Stored in the receiving set Rp.
[0029] As a preferred aspect, the evenly distributed set generation unit uses the data block names corresponding to the data block names in the transmission set to establish an evenly distributed set, which specifically includes the following steps:
[0030] For the digital node address Z in the transmission set Sd m Do the following:
[0031] Select the digital node addresses Z in the transmission set Sd one by one m , for the selected digital node address Z m , get the selected digital node address Z m The name of the data block stored in the corresponding digital node P n , according to the obtained data block name P n From the corresponding data storage information Gn Extract the data block size F from n , the extracted data block size F n Stored in the first candidate set γ1, the data block size F in the first candidate set γ1 n According to the corresponding data block, at the selected digital node address Z m The stacks in the corresponding digital nodes are arranged in order, and then the data block size F is selected one by one from the first candidate set γ1 n , until the " ”,γ1 r is the rth element in the first candidate set γ1, r=1, 2, 3...R, R is the total number of all elements in the first candidate set γ1, and the data block name P corresponding to the data block size selected from the first candidate set γ1 n Deposit into the equally divided set E m .
[0032] As a preferred aspect, a block transmission pair generation unit is used to form a block transmission pair according to a transmission set and a reception set, specifically including the following operation steps:
[0033] Get the digital node address Z in the transmission set Sd m and the digital node address Z in the receiving set Rp m The transmission time T hy , h is the number corresponding to the element in the transmission set Sd, h=1, 2, 3······H, H is the total number of elements in the transmission set Sd; y is the number corresponding to the element in the receiving set Rp, y=1, 2, 3······Y, Y is the total number of elements in the receiving set Rp;
[0034] Select the elements Sd in the transmission set Sd in turn h , element Sd h is the digital node address Z m , for the selected element Sd h , keep h unchanged, and traverse the transmission time T hy , the minimum transmission time T hy The corresponding element Rp in the receiving set Rp y With the selected element Sd h Composed of data block transmission pairs (Sd h , Rp y ).
[0035] As a preferred aspect, a first data block transmission set is formed according to the block transmission pair and the equal distribution set, a first data block transmission strategy is established by a first data block transmission strategy establishment unit, and the equal distribution set and the digital node storage ratio in the transmission set are updated, specifically including the following steps:
[0036] Get all data block transfer pairs (Sd h , Rp y ), for each data block transmission pair (Sd h , Rp y ), the element Sd h The corresponding digital node address is recorded as Z sd , the element Rp y The corresponding digital node address is recorded as Z Rp , perform the following operations to obtain the data block transmission pair (Sd h , Rp y ) in the element Sd h The corresponding evenly divided set E sd , from the evenly divided set E sd Select the data block names P one by one n , and according to the data block name P n Get the corresponding data block size F n , until the " ”,FE sd (ω) is the evenly distributed set E sd The data block size F corresponding to the ωth element in n , ω=1, 2, 3······W, W is the evenly distributed set E sd The total number of elements in will be removed from the evenly distributed set E sd The name of the data block selected in P n The first data block transmission set S1 is formed h , divide the set E sd The name of the data block selected in P n Delete all and divide the set E sd Update, and then transmit the first data block set S1 h With data block transmission pair (Sd h , Rp y ) Establish the first data block transmission strategy (Sd h , Rp y , S1 h ), according to the first data block transmission strategy (Sd h , Rp y , S1 h ) for the digital node address Z in the receiving set Rp Rp The corresponding digital node storage ratio is X Rp Update, the update formula is , where FS1 h (λ) is the first data block transmission set S1 h The data block size F of the λth element in n, λ=1, 2, 3...Λ, Λ is the first data block transmission set S1 h The total number of elements in .
[0037] As a preferred aspect, according to the evenly divided set and the digital node storage ratio updated by the first data block transmission strategy establishment unit, a second data block transmission strategy is established by a second data block transmission strategy establishment unit, specifically including the following steps:
[0038] Get all equally divided sets E m , select the equal distribution set E one by one m , for the selected equal distribution set E m , do the following:
[0039] ζ1: Determine the evenly divided set E m Is the number of elements in 0? If so, no operation is performed; otherwise, enter ζ2;
[0040] ζ2: Get the selected evenly divided set E m The corresponding digital node address Z m The corresponding element Sd in the transmission set Sd h and all elements Rp of the receiving set Rp y The transmission time T hy , for the selected E m The corresponding digital node address Z m The corresponding element Sd in the transmission set Sd h , keep h unchanged, and traverse the transmission time T hy , the transmission time T hy Arrange from small to large, and arrange the transmission time T hy The corresponding element Rp y Store them into the second candidate set γ2 in sequence;
[0041] ζ3: Let k = 1, k is used as a number to select an element in the second candidate set γ2;
[0042] ζ4: Select the kth element γ2 from the second candidate set γ2 k , the element γ2 k The corresponding digital node address is recorded as Z dx , from the evenly divided set E m Select the data block names P one by one n , and according to the data block name P n Get the corresponding data block size F n , until the " ”,FE m (d) is the evenly divided set E mThe dth element in, D is the evenly divided set E sd The total number of elements in will be removed from the evenly distributed set E m The name of the data block selected in P n The second data block transmission set S2 is formed h , divide the set E m The name of the data block selected in P n Delete all and divide the set E m Update and establish the second data block transmission strategy (Sd h ,γ2 k , S2 h ), according to the second data block transmission strategy (Sd h , Rp y , S2 h ) for the element γ2 in the second candidate set γ2 k The corresponding digital node address Z dx The corresponding digital node storage ratio is X dx Update, the update formula is , where FS2 h (v) is the second data block transmission set S2 h The data block size F of the vth element in n ;
[0043] ζ5: Determine the evenly divided set E m Is the number of elements in 0? If so, complete the selection of the evenly divided set E. m operation; otherwise, assign k+1 to k and return to ζ4.
[0044] As a preferred aspect, the balancing operation is performed by a digital node capacity balancing execution unit, specifically including the following steps:
[0045] All first data block transmission strategies (Sd h , Rp y , S1 h ) and the second data block transmission strategy (Sd h ,γ2 k , S2 h ) constitutes a data block transmission strategy table;
[0046] Get the addresses Z of all digital nodes in the third preset cycle m Number of visits ZH m , will satisfy the "ZH m <£" digital node address Z m Stored in the free set §, £ is the third preset value, for all digital node addresses Z in the free set § mThe first data block transmission strategy (Sd h , Rp y , S1 h ) and the second data block transmission strategy (Sd h ,γ2 k , S2 h ) is matched, and if the match is successful, the first data block transmission strategy (Sd h , Rp y , S1 h ) and the second data block transmission strategy (Sd h ,γ2 k , S2 h ) to move data blocks.
[0047] The present invention has the following advantages:
[0048] 1. The present invention obtains the storage ratio of digital nodes at preset intervals, and matches the digital nodes according to the obtained data, so that the digital nodes with high digital node storage ratio are connected with the digital nodes with low digital node storage ratio, first completing the simulation process of data block movement, and then according to the usage of the digital nodes, it can complete the storage equalization of data blocks in the digital nodes without affecting the use of the digital nodes, and balance the storage ratio of the digital nodes.
[0049] 2. The present invention improves the efficiency of obtaining storage addresses for new data and completing storage by counting the number of accesses to different data storage information and sorting the data storage information according to the number of accesses. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0051] Figure 1 A schematic diagram of the structure of a visual big data operation and maintenance management platform provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical scheme and advantages of the present invention clearer, some embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. However, it can be understood by those skilled in the art that in the various embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present invention. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present invention can also be implemented. Example
[0053] Figure 1 A visualized big data operation and maintenance management platform is provided in an embodiment of the present invention, and includes:
[0054] The access data table update module is used to update the access data table. The access data table includes several data storage information, including data block name, data block storage location and data block size. In HDFS distributed storage, there is a name node NameNode, corresponding to multiple digital nodes DataNode. Each digital node DataNode can store multiple data blocks. In order to ensure data security, each file can be configured with a number of copies, which is 3 by default, but these copies will not be stored in the same digital node DataNode. The access data table is stored in the NameNode. Build an HDFS. The client Client interacts with the NameNode and DataNode to access files in the HDFS and provide a large amount of data. The NameNode manages the DataNode and is composed of DataNodes of equal storage size. It can be regarded as a server connected to multiple computers with the same hard disk capacity. The server can obtain a large amount of data generated by users.
[0055] The digital node monitoring module is used to obtain the data block name and digital node storage ratio stored in the digital node, and determine whether to perform the digital node capacity balancing operation according to the digital node storage ratio. Since the storage capacity of the digital nodes is equal, if the storage ratio of the digital nodes is unbalanced, it means that the actual storage capacity is also unbalanced, and further capacity balancing operation is required;
[0056] A digital node capacity balancing module is used to perform a digital node capacity balancing operation, that is, to balance the digital node capacity according to the data block name stored in the digital node and the digital node storage ratio. The digital node capacity balancing module includes a digital node address splitting unit, an equal distribution set generating unit, a block transmission pair generating unit, a first data block transmission strategy establishing unit, a second data block transmission strategy establishing unit and a digital node capacity balancing executing unit;
[0057] An evenly distributed set generation unit, used to establish an evenly distributed set according to the data block names corresponding to the digital node addresses in the transmission set;
[0058] A block transmission pair generation unit: used to form a block transmission pair according to corresponding digital node addresses in a transmission set and a receiving set;
[0059] A first data block transmission strategy establishing unit, configured to establish a first data block transmission strategy according to a matching result of the block transmission pair and a first data block transmission set composed of digital block names stored in an evenly divided set, and update an evenly divided set and a digital node storage ratio in the transmission set;
[0060] A second data block transmission strategy establishing unit, configured to establish a second data block transmission strategy according to the evenly divided set and the digital node storage ratio updated by the first data block transmission strategy establishing unit;
[0061] A digital node capacity balancing execution unit is used to execute a balancing operation according to a first data block transmission strategy and a second data block transmission strategy.
[0062] Updating the access data table through the access data table update module specifically includes the following steps:
[0063] Obtain each data storage information G in the access data table within the first preset period n Number of visits H n , n=1, 2, 3······N, N is the total amount of data storage information; according to the number of accesses H n In descending order, store the corresponding data in the access data table G n Rearrange and update the access data table. When obtaining a large amount of data, the same type of data will be classified using the same data information. For example, the order data storage information of online shopping includes order establishment, order completion and order refund, so the number of accesses to the order data information is 9456. However, the store data storage information includes the number of followers and the shopping amount of the followers, and the number of accesses is 243. In this way, the order data storage information is arranged in front of the store data storage information in the access data table. When obtaining new data, traversing the access data table can more quickly find the data storage information with a large number of accesses and obtain the corresponding storage address.
[0064] The present invention improves the efficiency of acquiring storage addresses for new data and completing storage by counting the number of accesses to different data storage information and sorting the data storage information according to the number of accesses.
[0065] The digital node monitoring module is used to obtain the storage ratio of the digital node and determine whether to perform the digital node capacity balancing operation. The specific steps include:
[0066] Get the digital node storage ratio X m , m=1, 2, 3······M, M is the total number of digital nodes. For example, there are 8 digital nodes, and the storage proportions are 23%, 45%, 76%, 35%, 72%, 67%, 12% and 56% respectively; calculate the storage proportion difference δ between each pair of digital nodes ij , and satisfy i<j, i, j∈{1, 2, 3······M}, count all the ij |>A”, where A is the first threshold, and whether “ψ / M>B” is established is judged, and B is the second threshold, for example, A=35%, B=1. If “ψ / M>B” is established, ψ=9 is obtained after calculation, so “9 / 8>1” is established, and the digital node capacity balancing operation is performed. Otherwise, the digital node storage ratio is obtained again after an interval of the second preset period.
[0067] The digital node capacity balancing operation is performed by the digital node capacity balancing module, which specifically includes the following steps:
[0068] Get all digital node addresses Z through digital node address splitting unit m And the corresponding digital node storage ratio X m The corresponding digital node storage capacity Q m , based on the digital node storage ratio X m Split digital node address Z m , establish a transmission set Sd and a receiving set Rp;
[0069] The digital node address Z in the transmission set m The corresponding data block name P n The equal distribution set E is established by the equal distribution set generation unit m ;
[0070] The block transmission pair generation unit generates the transmission set Sd and the receiving set Rp according to the transmission time T hy Match and generate block transfer pairs (Sd h , Rp y ), h is the number corresponding to the element in the transmission set Sd, h=1, 2, 3······H, H is the total number of elements in the transmission set Sd, y is the number corresponding to the element in the reception set Rp, y=1, 2, 3······Y, Y is the total number of elements in the reception set Rp;
[0071] Get block transfer pair (Sd h , Rpy ) and the evenly divided set E m and establishes a first data block transmission strategy (Sd h , Rp y , S1 h ), and update the equal distribution set E in the transmission set m And the digital node storage ratio X m ;
[0072] Obtain the updated evenly distributed set E of the first data block transmission strategy establishment unit m And the digital node storage ratio X m , establish a second data block transmission strategy (Sd h ,γ2 k , S2 h );
[0073] Based on the first data block transmission strategy (Sd h , Rp y , S1 h ) and the second data block transmission strategy (Sd h ,γ2 k , S2 h ), the data blocks are moved through the digital node capacity balancing execution unit.
[0074] The digital node address is split into a transmission set and a receiving set by a digital node address splitting unit. The specific steps are as follows:
[0075] Get all digital node addresses Z m And the corresponding digital node storage ratio X m The corresponding digital node storage capacity Q m , calculate the average threshold , get the average interval (J-ε, J+ε), ε is the error ratio, for example ε=5%, the total capacity of all digital nodes is 4T, the ratios are 23%, 45%, 76%, 35%, 72%, 67%, 12% and 56% respectively, the digital node addresses are Z1, Z2, Z3, Z4, Z5, Z6, Z7 and Z8 respectively, then J=48.25%, the average interval is (43.25%, 53.25%), traverse all digital nodes storage ratio X m , judge "X m ≥J" is true, if "X m ≥J” holds true, then the corresponding digital node address Z m Stored in the transmission set Sd, then the transmission set Sd = (Z3, Z5, Z6 and Z8), if "X m≥J” is not true, then the corresponding digital node address Z m Stored in the receiving set Rp, then the receiving set Rp = (Z1, Z2, Z4 and Z7).
[0076] The equal distribution set generation unit uses the data block names corresponding to the data block names in the transmission set to establish the equal distribution set, which specifically includes the following steps:
[0077] For the digital node address Z in the transmission set Sd m Do the following:
[0078] Select the digital node addresses Z in the transmission set Sd one by one m , now we select Z8, which corresponds to the order data information, for the selected digital node address Z m , get the selected digital node address Z m The name of the data block stored in the corresponding digital node P n , for example, order data, according to the acquired data block name P n From the corresponding data storage information G n Extract the data block size F from n , F n = 200G, the extracted data block size F n Stored in the first candidate set γ1, the data block size F in the first candidate set γ1 n According to the corresponding data block, at the selected digital node address Z m The stack order in the corresponding digital node is arranged. It should be noted that the stack order has direction uniqueness and can arrange the related content in order to achieve the purpose of sequential processing. Then, the data block size F is selected one by one from the first candidate set γ1. n , the selected order data is 70G, until the " ”,γ1 r is the rth element in the first candidate set γ1, r=1, 2, 3······R, R is the total number of all elements in the first candidate set γ1. After extracting 200G of order data, the digital node storage ratio corresponding to the digital node address Z8 drops from 56% to 51.12%, which is within the average distribution interval (43.25%, 53.25%), thus satisfying the condition. The data block name P corresponding to the data block size selected from the first candidate set γ1 n Deposit into the equally divided set E m , so the evenly distributed set E corresponding to the digital node address Z8 m The name of the data block stored in P n For order information.
[0079] The block transmission pair generation unit forms a block transmission pair according to the transmission set and the reception set, which specifically includes the following operation steps:
[0080] Get the digital node address Z in the transmission set Sd m and the digital node address Z in the receiving set Rp m The transmission time T hy , h is the number corresponding to the element in the transmission set Sd, h=1, 2, 3······H, H is the total number of elements in the transmission set Sd; y is the number corresponding to the element in the receiving set Rp, y=1, 2, 3······Y, Y is the total number of elements in the receiving set Rp, for example, the transmission set (Z3, Z5, Z6 and Z8) and the receiving set (Z1, Z2, Z4 and Z7), there are 16 transmission times T hy ;
[0081] Select the elements Sd in the transmission set Sd in turn h , element Sd h is the digital node address Z m , for the selected element Sd h , keeping h unchanged, for example, for the selected digital node address Z8, the traversal transmission time T hy , the minimum transmission time T hy The corresponding element Rp in the receiving set Rp y With the selected element Sd h Composed of data block transmission pairs (Sd h , Rp y , (Z1, Z2, Z4 and Z7) with digital node address Z8 and the smallest transmission time corresponding to digital node address Z2, thus forming a data block transmission pair (Z8, Z2).
[0082] The first data block transmission set is formed according to the block transmission pairs and the equal distribution set, a first data block transmission strategy is established by a first data block transmission strategy establishment unit, and the equal distribution set and the digital node storage ratio in the transmission set are updated, specifically including the following steps:
[0083] Get all data block transfer pairs (Sd h , Rp y ), for each data block transmission pair (Sd h , Rp y ), now for (Z8, Z2), the element Sd h The corresponding digital node address is recorded as Z sd , the element Rp y The corresponding digital node address is recorded as Z Rp , perform the following operations to obtain the data block transmission pair (Sdh , Rp y ) in the element Sd h The corresponding evenly divided set E sd , from the evenly divided set E sd Select the data block names P one by one n , the selected result data block name P n For order information, and according to the data block name P n Get the corresponding data block size F n , 200G, until the " ”,FE sd (ω) is the evenly distributed set E sd The data block size F corresponding to the ωth element in n , ω=1, 2, 3······W, W is the evenly distributed set E sd The total number of elements in the order information is stored in the digital node corresponding to the digital node address Z2. The digital node storage ratio increases from 45% to 49.88%, which meets the judgment condition of less than 53.25%. sd The name of the data block selected in P n The first data block transmission set S1 is formed h , then the first data block transmission set S1 h There is only one data block named P n , and the data block name is order information, which will be evenly divided into set E sd The name of the data block selected in P n Delete all and divide the set E sd The number of elements in the evenly distributed set E8 corresponding to the digital node position information Z8 is 0, and then the first data block transmission set S1 is h With data block transmission pair (Sd h , Rp y ) Establish the first data block transmission strategy (Sd h , Rp y , S1 h ), the first data block transmission strategy generates a (Z8, Z2, order information), according to the first data block transmission strategy (Sd h , Rp y , S1 h ) for the digital node address Z in the receiving set Rp Rp The corresponding digital node storage ratio is X Rp Update, the update formula is , where FS1 h (λ) is the first data block transmission set S1 h The data block size F of the λth element inn , λ=1, 2, 3...Λ, Λ is the first data block transmission set S1 h The total number of elements in, after the update is completed, the digital node storage accounts for 49%, 49.88%, 76%, 50%, 46%, 52%, 12% and 51.12% respectively. The digital node addresses that need to be transmitted are Z4 and Z7, and after the update is completed, there are still 3 elements in the evenly divided set E3 corresponding to the digital node address Z3.
[0084] According to the evenly divided set and the digital node storage ratio updated by the first data block transmission strategy establishment unit, a second data block transmission strategy is established by a second data block transmission strategy establishment unit, specifically including the following steps:
[0085] Get all equally divided sets E m , select the equal distribution set E one by one m , for the selected equal distribution set E m , perform the following operations, knowing that the number of elements in the evenly divided set E8 is 0:
[0086] ζ1: Determine the evenly divided set E m Is the number of elements in 0? If so, no operation is performed, so no operation is performed on the evenly divided set E8; otherwise, enter ζ2. For example, if there are 3 elements in E3, namely, amount, address and logistics information, the next step needs to be performed;
[0087] ζ2: Get the selected evenly divided set E m The corresponding digital node address Z m The corresponding element Sd in the transmission set Sd h and all elements Rp of the receiving set Rp y The transmission time T hy , for the selected E m The corresponding digital node address Z m The corresponding element Sd in the transmission set Sd h , keep h unchanged, and traverse the transmission time T hy , the transmission time T hy Arrange from small to large, and arrange the transmission time T hy The corresponding element Rp y Stored in the second candidate set γ2 in sequence, then the digital node address Z7 needs to be stored in the second candidate set γ2;
[0088] ζ3: Let k = 1, k is used as a number to select an element in the second candidate set γ2;
[0089] ζ4: Select the kth element γ2 from the second candidate set γ2 k, the element γ2 k The corresponding digital node address is recorded as Z dx , from the evenly divided set E m Select the data block names P one by one n , and according to the data block name P n Get the corresponding data block size F n , the size of the amount, address and logistics information data blocks are 300G, 400G and 500G respectively, until the " ”,FE m (d) is the evenly divided set E m The dth element in, D is the evenly divided set E sd After the transmission is completed, the digital node storage corresponding to digital node address Z4 and digital node address Z7 accounts for 44.26% and 43.74% respectively, and will be removed from the evenly distributed set E m The name of the data block selected in P n The second data block transmission set S2 is formed h , then the second data block transmission set S2 h For (amount, address, logistics information), divide the set E m The name of the data block selected in P n Delete all and divide the set E m Update, there are no elements in the evenly divided set E3, and establish the second data block transmission strategy (Sd h ,γ2 k , S2 h ), the second data block transmission strategy is (Z4, Z7, amount, address, logistics information), according to the second data block transmission strategy (Sd h , Rp y , S2 h ) for the element γ2 in the second candidate set γ2 k The corresponding digital node address Z dx The corresponding digital node storage ratio is X dx Update, the update formula is , where FS2 h (v) is the second data block transmission set S2 h The data block size F of the vth element in n ;
[0090] ζ5: Determine the evenly divided set E m Is the number of elements in 0? If so, complete the selection of the evenly divided set E. m operation; otherwise, assign k+1 to k and return to ζ4.
[0091] The balancing operation is performed by the digital node capacity balancing execution unit, which specifically includes the following steps:
[0092] All first data block transmission strategies (Sd h , Rp y , S1 h ) and the second data block transmission strategy (Sd h ,γ2 k , S2 h ) constitutes a data block transmission strategy table, for example (Z4, Z7, amount, address, logistics information);
[0093] Get the addresses Z of all digital nodes in the third preset cycle m Number of visits ZH m The third preset period is 0.5 hours. During this 0.5 hour, the number of accesses ZH corresponding to the digital node address Z4 and the digital node address Z7 m 900 and 700 respectively, which will satisfy the "ZH m <£" digital node address Z m Stored in the free set §, £ is the third preset value, for example £ = 600, then "ZH m <£", for all digital node addresses Z in the free set § m The first data block transmission strategy (Sd h , Rp y , S1 h ) and the second data block transmission strategy (Sd h ,γ2 k , S2 h ) to complete the matching of (Z4, Z7, amount, address, logistics information). If the match is successful, the first data block transmission strategy (Sd h , Rp y , S1 h ) and the second data block transmission strategy (Sd h ,γ2 k , S2 h ) Move the data block so that the amount, address, and logistics information are moved from digital node address Z4 to digital node address Z7.
[0094] The present invention obtains the storage ratio of digital nodes at preset time intervals, and matches the digital nodes according to the obtained data, so that the digital nodes with high digital node storage ratio are connected with the digital nodes with low digital node storage ratio, first completing the simulation process of data block movement, and then according to the usage of the digital nodes, it can complete the storage equalization of data blocks in the digital nodes without affecting the use of the digital nodes, and balance the storage ratio of the digital nodes.
[0095] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention. Parts not described in detail in this specification belong to the prior art known to those skilled in the art.
Claims
1. A visual big data operation and maintenance management platform, characterized in that: include: The access data table update module is used to update the access data table, the access data table includes a number of data storage information, the data storage information includes the data block name, data block storage location and data block size; The digital node monitoring module is used to obtain the name of the data block stored in the digital node and the digital node storage ratio, and determine whether to perform the digital node capacity balancing operation according to the digital node storage ratio; A digital node capacity balancing module is used to perform a digital node capacity balancing operation, that is, to balance the digital node capacity according to the data block names stored in the digital nodes and the digital node storage ratio; The digital node capacity balancing module includes a digital node address splitting unit, which is used to split the digital node address into a transmission set and a receiving set according to the digital node storage ratio; An evenly distributed set generation unit, used to establish an evenly distributed set according to the data block names corresponding to the digital node addresses in the transmission set; A block transmission pair generation unit: used to form a block transmission pair according to corresponding digital node addresses in a transmission set and a receiving set; A first data block transmission strategy establishing unit, configured to establish a first data block transmission strategy according to a matching result of the block transmission pair and a first data block transmission set composed of digital block names stored in an evenly divided set, and update an evenly divided set and a digital node storage ratio in the transmission set; A second data block transmission strategy establishing unit, configured to establish a second data block transmission strategy according to the evenly divided set and the digital node storage ratio updated by the first data block transmission strategy establishing unit; A digital node capacity balancing execution unit is used to execute a balancing operation according to a first data block transmission strategy and a second data block transmission strategy.
2. A visual big data operation and maintenance management platform as claimed in claim 1, characterized in that: Updating the access data table through the access data table update module specifically includes the following steps: Obtain each data storage information G in the access data table within the first preset period n Number of visits H n , n=1, 2, 3······N, N is the total amount of data storage information; according to the number of accesses H n In descending order, store the corresponding data in the access data table G n Rearrange and update the access data table.
3. A visual big data operation and maintenance management platform as claimed in claim 2, characterized in that: The digital node monitoring module is used to obtain the storage ratio of the digital node and determine whether to perform the digital node capacity balancing operation. The specific steps include: Get the digital node storage ratio X m , m=1, 2, 3······M, M is the total number of digital nodes; Calculate the storage ratio difference δ between digital nodes ij , and satisfy i<j, i, j∈{1, 2, 3······M}, count all the ones that satisfy"|δ ij |>A”, where A is the first threshold, and whether "ψ / M>B" is established is judged. B is the second threshold. If "ψ / M>B" is established, the digital node capacity balancing operation is performed. Otherwise, the digital node storage ratio is obtained again after a second preset period.
4. A visual big data operation and maintenance management platform as claimed in claim 3, characterized in that: The digital node capacity balancing operation is performed by the digital node capacity balancing module, which specifically includes the following steps: Get all digital node addresses Z through digital node address splitting unit m And the corresponding digital node storage ratio X m The corresponding digital node storage capacity Q m , based on the digital node storage ratio X m Split digital node address Z m , establish a transmission set Sd and a receiving set Rp; The digital node address Z in the transmission set m The corresponding data block name P n The equal distribution set E is established by the equal distribution set generation unit m ; The block transmission pair generation unit generates the transmission set Sd and the receiving set Rp according to the transmission time T hy Match and generate block transfer pairs (Sd h , Rp y ), h is the number corresponding to the element in the transmission set Sd, h=1, 2, 3······H, H is the total number of elements in the transmission set Sd, y is the number corresponding to the element in the reception set Rp, y=1, 2, 3······Y, Y is the total number of elements in the reception set Rp; Get block transfer pair (Sd h , Rp y ) and the evenly divided set E m and establishes a first data block transmission strategy (Sd h , Rp y , S1 h ), S1 h The first data block transmission set is updated with the equal distribution set E in the transmission set. m And the digital node storage ratio X m ; Obtain the updated equal distribution set E of the first data block transmission strategy establishment unit m And the digital node storage ratio X m , establish a second data block transmission strategy (Sd h ,γ2 k , S2 h ), γ2 is the second candidate set, S2 h A first data block transmission set; Based on the first data block transmission strategy (Sd h , Rp y , S1 h ) and the second data block transmission strategy (Sd h ,γ2 k , S2 h ), the data blocks are moved through the digital node capacity balancing execution unit.
5. A visual big data operation and maintenance management platform as claimed in claim 4, characterized in that: The digital node address is split into a transmission set and a receiving set by a digital node address splitting unit. The specific steps are as follows: Get all digital node addresses Z m And the corresponding digital node storage ratio X m The corresponding digital node storage capacity Q m , calculate the average threshold , we get the average interval (J-ε, J+ε), where ε is the error proportion; Traverse all digital nodes storage ratio X m , judge "X m ≥J" is true, if "X m ≥J” holds true, then the corresponding digital node address Z m Stored in the transmission set Sd, if "X m ≥J” is not true, then the corresponding digital node address Z m Stored in the receiving set Rp.
6. A visual big data operation and maintenance management platform as claimed in claim 5, characterized in that: The equal distribution set generation unit uses the data block names corresponding to the data block names in the transmission set to establish the equal distribution set, which specifically includes the following steps: For the digital node address Z in the transmission set Sd m Do the following: Select the digital node addresses Z in the transmission set Sd one by one m , for the selected digital node address Z m , get the selected digital node address Z m The name of the data block stored in the corresponding digital node P n , according to the obtained data block name P n From the corresponding data storage information G n Extract the data block size F from n , the extracted data block size F n Stored in the first candidate set γ1, the data block size F in the first candidate set γ1 n According to the corresponding data block, at the selected digital node address Z m The stacks in the corresponding digital nodes are arranged in order, and then the data block size F is selected one by one from the first candidate set γ1 n , until satisfied" ”,γ1 r is the rth element in the first candidate set γ1, r=1, 2, 3...R, R is the total number of all elements in the first candidate set γ1, and the data block name P corresponding to the data block size selected from the first candidate set γ1 n Deposit into the equally divided set E m .
7. A visual big data operation and maintenance management platform as claimed in claim 6, characterized in that: The block transmission pair generation unit forms a block transmission pair according to the transmission set and the reception set, which specifically includes the following operation steps: Get the digital node address Z in the transmission set Sd m and the digital node address Z in the receiving set Rp m The transmission time T hy , h is the number corresponding to the element in the transmission set Sd, h=1, 2, 3······H, H is the total number of elements in the transmission set Sd; y is the number corresponding to the element in the receiving set Rp, y=1, 2, 3······Y, Y is the total number of elements in the receiving set Rp; Select the elements Sd in the transmission set Sd in turn h , element Sd h is the digital node address Z m , for the selected element Sd h , keep h unchanged, and traverse the transmission time T hy , the minimum transmission time T hy The corresponding element Rp in the receiving set Rp y With the selected element Sd h Composed of data block transmission pairs (Sd h , Rp y ).
8. A visual big data operation and maintenance management platform as claimed in claim 7, characterized in that: The first data block transmission set is formed according to the block transmission pairs and the equal distribution set, a first data block transmission strategy is established by a first data block transmission strategy establishment unit, and the equal distribution set and the digital node storage ratio in the transmission set are updated, specifically including the following steps: Get all data block transfer pairs (Sd h , Rp y ), for each data block transmission pair (Sd h , Rp y ), the element Sd h The corresponding digital node address is recorded as Z sd , the element Rp y The corresponding digital node address is recorded as Z Rp , perform the following operations to obtain the data block transmission pair (Sd h , Rp y ) in the element Sd h The corresponding evenly divided set E sd , from the evenly divided set E sd Select the data block names P one by one n , and according to the data block name P n Get the corresponding data block size F n , until satisfied" ”,FE sd (ω) is the evenly distributed set E sd The data block size F corresponding to the ωth element in n , ω=1, 2, 3······W, W is the evenly distributed set E sd The total number of elements in will be removed from the evenly distributed set E sd The name of the data block selected in P n The first data block transmission set S1 is formed h , divide the set E sd The name of the data block selected in P n Delete all and divide the set E sd Update, and then transmit the first data block set S1 h With data block transmission pair (Sd h , Rp y ) Establish the first data block transmission strategy (Sd h , Rp y , S1 h ), according to the first data block transmission strategy (Sd h , Rp y , S1 h ) for the digital node address Z in the receiving set Rp Rp The corresponding digital node storage ratio is X Rp Update, the update formula is , where FS1 h (λ) is the first data block transmission set S1 h The data block size F of the λth element in n , λ=1, 2, 3...Λ, Λ is the first data block transmission set S1 h The total number of elements in .
9. A visual big data operation and maintenance management platform as claimed in claim 8, characterized in that: According to the evenly divided set and the digital node storage ratio updated by the first data block transmission strategy establishment unit, a second data block transmission strategy is established by a second data block transmission strategy establishment unit, specifically including the following steps: Get all equally divided sets E m , select the equal distribution set E one by one m , for the selected equal distribution set E m , do the following: ζ1: Determine the evenly divided set E m Is the number of elements in 0? If so, no operation is performed; otherwise, enter ζ2; ζ2: Get the selected evenly divided set E m The corresponding digital node address Z m The corresponding element Sd in the transmission set Sd h and all elements Rp of the receiving set Rp y The transmission time T hy , for the selected E m The corresponding digital node address Z m The corresponding element Sd in the transmission set Sd h , keep h unchanged, and traverse the transmission time T hy , the transmission time T hy Arrange from small to large, and arrange the transmission time T hy The corresponding element Rp y Store them into the second candidate set γ2 in sequence; ζ3: Let k = 1, k is used as a number to select an element in the second candidate set γ2; ζ4: Select the kth element γ2 from the second candidate set γ2 k , the element γ2 k The corresponding digital node address is recorded as Z dx , from the evenly divided set E m Select the data block names P one by one n , and according to the data block name P n Get the corresponding data block size F n , until satisfied" ”,FE m (d) is the evenly divided set E m The dth element in, D is the evenly divided set E sd The total number of elements in will be removed from the evenly distributed set E m The name of the data block selected in P n The second data block transmission set S2 is formed h , divide the set E m The name of the data block selected in P n Delete all and divide the set E m Update and establish the second data block transmission strategy (Sd h ,γ2 k , S2 h ), according to the second data block transmission strategy (Sd h , Rp y , S2 h ) for the element γ2 in the second candidate set γ2 k The corresponding digital node address Z dx The corresponding digital node storage ratio is X dx Update, the update formula is , where FS2 h (v) is the second data block transmission set S2 h The data block size F of the vth element in n ; ζ5: Determine the evenly divided set E m Is the number of elements in 0? If so, complete the selection of the evenly divided set E. m operation; otherwise, assign k+1 to k and return to ζ4.
10. A visual big data operation and maintenance management platform as claimed in claim 9, characterized in that: The balancing operation is performed by the digital node capacity balancing execution unit, which specifically includes the following steps: All first data block transmission strategies (Sd h , Rp y , S1 h ) and the second data block transmission strategy (Sd h ,γ2 k , S2 h ) constitutes a data block transmission strategy table; Get the addresses Z of all digital nodes in the third preset cycle m Number of visits ZH m , will satisfy "ZH m <£" digital node address Z m Stored in the free set §, £ is the third preset value, for all digital node addresses Z in the free set § m The first data block transmission strategy (Sd h , Rp y , S1 h ) and the second data block transmission strategy (Sd h ,γ2 k , S2 h ) is matched, and if the match is successful, the first data block transmission strategy (Sd h , Rp y , S1 h ) and the second data block transmission strategy (Sd h ,γ2 k , S2 h ) to move data blocks.
