Data fragment intelligent supervision system and method based on block chain

Abstract

The invention relates to the field of data fragmentation management, in particular to a blockchain-based data fragmentation intelligent supervision system and method.The blockchain-based data fragmentation intelligent supervision system comprises a window reproduction module, a field memory module, a hash storage module, a blockchain module and a retrieval orientation module, and the window reproduction module is used for calculating a standard field with the highest reproduction rate; the field memory module is used for determining a field offset and generating and storing a random field, the hash storage module is used for distributing a storage position of new data and updating the random field, the block chain module is used for uploading the random field to a block chain and broadcasting the random field, and the retrieval orientation module is used for outputting a partition retrieval result according to a retrieval field. The method has less retrieval data volume, can reduce the storage space occupied by the data table, improves the fragmentation storage efficiency, allows the user to provide the retrieval field with any length of the data table head during retrieval, and improves the speed and safety of data fragmentation.

Description

Technical Field

[0001] This invention relates to the field of data sharding management, specifically to a blockchain-based intelligent monitoring system and method for data sharding. Background Technology

[0002] Data sharding refers to the process of dividing a large amount of data into multiple smaller fragments and distributing them across small databases for distributed storage. By sharding, large amounts of data can be effectively distributed and stored across different nodes, giving sharded storage better scalability and bandwidth capacity compared to single-server storage.

[0003] During data sharding, different node servers exhibit significant differences in data storage capacity, read / write efficiency, and access latency. The way data is allocated across nodes directly impacts sharding efficiency and storage consistency; an inappropriate allocation method can drastically increase retrieval and storage latency. Existing technologies typically determine allocation status by calculating server access frequency, but this method requires analyzing the properties of each data item individually, making it unsuitable for sharding large datasets.

[0004] Furthermore, existing sharding systems require users to provide complete primary keys when processing user search requests, and then search each shard database one by one to obtain the search results. However, in some cases, users may not be able to provide complete and accurate primary keys, and performing fuzzy searches will take a lot of time, affecting the efficiency of accessing sharded data. Summary of the Invention

[0005] The purpose of this invention is to provide a data sharding intelligent supervision system and method based on blockchain to solve the problems mentioned in the background art.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a blockchain-based intelligent data sharding monitoring system, comprising: a window reproduction module, a field memory module, a hash storage module, a blockchain module, and a retrieval targeting module;

[0007] The window reproduction module consists of data processing software, which is used to calculate the correlation of the data to be partitioned, multiply the correlation by a preset amplification factor and round it to obtain the window length, extract the data segment corresponding to the window length from each data, calculate the reproduction rate of the extracted data segment in other data, and output the data segment with the highest reproduction rate as a standard field.

[0008] The field memory module consists of partition management software and secondary memory set in the partition server. The partition management software feeds back standard fields to the partition server. The partition server determines the field offset based on the number and database processing capacity, generates random fields, and makes the Hamming distance between the random fields and the standard fields equal to the field offset. The random fields are stored in the secondary memory.

[0009] The hash storage module is used to process each field of the new data with a preset proportional hash function when new data is stored, to obtain the hash value sequence of the new data, calculate the hash distance between the hash value sequence and the random field in the memory of each shard database, store the data in the shard database with the lowest hash distance, calculate the suffix length according to the field characteristics, insert a field with a suffix length in the data table header as the shard key into the random field, and update the random field.

[0010] The blockchain module is used to upload the updated random field to the blockchain when the random field is updated, broadcast the update result to all partition servers, and broadcast the search result to the blockchain when the search result of the sharded database is valid during the data retrieval process, thereby stopping the search process of other partition servers.

[0011] The retrieval targeting module is used to obtain a retrieval field of arbitrary length through user interaction software, send the retrieval field to each shard database, and the shard database processes the random field into a first matrix with the same number of rows as the retrieval field. The retrieval field is used as the augmented part of the first matrix to obtain a second matrix. The length of the last row of the second matrix is ​​calculated, and the shard database with the smallest length of the last row of the second matrix is ​​selected as the requested targeting database. Data is retrieved from the targeting database and output.

[0012] Furthermore, the window reproduction module includes: a loop length unit and a window comparison unit;

[0013] The loop length unit is used to calculate the correlation of the data to be partitioned, and the length of the data segment truncation window is determined by the correlation.

[0014] The window comparison unit is used to extract data segments from the data to be partitioned using a sliding sampling window, and to calculate the recurrence rate of the data segments in each partitioned database.

[0015] Furthermore, the field memory module includes: random partitioning units and secondary memory units;

[0016] The random partitioning unit is used to obtain the data capacity of the sharded database, the amount of communication data within a unit period, and the amount of retrieval data within a unit period. The minimum value is then multiplied by a preset coefficient to obtain the field offset.

[0017] The secondary memory unit is used to generate and store random fields in secondary memory whose Hamming distance from the standard field is equal to the field offset.

[0018] Furthermore, the hash storage module includes: a field matching unit, an input storage unit, and a table header link unit;

[0019] The field matching unit is used to calculate the hash value sequence of the newly stored data and output the random field that is closest to the hash value sequence.

[0020] The storage unit is used to store new data into the sharded database corresponding to the output random field;

[0021] The header linking unit is used to extract fields from the new data header and insert them into the random fields corresponding to the sharded database where the new data is stored.

[0022] Furthermore, the blockchain module includes: an encrypted upload unit and a partitioned broadcast unit;

[0023] The encrypted upload unit is used to upload the updated random field to the blockchain when new data is stored in the sharded database;

[0024] The partition broadcasting unit is used to broadcast the results of new data storage and the results of retrieval fields in the partition database to all partition databases.

[0025] Furthermore, the retrieval targeting module includes: a retrieval interaction unit, a field augmentation unit, and a partition output unit;

[0026] The search interaction unit is used to provide a user-oriented interactive platform to obtain search fields provided by the user.

[0027] The field augmentation unit is used to construct an augmentation matrix by taking the user's search field and the random fields of each shard database as the rows and columns of the matrix, calculate the tail row length of the augmentation matrix, and designate the database corresponding to the matrix with the smallest tail row length as the targeted database.

[0028] The partition output unit is used to retrieve data containing the search field header in the targeted database and output the search results to the user interaction page.

[0029] A blockchain-based intelligent data sharding supervision method includes the following steps:

[0030] Step S1. Number each piece of data to be partitioned, calculate the correlation of the data to be partitioned, multiply the correlation by the preset amplification factor and round down to get the window length, slide and cut out a data segment equal to the window length from the data to be partitioned with the largest amount of data, calculate the recurrence rate of each cut data segment in other data, and take the data segment with the highest recurrence rate as the standard field.

[0031] Step S2. Each sharded database calculates the offset distance according to the database's working parameters and the field offset method, generates a random field, and makes the Hamming distance between the random field and the standard field equal to the offset distance of the sharded database;

[0032] Step S3. Use a preset hash function to process the data to be partitioned and the random fields of each database to obtain the hash values ​​of the primary key field and each random field of the data to be partitioned, and store the data to be partitioned into the shard database with the closest hash value.

[0033] Step S4. When new data is stored in the sharded database, a fixed length of data is extracted from the new data table header as the sharding key. The allocation key is appended to the random field of the sharded database to form a new random field. The updated random field is uploaded to the blockchain and broadcast to all sharded databases through the blockchain.

[0034] Step S5. The user provides the blockchain with a search field of arbitrary length in the header of the required data table. The sharded database processes the random field into a first matrix with the same number of rows as the search field. The search field is used as the augmented part of the first matrix to obtain a second matrix. The length of the last row of the second matrix is ​​calculated. The sharded database with the smallest length of the last row of the second matrix is ​​used as the target database for the request, and the data is retrieved and output from the target database.

[0035] Furthermore, step S1 includes:

[0036] Step S11. Traverse the data to be partitioned in the main server, number each data item, and record the numbering result as {W1, W2, ..., Wn}, where n is the number of data items to be partitioned, and Wn represents the data item to be partitioned with number n.

[0037] Step S12. Calculate the correlation coefficient of the data to be partitioned using the following formula:

[0038]

[0039] Where P is the correlation coefficient, m is the maximum field length of all data to be partitioned, TU is the alignment function that outputs the number of identical elements in the variable, and ri w1 ,ri wn 、...ri wn These represent the values ​​of the i-th data in W1, W2, ..., Wn, respectively.

[0040] Step S13. Calculate the window length L, where L = P·e, and e is a preset magnification factor. Obtain the data to be partitioned with the largest data volume as the representative data, and use L as the window length to extract fields from the representative data to obtain the set {D1, D2, ..., D...}. m-L}, where D m-L This represents the field consisting of the mL-th to m-th data points in the data set;

[0041] Step S14. Calculate the values ​​of fields D1, D2, ..., D respectively. m-L The probability of occurrence in all the data to be partitioned is used to determine the standard field D0, which is the field with the highest probability of occurrence.

[0042] Furthermore, step S2 includes:

[0043] Step S21. Based on the database's operating parameters, calculate the offset distance of each shard database using the field offset method. The operating parameters include: database data capacity, communication data volume per unit period, and retrieval data volume per unit period. The field offset method is as follows: select the minimum value among the operating parameters and output the offset distance k, k = MOD(U min ·Es), where MOD is the floor function, U min This represents the minimum value among the operating parameters, and Es is the preset amplification factor.

[0044] Step S22. Generate a random field of length m in each sharded database, and ensure that the generated random field satisfies the following condition: the Hamming distance between it and the standard field is equal to the offset distance of the sharded database.

[0045] Step S23. Divide the memory space of the sharded database into independent memory segments and store the random fields of the partitioned servers in the independent memory segments.

[0046] Furthermore, step S3 includes:

[0047] Step S31. Preset hash function F(x), input random field into hash function to obtain hash value {H1, H2, ..., Ht} of random field of each shard database, where t represents the number of shard databases and Ht represents the hash value of random field of t-th shard database;

[0048] Step S32. Arrange the data to be partitioned in numerical order, extract the first m data from the header of the data to be partitioned to obtain the primary key field, and process the primary key field using the hash function H(x) to obtain the hash value H0 of the primary key field;

[0049] Step S33. Calculate the difference between H1, H2, ... Ht and H0 respectively, select the minimum value, and use the shard database address corresponding to the minimum value as the storage address of the data to be partitioned, and store the data to be partitioned in it;

[0050] Step S34. Perform steps S32-S33 on each piece of data to be partitioned, and store all the data to be partitioned in the database to be partitioned.

[0051] Furthermore, step S4 includes:

[0052] Step S41. Whenever data is stored in the sharded database, extract the first q data from the header of the data table to form the sharding key, where q is a preset parameter and q≥t. Insert the sharding key into the end of the random field of the sharded database to form a new random field.

[0053] Step S42. Whenever a new random field is generated, the sharded database uploads the updated random field to the blockchain. When the data processing center detects the growth of the blockchain, it retrieves the update status of the random field from the blockchain and broadcasts the length of the new random field in all sharded databases.

[0054] Furthermore, step S5 includes:

[0055] Step S51. Build a user-oriented interactive platform. When a user needs to search, obtain the search field provided by the user, read the length b of the search field, and generate a first matrix A1 based on the length of the search field. A1 satisfies:

[0056]

[0057] Among them, R b AE represents the b-th data value in the random field of the sharded database, where AE is a preset placeholder data and AE≠0;

[0058] Step S52. Use the retrieved field as a row augmentation part to form a second matrix A2 with the first matrix, where A2 satisfies:

[0059]

[0060] Among them, G1, G2, ... G b Retrieve the 1st, 2nd, ... bth data values ​​in the field respectively;

[0061] Step S53. Simplify the second matrix A2 based on the last row, and output the number of elements with non-zero data values ​​in the last row of the simplified second matrix, which is recorded as the tail row length corresponding to the sharded database.

[0062] Step S54. Select the shard database with the smallest tail length as the target database for the retrieval behavior, retrieve the corresponding data in the target database, and output the retrieval results.

[0063] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0064] 1. This invention can use the highest reproducibility field of data as the standard field to generate partition fields in the sharded database, so that the Hamming distance between the partition field and the standard field is related to the data storage capacity of the server. This allows for rapid data allocation among different partition servers, avoiding data skew and excessive load on a single node, and improving the speed and security of data sharding.

[0065] 2. This invention can calculate the hash distance between the stored data and each partition field when data is stored, and store the data in the shard database with the lowest hash distance. The data field is extracted from the table header and used as the shard key after the partition field. The modified partition field is stored in the blockchain to avoid data key loss. Compared with primary key storage, it has less data to retrieve, can extract the data shard position faster, reduce the storage space occupied by the data table, save storage costs, and improve shard storage efficiency.

[0066] 3. This invention allows users to provide search fields of arbitrary length in the data table header when searching. Each node uses the search field as an augmented part of the partition field matrix, calculates the tail row length of the augmented matrix, and uses the partition with the smallest tail row length as the requested partition. This avoids sequential searching by each node, saves the computational load of block orientation, and reduces the request processing latency of the partition server. Attached Figure Description

[0067] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0068] Figure 1 This is a schematic diagram of the structure of a blockchain-based intelligent data sharding monitoring system according to the present invention;

[0069] Figure 2 This is a schematic diagram illustrating the steps of a blockchain-based intelligent data sharding supervision method according to the present invention. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] Please see Figure 1 The present invention provides a technical solution: a data sharding intelligent supervision system based on blockchain, comprising: a window reproduction module, a field memory module, a hash storage module, a blockchain module, and a retrieval targeting module;

[0072] The window reproduction module consists of data processing software, which is used to calculate the correlation of the data to be partitioned, multiply the correlation by a preset amplification factor and round it to obtain the window length, extract the data segment corresponding to the window length from each data, calculate the reproduction rate of the extracted data segment in other data, and output the data segment with the highest reproduction rate as a standard field.

[0073] The window reproduction module includes: a loop length unit and a window comparison unit;

[0074] The loop length unit is used to calculate the correlation of the data to be partitioned, and the length of the data segment truncation window is determined by the correlation.

[0075] The window comparison unit is used to extract data segments from the data to be partitioned using a sliding sampling window, and to calculate the recurrence rate of the data segments in each partitioned database.

[0076] The field memory module consists of partition management software and secondary memory set in the partition server. The partition management software feeds back standard fields to the partition server. The partition server determines the field offset based on the number and database processing capacity, generates random fields, and makes the Hamming distance between the random fields and the standard fields equal to the field offset. The random fields are stored in the secondary memory.

[0077] The field memory module includes: random partition units and secondary memory units;

[0078] The random partitioning unit is used to obtain the data capacity of the sharded database, the amount of communication data within a unit period, and the amount of retrieval data within a unit period. The minimum value is then multiplied by a preset coefficient to obtain the field offset.

[0079] The secondary memory unit is used to generate and store random fields in secondary memory whose Hamming distance from the standard field is equal to the field offset.

[0080] The hash storage module is used to process each field of the new data with a preset proportional hash function when new data is stored, to obtain the hash value sequence of the new data, calculate the hash distance between the hash value sequence and the random field in the memory of each shard database, store the data in the shard database with the lowest hash distance, calculate the suffix length according to the field characteristics, insert a field with a suffix length in the data table header as the shard key into the random field, and update the random field.

[0081] The hash storage module includes: a field matching unit, an input storage unit, and a table header link unit;

[0082] The field matching unit is used to calculate the hash value sequence of the newly stored data and output the random field that is closest to the hash value sequence.

[0083] The storage unit is used to store new data into the sharded database corresponding to the output random field;

[0084] The header linking unit is used to extract fields from the new data header and insert them into the random fields corresponding to the sharded database where the new data is stored.

[0085] The blockchain module is used to upload the updated random field to the blockchain when the random field is updated, broadcast the update result to all partition servers, and broadcast the search result to the blockchain when the search result of the sharded database is valid during the data retrieval process, thereby stopping the search process of other partition servers.

[0086] The blockchain module includes: an encrypted upload unit and a partitioned broadcast unit;

[0087] The encrypted upload unit is used to upload the updated random field to the blockchain when new data is stored in the sharded database;

[0088] The partition broadcasting unit is used to broadcast the results of new data storage and the results of retrieval fields in the partition database to all partition databases.

[0089] The retrieval targeting module is used to obtain a retrieval field of arbitrary length through user interaction software, send the retrieval field to each shard database, and the shard database processes the random field into a first matrix with the same number of rows as the retrieval field. The retrieval field is used as the augmented part of the first matrix to obtain a second matrix. The length of the last row of the second matrix is ​​calculated, and the shard database with the smallest length of the last row of the second matrix is ​​selected as the requested targeting database. Data is retrieved from the targeting database and output.

[0090] The search targeting module includes: a search interaction unit, a field augmentation unit, and a partition output unit;

[0091] The search interaction unit is used to provide a user-oriented interactive platform to obtain search fields provided by the user.

[0092] The field augmentation unit is used to construct an augmentation matrix by taking the user's search field and the random fields of each shard database as the rows and columns of the matrix, calculate the tail row length of the augmentation matrix, and designate the database corresponding to the matrix with the smallest tail row length as the targeted database.

[0093] The partition output unit is used to retrieve data containing the search field header in the targeted database and output the search results to the user interaction page.

[0094] like Figure 2 As shown, a data sharding intelligent supervision method based on blockchain includes the following steps:

[0095] Step S1. Number each piece of data to be partitioned, calculate the correlation of the data to be partitioned, multiply the correlation by the preset amplification factor and round down to get the window length, slide and cut out a data segment equal to the window length from the data to be partitioned with the largest amount of data, calculate the recurrence rate of each cut data segment in other data, and take the data segment with the highest recurrence rate as the standard field.

[0096] Step S1 includes:

[0097] Step S11. Traverse the data to be partitioned in the main server, number each data item, and record the numbering result as {W1, W2, ..., Wn}, where n is the number of data items to be partitioned, and Wn represents the data item to be partitioned with number n.

[0098] Step S12. Calculate the correlation coefficient of the data to be partitioned using the following formula:

[0099]

[0100] Where P is the correlation coefficient, m is the maximum field length of all data to be partitioned, TU is the alignment function that outputs the number of identical elements in the variable, and ri w1 ,ri wn 、...ri wn These represent the values ​​of the i-th data in W1, W2, ..., Wn, respectively.

[0101] Step S13. Calculate the window length L, where L = P·e, and e is a preset magnification factor. Obtain the data to be partitioned with the largest data volume as the representative data, and use L as the window length to extract fields from the representative data to obtain the set {D1, D2, ..., D...}. m-L}, where D m-L This represents the field consisting of the mL-th to m-th data points in the data set;

[0102] Step S14. Calculate the values ​​of fields D1, D2, ..., D respectively. m-L The probability of occurrence in all the data to be partitioned is used to determine the standard field D0, which is the field with the highest probability of occurrence.

[0103] Step S2. Each sharded database calculates the offset distance according to the database's working parameters and the field offset method, generates a random field, and makes the Hamming distance between the random field and the standard field equal to the offset distance of the sharded database;

[0104] Step S2 includes:

[0105] Step S21. Based on the database's operating parameters, calculate the offset distance of each shard database using the field offset method. The operating parameters include: database data capacity, communication data volume per unit period, and retrieval data volume per unit period. The field offset method is as follows: select the minimum value among the operating parameters and output the offset distance k, k = MOD(U min ·Es), where MOD is the floor function, U min This represents the minimum value among the operating parameters, and Es is the preset amplification factor.

[0106] Step S22. Generate a random field of length m in each sharded database, and ensure that the generated random field satisfies the following condition: the Hamming distance between it and the standard field is equal to the offset distance of the sharded database.

[0107] Step S23. Divide the memory space of the sharded database into independent memory segments and store the random fields of the partitioned servers in the independent memory segments.

[0108] Step S3. Use a preset hash function to process the data to be partitioned and the random fields of each database to obtain the hash values ​​of the primary key field and each random field of the data to be partitioned, and store the data to be partitioned into the shard database with the closest hash value.

[0109] Step S3 includes:

[0110] Step S31. Preset hash function F(x), input random field into hash function to obtain hash value {H1, H2, ..., Ht} of random field of each shard database, where t represents the number of shard databases and Ht represents the hash value of random field of t-th shard database;

[0111] Step S32. Arrange the data to be partitioned in numerical order, extract the first m data from the header of the data to be partitioned to obtain the primary key field, and process the primary key field using the hash function H(x) to obtain the hash value H0 of the primary key field;

[0112] Step S33. Calculate the difference between H1, H2, ... Ht and H0 respectively, select the minimum value, and use the shard database address corresponding to the minimum value as the storage address of the data to be partitioned, and store the data to be partitioned in it;

[0113] Step S34. Perform steps S32-S33 on each piece of data to be partitioned, and store all the data to be partitioned in the database to be partitioned.

[0114] Step S4. When new data is stored in the sharded database, a fixed length of data is extracted from the new data table header as the sharding key. The allocation key is appended to the random field of the sharded database to form a new random field. The updated random field is uploaded to the blockchain and broadcast to all sharded databases through the blockchain.

[0115] Step S4 includes:

[0116] Step S41. Whenever data is stored in the sharded database, extract the first q data from the header of the data table to form the sharding key, where q is a preset parameter and q≥t. Insert the sharding key into the end of the random field of the sharded database to form a new random field.

[0117] Step S42. Whenever a new random field is generated, the sharded database uploads the updated random field to the blockchain. When the data processing center detects the growth of the blockchain, it retrieves the update status of the random field from the blockchain and broadcasts the length of the new random field in all sharded databases.

[0118] Step S5. The user provides the blockchain with a search field of arbitrary length in the header of the required data table. The sharded database processes the random field into a first matrix with the same number of rows as the search field. The search field is used as the augmented part of the first matrix to obtain a second matrix. The length of the last row of the second matrix is ​​calculated. The sharded database with the smallest length of the last row of the second matrix is ​​used as the target database for the request, and the data is retrieved and output from the target database.

[0119] Step S5 includes:

[0120] Step S51. Build a user-oriented interactive platform. When a user needs to search, obtain the search field provided by the user, read the length b of the search field, and generate a first matrix A1 based on the length of the search field. A1 satisfies:

[0121]

[0122] Among them, R b AE represents the b-th data value in the random field of the sharded database, where AE is a preset placeholder data and AE≠0;

[0123] Step S52. Use the retrieved field as a row augmentation part to form a second matrix A2 with the first matrix, where A2 satisfies:

[0124]

[0125] Among them, G1, G2, ... G b Retrieve the 1st, 2nd, ... bth data values ​​in the field respectively;

[0126] Step S53. Simplify the second matrix A2 based on the last row, and output the number of elements with non-zero data values ​​in the last row of the simplified second matrix, which is recorded as the tail row length corresponding to the sharded database.

[0127] Step S54. Select the shard database with the smallest tail length as the target database for the retrieval behavior, retrieve the corresponding data in the target database, and output the retrieval results.

[0128] Example: In a sharding task, there are 5 data entries to be partitioned and 3 shard databases. The correlation coefficient P of the 5 data entries to be partitioned is 0.4, and the amplification factor e is 10. Then, the first 4 data entries to be partitioned are selected as data segments. Among them, the data segment

[1011] appears the most frequently.

[1011] is used as the standard field. The Hamming distances between the 3 shard databases and

[1011] are 1, 2 and 3, respectively. Then, the random fields are

[1010] ,

[1000] and

[1100] , respectively.

[0129] When the selected hash function is a summation function, the hash value of the first piece of data to be partitioned is 1, and the hash values ​​of the random fields are 2, 1, and 2 respectively. Then, the data to be partitioned is stored in sharded database 2, and the first 4 pieces of data are appended to the random field of sharded database 2. The updated random field is then uploaded to the blockchain to complete the sharded storage of the data.

[0130] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0131] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data sharding intelligent supervision method based on blockchain, characterized in that, The method includes the following steps: Step S1. Number each piece of data to be partitioned, calculate the correlation of the data to be partitioned, multiply the correlation by the preset amplification factor and round down to get the window length, slide and cut out a data segment equal to the window length from the data to be partitioned with the largest amount of data, calculate the recurrence rate of each cut data segment in other data, and take the data segment with the highest recurrence rate as the standard field. Step S2. Each sharded database calculates the offset distance according to the database's working parameters and the field offset method, generates a random field, and makes the Hamming distance between the random field and the standard field equal to the offset distance of the sharded database; Step S3. Use a preset hash function to process the data to be partitioned and the random fields of each database to obtain the hash values ​​of the primary key field and each random field of the data to be partitioned, and store the data to be partitioned into the shard database with the closest hash value. Step S4. When new data is stored in the sharded database, a fixed length of data is extracted from the new data table header as the sharding key. The allocation key is appended to the random field of the sharded database to form a new random field. The updated random field is uploaded to the blockchain and broadcast to all sharded databases through the blockchain. Step S5. The user provides the blockchain with a search field of arbitrary length in the header of the required data table. The sharded database processes the random field into a first matrix with the same number of rows as the search field. The search field is used as the augmented part of the first matrix to obtain a second matrix. The length of the last row of the second matrix is ​​calculated. The sharded database with the smallest length of the last row of the second matrix is ​​used as the target database for the request, and the data is retrieved and output from the target database.

2. The data sharding intelligent supervision method based on blockchain according to claim 1, characterized in that: Step S1 includes: Step S11. Traverse the data to be partitioned in the main server, number each data item, and record the numbering result as {W1, W2, ..., Wn}, where n is the number of data items to be partitioned, and Wn represents the data item to be partitioned with number n. Step S12. Calculate the correlation coefficient of the data to be partitioned using the following formula: ； Where P is the correlation coefficient, m is the maximum field length of all data to be partitioned, TU is the alignment function that outputs the number of identical elements in the variable, and ri w1 ,ri wn 、...ri wn These represent the values ​​of the i-th data in W1, W2, ..., Wn, respectively. Step S13. Calculate the window length L, where L = P·e, and e is a preset magnification factor. Obtain the data to be partitioned with the largest data volume as the representative data, and use L as the window length to extract fields from the representative data to obtain the set {D1, D2, ..., D...}. m-L }, where D m-L This represents the field consisting of the mL-th to m-th data points in the data set; Step S14. Calculate the values ​​of fields D1, D2, ..., D respectively. m-L The probability of occurrence in all the data to be partitioned is used to determine the standard field D0, which is the field with the highest probability of occurrence.

3. The data sharding intelligent supervision method based on blockchain according to claim 2, characterized in that: Step S2 includes: Step S21. Based on the database's operating parameters, calculate the offset distance of each shard database using the field offset method. The operating parameters include: database data capacity, communication data volume per unit period, and retrieval data volume per unit period. The field offset method is as follows: select the minimum value among the operating parameters and output the offset distance k, k = MOD(U min ·Es), where MOD is the floor function, U min This represents the minimum value among the operating parameters, and Es is the preset amplification factor. Step S22. Generate a random field of length m in each sharded database, and ensure that the generated random field satisfies the following condition: the Hamming distance between it and the standard field is equal to the offset distance of the sharded database. Step S23. Divide the memory space of the sharded database into independent memory segments and store the random fields of the partitioned servers in the independent memory segments.

4. The data sharding intelligent supervision method based on blockchain according to claim 3, characterized in that: Step S3 includes: Step S31. Preset hash function F(x), input random field into hash function to obtain hash value {H1, H2, ..., Ht} of random field of each shard database, where t represents the number of shard databases and Ht represents the hash value of random field of t-th shard database; Step S32. Arrange the data to be partitioned in numerical order, extract the first m data from the header of the data to be partitioned to obtain the primary key field, and process the primary key field using the hash function H(x) to obtain the hash value H0 of the primary key field; Step S33. Calculate the difference between H1, H2, ... Ht and H0 respectively, select the minimum value, and use the shard database address corresponding to the minimum value as the storage address of the data to be partitioned, and store the data to be partitioned in it; Step S34. Perform steps S32-S33 on each piece of data to be partitioned, and store all the data to be partitioned in the database to be partitioned; Step S4 includes: Step S41. Whenever data is stored in the sharded database, extract the first q data from the header of the data table to form a sharding key, where q is a preset parameter and q≥t. Insert the sharding key into the end of the random field of the sharded database to form a new random field. Step S42. Whenever a new random field is generated, the sharded database uploads the updated random field to the blockchain. When the data processing center detects the growth of the blockchain, it retrieves the update status of the random field from the blockchain and broadcasts the length of the new random field in all sharded databases.

5. The data sharding intelligent supervision method based on blockchain according to claim 4, characterized in that: Step S5 includes: Step S51. Build a user-oriented interactive platform. When a user needs to search, obtain the search field provided by the user, read the length b of the search field, and generate a first matrix A1 based on the length of the search field. A1 satisfies: ； Among them, R b AE represents the b-th data value in the random field of the sharded database, where AE is a preset placeholder data and AE≠0; Step S52. Use the retrieved field as a row augmentation part to form a second matrix A2 with the first matrix, where A2 satisfies: ； Among them, G1, G2, ... G b Retrieve the 1st, 2nd, ... bth data values ​​in the field respectively; Step S53. Simplify the second matrix A2 based on the last row, and output the number of elements with non-zero data values ​​in the last row of the simplified second matrix, which is recorded as the tail row length corresponding to the sharded database. Step S54. Select the shard database with the smallest tail length as the target database for the retrieval behavior, retrieve the corresponding data in the target database, and output the retrieval results.

6. A data sharding intelligent monitoring system based on blockchain, characterized in that, The system includes the following modules: window reproduction module, field memory module, hash storage module, blockchain module, and retrieval targeting module; The window reproduction module consists of data processing software, which is used to calculate the correlation of the data to be partitioned, multiply the correlation by a preset amplification factor and round it to obtain the window length, extract the data segment corresponding to the window length from each data, calculate the reproduction rate of the extracted data segment in other data, and output the data segment with the highest reproduction rate as a standard field. The field memory module consists of partition management software and secondary memory set in the partition server. The partition management software feeds back standard fields to the partition server. The partition server determines the field offset based on the number and database processing capacity, generates random fields, and makes the Hamming distance between the random fields and the standard fields equal to the field offset. The random fields are stored in the secondary memory. The hash storage module is used to process each field of the new data with a preset proportional hash function when new data is stored, to obtain the hash value sequence of the new data, calculate the hash distance between the hash value sequence and the random field in the memory of each shard database, store the data in the shard database with the lowest hash distance, calculate the suffix length according to the field characteristics, insert a field with a suffix length in the data table header as the shard key into the random field, and update the random field. The blockchain module is used to upload the updated random field to the blockchain when the random field is updated, broadcast the update result to all partition servers, and broadcast the search result to the blockchain when the search result of the sharded database is valid during the data retrieval process, thereby stopping the search process of other partition servers. The retrieval targeting module is used to obtain a retrieval field of arbitrary length through user interaction software, send the retrieval field to each shard database, and the shard database processes the random field into a first matrix with the same number of rows as the retrieval field. The retrieval field is used as the augmented part of the first matrix to obtain a second matrix. The length of the last row of the second matrix is ​​calculated, and the shard database with the smallest length of the last row of the second matrix is ​​selected as the requested targeting database. Data is retrieved from the targeting database and output.

7. The data sharding intelligent supervision system based on blockchain according to claim 6, characterized in that: The window reproduction module includes: a loop length unit and a window comparison unit; The loop length unit is used to calculate the correlation of the data to be partitioned, and the length of the data segment truncation window is determined by the correlation. The window comparison unit is used to extract data segments from the data to be partitioned using a sliding sampling window, and to calculate the recurrence rate of the data segments in each partitioned database. The field memory module includes: random partition units and secondary memory units; The random partitioning unit is used to obtain the data capacity of the sharded database, the amount of communication data within a unit period, and the amount of retrieval data within a unit period. The minimum value is then multiplied by a preset coefficient to obtain the field offset. The secondary memory unit is used to generate and store random fields in secondary memory whose Hamming distance from the standard field is equal to the field offset.

8. The data sharding intelligent supervision system based on blockchain according to claim 7, characterized in that: The hash storage module includes: a field matching unit, an input storage unit, and a table header link unit; The field matching unit is used to calculate the hash value sequence of the newly stored data and output the random field that is closest to the hash value sequence. The storage unit is used to store new data into the sharded database corresponding to the output random field; The header linking unit is used to extract fields from the new data header and insert them into the random fields corresponding to the sharded database where the new data is stored.

9. A data sharding intelligent monitoring system based on blockchain according to claim 8, characterized in that: The blockchain module includes: an encrypted upload unit and a partitioned broadcast unit; The encrypted upload unit is used to upload the updated random field to the blockchain when new data is stored in the sharded database; The partition broadcasting unit is used to broadcast the results of new data storage and the results of retrieval fields in the partition database to all partition databases.

10. A data sharding intelligent monitoring system based on blockchain according to claim 9, characterized in that: The search targeting module includes: a search interaction unit, a field augmentation unit, and a partition output unit; The search interaction unit is used to provide a user-oriented interactive platform to obtain search fields provided by the user. The field augmentation unit is used to construct an augmentation matrix by taking the user's search field and the random fields of each shard database as the rows and columns of the matrix, calculate the tail row length of the augmentation matrix, and designate the database corresponding to the matrix with the smallest tail row length as the targeted database. The partition output unit is used to retrieve data containing the search field header in the targeted database and output the search results to the user interaction page.

Images