Incremental calculation-based hierarchical storage architecture sampling method for blockchain

Abstract

The application provides a kind of incremental calculation-based hierarchical storage architecture sampling method of blockchain, comprising: obtaining original data in low-layer blockchain;Based on the original data, generate a first feature vector;The first feature vector generated based on the first feature vector and the first variance corresponding to the first feature vector are stored to high-layer blockchain;Obtain the update data in low-layer blockchain;Based on the first feature vector and the first variance, the update data is clustered according to data distribution change;Based on the clustering result, the update data of the low-layer blockchain is integrated in high-layer blockchain using incremental calculation method.Integrate the update data of the low-layer blockchain in the high-layer blockchain using the incremental calculation method, ensure the coverage of the high-layer blockchain to the low-layer data blockchain as a whole, and at the same time, select appropriate samples to replace the original sampling results when the low-layer data is updated, thereby helping to optimize the hierarchical multi-modal data storage architecture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of blockchain, in particular to a sampling method and device for hierarchical storage architecture of blockchain based on incremental calculation. BACKGROUND

[0002] In the blockchain architecture, due to the limited amount of information that can be transmitted by transactions, a large amount of multimedia hypertext information still needs to be stored in the distributed storage engine of each node, and the index corresponding to the information is circulated in the blockchain, and then the data is obtained from the distributed engine according to the index by different nodes. However, due to the need of hierarchical management of the blockchain, the data on the lower layer blockchain will be summarized in the higher layer blockchain, resulting in too large data on the higher layer blockchain.

[0003] In order to ensure the coverage of the data on the higher layer blockchain to the whole lower layer blockchain, the traditional technology generally samples the original data and the updated data every time new data is input, which is not effective, so that it cannot guarantee that the appropriate sample is selected to replace the original sampling result in time when the lower layer data is updated.

[0004] The above problems are urgent to be solved. SUMMARY

[0005] The present application overcomes the above-mentioned shortcomings of the prior art, based on the advantages of incremental calculation, and provides a sampling method and device for hierarchical storage architecture of blockchain based on incremental calculation.

[0006] The technical scheme adopted by the present application to solve its technical problems is: a sampling method for hierarchical storage architecture of blockchain based on incremental calculation, comprising: obtaining original data in a lower layer blockchain; generating a first feature vector based on the original data; storing a first feature vector generated based on the first feature vector and a first variance corresponding to the first feature vector to a higher layer blockchain; obtaining updated data in the lower layer blockchain; clustering the updated data according to data distribution changes based on the first feature vector and the first variance; integrating the updated data of the lower layer blockchain in the higher layer blockchain by using an incremental calculation method based on the clustering result.

[0007] Further, the original data and the updated data are multi-modal hypertext data, and the multi-modal hypertext data includes one or a combination of graphic-text combined data, video-text combined data and audio-text combined data.

[0008] Further, the generating of the first feature vector based on the original data includes: obtaining a unified feature model by using a multi-modal joint learning method; and using the feature model to represent all low-layer multi-modal hypertext data as a first feature vector of a unified length.

[0009] Further, the storing, to the high-level blockchain, of the first feature vector generated based on the first representation vector and the first variance corresponding to the first feature vector comprises: calculating cosine similarities between the first representation vectors two by two; clustering all the first representation vectors into a plurality of first sub-sub-sets with higher internal similarity according to the cosine similarities; calculating a first variance between the representation vectors in each first sub-set; taking a representation vector corresponding to a center point of each first sub-set as a first feature vector; and storing the first feature vector and the first variance in the high-level blockchain.

[0010] Further, the clustering, according to data distribution changes, of the update data based on the first feature vector and the first variance comprises: obtaining a second representation vector based on the update data; generating a first probability corresponding to the second representation vector based on the second representation vector, the first feature vector and the first variance; and clustering the second representation vectors corresponding to the first probabilities less than a first threshold into a new set.

[0011] Further, the generating of the first probability corresponding to the second representation vector based on the second representation vector, the first feature vector and the first variance comprises: inputting the second representation vector, the first feature vector and the first variance into a Gaussian distribution model to output the first probability.

[0012] Further, the integrating, in the high-level blockchain, of the update data of the low-level blockchain based on the clustering result by using an incremental calculation method comprises: calculating a second variance between a second representation vector corresponding to a center point of the new set and the second representation vectors in the new set; generating a second probability of the first feature vector based on the second representation vector corresponding to the center point of the new set and the second variance; updating the first variance corresponding to the first feature vector saved in the high-level blockchain based on the second probability; and saving the second variance and the second representation vector corresponding to the center point of the new set into the high-level blockchain.

[0013] The application further provides a sampling device of a blockchain hierarchical storage architecture based on incremental calculation, which comprises: an original data acquisition module, configured to acquire original data in a low-level blockchain; a first representation vector generation module, configured to generate a first representation vector based on the original data; a storage module, configured to store, to a high-level blockchain, a first feature vector generated based on the first representation vector and a first variance corresponding to the first feature vector; an update data acquisition module, configured to acquire update data in the low-level blockchain; a clustering module, configured to cluster the update data according to data distribution changes based on the first feature vector and the first variance; and an integrating module, configured to integrate, in the high-level blockchain, the update data of the low-level blockchain based on the clustering result by using an incremental calculation method.

[0014] The application also provides a computer readable storage medium, wherein one or more instructions are stored in the computer readable storage medium, and the processor of the risk analysis device in the one or more instructions is executed to implement the sampling method of the hierarchical storage architecture of the blockchain based on the incremental calculation.

[0015] The application also provides an electronic device, comprising a memory and a processor, wherein at least one program instruction is stored in the memory, and the processor is loaded and executed to implement the sampling method of the hierarchical storage architecture of the blockchain based on the incremental calculation.

[0016] The application has the following beneficial effects: the sampling method of the hierarchical storage architecture of the blockchain based on the incremental calculation comprises the following steps: obtaining original data in a low-layer blockchain; generating a first feature vector based on the original data; storing a first feature vector generated based on the first feature vector and a first variance corresponding to the first feature vector into a high-layer blockchain; obtaining updated data in the low-layer blockchain; clustering the updated data according to data distribution changes based on the first feature vector and the first variance; and integrating the updated data of the low-layer blockchain into the high-layer blockchain by using an incremental calculation method based on the clustering result. In the process of sampling data of the blockchain, after the original data is sampled, the original data sampled data and new data are combined to be sampled again each time new data is input, and all data in the original data does not need to be resampled, so that the updated data of the low-layer blockchain is integrated into the high-layer blockchain by using the incremental calculation method, the coverage of the high-layer blockchain data to the low-layer data blockchain is ensured, and appropriate samples are selected in time to replace the original sampling result when the low-layer data is updated, thereby helping to optimize the hierarchical multi-modal data storage architecture. BRIEF DESCRIPTION OF DRAWINGS

[0017] The application will be further described below in combination with the drawings and embodiments.

[0018] Figure 1 FIG. 1 is a flowchart of the sampling method of the hierarchical storage architecture of the blockchain based on the incremental calculation provided by the embodiments of the application.

[0019] Figure 2 FIG. 2 is a principle block diagram of the sampling device of the hierarchical storage architecture of the blockchain based on the incremental calculation provided by the embodiments of the application.

[0020] Figure 3 FIG. 3 is a partial block diagram of the electronic device provided by the embodiments of the application. DETAILED DESCRIPTION

[0021] Before any examples embodiments are described in further detail, it should be noted that some example embodiments are described as processes depicted as flow diagrams. Although the processes are described in a particular sequential order, many of the processes can be performed concurrently, in parallel, or simultaneously. In addition, the order of the processes can be re-arranged. The processes can be terminated when their functions are completed, but the processes can also have additional steps not included in the figure. The processes can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0022] It should be understood that, although terms such as "first," "second," and the like can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element without departing from the scope of the example embodiments. The term "and / or" as used herein encompasses any and all combinations of one or more of the associated associated items.

[0023] The application will now be described in detail by reference to the drawings. The drawings are schematic representations of the application, and as such, they show only the essential structures of the application, and they are not to scale. Thus, for purposes of clarity, not every component is shown. In the drawings:

[0024] Example 1

[0025] Referring to Figure 1 The sampling method based on incremental calculation of the hierarchical blockchain storage architecture ensures the coverage of high-level blockchain data on the low-level data blockchain as a whole, and timely selects appropriate samples to replace the original sampling results when the low-level data is updated, thereby helping to optimize the hierarchical multi-modal data storage architecture.

[0026] As an example, the sampling method based on incremental calculation of the hierarchical blockchain storage architecture includes the following steps:

[0027] S110: Obtain original data in a low-level blockchain.

[0028] As an example, the original data is multi-modal hypertext data, and the multi-modal hypertext data includes one or a combination of picture-text combined data, picture-text combined data, and sound-text combined data.

[0029] S120: Generate a first feature vector based on the original data.

[0030] As an example, the generation of the first feature vector based on the original data includes: obtaining a unified feature model using a multi-modal joint learning method; and using the feature model to represent all low-level multi-modal hypertext data as a first feature vector of a unified length.

[0031] Specifically, all low-layer multi-modal hypertext data is integrated, a unified representation model M is obtained by using a multi-modal joint learning method, and all low-layer multi-modal hypertext data is represented into a unified length representation vector V (a first representation vector) by using the model M. The length of the output vector of the multi-modal is set by a user, for example, a technical person can set the length of the output vector to 1000, so that the image, video, and audio can all be expressed into a 1000-length vector. The multi-modal joint learning method is mature in the prior art, and will not be described in detail here. Because the vectors have the same length, similarity can be compared, and when the vectors are different, it is impossible to compare the similarity between a video and an image. Therefore, the lengths of all vectors are unified here, which facilitates the smooth progress of subsequent steps.

[0032] S130: store the first feature vector generated based on the first representation vector and the first variance corresponding to the first feature vector to a high-layer blockchain.

[0033] As an example, the storing of the first feature vector generated based on the first representation vector and the first variance corresponding to the first feature vector to a high-layer blockchain includes: calculating the cosine similarity between the first representation vectors two by two; clustering all first representation vectors into a plurality of first sub-sets with high internal similarity according to the cosine similarity; calculating the first variance between the representation vectors in each first sub-set; taking the representation vector corresponding to the center point of each first sub-set as a first feature vector; and storing the first feature vector and the first variance in the high-layer blockchain.

[0034] Specifically, the cosine similarity between the first representation vectors V two by two is calculated, and all representation vectors are clustered into k first sub-sets with high internal similarity according to the cosine similarity by using a k-center algorithm. The cosine similarity means that the cosine value of the angle between two vectors in a vector space is used as a measure of the difference between the two individuals, and the closer the value is to 1, the closer the angle is to 0°, that is, the two vectors are more similar, which is called cosine similarity. It measures the similarity of the proportions of two variables in each direction (attribute). Therefore, by calculating the cosine similarity between two vectors, it can be determined whether the two vectors are similar. When the two vectors are similar enough, the similar vectors are clustered into a sub-set. The feature vectors {v1, v2, …, vk} of the k center points and the corresponding variances {s1, s2, …, sk} are saved in the high-layer blockchain. That is, by this technical solution, representative data in similar data in the low-layer blockchain can be transmitted to the high-layer blockchain for saving, instead of storing all data obtained in the low-layer blockchain to the high-layer blockchain, so as to reduce the burden of the high-layer blockchain.

[0035] Specifically, if the obtained first feature vector is and the cosine similarity calculation is performed on the , three sub-sets with high internal similarity are obtained (Set ), Set ) and Set , wherein has high similarity, has high similarity has high similarity. The variance between vectors in each first sub-set is calculated, that is, the variances of the above three sets (Set ), Set ) and Set ) are calculated to obtain variances , the vectors corresponding to the center points of the three sub-sets , and are saved to the high-level blockchain. Among them , and are the first feature vectors. It should be noted that the calculation of cosine similarity and k-center algorithm described above are very mature in the prior art, and will not be described here.

[0036] S140: Obtain update data in the low-level blockchain.

[0037] As an example, the update data is multi-modal hypertext data, and the multi-modal hypertext data includes one or a combination of picture-text combined data, picture-text combined data and audio-text combined data.

[0038] S150: Cluster the update data according to data distribution changes based on the first feature vector and the first variance.

[0039] As an example, the clustering of the update data according to data distribution changes based on the first feature vector and the first variance includes: obtaining a second feature vector based on the update data; generating a first probability corresponding to the second feature vector based on the second feature vector, the first feature vector and the first variance; and clustering the second feature vector corresponding to the first probability less than the first threshold value into a new set.

[0040] ​Specifically, based on the above-mentioned method of obtaining the first feature vector, the second feature vector Y is obtained here, the first feature vector {v1, v2, …, vk} and the corresponding variance {s1, s2, …, sk} of the k center points saved in the high-level blockchain are used to calculate the probability p of each update data feature vector, and all feature vectors with a generated probability p less than 0.05 (first threshold) are integrated into a new cluster C. More specifically, the second feature vector Y, the first feature vector, and the first variance are input into the Gaussian distribution model, which outputs the probability value of the category to which the second feature vector belongs. When the probability value is too small, it proves that the second feature vector is a new vector, and when the probability value is large, it proves that the second feature vector already exists in the high-level blockchain, so it is not necessary to sample it again, and then only the new data needs to be sampled.

[0041] For example, there are 10 second feature vectors, and the second feature vector is After inputting the second feature vector and the above-mentioned first feature vector and first variance into the Gaussian model, a probability value is outputted. If the probability value of the second feature vector is less than 0.05, , then the new set C is .

[0042] S160: Based on the clustering result, the update data of the low-level blockchain is integrated into the high-level blockchain using an incremental calculation method.

[0043] As an example, the method of integrating the update data of the low-level blockchain into the high-level blockchain based on the clustering result includes: calculating the second feature vector corresponding to the center point of the new set and the second variance between the second feature vectors in the new set; generating a second probability corresponding to the second feature vector based on the second feature vector and the second variance; updating the variance corresponding to the second probability saved in the high-level blockchain; saving the updated variance and the second feature vector corresponding to the center point of the second sub-set to the high-level blockchain.

[0044] Specifically, for the above-mentioned obtained set C, the center point and the second variance are calculated, and based on the above-mentioned method, the Gaussian distribution is used to calculate the probability p1, p2, …, pk of the vector (i.e. the first feature vector) of the k center points saved in the high-level blockchain generated by the second feature vector corresponding to the center point of the set C and the second variance. The i-th first variance saved in the high-level blockchain is updated to , and the second feature vector and the second variance corresponding to the center point are saved in the high-level blockchain.

[0045] Based on the above method, in the process of sampling the data of the blockchain, after sampling the original data, the data after sampling the original data and the new data are combined and sampled again each time new data is input, without re-sampling all the data in the original data. By using the incremental calculation method, the updated data of the lower layer blockchain is integrated into the higher layer blockchain, ensuring the coverage of the higher layer blockchain data to the lower layer data blockchain as a whole, and timely selecting appropriate samples to replace the original sampling results when the lower layer data is updated, thereby helping to optimize the hierarchical multi-modal data storage architecture.

[0046] Embodiment 2

[0047] Referring to Figure 2 The embodiment provides a kind of based on incremental calculation's blockchain hierarchical storage architecture sampling device, comprising:

[0048] The original data acquisition module 210 is used to acquire the original data in the lower layer blockchain, and the original data is multi-modal hypertext data.

[0049] The first feature vector generation module 220 is used to generate a first feature vector based on the original data.

[0050] As an example, the first feature vector generation module 220 includes: an acquisition representation model unit adapted to acquire a unified representation model using a multi-modal joint learning method; and a representation unit adapted to represent all lower layer multi-modal hypertext data as a first feature vector of a uniform length using the representation model.

[0051] The storage module 230 is used to store the first feature vector generated based on the first feature vector and the first variance corresponding to the first feature vector to the higher layer blockchain.

[0052] As an example, the storage module 230 includes: a calculation unit adapted to calculate the cosine similarity between the first feature vectors; a clustering unit adapted to cluster all first feature vectors into a plurality of first sub-sets with high internal similarity according to the cosine similarity; a first variance calculation unit adapted to calculate the first variance between the feature vectors in each first sub-set; a unit adapted to take the feature vector corresponding to the center point of each first sub-set as the first feature vector; and a storage unit adapted to store the first feature vector and the first variance in the higher layer blockchain.

[0053] The updated data acquisition module 240 is used to acquire updated data in the lower layer blockchain.

[0054] The clustering module 250 is used to acquire updated data in the lower layer blockchain and cluster the updated data according to data distribution changes based on the first feature vector and the first variance.

[0055] As an example, the clustering module 240 comprises: a second feature vector acquisition unit adapted to acquire a second feature vector based on the update data; a first probability generation unit adapted to generate a first probability corresponding to the second feature vector based on the second feature vector and the first feature vector and the first variance; and a clustering unit adapted to cluster the second feature vector corresponding to the first probability less than a first threshold into a new set.

[0056] The integration module 260 is configured to integrate the update data of the low-level blockchain into the high-level blockchain using an incremental calculation method based on the clustering result.

[0057] As an example, the integration module 260 comprises: a second variance calculation unit configured to calculate a second variance between the second feature vector corresponding to the center point of the new set and the second feature vectors in the new set; a second probability generation unit configured to generate a second probability of the first feature vector based on the second feature vector corresponding to the center point of the new set and the second variance; an update unit configured to update the first variance corresponding to the first feature vector saved in the high-level blockchain based on the second probability; and a saving unit configured to save the second feature vector corresponding to the center point of the new set and the second variance into the high-level blockchain.

[0058] Embodiment 3

[0059] The embodiment of the present application also provides a storage medium, wherein the storage medium stores the sampling method for the incremental calculation-based blockchain hierarchical storage architecture, and the distributed computing resource scheduler is executed by a processor to implement the steps of the distributed computing resource scheduling method as described above. Since the storage medium adopts all the technical solutions of the above embodiments, it has all the beneficial effects brought by the technical solutions of the above embodiments, which will not be repeated here.

[0060] Embodiment 4

[0061] Please refer to Figure 3 The embodiment of the present application also provides an electronic device, which comprises a memory and a processor, wherein the memory stores at least one program instruction, and the processor loads and executes the at least one program instruction to implement the sampling method for the incremental calculation-based blockchain hierarchical storage architecture provided in the embodiment 1.

[0062] The memory 302 and the processor 301 are connected by a bus, which can include any number of interconnecting buses and bridges needed to support various components. The bus can also include various other components, such as a memory bridge, a processor bus, a peripheral bus, and the like. The bus interfaces between the bus and the transceiver. The transceiver can be a single device or multiple devices, such as a plurality of receivers and transmitters, that provide means for communicating with various other apparatus over a transmission medium. Data processed by the processor 301 is transmitted over a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor 301.

[0063] The processor 301 is responsible for managing the bus and general processing, and can also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory 302 can be used to store data used by the processor 301 during execution of operations.

[0064] Based on the above ideal embodiments according to the present application, through the above description, relevant personnel can make various changes and modifications without deviating from the scope of the present application. The technical scope of the present application is not limited to the content of the specification, and must be determined according to the scope of the claims.

Claims

1. A sampling method for a hierarchical storage architecture of a blockchain based on incremental calculation, characterized in that, The method comprises: obtaining original data in a low-layer blockchain; generating a first feature vector based on the original data; storing the first feature vector generated based on the first feature vector and a first variance corresponding to the first feature vector to a high-layer blockchain; obtaining updated data in the low-layer blockchain; clustering the updated data according to data distribution changes based on the first feature vector and the first variance; integrating the updated data of the low-layer blockchain in the high-layer blockchain based on the clustering result by using an incremental calculation method; the obtaining of the updated data in the low-layer blockchain and the clustering of the updated data according to data distribution changes based on the first feature vector and the first variance comprise: obtaining a second feature vector based on the updated data; generating a first probability corresponding to the second feature vector based on the second feature vector and the first feature vector and the first variance; clustering the second feature vector corresponding to the first probability less than a first threshold value into a new set; the integrating of the updated data of the low-layer blockchain in the high-layer blockchain based on the clustering result by using the incremental calculation method comprises: calculating a second variance between a second feature vector corresponding to a center point of the new set and the second feature vectors in the new set; generating a second probability of the first feature vector based on the second feature vector corresponding to the center point of the new set and the second variance; updating the first variance corresponding to the first feature vector saved in the high-layer blockchain based on the second probability; saving the second variance and the second feature vector corresponding to the center point of the new set to the high-layer blockchain.

2. The incremental calculation-based hierarchical storage architecture sampling method of claim 1, wherein, The original data and the updated data are multi-modal hyper-text data, and the multi-modal hyper-text data comprise one or a combination of picture-text combined data, video-text combined data and audio-text combined data.

3. The incremental computation based blockchain hierarchical storage architecture sampling method of claim 1, wherein, The generating of the first feature vector based on the original data comprises: obtaining a unified feature model by using a multi-modal joint learning method; characterizing all low-layer multi-modal hyper-text data into first feature vectors of a unified length by using the feature model.

4. The incremental computation based blockchain hierarchical storage architecture sampling method of claim 1, wherein, The storing of the first feature vector generated based on the first feature vector and the first variance corresponding to the first feature vector to the high-layer blockchain comprises: calculating cosine similarity between the first feature vectors two by two; clustering all first feature vectors into a plurality of first sub-sets with high internal similarity according to the cosine similarity; calculating a first variance between the feature vectors in each first sub-set; taking a feature vector corresponding to a center point of each first sub-set as a first feature vector; storing the first feature vector and the first variance in the high-layer blockchain.

5. The incremental computation based blockchain hierarchical storage architecture sampling method of claim 1, wherein, The generating of a first probability corresponding to the second feature vector based on the second feature vector and the first feature vector and the first variance comprises: inputting the second feature vector, the first feature vector and the first variance into a Gaussian distribution model to output the first probability.

6. A blockchain hierarchical storage architecture sampling device based on incremental calculation, characterized in that, The device comprises: an original data obtaining module configured to obtain original data in a low-layer blockchain; a first feature vector generating module configured to generate a first feature vector based on the original data; a storage module configured to store a first feature vector generated based on the first representation vector and a first variance corresponding to the first feature vector to a high-level blockchain; an update data acquisition module configured to acquire update data in a low-level blockchain; a clustering module configured to cluster the update data according to data distribution changes based on the first feature vector and the first variance, including acquiring a second representation vector based on the update data, generating a first probability corresponding to the second representation vector based on the second representation vector, the first feature vector and the first variance, and clustering a second representation vector corresponding to a first probability less than a first threshold into a new set; an integration module configured to integrate the update data of the low-level blockchain in the high-level blockchain based on the clustering result using an incremental calculation method, including calculating a second representation vector corresponding to a center point of the new set and a second variance between second representation vectors in the new set, generating a second probability of the first feature vector based on the second representation vector corresponding to the center point of the new set and the second variance, updating the first variance corresponding to the first feature vector saved by the high-level blockchain based on the second probability, and saving the second variance and the second representation vector corresponding to the center point of the new set to the high-level blockchain.

7. A computer-readable storage medium having stored therein at least one instruction, wherein The computer instructions are used to enable the computer to execute the incremental calculation-based blockchain hierarchical storage architecture sampling method of any one of claims 1 to 5.

8. An electronic device, comprising: comprising: a memory and a processor; at least one program instruction is stored in the memory; the processor loads and executes the at least one program instruction to realize the incremental calculation-based blockchain hierarchical storage architecture sampling method of any one of claims 1 to 5.

Images