A big data real-time processing system and method

Abstract

The present invention discloses a big data real-time processing system and method, which relates to the technical field of big data processing, and comprises a data segmentation module, a data storage module, a data reading module and a data backup module; the data segmentation module utilizes a segmentation strategy to segment the big data entering the blockchain in real time, extracts keywords, and distinguishes the frequency of data blocks according to historical query records; the data storage module stores data blocks through storage contracts, encrypts their identifiers, and establishes mapping relationships; the data reading module screens data blocks based on a private key set when a user queries, and determines the sending order through keyword matching; the data backup module generates a backup after storage, and performs different update strategies for data blocks of different frequencies, and adaptively modifies the original data using a hash value; combined with blockchain technology, it integrates multiple advantages such as real-time processing, efficient storage, secure reading and flexible backup, and provides a comprehensive and efficient solution for big data processing.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a system and method for real-time processing of big data. Background Art

[0002] With the rapid development of information technology, big data has become an indispensable part of modern society. All walks of life are constantly generating and accumulating massive amounts of data, which contain huge value and potential. However, how to efficiently process, store and manage these big data has become a major challenge facing enterprises and research institutions. In particular, in scenarios where real-time processing and analysis of big data are required, traditional data processing systems often cannot meet the needs. Therefore, the development of an efficient and reliable big data real-time processing system is particularly important.

[0003] In the field of big data processing, traditional batch processing systems and stream processing systems each have their limitations: batch processing systems are usually suitable for offline processing of large-scale data, but cannot meet real-time requirements; and although stream processing systems can process real-time data streams, their performance and efficiency are often unsatisfactory when dealing with complex data analysis and queries; in addition, the storage and management of big data is also a thorny issue. How to effectively block, store and read data to improve data processing efficiency and response speed is a key issue that big data processing systems need to solve. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In response to the technical problems in the background technology, the present invention proposes a big data real-time processing system and method, which effectively solves the limitations of traditional big data processing systems in real-time, performance efficiency and big data storage management through real-time segmentation, efficient keyword extraction, optimized storage and reading mechanism, and intelligent data backup and update strategy; thereby solving the technical problems recorded in the background technology.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] A big data real-time processing system, comprising:

[0009] The data segmentation module, when detecting that big data enters the blockchain network, performs real-time segmentation of the big data based on the first segmentation strategy and the second segmentation strategy in the segmentation contract; extracts keywords from each data block based on the jieba word segmentation tool and the TF-IDF feature extraction algorithm; and divides the data blocks into high-frequency data blocks and low-frequency data blocks based on the historical query record data of the data query platform in the past week;

[0010] The data storage module performs storage operations on each data block after segmentation based on the storage contract and generates a block identifier for each data block; performs encryption operations on the block identifier of each data block and stores the mapping relationship between the block identifier of each data block and the node location where it is stored in a distributed hash table;

[0011] Data reading module: when a user performs a data query operation, the accessible data blocks are filtered out based on the user's private key set, and the query keywords entered by the user are matched with the keywords of the filtered data blocks based on the reading contract. The sending order of the data corresponding to the data blocks is determined according to the number of query keywords and the order of keyword matching, and displayed in the data query platform in the sending order;

[0012] The data backup module, after completing the storage of data blocks, backs up all data blocks based on the backup contract, generates the same number of backup data blocks as the original data blocks, and the corresponding block identifiers; when monitoring the user to perform data modification operations, if the data block to be modified is a high-frequency data block, the backup data block is updated in real time; if the data block to be modified is a low-frequency data block, the backup data block is updated non-real time; the original data is adaptively modified based on the hash value in the block identifier.

[0013] Specifically, the first block strategy is as follows: in the process of big data passing through the big data processing interface, the cumulative amount of data passing through is recorded in real time; when it is monitored that the cumulative amount of data passing through exceeds the set block data flow threshold, a block operation is performed, that is, the big data is split at the data passing through the big data processing interface at this time, and the data that has passed through the big data processing interface after the split is packaged into a data block, and stored in the buffer block area; after the block operation is performed, the cumulative amount of data passing through is cleared, and the first block strategy is repeated; if the remaining amount of data is not enough to exceed the block data flow threshold, the remaining data is packaged into a data block.

[0014] Specifically, the second block strategy is as follows: divide the content of each data block in the buffer block area based on the content category to obtain a number of intermediate data blocks, and update the number of pointers and the data blocks they point to; the content category includes data type, data subject, data source, and business process link, and in a data block, the continuous data of the same data type, the same data subject, the same data source, and the same business process link are retained in the same intermediate data block;

[0015] Based on the Jieba word segmentation tool, all the words in each intermediate data block are extracted, and stop words are screened out from the extracted words, including meaningless words and punctuation marks; the importance index of the remaining words in the corresponding intermediate data block is calculated based on the TF-IDF feature extraction algorithm; the words with the top three importance indexes in each intermediate data block are used as keywords of the intermediate data block.

[0016] Further, historical query record data of the data query platform in the past week is obtained, and the number of accesses to the data corresponding to each intermediate data block is counted respectively, and the total number of accesses to each intermediate data block in the past week is obtained. If there is a data access corresponding to multiple intermediate data blocks, the number of accesses to these intermediate data blocks is increased by one at the same time; a frequency judgment standard value is set, and intermediate data blocks whose total access frequency is not less than the frequency judgment standard value are stored in a high-frequency access buffer area, and intermediate data blocks whose total access frequency is less than the frequency judgment standard value are stored in a low-frequency access buffer area;

[0017] For the intermediate data blocks in the high-frequency access buffer area and the low-frequency access buffer area, merge the intermediate data blocks with the same keywords, the same total access frequency and continuous pointers, and update the pointers;

[0018] The middle data blocks in the low-frequency access buffer area are recorded as low-frequency data blocks, and the middle data blocks in the high-frequency access buffer area are recorded as high-frequency data blocks.

[0019] Specifically, the block identifier is obtained by combining a hash value generated for each data block, a category of each data block, a pointer of each data block to the next data block, and a keyword of each data block;

[0020] Among them, the category of each data block represents the high frequency or low frequency of the data block; the first keyword, the second keyword and the third keyword respectively represent the top three words in importance index in each data block; the block identifiers of all data blocks are recorded, and the recorded results are stored in a new data block, recorded as an identifier data block.

[0021] Specifically, when a user performs a data query operation, the query keyword input by the user is combined with the user's private key set to form a request string, and the user's private key set stores all RSA private keys in the user's account; a transmission interface for reading the contract is defined in the storage contract, and when the request string is formed, the request string is transmitted to the storage contract through the transmission interface;

[0022] Based on the character comparison function in the storage contract, the received request string and all the filtered block identifiers are split, specifically: the query keyword in the request string is split out, and the first keyword, the second keyword and the third keyword in the block identifier are split out. The query keyword, the first keyword, the second keyword and the third keyword all exist in the form of strings.

[0023] Further, if there is only one query keyword, the query keyword is sequentially matched with all the first keywords, the second keywords, and the third keywords, and the block identifiers corresponding to the matched keywords are sequentially placed in the read queue according to the order of query keyword matching, wherein the matching order is in the order of the first keyword, the second keyword, and the third keyword; if the matching order of multiple block identifiers is the same, the two block identifiers are sequentially placed in the read queue in descending order according to the total access frequency of the data blocks corresponding to the multiple block identifiers;

[0024] If there are multiple query keywords, the query keywords are split out one by one, and the split query sub-keywords are matched with the first keyword, the second keyword and the third keyword one by one; the corresponding block identifiers are put into the reading queue in sequence according to the order of query sub-keyword matching, that is, the block identifiers that match the keywords with a higher order in the block identifier are put into the reading queue first; if two block identifiers match the same number of keywords with the query keywords, and the order of these keywords is also the same, then according to the total access frequency of the data blocks corresponding to the two block identifiers, the two block identifiers are put into the reading queue in sequence from large to small.

[0025] Furthermore, when the storage of the data blocks is completed, the backup function in the backup contract is called to copy all the stored data blocks, and the hash values ​​in the block identifiers of these backup data blocks are the same as the hash values ​​in the block identifiers of the corresponding data blocks;

[0026] When a user is detected to be performing a data modification operation, the backup data block is updated based on the category of the modified data block;

[0027] When the data block operated by the user is a high-frequency data block, a new data block is generated in real time based on the modified result of the user on the data query platform. The newly generated data block replaces the original backup data block through the storage location of the recorded backup data block, and the storage node location does not change;

[0028] When the data block operated by the user is a low-frequency data block, only when it is determined that the user has ended access to the data block, a data modification signal is sent to the backup contract, and the backup data block is updated in the same way as the high-frequency data block.

[0029] Furthermore, when the backup data block is updated, the hash value corresponding to the content in the backup data block will also change, and then the block identifier of the backup data block will also change; when the backup contract detects that the block identifier of the backup data block has changed, the block identifiers of the backup data block before and after the change are sent to the storage contract, the original data is found based on the hash value of the block identifier before the change, and the original data is updated based on the hash value of the block identifier after the change. The original data is the data block stored in the data storage module.

[0030] A method for real-time processing of big data, comprising:

[0031] Step 1: When big data is detected entering the blockchain network, the big data is segmented in real time based on the first segmentation strategy and the second segmentation strategy in the segmentation contract; keywords are extracted from each data block based on the jieba word segmentation tool and the TF-IDF feature extraction algorithm; data blocks are divided into high-frequency data blocks and low-frequency data blocks based on the historical query record data of the data query platform in the past week;

[0032] Step 2: Perform storage operations on each data block after block division based on the storage contract, and generate a block identifier for each data block; perform encryption operations on the block identifier of each data block, and store the mapping relationship between the block identifier of each data block and the node location where it is stored in a distributed hash table;

[0033] Step 3: When the user performs a data query operation, the accessible data blocks are filtered out based on the user's private key set, and the query keywords entered by the user are matched with the keywords of the filtered data blocks based on the reading contract. The sending order of the data corresponding to the data blocks is determined according to the number of query keywords and the order of keyword matching, and displayed in the data query platform in the sending order;

[0034] Step 4: After the storage of the data blocks is completed, all data blocks are backed up based on the backup contract to generate the same number of backup data blocks as the original data blocks, as well as the corresponding block identifiers; when the user is detected to be performing a data modification operation, if the data block to be modified is a high-frequency data block, the backup data block is updated in real time; if the data block to be modified is a low-frequency data block, the backup data block is updated non-real time; the original data is adaptively modified based on the hash value in the block identifier.

[0035] (III) Beneficial effects

[0036] The present invention provides a big data real-time processing system and method, which has the following beneficial effects:

[0037] 1. By monitoring the big data entering the blockchain network in real time, and performing efficient and accurate block processing on the big data based on the dual block strategy, it not only ensures the real-time and integrity of the data, but also realizes efficient indexing and classification of data blocks through content division and keyword extraction; combined with historical query records, the data blocks are divided into high-frequency and low-frequency, which helps to optimize storage and query efficiency, so that high-frequency data can be accessed faster in the future, and low-frequency data can reduce unnecessary storage overhead, thereby improving the performance and response speed of the entire big data processing system;

[0038] 2. Generate a unique hash value for the data block through the SHA2-256 algorithm, and combine the category, pointer and keyword of the data block to form a block identifier, ensuring the uniqueness and traceability of the data; encryption operations and control of user access rights enhance data security; store the mapping relationship between the block identifier and the data block storage location in a distributed hash table, achieving efficient retrieval and positioning of data; this storage method not only improves the storage efficiency of data, but also ensures the reliability and persistence of data in the blockchain network, providing a solid storage foundation for real-time processing of big data;

[0039] 3. The accessible data blocks are screened out through the user's private key set, and the sending order of the data blocks is determined based on the keyword matching algorithm, realizing accurate data query and fast response; it not only supports the matching of a single query keyword, but also can handle complex query requirements of multiple query keywords, thereby improving the flexibility and accuracy of the query; by displaying the results in the data query platform according to the sending order, users can intuitively obtain the required data, improving user experience and query efficiency, and meeting the requirements for data reading speed and accuracy in real-time processing of big data;

[0040] 4. When the user modifies the data, the module can adopt corresponding update strategies according to the high-frequency or low-frequency characteristics of the data block, thereby reducing unnecessary backup overhead while ensuring data integrity; the original data is adaptively modified through the hash value in the block identifier to achieve seamless data update and synchronization; this backup mechanism not only improves the data recovery capability, but also adopts different update methods based on different data, providing reliable data protection for real-time processing of big data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of the structure of a big data real-time processing system provided by the present invention;

[0042] Figure 2 A flowchart of the steps of a real-time big data processing method provided by the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] refer to Figure 1 The present invention provides a big data real-time processing system, comprising:

[0045] The data segmentation module, when detecting that big data enters the blockchain network, performs real-time segmentation of the big data based on the first segmentation strategy and the second segmentation strategy in the segmentation contract; extracts keywords from each data block based on the jieba word segmentation tool and the TF-IDF feature extraction algorithm; and divides the data blocks into high-frequency data blocks and low-frequency data blocks based on the historical query record data of the data query platform in the past week;

[0046] The data segmentation module includes the following steps:

[0047] Step 101: Set up a block contract in the blockchain network to perform block operations on big data, wherein the block contract includes multiple big data processing interfaces, each of which is used to process continuous data from the same data source. Only when all the big data from the data source is processed, the corresponding big data processing interface can reprocess continuous data from other data sources;

[0048] Step 102: When new big data is detected entering the blockchain network, the block contract will allocate an idle big data processing interface and make the data pass through the allocated big data processing interface evenly. When the data passes through the big data processing interface, the data is pre-processed. The pre-processing operation includes data cleaning, including integrity verification, format verification, deduplication processing and missing value processing;

[0049] Step 103, define the first block strategy in the block contract, specifically: in the process of big data passing through the big data processing interface, record the accumulated data volume in real time; the system administrator sets the block data flow threshold, and when it is monitored that the accumulated data volume exceeds the set block data flow threshold, execute the block operation, that is, split the big data at the data passing through the big data processing interface at this time, package the split data that has passed through the big data processing interface into a data block, and store it in the buffer block area; after executing the block operation, clear the accumulated data volume, and repeat the above-mentioned first block strategy; if the remaining data volume is not enough to exceed the block data flow threshold, then package the remaining data volume into a data block;

[0050] According to the time when the data block enters the buffer block area, a pointer to the next data block is generated for each split data block, and a directed acyclic graph is obtained, which represents the original order of the big data;

[0051] Step 104: define a second block strategy in the block contract, specifically: the system administrator divides the content of each data block in the buffer block area based on the content category to obtain a number of intermediate data blocks, and updates the number of pointers and the data blocks they point to; the content category includes data type, data subject, data source, and business process link. In a data block, data of the same data type, the same data subject, the same data source, and the same business process link need to be retained in the same intermediate data block;

[0052] Extract all words in each intermediate data block based on the Jieba word segmentation tool, and filter out stop words from the extracted words based on a predefined stop word list, wherein the stop words include meaningless words such as conjunctions and modal particles and punctuation marks; calculate the importance index of the remaining words in the corresponding intermediate data block based on the TF-IDF feature extraction algorithm, the higher the importance index, the more representative the word is in the intermediate data block; wherein the TF-IDF feature extraction algorithm evaluates the weight of a word by multiplying the statistical word frequency (TF) and the inverse document frequency (IDF); in each intermediate data block, the words with the top three importance indexes are used as keywords of the intermediate data block;

[0053] Obtain the historical query record data of the data query platform in the past week, count the number of accesses to the data corresponding to each intermediate data block, and obtain the total access frequency of each intermediate data block in the past week. If there is a data access corresponding to multiple intermediate data blocks, the number of accesses to these intermediate data blocks is increased by one at the same time; the system administrator sets the frequency judgment standard value, and stores the intermediate data blocks whose total access frequency is not less than the frequency judgment standard value in the high-frequency access buffer area, and stores the intermediate data blocks whose total access frequency is less than the frequency judgment standard value in the low-frequency access buffer area;

[0054] For the intermediate data blocks in the high-frequency access buffer area and the low-frequency access buffer area, merge the intermediate data blocks with the same keywords, the same total access frequency and continuous pointers, and update the pointers, that is, if the two intermediate data blocks before and after the pointer are both in the high (low) frequency access buffer area, and the keywords and total access frequency of the two intermediate data blocks are the same, then merge the two intermediate data blocks;

[0055] The middle data blocks in the low-frequency access buffer area are recorded as low-frequency data blocks, and the middle data blocks in the high-frequency access buffer area are recorded as high-frequency data blocks.

[0056] By real-time monitoring of big data entering the blockchain network, and based on the dual-block strategy, efficient and accurate block processing of big data is carried out; not only the real-time and integrity of the data are ensured, but also through content division and keyword extraction, efficient indexing and classification of data blocks are achieved; combined with historical query records, data blocks are divided into high-frequency and low-frequency, which helps to optimize storage and query efficiency, so that high-frequency data can be accessed faster in the future, and low-frequency data can reduce unnecessary storage overhead, thereby improving the performance and response speed of the entire big data processing system.

[0057] The data storage module performs storage operations on each data block after segmentation based on the storage contract and generates a block identifier for each data block; performs encryption operations on the block identifier of each data block and stores the mapping relationship between the block identifier of each data block and the node location where it is stored in a distributed hash table;

[0058] The data storage module includes the following steps:

[0059] Step 201: Establish a storage contract in the blockchain network. For the low-frequency data blocks and high-frequency data blocks obtained by block division, perform hash operations on these data blocks based on the SHA2-256 algorithm defined in the storage contract to obtain a unique hash value for each data block. The hash value of each data block is generated based on the content of each data block. If the data in the data block changes, the corresponding hash value will also change.

[0060] Step 202: Combine the hash value generated by each data block, the category of each data block, the pointer of each data block to the next data block, and the keyword of each data block to form a string, specifically in the form of "'category of each data block' + 'hash value generated by each data block' + '|' + 'pointer of each data block to the next data block' + '|' + 'first keyword' + 'second keyword' + 'third keyword'", and record this string as a block identifier; wherein the category of each data block indicates the high frequency or low frequency of the data block, the character "1" is used to indicate a high frequency data block, and the character "0" is used to indicate a low frequency data block; the first keyword, the second keyword, and the third keyword respectively indicate the top three words in importance index in each data block; record the block identifiers of all data blocks, and store the recorded results in a new data block, recorded as an identifier data block;

[0061] Step 203, encrypting the block identifier of each data block, specifically: based on the identity of the user in the data query platform, screening out accessible users for each data block, assigning a pair of RSA public keys and RSA private keys to each data block based on the RSA encryption algorithm, putting the RSA public key of each data block into the access tag of the corresponding block identifier, and storing the RSA private key of each data block into the account of the accessible user of the corresponding data block; when accessing the data of the data block, it is necessary to first compare the RSA private key of the user with the RSA public key of the data block to be accessed to see if they match, thereby achieving access control to the data;

[0062] Step 204: The storage contract allocates an area in the blockchain network, and stores all high-frequency data blocks and low-frequency data blocks on different nodes in the area. The mapping relationship between the block identifier of each data block and the node location where it is stored is stored is stored in a distributed hash table. By using the hash value in the block identifier and then querying the corresponding data block node in the distributed hash table, the location where the data block is stored can be obtained.

[0063] The unique hash value of the data block is generated by the SHA2-256 algorithm, and the block identifier is formed by combining the category, pointer and keyword of the data block, ensuring the uniqueness and traceability of the data; encryption operations and control of user access rights enhance data security; the mapping relationship between the block identifier and the data block storage location is stored in a distributed hash table, realizing efficient retrieval and positioning of data; this storage method not only improves the storage efficiency of data, but also ensures the reliability and persistence of data in the blockchain network, providing a solid storage foundation for real-time processing of big data.

[0064] Data reading module: when a user performs a data query operation, the accessible data blocks are filtered out based on the user's private key set, and the query keywords entered by the user are matched with the keywords of the filtered data blocks based on the reading contract. The sending order of the data corresponding to the data blocks is determined according to the number of query keywords and the order of keyword matching, and displayed in the data query platform in the sending order;

[0065] The data reading module includes the following steps:

[0066] Step 301: Establish a read contract in the blockchain network. When a user performs a data query operation in the data query platform, the query keyword input by the user is combined with the user's private key set to form a request string. The user's private key set stores all RSA private keys in the user account. A transmission interface for the read contract is defined in the storage contract. When the request string is formed, the request string is transmitted to the storage contract through the transmission interface.

[0067] Step 302: After receiving the request string passed in by the read contract, obtain the user's private key set, and filter out all the block identifiers of the public keys corresponding to the private keys held by the user from the identifier data block based on the RSA decryption algorithm in the storage contract;

[0068] Step 303: Based on the character comparison function in the storage contract, the received request string and all the block identifiers screened out in step 302 are split and matched, specifically: the query keyword in the request string is split out, and the first keyword, the second keyword, and the third keyword in the block identifier are split out, and the query keyword, the first keyword, the second keyword, and the third keyword all exist in the form of strings;

[0069] If there is only one query keyword, the query keyword is matched with all the first keywords, the second keywords and the third keywords in turn, and the block identifiers corresponding to the matched keywords are put into the reading queue in turn according to the order of query keyword matching (in the order of the first keyword, the second keyword and the third keyword), and the matching means that the two characters are completely equal or the first, the second or the third keyword contains the complete query keyword; for example: when the query keyword matches the second keyword in the block identifier A and the first keyword in the block identifier B, the block identifier B and the block identifier A are put into the reading queue respectively according to the order of the first keyword and the second keyword matching; if the first keywords in the block identifier A and the block identifier B both match the query keyword, the two block identifiers are put into the reading queue in order from large to small according to the total access frequency of the data blocks corresponding to the two block identifiers;

[0070] If there are multiple query keywords, the query keywords are split out one by one, and the split query sub-keywords are matched with the first keyword, the second keyword and the third keyword one by one. If the query sub-keyword matches any one of the first keyword, the second keyword and the third keyword, the corresponding block identifier is put into the read set and the specific keyword matching situation is marked; the corresponding block identifiers are put into the read queue in sequence according to the order of query sub-keyword matching, that is, the block identifier that first matches the keyword with the front order in the block identifier is put into the read queue first; if two block identifiers match the same number of keywords with the query keyword, and the order of these keywords is also the same, the two block identifiers are put into the read queue in sequence from large to small according to the total access frequency of the data blocks corresponding to the two block identifiers;

[0071] For example, when the query subkeyword in the query keyword matches the first keyword in the block identifier C, matches the first keyword and the second keyword in the block identifier D, and matches the second keyword and the third keyword in the block identifier E, the block identifier C and the block identifier D that match the first keyword are first screened out, and because the second keyword in the block identifier D matches the query keyword, the two block identifiers are placed in the read queue in the order of block identifier D and block identifier C, and the first keyword in the block identifier E does not match the query keyword, so the block identifier E is finally placed in the read queue;

[0072] Step 304: Take out the block identifiers in the read queue in turn, compare the hash value in the block identifier with the hash value in the distributed hash table based on the character comparison function in the storage contract to query the node location where the corresponding data block is stored in the blockchain network, and send the data of the corresponding data block to the user of the data query platform in the order in the read queue, and display it in the data query platform in the order of sending.

[0073] Accessible data blocks are filtered out through the user's private key set, and the sending order of data blocks is determined based on the keyword matching algorithm, realizing accurate data query and rapid response; it not only supports the matching of a single query keyword, but also can handle complex query requirements of multiple query keywords, thereby improving the flexibility and accuracy of the query; by displaying the results in the data query platform in the order of sending, users can intuitively obtain the required data, improving user experience and query efficiency, and meeting the requirements for data reading speed and accuracy in real-time processing of big data.

[0074] The data backup module, after completing the storage of data blocks, performs backup operations on all data blocks based on the backup contract, generates the same number of backup data blocks as the original data blocks, and the corresponding block identifiers; when monitoring the user to perform data modification operations, if the data block to be modified is a high-frequency data block, the backup data block is updated in real time; if the data block to be modified is a low-frequency data block, the backup data block is updated in non-real time; the original data is adaptively modified based on the hash value in the block identifier;

[0075] The data backup module includes the following steps:

[0076] Step 401: Establish a backup contract in the blockchain network. After the data blocks are stored, call the backup function in the backup contract to copy all the stored data blocks, store the backup data blocks in a new area, and record the storage node location of each backup data block based on the distributed hash table. The hash values ​​in the block identifiers of these backup data blocks are the same as the hash values ​​in the block identifiers of the corresponding data blocks, but the pointers to the next data blocks are different because the node locations where the data blocks are stored are different, and thus the block identifiers of the backup data blocks are different from the block identifiers of the corresponding data blocks.

[0077] Step 402: When the data in the data block is sent to the user, the backup contract locates the backup node of the data block, starts the monitoring event on the data query platform, and determines whether the backup data block needs to be updated. Specifically, after the user receives the data, he can modify the data on the data query platform. The monitoring event on the data query platform can monitor the user's operation information in real time.

[0078] When the data block operated by the user is a high-frequency data block, the backup data block is updated in real time based on the monitoring event of the data query platform and the data block update function of the backup contract. When the user is detected to have modified the data, such as adding, modifying, deleting data or revoking the operation, a data modification signal is sent to the backup contract; after receiving the signal, the backup contract generates a new data block in real time based on the modified result of the user on the data query platform, and replaces the original backup data block with the newly generated data block through the storage location of the recorded backup data block, and the storage node location does not change;

[0079] When the data block operated by the user is a low-frequency data block, the backup data block is updated in non-real time based on the monitoring event of the data query platform and the data block update function of the backup contract. Only when it is determined that the user has ended access to the data block, a data modification signal is sent to the backup contract, and the backup data block is updated in the same way as above;

[0080] Step 403, when the backup data block is updated, the hash value corresponding to the content in the backup data block will also change, and then the block identifier of the backup data block will also change; when the backup contract detects that the block identifier of the backup data block has changed, the block identifiers of the backup data block before and after the change are sent to the storage contract, the original data is found based on the hash value of the block identifier before the change, and the original data is updated based on the hash value of the block identifier after the change. The original data is the data block stored in the data storage module.

[0081] When users modify data, the module can adopt corresponding update strategies according to the high-frequency or low-frequency characteristics of the data block, thereby reducing unnecessary backup overhead while ensuring data integrity; the original data is adaptively modified through the hash value in the block identifier to achieve seamless data update and synchronization; this backup mechanism not only improves the data recovery capability, but also adopts different update methods based on different data, providing reliable data protection for real-time processing of big data.

[0082] refer to Figure 2 The present invention also provides a method for real-time processing of big data, comprising:

[0083] Step 1: When big data is detected entering the blockchain network, the big data is segmented in real time based on the first segmentation strategy and the second segmentation strategy in the segmentation contract; keywords are extracted from each data block based on the jieba word segmentation tool and the TF-IDF feature extraction algorithm; data blocks are divided into high-frequency data blocks and low-frequency data blocks based on the historical query record data of the data query platform in the past week;

[0084] Step 2: Perform storage operations on each data block after block division based on the storage contract, and generate a block identifier for each data block; perform encryption operations on the block identifier of each data block, and store the mapping relationship between the block identifier of each data block and the node location where it is stored in a distributed hash table;

[0085] Step 3: When the user performs a data query operation, the accessible data blocks are filtered out based on the user's private key set, and the query keywords entered by the user are matched with the keywords of the filtered data blocks based on the reading contract. The sending order of the data corresponding to the data blocks is determined according to the number of query keywords and the order of keyword matching, and displayed in the data query platform in the sending order;

[0086] Step 4: After the storage of the data blocks is completed, all data blocks are backed up based on the backup contract to generate the same number of backup data blocks as the original data blocks, as well as the corresponding block identifiers; when the user is detected to be performing a data modification operation, if the data block to be modified is a high-frequency data block, the backup data block is updated in real time; if the data block to be modified is a low-frequency data block, the backup data block is updated non-real time; the original data is adaptively modified based on the hash value in the block identifier.

[0087] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer storage medium or transmitted via a computer storage medium.

[0088] Computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. Computer storage media can be any available media that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state drives (SSDs)).

[0089] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A big data real-time processing system, characterized by: include: The data block module, when detecting that big data enters the blockchain network, blocks the big data in real time based on the first block strategy and the second block strategy in the block contract; Extract keywords from each data block based on the Jieba word segmentation tool and the TF-IDF feature extraction algorithm; divide the data blocks into high-frequency data blocks and low-frequency data blocks based on the historical query records of the data query platform in the past week; The data storage module performs storage operations on each data block after segmentation based on the storage contract and generates a block identifier for each data block; Perform encryption operation on the block identifier of each data block, and store the mapping relationship between the block identifier of each data block and the node location where it is stored in a distributed hash table; Data reading module: when a user performs a data query operation, the accessible data blocks are filtered out based on the user's private key set, and the query keywords entered by the user are matched with the keywords of the filtered data blocks based on the reading contract. The sending order of the data corresponding to the data blocks is determined according to the number of query keywords and the order of keyword matching, and displayed in the data query platform in the sending order; The data backup module, after completing the storage of data blocks, performs a backup operation on all data blocks based on the backup contract, generates the same number of backup data blocks as the original data blocks, and the corresponding block identifiers; When it is detected that a user is performing a data modification operation, if the data block being modified is a high-frequency data block, the backup data block is updated in real time; If the data block to be modified is a low-frequency data block, the backup data block is updated in non-real time; The original data is adapted based on the hash value in the chunk identifier.

2. A big data real-time processing system as claimed in claim 1, characterized in that: The first block strategy is specifically as follows: in the process of big data passing through the big data processing interface, the accumulated amount of data passing through is recorded in real time; when it is monitored that the accumulated amount of data passing through exceeds the set block data flow threshold, a block operation is performed, that is, the big data is segmented at the data passing through the big data processing interface at this time, and the data that has passed through the big data processing interface after segmentation is packaged into a data block and stored in the buffer block area; After the block operation is executed, the accumulated amount of data is cleared and the first block strategy is repeated; If the remaining data amount is not enough to exceed the block data flow threshold, the remaining data amount is packaged into a data block.

3. A big data real-time processing system as claimed in claim 2, characterized in that: The second block strategy is specifically as follows: divide the content of each data block in the buffer block area based on the content category to obtain a plurality of intermediate data blocks, and update the number of pointers and the data blocks pointed to; the content category includes data type, data subject, data source, and business process link, and in one data block, the continuous data of the same data type, the same data subject, the same data source, and the same business process link are retained in the same intermediate data block; Using the Jieba word segmentation tool, all words in each intermediate data block are extracted, and stop words are filtered out from the extracted words. Stop words include meaningless words and punctuation marks. The importance index of the remaining words in the corresponding intermediate data block is calculated based on the TF-IDF feature extraction algorithm. The top three words in importance index in each intermediate data block are used as keywords of the intermediate data block.

4. A big data real-time processing system as claimed in claim 3, characterized in that: Obtain the historical query record data of the data query platform in the past week, count the number of accesses to the data corresponding to each intermediate data block, and obtain the total access frequency of each intermediate data block in the past week. If there is a data access corresponding to multiple intermediate data blocks, then the number of accesses to these intermediate data blocks is increased by one at the same time; set the frequency judgment standard value, store the intermediate data blocks whose total access frequency is not less than the frequency judgment standard value in the high-frequency access buffer area, and store the intermediate data blocks whose total access frequency is less than the frequency judgment standard value in the low-frequency access buffer area; For the intermediate data blocks in the high-frequency access buffer area and the low-frequency access buffer area, merge the intermediate data blocks with the same keywords, the same total access frequency and continuous pointers, and update the pointers; The middle data blocks in the low-frequency access buffer area are recorded as low-frequency data blocks, and the middle data blocks in the high-frequency access buffer area are recorded as high-frequency data blocks.

5. A big data real-time processing system as claimed in claim 4, characterized in that: The block identifier is obtained by combining a hash value generated for each data block, a category of each data block, a pointer of each data block to the next data block, and a keyword of each data block; Among them, the category of each data block represents the high frequency or low frequency of the data block; the first keyword, the second keyword and the third keyword respectively represent the top three words in importance index in each data block; the block identifiers of all data blocks are recorded, and the recorded results are stored in a new data block, recorded as an identifier data block.

6. A big data real-time processing system as claimed in claim 5, characterized in that: When a user performs a data query operation, the query keyword entered by the user is combined with the user's private key set to form a request string, in which all RSA private keys in the user's account are stored; a transmission interface for reading the contract is defined in the storage contract, and when the request string is formed, the request string is transmitted to the storage contract through the transmission interface; Based on the character comparison function in the storage contract, the received request string and all the filtered block identifiers are split, specifically: the query keyword in the request string is split out, and the first keyword, the second keyword and the third keyword in the block identifier are split out. The query keyword, the first keyword, the second keyword and the third keyword all exist in the form of strings.

7. A big data real-time processing system as claimed in claim 6, characterized in that: If there is only one query keyword, the query keyword is matched with all the first keywords, the second keywords and the third keywords in turn, and the block identifiers corresponding to the matched keywords are put into the read queue in turn according to the order of query keyword matching, and the matching order is in the order of the first keyword, the second keyword, and the third keyword; if the matching order of multiple block identifiers is the same, the two block identifiers are put into the read queue in turn according to the total access frequency of the data blocks corresponding to the multiple block identifiers in descending order; If there are multiple query keywords, the query keywords are split out one by one, and the split query sub-keywords are matched with the first keyword, the second keyword and the third keyword one by one; the corresponding block identifiers are put into the reading queue in sequence according to the order of query sub-keyword matching, that is, the block identifiers that match the keywords with a higher order in the block identifier are put into the reading queue first; if two block identifiers match the same number of keywords with the query keywords, and the order of these keywords is also the same, then according to the total access frequency of the data blocks corresponding to the two block identifiers, the two block identifiers are put into the reading queue in sequence from large to small.

8. A big data real-time processing system as claimed in claim 7, characterized in that: When the storage of the data block is completed, the backup function in the backup contract is called to copy all the stored data blocks. The hash values ​​in the block identifiers of these backup data blocks are the same as the hash values ​​in the block identifiers of the corresponding data blocks. When a user is detected to be performing a data modification operation, the backup data block is updated based on the category of the modified data block; When the data block operated by the user is a high-frequency data block, a new data block is generated in real time based on the modified result of the user on the data query platform. The newly generated data block replaces the original backup data block through the storage location of the recorded backup data block, and the storage node location does not change; When the data block operated by the user is a low-frequency data block, only when it is determined that the user has ended access to the data block, a data modification signal is sent to the backup contract, and the backup data block is updated in the same way as the high-frequency data block.

9. A big data real-time processing system as claimed in claim 8, characterized in that: When the backup data block is updated, the hash value corresponding to the content in the backup data block will also change, and then the block identifier of the backup data block will also change; when the backup contract detects that the block identifier of the backup data block has changed, the block identifiers of the backup data block before and after the change are sent to the storage contract, the original data is found based on the hash value of the block identifier before the change, and the original data is updated based on the hash value of the block identifier after the change. The original data is the data block stored in the data storage module.

10. A method for real-time processing of big data, characterized in that: include: Step 1: When big data is detected entering the blockchain network, the big data is segmented in real time based on the first segmentation strategy and the second segmentation strategy in the segmentation contract; keywords are extracted from each data block based on the jieba word segmentation tool and the TF-IDF feature extraction algorithm; data blocks are divided into high-frequency data blocks and low-frequency data blocks based on the historical query record data of the data query platform in the past week; Step 2: Perform storage operations on each data block after segmentation based on the storage contract, and generate a block identifier for each data block; Perform encryption operation on the block identifier of each data block, and store the mapping relationship between the block identifier of each data block and the node location where it is stored in a distributed hash table; Step 3: When the user performs a data query operation, the accessible data blocks are filtered out based on the user's private key set, and the query keywords entered by the user are matched with the keywords of the filtered data blocks based on the reading contract. The sending order of the data corresponding to the data blocks is determined according to the number of query keywords and the order of keyword matching, and displayed in the data query platform in the sending order; Step 4: After the storage of the data blocks is completed, all data blocks are backed up based on the backup contract to generate the same number of backup data blocks as the original data blocks, as well as the corresponding block identifiers; When it is detected that a user is performing a data modification operation, if the data block being modified is a high-frequency data block, the backup data block is updated in real time; If the data block to be modified is a low-frequency data block, the backup data block is updated in non-real time; The original data is adapted based on the hash value in the chunk identifier.

Images