Data processing method and device, equipment and storage medium
By synchronizing blockchain data in real time to the database and determining full and incremental indexes, matching the target index, the problems of low blockchain data retrieval efficiency and data consistency are solved, and efficient retrieval and consistency synchronization are achieved.
Patent Information
- Application Number
- CN202310563068.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-18
- Publication Date
- 2024-11-22
AI Technical Summary
In the prior art, when querying blockchain data from the storage layer through query chain codes deployed on nodes, the search efficiency is inefficient and cannot meet the search requirements in a high concurrency environment. At the same time, there are data consistency problems.
Synchronize blockchain data to the database in real time, determine the data table and index, match the target index through full index and incremental index, respond to search requests to improve retrieval efficiency and ensure data consistency.
It improves the search efficiency of blockchain data, solves the contradiction between search efficiency and data consistency, reduces the number of database calls and traversal rows, and reduces the time required for retrieval.
Smart Images

Figure CN119003581A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the fields of big data and blockchain technology, and in particular, to a data processing method, device, equipment and storage medium. Background Art
[0002] With the rapid development of the Internet, cooperation between users is becoming more and more frequent. In cooperation, data is mostly stored in centralized databases, which often face trust mechanism problems. Blockchain technology, due to its decentralized and tamper-proof characteristics, has good decentralization and trust-free capabilities. Alliance chain refers to a blockchain managed by multiple institutions or organizations, each of which runs one or more nodes. The alliance chain controls the entry of institutions or organizations that are members of the alliance through identity permissions. The data in the alliance chain only allows institutions or organizations that are members of the alliance to read and send transactions, and jointly record transaction data.
[0003] When users retrieve blockchain data stored in the consortium chain, they can query the storage layer for relevant blockchain data through the query chain code deployed on each node.
[0004] In the process of implementing the present invention, the inventors found that there are at least the following technical problems in the prior art:
[0005] Querying relevant blockchain data from the storage layer through the query chain code deployed on each node involves data retrieval in each block of the blockchain, and the retrieval efficiency is low. Summary of the invention
[0006] The present invention provides a data processing method, device, server and storage medium to improve the retrieval efficiency of blockchain data and solve the contradiction between retrieval efficiency and data consistency.
[0007] In a first aspect, an embodiment of the present invention provides a data processing method, including:
[0008] Synchronize blockchain data to a database in real time, determine the data table corresponding to the blockchain data, and determine the full index based on the data table;
[0009] Obtaining change data of the data table corresponding to the blockchain data, and determining an incremental index according to the change data;
[0010] Determine a target index according to the incremental index and the full index;
[0011] In response to the received search request, a search is performed in the target index based on the search request to obtain a search result.
[0012] In a second aspect, an embodiment of the present invention further provides a data processing device, including:
[0013] A full index determination module is used to synchronize blockchain data to a database in real time, determine a data table corresponding to the blockchain data, and determine a full index based on the data table;
[0014] An incremental index determination module, used to obtain the change data of the data table corresponding to the blockchain data, and determine the incremental index according to the change data;
[0015] A target index determination module, used to determine a target index according to the incremental index and the full index;
[0016] The retrieval module is used to respond to the received retrieval request and perform a search in the target index based on the retrieval request to obtain a retrieval result.
[0017] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:
[0018] one or more processors;
[0019] a storage device for storing one or more programs,
[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of the first aspects.
[0021] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the data processing method as described in any one of the first aspects.
[0022] The embodiments of the above invention have the following advantages or beneficial effects:
[0023] The data processing method provided by the embodiment of the present invention includes: synchronizing blockchain data to a database in real time, determining a data table corresponding to the blockchain data, and determining a full index according to the data table; obtaining change data of the data table corresponding to the blockchain data, and determining an incremental index according to the change data; determining a target index according to the incremental index and the full index; and in response to a received search request, searching in the target index based on the search request to obtain a search result. The above technical solution synchronizes blockchain data to a database in real time, ensures the consistency of the blockchain data stored in the database and the blockchain data stored in the blockchain, determines the data table storing blockchain data in the database, that is, the data table corresponding to the blockchain data, and then determines the index corresponding to the data table through a pre-designed index implementation class, that is, the full index, to achieve the determination of the full index of the data table storing blockchain data in the database. Obtain the change data of the data table corresponding to the blockchain data, and determine the incremental index according to the change data. The change data can be updated in real time with the blockchain data, reflecting the update of the blockchain data, and the incremental index determined according to the change data can also reflect the update of the blockchain data. The target index is determined according to the full index and the incremental index, and the target index can also be updated in real time with the blockchain data, which is more matched with the real-time blockchain data. In response to the received search request, the target row is determined in the target index based on the search conditions corresponding to the search request, and then the target row in the database is traversed according to the search conditions corresponding to the search request to obtain the search result, thereby reducing the number of database calls and the number of rows that the database needs to traverse, shortening the time required for the search, improving the search efficiency, and resolving the contradiction between the search efficiency and the data consistency. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention;
[0025] Figure 2 A flowchart of another data processing method provided by an embodiment of the present invention;
[0026] Figure 3 A flowchart of step 210 in another data processing method provided by an embodiment of the present invention;
[0027] Figure 4 A comparison chart of service response time for retrieving asset information provided by an embodiment of the present invention;
[0028] Figure 5 A flowchart of another data processing method provided by an embodiment of the present invention;
[0029] Figure 6 A schematic diagram of a data processing system provided by an embodiment of the present invention;
[0030] Figure 7 A schematic diagram of another data processing system provided by an embodiment of the present invention;
[0031] Figure 8 A schematic diagram of the structure of a data processing device provided by an embodiment of the present invention;
[0032] Fig. 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0033] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0034] It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe various operations (or steps) as sequential processes, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.
[0035] In the development of blockchain technology, in order to adapt to the needs of different application scenarios, blockchain technology is usually divided into public chain, alliance chain and private chain according to different characteristics. Among them, the public chain refers to a blockchain that can be read by anyone in the world, anyone can send transactions and transactions can be effectively confirmed, and anyone can participate in the consensus process. The public chain is open to any node, and any node can enter without identity verification; the private chain refers to a blockchain with write permissions only in one organization or institution. The private chain has the strongest identity permission restrictions, is mostly used for internal work, the data is not open to the public, and has the highest performance; the alliance chain refers to a blockchain managed by multiple institutions or organizations, and each institution or organization runs one or more nodes. The alliance chain controls the entry of institutions or organizations that are members of the alliance through identity permissions. The data in the alliance chain only allows institutions or organizations that are members of the alliance to read and send transactions, and jointly record transaction data.
[0036] Among them, Hyperledger Fabric has become the representative of alliance chain with its identity verification and access control technology. Hyperledger Fabric has a complete member identity management mechanism. At the same time, due to its decentralized, trustless, tamper-proof, open and transparent characteristics, it can solve the trust, privacy and security issues in the field of alliance cooperation to a certain extent.
[0037] Due to the technical characteristics of blockchain, it is always difficult to strike a balance between data consistency and retrieval efficiency. When searching for data on the blockchain, it is often necessary to traverse the entire chain to determine the search results.
[0038] Hyperledger Fabric can improve retrieval efficiency by storing world state in built-in databases such as CouchDB, but this solution still cannot meet retrieval needs well.
[0039] In addition, the process of querying blockchain data involves the application layer, storage layer, and contract layer in the blockchain protocol. When users retrieve blockchain data stored in the consortium chain, they can query the storage layer for relevant blockchain data through the query chain code deployed on each node. Specifically, the chain code written in the contract layer can be called. The chain code contains multiple read and write functions for blockchain data. Each read and write function consists of two parts: the blockchain system chain code and the judgment logic. The system chain code encapsulates the functions for operating the channel, that is, the system chain code includes the query chain code (QSCC) for querying blockchain data and the verification chain code (VSCC) for verifying the correctness of blockchain data. Querying the storage layer for relevant blockchain data through the query chain code deployed on the node involves data retrieval in each block of the blockchain. The retrieval efficiency is low and cannot meet the retrieval requirements in a high-concurrency environment.
[0040] The low retrieval efficiency and poor data consistency of blockchain data are the problems that blockchain technology needs to solve. Most of the existing solutions improve retrieval efficiency through node verification and third-party databases, but it is difficult to balance retrieval efficiency and data consistency.
[0041] Therefore, this application proposes a data processing method to improve the retrieval efficiency of blockchain data on the premise of solving data consistency, and better solve the contradiction between retrieval efficiency and data consistency.
[0042] The data processing method proposed in this application will be described in detail below with reference to diagrams and embodiments.
[0043] Figure 1The flowchart of a data processing method provided by an embodiment of the present invention is applicable to situations where the contradiction between retrieval efficiency and data consistency needs to be resolved. The method can be executed by a data processing device, which can be implemented by software and / or hardware. Figure 1 The method specifically comprises the following steps:
[0044] Step 110: synchronize the blockchain data to the database in real time, determine the data table corresponding to the blockchain data, and determine the full index based on the data table.
[0045] Specifically, synchronizing the blockchain data stored in the blockchain to the database can be understood as sending the blockchain data to the database. The database stores the blockchain data in at least one data table of the database, and the at least one data table storing the blockchain data is the data table corresponding to the blockchain data.
[0046] In order to ensure the consistency between the blockchain data stored in the database and the blockchain data stored in the blockchain, the blockchain data can also be synchronized to the database through the broadcast information broadcast by the endorsement node. The broadcast information here can be understood as real-time blockchain data and / or update information of real-time blockchain data, and the update information can be at least one of adding information, deleting information, and modifying information. When the broadcast information is real-time blockchain data, the real-time blockchain data is synchronized to the database in real time, which can ensure the consistency of the blockchain data stored in the database and the blockchain data stored in the blockchain; when the broadcast information is update information of real-time blockchain data, the blockchain data stored in the database is synchronized in real time based on the update information, which can also ensure the consistency of the blockchain data stored in the database and the blockchain data stored in the blockchain.
[0047] Then, the full index corresponding to the data table can be determined. Through the pre-designed index implementation class, the data in the static file corresponding to the data table is constructed into the index corresponding to the data table, that is, the full index.
[0048] The database index can be understood as a structure that sorts the values of one or more columns in the database table. It is an identifier added to the data table to improve the database search efficiency. When the database searches according to a search request, the default scanning method is to scan the entire table according to the search conditions corresponding to the search request, and add the data that meets the search conditions to the search results. After adding an index to the data table, when the database searches according to the search request, it can first locate the data row in the index, and then add the data in the data row that meets the search conditions to the search results. When searching for data in a database with an added index, the number of data rows that need to be traversed is significantly reduced, which can improve data retrieval efficiency.
[0049] In an embodiment of the present invention, the blockchain data is synchronized to the database in real time to ensure the consistency of the blockchain data stored in the database and the blockchain data stored in the blockchain, determine the data table storing the blockchain data in the database, that is, the data table corresponding to the blockchain data, and then determine the index corresponding to the data table through a pre-designed index implementation class, that is, the full index, to achieve the full index of the data table storing the blockchain data in the database.
[0050] Step 120: Obtain change data of the data table corresponding to the blockchain data, and determine an incremental index based on the change data.
[0051] The changed data is binary log data, which is a binary log that records changes in the data table structure in the database and modifications to the data stored in the database.
[0052] Specifically, the change data of the database can be obtained through the master-slave synchronization mechanism of the database, that is, the change data of the data table corresponding to the blockchain data can be obtained. After obtaining the change data of the data table corresponding to the blockchain data, the change data can be converted into an incremental index according to a pre-set information processing template. Specifically, the change data can be converted into a change object, and then the change object can be further converted into an index object, which is an incremental index.
[0053] In an embodiment of the present invention, the acquired change data can reflect the update status of the data table storing the blockchain data, and the incremental index determined based on the change data can also reflect the update status of the data table storing the blockchain data. The incremental index provides a data basis for determining a target index with a higher degree of match with the real-time blockchain data.
[0054] Step 130: Determine a target index based on the incremental index and the full index.
[0055] Specifically, in order to determine a target index that better matches the real-time blockchain data, the incremental index determined by the change data can be added to the full index determined by the blockchain data to obtain the target index. Compared with the full index, the target index better matches the real-time blockchain data.
[0056] In practical applications, after determining the target index, the target index can be stored in the internal cache of the server.
[0057] In an embodiment of the present invention, a target index that better matches real-time blockchain data is determined.
[0058] Step 140: In response to the received search request, search the target index based on the search request to obtain a search result.
[0059] Specifically, in response to a received search request, a target row can be determined in a target index based on the search condition corresponding to the search request, and then the target row in the database can be traversed according to the search condition corresponding to the search request to obtain a search result. Since the target index is stored in the internal cache of the server, it is not necessary to call the database when determining the target row in the target index according to the search request, thereby reducing the number of database calls. The database only needs to be called when determining the search result in the target row, thereby reducing the number of rows that the database needs to traverse.
[0060] In the embodiment of the present invention, in response to a received search request, a target row is first determined in the target index according to the search request, and then a search result is determined in the target row. Using the target index to determine the search result reduces the number of database calls and the number of rows that the database needs to traverse, reduces the time required for the search, and improves the search efficiency.
[0061] The data processing method provided by the embodiment of the present invention includes: synchronizing blockchain data to a database in real time, determining a data table corresponding to the blockchain data, and determining a full index according to the data table; obtaining change data of the data table corresponding to the blockchain data, and determining an incremental index according to the change data; determining a target index according to the incremental index and the full index; and in response to a received search request, searching in the target index based on the search request to obtain a search result. The above technical solution synchronizes blockchain data to a database in real time, ensures the consistency of the blockchain data stored in the database and the blockchain data stored in the blockchain, determines the data table storing blockchain data in the database, that is, the data table corresponding to the blockchain data, and then determines the index corresponding to the data table through a pre-designed index implementation class, that is, the full index, to achieve the determination of the full index of the data table storing blockchain data in the database. Obtain the change data of the data table corresponding to the blockchain data, and determine the incremental index according to the change data. The change data can be updated in real time with the blockchain data, reflecting the update of the blockchain data, and the incremental index determined according to the change data can also reflect the update of the blockchain data. The target index is determined according to the full index and the incremental index, and the target index can also be updated in real time with the blockchain data, which is more matched with the real-time blockchain data. In response to the received search request, the target row is determined in the target index based on the search conditions corresponding to the search request, and then the target row in the database is traversed according to the search conditions corresponding to the search request to obtain the search result, thereby reducing the number of database calls and the number of rows that the database needs to traverse, shortening the time required for the search, improving the search efficiency, and resolving the contradiction between the search efficiency and the data consistency.
[0062] Figure 2This is a flowchart of another data processing method provided by an embodiment of the present invention. The embodiment of the present invention can be applied to situations where the contradiction between retrieval efficiency and data consistency needs to be resolved. The embodiment of the present invention is specific based on the above embodiment, and the explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 2 , the data processing method provided by the embodiment of the present invention includes:
[0063] Step 210: Synchronize blockchain data to the database in real time.
[0064] The blockchain data may be data stored in Hyperledger Fabric, and the database may be a MySQL database.
[0065] Specifically, while the server is initialized based on the initialization function, the synchronization function can be triggered to synchronize the blockchain data stored in the Hyperledger Fabric to the MySQL database.
[0066] Network fluctuations may cause the loss of blockchain data synchronized to the MySQL database. Therefore, the embodiment of the present invention can also obtain blockchain data through endorsement nodes. The endorsement nodes can further broadcast the acquired blockchain data, and synchronize the blockchain data to the database again according to the blockchain data broadcast by the endorsement nodes, thereby avoiding the loss of blockchain data and improving the consistency of the blockchain data stored in the database with the blockchain data stored in Hyperledger Fabric.
[0067] Figure 3 This is a flowchart of step 210 in another data processing method provided by an embodiment of the present invention. In one implementation, step 210 may specifically include:
[0068] Step 2110: Obtain the blockchain data stored in the blockchain based on the endorsement node, and synchronize the blockchain data to the database in real time according to the broadcast information broadcast by the endorsement node.
[0069] Among them, the broadcast information can be understood as blockchain data and / or update information of blockchain data, and the update information can be at least one of adding information, deleting information, and modifying information.
[0070] The endorsement node can read the blockchain data stored in Hyperledger Fabric and broadcast the blockchain data as broadcast information. The endorsement node can specifically read the blockchain data stored in Hyperledger Fabric through the chain code. Then, the blockchain data is synchronized to the database in real time according to the broadcast information broadcast by the endorsement node.
[0071] When updating the blockchain data stored in Hyperledger Fabric, it is necessary to submit a transaction by calling the chain code. Specifically, the update request can be sent to the endorsement node, and the endorsement node calls the relevant chain code to endorse the transaction. After the endorsement is completed, the transaction is submitted to the sorting node. After the sorting node verifies the transaction, it can update the blockchain data stored in the blockchain according to the update information contained in the update request. The endorsement node can also send the update information to other endorsement nodes, and all endorsement nodes can broadcast the update information as broadcast information. Then, the blockchain data is synchronized in real time according to the broadcast information broadcast by the endorsement node, so as to synchronize the real-time blockchain data to the database.
[0072] In practical applications, after the blockchain data stored in the blockchain is synchronized to the database based on a common data synchronization method, the blockchain data can be synchronized in real time according to the update information contained in the update request received by the endorsement node.
[0073] The blockchain data is synchronized in real time through the common data synchronization method and the update information contained in the update request received by the endorsement node, ensuring the consistency of the blockchain data stored in the MySQL database and the blockchain data stored in the Hyperledger Fabric.
[0074] Step 2120: Determine the theoretical height of the blockchain data synchronized to the database in real time based on the broadcast information broadcast by the endorsement node.
[0075] Specifically, when the broadcast information is blockchain data, the broadcast information may include the blockchain data, and therefore, the theoretical height of the blockchain data may be determined based on the broadcast information. When the broadcast information is update information of the blockchain data and the update information is increase information, the broadcast information may include the amount of increase in the blockchain data, and therefore, the increase in the height of the blockchain data may be determined based on the broadcast information, and further, the theoretical height of the blockchain data may be determined based on the existing height and the increase in the height of the blockchain data.
[0076] It is possible to determine the theoretical height of blockchain data, which can reflect the theoretical data volume of blockchain data.
[0077] Step 2130: When it is determined that the actual height of the blockchain data synchronized to the database in real time is inconsistent with the theoretical height, it is determined that data loss exists in the blockchain data synchronized to the database in real time.
[0078] Specifically, after determining the actual height of the blockchain data, the actual height of the blockchain data is compared with the theoretical height. When it is determined that the actual height of the blockchain data is inconsistent with the theoretical height, it can be determined that there is data loss in the blockchain data synchronized to the database in real time. When it is determined that the actual height of the blockchain data is consistent with the theoretical height, it can be determined that there is no data loss in the blockchain data synchronized to the database in real time. At this time, the blockchain data synchronized to the database in real time is consistent with the blockchain data stored in the blockchain.
[0079] According to the actual height and theoretical height of the blockchain data synchronized to the database in real time, it is possible to determine whether there is data loss in the blockchain data synchronized to the database in real time.
[0080] Step 2140: Return to execute the process of synchronizing the blockchain data to the database in real time according to the broadcast information broadcast by the endorsement node, until it is determined that there is no data loss in the blockchain data synchronized to the database in real time.
[0081] Specifically, when it is determined that there is data loss in the blockchain data synchronized to the database in real time, the endorsement node can rebroadcast the information, and then continue to synchronize the blockchain data based on the broadcast information until there is no data loss in the blockchain data synchronized to the database in real time, further ensuring the consistency of the blockchain data synchronized to the database in real time and the blockchain data stored in the blockchain.
[0082] Furthermore, in the process of synchronizing the blockchain data to the database in real time according to the broadcast information broadcast by the endorsement node, it also includes: storing the broadcast information in a message queue.
[0083] Specifically, the broadcast information broadcasted by the endorsement node can be stored in the message queue to ensure that the blockchain data and / or the updated data of the blockchain data will not be lost. At the same time, the blockchain data and / or the blockchain data can also be transmitted in order through the confirmation mechanism.
[0084] In an embodiment of the present invention, the blockchain data stored in the blockchain is synchronized to the database in real time, and the consistency of the blockchain data stored in the database and the blockchain data stored in the blockchain is ensured.
[0085] Step 220: Determine a data table corresponding to the blockchain data, and determine a full index based on the data table.
[0086] In one implementation, determining the full index according to the data table includes:
[0087] The preset index implementation class is called to process the static file corresponding to the data table to obtain the full index.
[0088] Specifically, the database can store blockchain data in at least one data table of the database. For example, the blockchain data can be stored in at least one data table of the database according to the blocks stored in the blockchain data. Then, the at least one data table storing the blockchain data can be determined as the data table corresponding to the blockchain data. A static file corresponding to the data table is generated according to the data table corresponding to the blockchain data. Through a pre-designed index implementation class, the data in the static file corresponding to the data table is constructed as an index corresponding to the data table, that is, a full index.
[0089] In an embodiment of the present invention, a data table storing blockchain data in a database, that is, a data table corresponding to blockchain data, is determined, a static file corresponding to the data table is generated according to the data table corresponding to blockchain data, and the data in the static file corresponding to the data table is determined to be constructed as an index corresponding to the data table, that is, a full index, through a pre-designed index implementation class, to achieve the full index of the data table storing blockchain data in the database. The index corresponding to the data table can improve the efficiency of data retrieval on the data table, therefore, the full index of the data table synchronized to the database is determined, which improves the data retrieval efficiency of the blockchain data stored in the data table.
[0090] Step 230: Obtain change data of the data table corresponding to the blockchain data, and determine an incremental index based on the change data.
[0091] In one implementation, step 230 may specifically include:
[0092] The change data of the data table corresponding to the blockchain data is obtained from the database through the master-slave synchronization mechanism of the database; and the change data is converted into the incremental index according to a preset information processing template.
[0093] Specifically, the server has a built-in monitoring module (mysql-binlog-connector-java), which can disguise the microservice instance as a slave database of the MySQL database. When the server is initialized, the monitoring component can be started synchronously, and the monitoring module can disguise the module used to build the target index in the server as a slave database of the MySQL database. Then, the change data of the MySQL database can be obtained through the master-slave synchronization mechanism of the MySQL database, that is, the change data of the data table corresponding to the blockchain data can be obtained. After obtaining the change data of the data table corresponding to the blockchain data, the change data can be converted into an incremental index according to the pre-set information processing template.
[0094] Further, converting the change data into the incremental index includes:
[0095] After the change data is converted into a change object, an incremental index is determined according to the change object.
[0096] Specifically, after obtaining the change data, the change data may be filtered to filter out the interference data, and then the filtered change data may be converted into a change object, and then the change object may be converted into an incremental index.
[0097] In an embodiment of the present invention, change data of a data table corresponding to blockchain data is obtained from a database through a master-slave synchronization mechanism of the database. The change data can reflect the update status of the data table storing blockchain data. The incremental index determined based on the change data can also reflect the update status of the data table storing blockchain data. The incremental index provides a data basis for determining a target index with a higher degree of match with the blockchain data.
[0098] Step 240: Determine a target index based on the incremental index and the full index.
[0099] In one implementation, step 240 may specifically include:
[0100] The incremental index is added to the full index to obtain the target index.
[0101] Specifically, as time goes by, the blockchain data stored in the blockchain may change, and the blockchain data synchronized to the database may also change. In order to ensure data consistency and improve retrieval efficiency, it is necessary to determine a target index that better matches the blockchain data. Specifically, after determining the full index based on the blockchain data and the incremental index based on the changed data, the incremental index can be added to the full index to obtain the target index.
[0102] Likewise, after determining the target index, the target index may be stored in an internal cache of the server.
[0103] In an embodiment of the present invention, a target index that better matches real-time blockchain data is determined.
[0104] Step 250: In response to the received search request, search the target index based on the search request to obtain a search result.
[0105] Specifically, after receiving the search request, the server can determine the target row in the target index in the internal cache of the server based on the search conditions corresponding to the search request in response to the received search request, and then traverse the target row in the MySQL database according to the search conditions corresponding to the search request to obtain the search result. The target index is stored in the internal cache of the server. When determining the target row in the target index according to the search request, there is no need to call the database, which reduces the number of database calls, alleviates the pressure on the MySQL database, and improves the stability of the server. The database needs to be called only when the search result is determined in the target row, which reduces the number of rows that the database needs to traverse, reduces the time required for a single search, and improves the search efficiency.
[0106] It should be noted that when multiple servers need to retrieve blockchain data at the same time, the target index corresponding to each server can be constructed. After receiving the search request, each server responds to the received search request, determines the corresponding target index in its internal cache, determines the target row in the target index based on the search condition corresponding to the search request, and traverses the target row in the database according to the search condition corresponding to the search request to obtain the search result. Through the multi-instantiation mechanism of microservices, high-concurrency search requests are diverted and executed concurrently in different servers, which greatly improves the query efficiency and meets the search requirements in a high-concurrency environment.
[0107] Figure 4 A comparison chart of the service response time for retrieving asset information provided by an embodiment of the present invention, such as Figure 4 As shown, the horizontal axis represents the number of retrievals, and the vertical axis represents the total service response time. The upper half of the broken line shows the change of the total service response time when the asset information is directly retrieved from the blockchain data as the number of retrievals increases. The lower half shows the change of the total service response time when the asset information is retrieved from the blockchain data based on the data processing method proposed in the embodiment of the present invention as the number of retrievals increases. It can be clearly seen that the retrieval efficiency is improved by 1900% when the asset information is retrieved from the blockchain data using the data processing method proposed in the embodiment of the present invention.
[0108] In the embodiment of the present invention, in response to a received search request, a target row is first determined in the target index according to the search request, and then a search result is determined in the target row. Using the target index to determine the search result reduces the number of database calls and the number of rows that the database needs to traverse, reduces the time required for the search, and improves the search efficiency.
[0109] The data processing method provided by the embodiment of the present invention includes: synchronizing blockchain data to a database in real time; determining a data table corresponding to the blockchain data, and determining a full index based on the data table; obtaining change data of the data table corresponding to the blockchain data, and determining an incremental index based on the change data; determining a target index based on the incremental index and the full index; and in response to a received retrieval request, searching in the target index based on the retrieval request to obtain a retrieval result. The above technical solution synchronizes blockchain data in real time through ordinary data synchronization methods and update information contained in update requests received by endorsement nodes, determines the actual height and theoretical height of blockchain data, and when it is determined that there is data loss in blockchain data synchronized to the database in real time through inconsistency between the actual height and the theoretical height, continues to synchronize blockchain data in real time according to the update information until it is determined that there is no data loss in blockchain data synchronized to the database in real time, and synchronizes blockchain data stored in the blockchain to the database in real time, and ensures the consistency of blockchain data stored in the database and blockchain data stored in the blockchain, determines the data table storing blockchain data in the database, that is, the data table corresponding to blockchain data, generates static files corresponding to the data table according to the data table corresponding to blockchain data, determines the data in the static file corresponding to the data table to be constructed as the index corresponding to the data table, that is, the full index, through the pre-designed index implementation class, and realizes the determination of the full index of the data table storing blockchain data in the database. The change data of the data table corresponding to the blockchain data is obtained from the database through the master-slave synchronization mechanism of the database, and the change data can be updated in real time with the blockchain data to reflect the update of the blockchain data. The incremental index determined according to the change data can also reflect the update of the blockchain data. The target index can be obtained by adding the incremental index to the full index. The target index can also be updated in real time with the blockchain data to better match the real-time blockchain data. In response to the received search request, the target row is determined in the target index based on the search conditions corresponding to the search request, and then the target row in the database is traversed according to the search conditions corresponding to the search request to obtain the search result, which reduces the number of database calls and the number of rows that the database needs to traverse, reduces the time required for search, improves search efficiency, and solves the contradiction between search efficiency and data consistency.
[0110] Figure 5 The following is a flowchart of another data processing method provided by an embodiment of the present invention, which exemplarily provides one implementation method. Figure 6 A schematic diagram of a data processing system provided by an embodiment of the present invention is shown in FIG. Figure 6As shown, the data processing system includes: a data processing module, a MySQL database, an index building module, a data synchronization module and a consortium chain network. Among them, the data processing module has data retrieval requirements, the database stores blockchain data based on at least one data table, the index building module is used to build a target index based on the blockchain data, and the data synchronization module synchronizes the blockchain data stored in the blockchain to the database through the consortium chain network. Figure 7 A schematic diagram of another data processing system provided by an embodiment of the present invention, such as Figure 7 As shown, the data processing system includes: an application layer service instance, a third-party data storage center (MySQL database), a query service data synchronization module, a blockchain data synchronization module and a Hyperledger Fabric network. Among them, service instances are deployed in the application layer service instance, for example, "query service instance 1" and other service instances are deployed, and "query service instance 1" has data retrieval requirements; the third-party data storage center stores blockchain data based on at least one data table; the query service data synchronization module is used to build a target index based on the blockchain data stored in the third-party data storage center. Specifically, the query service data synchronization module can form a static data file based on the blockchain data stored in the MySQL database, and build a full index based on the static data file. The query service data synchronization module can also disguise the query service data synchronization module as a slave server of the MySQL database based on the second monitoring component to monitor the change data (binlog data) of the MySQL database, and process the filtered change data based on the information processing template to obtain an incremental index. Then, the query service data synchronization module determines the target index based on the full index and the incremental index; the Hyperledger Fabric network includes endorsement nodes (Peer nodes), and the blockchain data synchronization module is connected to the Hyperledger The endorsement node in the Fabric network synchronizes the blockchain data stored in the blockchain to the database. The first listening component (Peer-Listener) in the blockchain data synchronization module can listen to the broadcast information of the endorsement node, process the broadcast information based on the information processing template, and update the blockchain data according to the processed broadcast information. At the same time, the broadcast information can also be stored in the message queue to serialize the broadcast information while ensuring that the blockchain data and / or the updated data of the blockchain data will not be lost.
[0111] Figure 5 The data processing method shown can be implemented by Figure 6 or Figure 7 The data processing system shown is implemented as Figure 5 As shown, the implementation of the data processing method is based on Figure 6 The data processing system shown, when implemented, includes:
[0112] Step 510: Synchronize the blockchain data to the MySQL database in real time through the alliance chain network and data synchronization module.
[0113] In one embodiment, in the process of synchronizing blockchain data to a database through a consortium chain network and a data synchronization module, the process further includes:
[0114] When an update request for blockchain data is received, the endorsement node in the alliance chain network broadcasts the update information contained in the update request as broadcast information, and the first listening component in the data synchronization module receives the broadcast information broadcast by the endorsement node, and updates the blockchain data according to the broadcast information, so as to synchronize the blockchain data stored in the blockchain to the MySQL database in real time, thereby ensuring the consistency of the blockchain data stored in the MySQL database with the blockchain data stored in the blockchain.
[0115] The data synchronization module can also determine the theoretical height of the blockchain data synchronized to the MySQL database based on the broadcast information, and after determining the actual height of the blockchain data synchronized to the MySQL database, determine whether the actual height is consistent with the theoretical height. If the actual height is inconsistent with the theoretical height, it is determined that there is data loss in the blockchain data synchronized to the MySQL database in real time. At this time, the alliance chain network can broadcast update information again based on the endorsement node until there is no data loss in the blockchain data synchronized to the MySQL database in real time.
[0116] Step 520: Determine a data table of blockchain data stored in a database through an index construction module.
[0117] Step 530: Generate a static file corresponding to the data table through the index construction module, and construct a full index based on the data in the static file.
[0118] Step 540: The index construction module is disguised as a slave database of the MySQL database through the second listening component in the index construction module, and the change data of the data table corresponding to the blockchain data is synchronized to the index construction module through the master-slave synchronization mechanism of the MySQL database.
[0119] Step 550: Call a preset information processing template through the index building module to convert the change data into a change object, and determine an incremental index based on the change object.
[0120] Step 560: Add the incremental index to the full index through the index building module to obtain the target index.
[0121] Step 570: In response to the received search request, the data processing module determines the target row in the target index, and traverses the target row in the database according to the search condition corresponding to the search request to obtain the search result.
[0122] The implementation of the data processing method provided by the embodiment of the present invention has the corresponding functional modules and beneficial effects of executing the data processing method provided by any embodiment of the present invention.
[0123] Figure 8 The present invention provides a schematic diagram of the structure of a data processing device. The device and the data processing methods of the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the data processing device, reference can be made to the embodiments of the above data processing methods.
[0124] The specific structure of the data processing device is as follows: Figure 8 As shown, including:
[0125] A full index determination module 810 is used to synchronize blockchain data to a database in real time, determine a data table corresponding to the blockchain data, and determine a full index based on the data table;
[0126] An incremental index determination module 820 is used to obtain the change data of the data table corresponding to the blockchain data, and determine the incremental index according to the change data;
[0127] A target index determination module 830 is used to determine a target index according to the incremental index and the full index;
[0128] The retrieval module 840 is used to respond to the received retrieval request and perform a search in the target index based on the retrieval request to obtain a retrieval result.
[0129] Based on the above embodiment, the full index determination module 810 is specifically used to:
[0130] The blockchain data stored in the blockchain is obtained based on the endorsement node, and the blockchain data is synchronized to the database in real time according to the broadcast information broadcast by the endorsement node; the theoretical height of the blockchain data synchronized to the database in real time is determined according to the broadcast information broadcast by the endorsement node; when it is determined that the actual height of the blockchain data synchronized to the database in real time is inconsistent with the theoretical height, it is determined that there is data loss in the blockchain data synchronized to the database in real time; return to execute the synchronization of the blockchain data to the database in real time according to the broadcast information broadcast by the endorsement node until it is determined that there is no data loss in the blockchain data synchronized to the database in real time; determine the data table corresponding to the blockchain data, call the preset index implementation class to process the static file corresponding to the data table, and obtain the full index.
[0131] Based on the above embodiment, the incremental index determination module 820 is specifically configured to:
[0132] The change data of the data table corresponding to the blockchain data is obtained from the database through the master-slave synchronization mechanism of the database; and the change data is converted into the incremental index according to a preset information processing template.
[0133] In one implementation, converting the change data into the incremental index includes:
[0134] After the change data is converted into a change object, an incremental index is determined according to the change object.
[0135] Based on the above embodiment, the target index determination module 830 is specifically used for:
[0136] The incremental index is added to the full index to obtain the target index.
[0137] The data processing device provided in the embodiment of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the data processing method.
[0138] It is worth noting that in the embodiment of the above-mentioned data processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0139] Fig. 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Fig. 9 A block diagram of an exemplary computer device 9 suitable for use in implementing embodiments of the present invention is shown. Fig. 9 The computer device 9 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0140] like Fig. 9 As shown, the computer device 9 is in the form of a general-purpose computing electronic device. The components of the computer device 9 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).
[0141] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.
[0142] The computer device 9 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 9, including volatile and non-volatile media, removable and non-removable media.
[0143] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 9 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Fig. 9 not shown, usually called a "hard drive"). Although Fig. 9 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.
[0144] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0145] The computer device 9 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the computer device 9, and / or any device that enables the computer device 9 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the computer device 9 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 20. Fig. 9 As shown, the network adapter 20 communicates with other modules of the computer device 9 via the bus 18. It should be understood that although Fig. 9 Not shown, other hardware and / or software modules may be used in conjunction with computer device 9, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0146] The processing unit 16 executes various functional applications and page displays by running the programs stored in the system memory 28, for example, implementing the data processing method provided by the embodiment of the present invention, which includes:
[0147] Synchronize blockchain data to a database in real time, determine the data table corresponding to the blockchain data, and determine the full index based on the data table;
[0148] Obtaining change data of the data table corresponding to the blockchain data, and determining an incremental index according to the change data;
[0149] Determine a target index according to the incremental index and the full index;
[0150] In response to the received search request, a search is performed in the target index based on the search request to obtain a search result.
[0151] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data processing method provided by any embodiment of the present invention.
[0152] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, for example, a data processing method provided by an embodiment of the present invention is implemented. The method includes:
[0153] Synchronize blockchain data to a database in real time, determine the data table corresponding to the blockchain data, and determine the full index based on the data table;
[0154] Obtaining change data of the data table corresponding to the blockchain data, and determining an incremental index according to the change data;
[0155] Determine a target index according to the incremental index and the full index;
[0156] In response to the received search request, a search is performed in the target index based on the search request to obtain a search result.
[0157] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0158] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0159] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0160] Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0161] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0162] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A data processing method, characterized in that: include: Synchronize blockchain data to a database in real time, determine the data table corresponding to the blockchain data, and determine the full index based on the data table; Obtaining change data of the data table corresponding to the blockchain data, and determining an incremental index according to the change data; Determine a target index according to the incremental index and the full index; In response to the received search request, a search is performed in the target index based on the search request to obtain a search result.
2. The data processing method according to claim 1, characterized in that: Determine the full index according to the data table, including: The preset index implementation class is called to process the static file corresponding to the data table to obtain the full index.
3. The data processing method according to claim 1, characterized in that: Obtaining change data of the data table corresponding to the blockchain data, and determining an incremental index according to the change data, including: Obtaining the change data of the data table corresponding to the blockchain data from the database through the master-slave synchronization mechanism of the database; The changed data is converted into the incremental index according to a preset information processing template.
4. The data processing method according to claim 3, characterized in that: Converting the change data into the incremental index includes: After the change data is converted into a change object, an incremental index is determined according to the change object.
5. The data processing method according to claim 1, characterized in that: Determining a target index according to the incremental index and the full index includes: The incremental index is added to the full index to obtain the target index.
6. The data processing method according to claim 1, characterized in that: Synchronize blockchain data to the database in real time, including: Acquire the blockchain data stored in the blockchain based on the endorsement node, and synchronize the blockchain data to the database in real time according to the broadcast information broadcast by the endorsement node; When it is determined that there is data loss in the blockchain data synchronized to the database in real time, return to execute synchronization of the blockchain data to the database according to the broadcast information broadcast by the endorsement node until it is determined that there is no data loss in the blockchain data synchronized to the database.
7. The data processing method according to claim 6, characterized in that: After synchronizing the blockchain data to the database in real time according to the broadcast information broadcast by the endorsement node, it also includes: Determine the theoretical height of the blockchain data synchronized to the database in real time according to the broadcast information broadcast by the endorsement node; When it is determined that the actual height and theoretical height of the blockchain data synchronized to the database in real time are inconsistent, it is determined that data loss exists in the blockchain data synchronized to the database in real time.
8. A data processing device, characterized in that: include: A full index determination module is used to determine the data table corresponding to the blockchain data after synchronizing the blockchain data to the database in real time, and determine the full index according to the data table; An incremental index determination module, used to obtain the change data of the data table corresponding to the blockchain data, and determine the incremental index according to the change data; A target index determination module, used to determine a target index according to the incremental index and the full index; The retrieval module is used to respond to the received retrieval request and perform a search in the target index based on the retrieval request to obtain a retrieval result.
9. A computer device, characterized in that: The computer device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any one of claims 1 to 7.
10. A storage medium containing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the data processing method according to any one of claims 1 to 7.
Citation Information
Patent Citations
A block chain data-based offline synchronization method and device
CN109558411A
Database log synchronization method and device, computer equipment and readable storage medium
CN111008246A
Data synchronization method and device based on blockchain and computer readable storage medium
CN111475575A
Credit data offline synchronization method, device and storage method based on block chain
CN112905611A
Novel on-chain data high-speed retrieval model
CN113961598A
Cited By
Block chain-based area transmission system and method
CN119583564A
