Data synchronization method and device, electronic device and computer readable medium

Abstract

The present invention discloses a data synchronization method and device, an electronic device and a computer-readable medium, and relates to the field of computer technology. The method comprises: using a data capture component to capture the transaction data of a first system, and pushing the transaction data of the first system to a message middleware cluster of the first system; performing hash processing on the transaction data of the first system to obtain the on-chain evidence data of the transaction data of the first system, and storing the on-chain evidence data of the transaction data of the first system in a blockchain network, so that the second system can obtain the on-chain evidence data of the transaction data of the first system from the blockchain network; after receiving the data synchronization request of the second system, synchronizing the transaction data of the first system to the second system through the message middleware cluster of the first system. This implementation method can actively and quasi-real-timely acquire transaction data within an organization, avoid data loss, ensure that the captured data cannot be tampered with and is traceable, and provide a reliable data source for data analysis.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data synchronization method and device, an electronic device, and a computer-readable medium. Background Art

[0002] In the financial technology industry, regulatory probes, external perception, information sharing and reporting are used to collect and analyze financial innovation risks in real time, detect and comprehensively evaluate potential risks in advance, and create a safer, more convenient and transparent financial system. Regulatory probes are a new type of financial regulatory tool. The core idea is to analyze target messages in network traffic to capture transaction data within institutions and enhance the automation, intelligence and accuracy of supervision. The existing regulatory probe solution will lose TCP (transport layer protocol) data packets when the network is unreliable, and the bypass capture of network traffic cannot guarantee that the retransmission of data packets can be fully captured by the probe, so there is a certain probability of data loss.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] There may be cases of data loss, and there is no guarantee that the captured data will not be tampered with, and traceability cannot be guaranteed. The availability, disaster tolerance, and scalability of the probe are all poor. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data synchronization method and device, which can actively and quasi-real-time acquire transaction data within an institution, avoid data loss, ensure that the captured data cannot be tampered with and is traceable, and provide a reliable data source for regulatory agencies to conduct data analysis. The probe has high availability, disaster recovery and strong expansion capabilities.

[0006] To achieve the above objective, according to one aspect of an embodiment of the present invention, a data synchronization method is provided.

[0007] A data synchronization method comprises: using a data capture component to capture transaction data of a first system, and pushing the transaction data of the first system to a message middleware cluster of the first system; performing hash processing on the transaction data of the first system to obtain on-chain evidence data of the transaction data of the first system, and storing the on-chain evidence data of the transaction data of the first system in a blockchain network, so that a second system can obtain the on-chain evidence data of the transaction data of the first system from the blockchain network; after receiving a data synchronization request issued by the second system according to the on-chain evidence data of the transaction data of the first system, synchronizing the transaction data of the first system to the second system through the message middleware cluster of the first system.

[0008] Optionally, the using the data capture component to capture the transaction data of the first system includes: using a probe program deployed at a gateway application layer of the first system to capture the transaction data of the first system.

[0009] Optionally, the transaction data of the first system is captured by using a probe program deployed at the gateway application layer of the first system, including: copying traffic from the network card of the application of the first system to a physical machine, a virtual machine or a container, the copied traffic including the transaction data of the first system, and the application is used to execute transactions with the first system to obtain the transaction data of the first system.

[0010] Optionally, the use of the data capture component to capture the transaction data of the first system includes: using a probe program deployed in at least two probe servers of the first system to capture the transaction data of the first system, the at least two probe servers being arranged in the same Internet data center of the first system and using the same probe program configuration.

[0011] Optionally, the use of the data capture component to capture the transaction data of the first system includes: using a probe program in a physical machine, virtual machine or container where an application of the first system is located to capture the transaction data of the first system, and the application is used to execute transactions with the first system to obtain the transaction data of the first system.

[0012] Optionally, when the probe program is deployed in a docker container (a product implementation of container technology), an image file of the probe program is run in the container to capture transaction data of the first system, and the image file includes a basic image made based on busybox (a software toolbox that integrates Linux commands and tools) of the docker container.

[0013] Optionally, the data capture component is used to capture the transaction data of the first system, and the transaction data of the first system is pushed to the message middleware cluster of the first system, including: monitoring the message data of the first system during transactions through the probe program, and extracting target message data of the specified address and protocol from the message data, parsing the target message data into transaction data in CSV (comma separated value) format, and pushing the transaction data in CSV format to the message middleware cluster of the first system.

[0014] Optionally, parsing the target message data into transaction data in CSV format includes: storing the extracted target message data in a shared memory, reading the target message data from the shared memory and performing message reorganization to obtain a combined TCP data stream, parsing the TCP data stream according to the HTTP protocol to obtain JSON format data, extracting transaction data of a preset business from the JSON format data, and rearranging the transaction data of the preset business into the CSV format to obtain transaction data in the CSV format.

[0015] Optionally, the extracted target message data is stored in a plurality of shared memories, and the message reassembly, data parsing and arrangement operations are performed in a multi-process manner.

[0016] Optionally, pushing the transaction data of the first system to the message middleware cluster of the first system includes: dividing the transaction data of the first system into batches, and pushing the obtained transaction data of the batches and corresponding batch information to the message middleware cluster of the first system.

[0017] Optionally, dividing the transaction data of the first system into batches, and pushing the obtained batches of transaction data and corresponding batch information to the message middleware cluster of the first system includes: sending the transaction data of the first system to the uplink service of the first system one by one or in batches, and the uplink service determining the batch information of the transaction data of the first system, and saving the transaction data and corresponding batch information of the first system to a cache, and after the number of transaction data in the cache reaches a first number, pushing the first number of transaction data and the corresponding batch information as a record to the message middleware cluster of the first system.

[0018] Optionally, hash processing is performed on the transaction data of the first system to obtain on-chain evidence data of the transaction data of the first system, and the on-chain evidence data of the transaction data of the first system is stored in the blockchain network, including: hash processing is performed on the obtained batch of transaction data to obtain a hash value of the obtained batch of transaction data; the hash value of the obtained batch of transaction data and the corresponding batch information are stored in the blockchain network, so that the second system can issue the data synchronization request according to the hash value and the corresponding batch information.

[0019] Optionally, the first system includes multiple message middleware clusters; pushing the transaction data of the first system to the message middleware cluster of the first system includes: pushing the transaction data of the first system to each currently existing message middleware cluster of the first system.

[0020] According to another aspect of an embodiment of the present invention, a data synchronization method is provided.

[0021] A data synchronization method comprises: obtaining on-chain evidence data of transaction data of a first system from a blockchain network, wherein the on-chain evidence data is obtained by the first system performing hash processing on the captured transaction data; determining the transaction data of the first system to be synchronized to a second system according to the on-chain evidence data of the transaction data of the first system; and synchronizing the transaction data of the first system to be synchronized to the second system through a message middleware cluster of the second system.

[0022] Optionally, synchronizing the transaction data of the first system to be synchronized to the second system through the message middleware cluster of the second system includes: distributing the transaction data of the first system to be synchronized to multiple data acquisition components for parsing through the message middleware cluster of the second system, and storing various types of data obtained from the parsing into corresponding storage locations in the second system.

[0023] According to yet another aspect of an embodiment of the present invention, a data synchronization device is provided.

[0024] A data synchronization device comprises: a transaction data capture module, which is used to capture the transaction data of a first system by using a data capture component, and push the transaction data of the first system to a message middleware cluster of the first system; an on-chain evidence module, which is used to perform hash processing on the transaction data of the first system to obtain the on-chain evidence data of the transaction data of the first system, and store the on-chain evidence data of the transaction data of the first system in a blockchain network, so that a second system can obtain the on-chain evidence data of the transaction data of the first system from the blockchain network; and a transaction data synchronization module, which is used to synchronize the transaction data of the first system to the second system through the message middleware cluster of the first system after receiving a data synchronization request issued by the second system based on the on-chain evidence data of the transaction data of the first system.

[0025] Optionally, the transaction data capture module is further used to: capture the transaction data of the first system using a probe program deployed at the gateway application layer of the first system.

[0026] Optionally, the transaction data capture module is also used to: copy traffic from the network card of the application of the first system to a physical machine, a virtual machine or a container, the copied traffic includes the transaction data of the first system, and the application is used to execute transactions with the first system to obtain the transaction data of the first system.

[0027] Optionally, the transaction data capture module is also used to: capture the transaction data of the first system using a probe program deployed in at least two probe servers of the first system, wherein the at least two probe servers are arranged in the same Internet data center of the first system and use the same probe program configuration.

[0028] Optionally, the transaction data capture module is also used to: use a probe program in the physical machine, virtual machine or container where the application of the first system is located to capture the transaction data of the first system, and the application is used to execute transactions in the first system to obtain the transaction data of the first system.

[0029] Optionally, the transaction data capture module is also used to: when the probe program is deployed in a docker container (a product implementation of container technology), run the image file of the probe program in the container to capture the transaction data of the first system, and the image file includes a basic image made based on busybox of the docker container.

[0030] Optionally, the transaction data capture module is also used to: monitor the message data of the first system during transactions through the probe program, and extract target message data of the specified address and protocol from the message data, parse the target message data into transaction data in CSV format, and push the transaction data in CSV format to the message middleware cluster of the first system.

[0031] Optionally, the transaction data capture module includes a parsing submodule, which is used to: store the extracted target message data into a shared memory, read the target message data from the shared memory and reassemble the message to obtain a combined TCP (transport layer protocol) data stream, parse the TCP data stream according to the HTTP protocol to obtain JSON format data, extract transaction data of a preset business from the JSON (a data representation format, JavaScript Object Notation) format data, and rearrange the transaction data of the preset business into the CSV format to obtain transaction data in the CSV format.

[0032] Optionally, the extracted target message data is stored in a plurality of shared memories, and the parsing submodule is further used to: perform the message reassembly, the data parsing and the arrangement operations in a multi-process manner.

[0033] Optionally, the transaction data capture module includes a push submodule, which is used to: divide the transaction data of the first system into batches, and push the obtained batch transaction data and corresponding batch information to the message middleware cluster of the first system.

[0034] Optionally, the push submodule is also used to: send the transaction data of the first system one by one or in batches to the uplink service of the first system, and the uplink service determines the batch information of the transaction data of the first system, and saves the transaction data and the corresponding batch information of the first system to the cache, and after the number of transaction data in the cache reaches a first number, push the first number of transaction data and the corresponding batch information as a record to the message middleware cluster of the first system.

[0035] Optionally, the on-chain evidence storage module is also used to: perform hash processing on the obtained batch of transaction data to obtain a hash value of the obtained batch of transaction data; store the hash value of the obtained batch of transaction data and the corresponding batch information in the blockchain network, so that the second system can issue the data synchronization request based on the hash value and the corresponding batch information.

[0036] Optionally, the first system includes multiple message middleware clusters; and the transaction data capture module is further used to: push the transaction data of the first system to each currently surviving message middleware cluster of the first system.

[0037] According to yet another aspect of an embodiment of the present invention, a data synchronization device is provided.

[0038] A data synchronization device comprises: an on-chain evidence data acquisition module, used to obtain on-chain evidence data of transaction data of a first system from a blockchain network, wherein the on-chain evidence data is obtained by the first system performing hash processing on the captured transaction data; a to-be-synchronized data determination module, used to determine the transaction data of the first system to be synchronized to a second system based on the on-chain evidence data of the transaction data of the first system; and a data synchronization execution module, used to synchronize the transaction data of the first system to be synchronized to the second system through a message middleware cluster of the second system.

[0039] Optionally, the data synchronization execution module is also used to: distribute the transaction data of the first system to be synchronized to multiple data acquisition components for parsing through the message middleware cluster of the second system, and store the various types of data obtained from the parsing into corresponding storage locations in the second system.

[0040] According to yet another aspect of the embodiments of the present invention, an electronic device is provided.

[0041] An electronic device includes: one or more processors; and a memory for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the data synchronization method provided in an embodiment of the present invention.

[0042] According to yet another aspect of an embodiment of the present invention, a computer-readable storage medium is provided.

[0043] A computer readable medium stores a computer program, which, when executed by a processor, implements a data synchronization method provided by an embodiment of the present invention.

[0044] One embodiment of the above invention has the following advantages or beneficial effects: using a data capture component to capture the transaction data of the first system, and pushing the transaction data of the first system to the message middleware cluster of the first system; performing hash processing on the transaction data of the first system to obtain the on-chain evidence data of the transaction data of the first system, and storing the on-chain evidence data of the transaction data of the first system in the blockchain network, so that the second system can obtain the on-chain evidence data of the transaction data of the first system from the blockchain network; after receiving the data synchronization request issued by the second system according to the on-chain evidence data of the transaction data of the first system, the transaction data of the first system is synchronized to the second system through the message middleware cluster of the first system. It can actively and quasi-real-timely acquire the transaction data within the institution, avoid data loss, ensure that the captured data cannot be tampered with and is traceable, and provide a reliable data source for regulatory agencies to conduct data analysis, and can use the probe program deployed in at least two probe servers of the first system to capture the transaction data of the first system, at least two probe servers are set in the same Internet data center of the first system, and use the same probe program configuration, the probe is highly available, and has strong disaster tolerance and expansion capabilities.

[0045] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0047] Figure 1 is a schematic diagram of main steps of a data synchronization method according to an embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of probe server deployment according to an embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of the workflow of a probe program according to an embodiment of the present invention;

[0050] Figure 4 It is a schematic diagram of a function library and dependency relationship called in a workflow of a probe program according to an embodiment of the present invention;

[0051] Figure 5 is a schematic diagram of a multi-process processing flow of message parsing according to an embodiment of the present invention;

[0052] Figure 6 is a schematic diagram of main steps of a data synchronization method according to another embodiment of the present invention;

[0053] Figure 7 is a logical diagram of blockchain data access according to an embodiment of the present invention;

[0054] Figure 8 is a schematic diagram of programming of a data acquisition component according to an embodiment of the present invention;

[0055] Fig. 9 is a schematic diagram of the overall architecture of data synchronization according to an embodiment of the present invention;

[0056] Fig.10 is a schematic diagram of a full-link process of data synchronization according to an embodiment of the present invention;

[0057] Fig.11 It is a deployment diagram of a probe program, a Kafka cluster, and an on-chain service according to an embodiment of the present invention;

[0058] Fig.12 is a data flow diagram of an uplink service according to an embodiment of the present invention;

[0059] Fig.13 is a schematic diagram of a blockchain architecture according to an embodiment of the present invention;

[0060] Fig.14 is a schematic diagram of a message middleware cluster deployment according to an embodiment of the present invention;

[0061] Fig.15 is a schematic diagram of main modules of a data synchronization device according to an embodiment of the present invention;

[0062] Fig.16 is a schematic diagram of main modules of a data synchronization device according to another embodiment of the present invention;

[0063] Fig.17 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0064] Fig.18 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0066] Figure 1 FIG. 1 is a schematic diagram of the main steps of a data synchronization method according to an embodiment of the present invention. Figure 1 As shown, a data synchronization method according to an embodiment of the present invention mainly includes the following steps S101 to S103. The data synchronization method according to this embodiment is executed on the first system side.

[0067] Step S101: using a data capture component to capture transaction data of a first system, and pushing the transaction data of the first system to a message middleware cluster of the first system;

[0068] Step S102: Perform hash processing on the transaction data of the first system to obtain on-chain evidence data of the transaction data of the first system, and store the on-chain evidence data of the transaction data of the first system in the blockchain network, so that the second system can obtain the on-chain evidence data of the transaction data of the first system from the blockchain network;

[0069] The blockchain network includes blockchain nodes located on the first system side and the second system side respectively.

[0070] Step S103: After receiving a data synchronization request from the second system based on the on-chain evidence data of the transaction data of the first system, the transaction data of the first system is synchronized to the second system through the message middleware cluster of the first system.

[0071] Specifically, the transaction data of the first system is synchronized to the message middleware cluster of the second system.

[0072] In one embodiment, in a financial regulatory compliance scenario, the first system may be an institution-side system, where the institution is a digital currency operating institution, such as commercial banks or other financial business institutions; and the second system may be a central bank-side system, where the central bank is a central bank.

[0073] The data capture component can be a probe. A probe is a component deployed in a system in the form of software or hardware to capture target data. For a software-based probe, it can be called a probe program. The advantages of a regulatory probe are that it has a low degree of intrusion into a specific system, is flexible in setting, and can deeply reflect the internal operating mechanism of the regulated object.

[0074] A message middleware cluster, such as a Kafka cluster.

[0075] The data synchronization method of the embodiment of the present invention can be applied to financial regulatory compliance scenarios. The production and consumption of data are decoupled through the message middleware, and the unforgeable and traceable characteristics of the blockchain are used to ensure that the data captured by the regulatory probe is tamper-proof and non-repudiation. Finally, the data is aggregated in the big data platform, and probe technology is used in conjunction with big data technology to actively capture transaction data within the institution and generate analysis reports.

[0076] In the financial regulatory compliance scenario, the transaction data sources of the First System are mainly the First System's mobile APP, web pages (online banking) and the institution's internal counter system business data.

[0077] In one embodiment, using the data capture component to capture the transaction data of the first system may specifically include: using a probe program deployed at the gateway application layer of the first system to capture the transaction data of the first system.

[0078] Specifically, the probe program deployed at the gateway application layer of the first system is used to capture the transaction data of the first system, including: copying the traffic from the network card of the application of the first system to the physical machine, virtual machine or container, the copied traffic includes the transaction data of the first system, and the application is used for the first system to execute transactions to obtain the transaction data of the first system. Specifically, iptables can be used to copy traffic, and traffic replication within the same network segment or across network segments can be performed. The main function of iptables is to realize the control of network data packets entering and exiting the device and forwarding.

[0079] In another embodiment, using the data capture component to capture the transaction data of the first system may specifically include: using probe programs deployed in at least two probe servers of the first system to capture the transaction data of the first system. At least two probe servers are set in the same Internet data center (IDC) of the first system, each probe server has the same technical specification configuration, each probe server deploys a set of probe levels, and uses the same probe program configuration, and each probe server is connected to the message middleware cluster of the first system. Figure 2 is a schematic diagram of probe server deployment according to an embodiment of the present invention. Figure 2 In this example, two probe servers are used, namely probe servers A and B, and the message middleware cluster of the first system is the Kafka cluster. Each organization deploys at least two probe servers in one IDC to ensure high availability and strong disaster recovery capabilities.

[0080] In another embodiment, using the data capture component to capture the transaction data of the first system includes: using a probe program in a physical machine, virtual machine or container where an application of the first system is located to capture the transaction data of the first system, and the application is used for the first system to execute transactions to obtain the transaction data of the first system. In this way, the probe program can be directly started in the physical machine, virtual machine or container where the application is located, and the probe is directly started because it uses a containerized deployment method.

[0081] When the probe program is deployed in a docker container, an image file of the probe program is run in the container to capture transaction data of the first system, and the image file includes a basic image made based on busybox of the docker container.

[0082] In order to achieve cross-platform deployment, the probe program can be containerized. The probe image file can be run on different operating system platforms by running the probe image file when using the docker container runtime. In order to minimize the dependency of the probe program and reduce the size of the image file, busybox can be selected as the basic image. The image size is about 5MB. With the probe program, the image size is controlled at about 17MB, which is convenient for deployment and image management. Use the docker build command to generate a probe image file. This file contains the probe program (probe), the library used (lib), and the program files such as the automatic timing task (restart_probe.cron). After the probe is containerized, the traffic can be shared by using the traffic replication method. By adding probes and configuring traffic, the horizontal processing capacity can be improved.

[0083] In one embodiment, a data capture component is used to capture transaction data of the first system, and the transaction data of the first system is pushed to the message middleware cluster of the first system. Specifically, it may include: monitoring the message data of the first system during transactions through a probe program, and extracting target message data of the specified address and protocol from the message data, parsing the target message data into transaction data in CSV format, and pushing the transaction data in CSV format to the message middleware cluster of the first system.

[0084] Parsing the target message data into transaction data in CSV format may specifically include: storing the extracted target message data into a shared memory, reading the target message data from the shared memory and reassembling the message to obtain a combined TCP data stream, parsing the TCP data stream according to the HTTP protocol to obtain JSON format data, extracting transaction data of a preset service from the JSON format data, and reorganizing the transaction data of the preset service into CSV format to obtain transaction data in CSV format. The preset service is specified according to demand.

[0085] The probe program relies on BPF (Berkeley Packet Filter) to copy the data packet to the user program for processing. The workflow of the probe program includes the steps of data acquisition, data reorganization, data parsing, and data sending, such as Figure 3 As shown, it is a schematic diagram of the workflow of the probe program of an embodiment of the present invention. The data acquisition step is mainly based on the libpcap library, monitoring the message data in the port communication, and all message data (i.e., target message data) of the specified address and protocol can be extracted, and the TCP packet is filtered. Create a shared memory, and send the proposed target message data to this block of shared memory. This shared memory is maintained by a semaphore, and the semaphore controls the shared memory to support a maximum of 32,000 reads (read-only or write-only times). The data reorganization step is to carry out message reorganization, and the existence mechanism of the shared memory is to provide a buffered data pool for message reorganization. After the data in the shared memory is read out, the corresponding function in the libnids library can be called to complete the reorganization and sorting of the TCP message. After a complete TCP data stream combination is completed, the data can be parsed according to the HTTP protocol to obtain messages in the upstream and downstream JSON formats, and then the next step of data parsing steps is carried out. The data parsing step is to parse the message, analyze the message data (JSON format data), and use the cjson library to filter out the required business transaction data (i.e., the transaction data of the preset business), remove other message data, and rearrange the data into a CSV format to obtain transaction data in CSV format. The data sending step mainly sets up the message middleware, and sends the sorted data information to the message middleware, waiting for the blockchain end to read the chain. The function library and dependency relationship called in the workflow of the probe program in one embodiment of the present invention are as follows: Figure 4 As shown in the figure, libpcap is a data packet capture library, cjson is a Json format parsing library, libnids is a network intrusion monitoring system library, libnet is a small interface library that provides low-level network data packet construction, processing and sending functions, and librdkafka is a high-performance client of Apache Kafka implemented in C language, which provides an efficient and reliable client for the production and use of Kafka.

[0086] The extracted target message data can be stored in multiple shared memories, and the message reassembly, data parsing and arrangement operations can be performed through a multi-process approach.

[0087] Since BPF is based on a single-process model, it is possible to consider multi-process processing at the message parsing level, which can horizontally expand the message parsing capability and fully utilize the processing power of multi-core CPUs. Figure 5As shown, Figure 5 In this example, three shared memories are taken as examples, namely shared memory 1 to shared memory 3. The data reassembly step is included in the message parsing step. Figure 5 Not shown in FIG.

[0088] Pushing the transaction data of the first system to the message middleware cluster of the first system may specifically include: dividing the transaction data of the first system into batches, and pushing the obtained transaction data of the batches and corresponding batch information to the message middleware cluster of the first system.

[0089] Specifically, the transaction data of the first system is divided into batches, and the obtained batches of transaction data and corresponding batch information are pushed to the message middleware cluster of the first system, including: sending the transaction data of the first system to the upchain service of the first system one by one or in batches, and the upchain service determines the batch information of the transaction data of the first system, and saves the transaction data and the corresponding batch information of the first system to the cache. After the number of transaction data in the cache reaches a first number, the first number of transaction data and the corresponding batch information are pushed as a record to the message middleware cluster of the first system.

[0090] The transaction data of the first system is hashed to obtain the on-chain evidence data of the transaction data of the first system, and the on-chain evidence data of the transaction data of the first system is stored in the blockchain network, which may specifically include: hashing the obtained batch of transaction data to obtain the hash value of the obtained batch of transaction data; storing the hash value of the obtained batch of transaction data and the corresponding batch information in the blockchain network, so that the second system can issue a data synchronization request according to the hash value and the corresponding batch information. The on-chain evidence data may include the hash value of the obtained batch of transaction data and the corresponding batch information.

[0091] In one embodiment, the first system includes multiple message middleware clusters; the transaction data of the first system is pushed to the message middleware cluster of the first system, and specifically, the transaction data of the first system can be pushed to each currently existing message middleware cluster of the first system.

[0092] Figure 6 FIG. 1 is a schematic diagram of the main steps of a data synchronization method according to another embodiment of the present invention. The data synchronization method of this embodiment is executed on the second system side. Figure 6 As shown, the data synchronization method of this embodiment mainly includes the following steps S601 to S603.

[0093] Step S601: Obtaining on-chain evidence data of the transaction data of the first system from the blockchain network, where the on-chain evidence data is obtained by the first system performing hash processing on the captured transaction data;

[0094] Step S602: Determine the transaction data of the first system to be synchronized to the second system according to the on-chain evidence data of the transaction data of the first system;

[0095] Step S603: Synchronize the transaction data of the first system to be synchronized to the second system through the message middleware cluster of the second system.

[0096] The transaction data of the first system to be synchronized is synchronized to the second system through the message middleware cluster of the second system, specifically including: distributing the transaction data of the first system to be synchronized to multiple data acquisition components for parsing through the message middleware cluster of the second system, and storing various types of data obtained from the parsing into corresponding storage locations in the second system.

[0097] The various types of data obtained through analysis include business occurrence time, data type, operating organization and other information.

[0098] In the financial regulatory compliance scenario, the regulatory probe program collects the transaction data of the institution and reports it to the blockchain platform (i.e., blockchain network). After the blockchain platform completes the decryption and message conversion operations, it sends each message information to the big data platform through the message middleware Kafka. The big data platform consumes the data in Kafka and completes the HDFS (a distributed file system) storage operation based on the business occurrence time, data type, operating organization and other information.

[0099] A logical diagram of blockchain data access according to an embodiment of the present invention is shown in FIG. Figure 7 As shown, the data collection component used by the data consumer end can be a Flume component. Based on the data type and the operating organization Topic, the data is routed and distributed to different Flume Agents for processing, and the business occurrence time, Topic organization code, business type and other information are parsed in Flume. The parsed information is stored in different locations in the HDFS storage system of the second system. Topic is the basic unit of Kafka data writing operation, representing a type of data.

[0100] A schematic diagram of the programming of a data acquisition component according to an embodiment of the present invention is shown in FIG. Figure 8 As shown, the data collection components specifically include Flume Agent1, Flume Agent2, Flume Agent3, etc., which correspond to transaction data of a type (such as DataType1, DataType2, DataType3, etc.). The following functions are required in the data collection component Flume Agent:

[0101] Source: defines Kafka data source information, including IP, Topic, batch collection configuration, etc.

[0102] Interceptor: parses information such as business occurrence time and organization code, defines exception specifications, and marks abnormal data;

[0103] Selector: Diverts data channels according to interceptor tag information;

[0104] Channel: defines the data channel type; this type of program uses the "File" type, and the data is stored locally to ensure data security;

[0105] Sink: defines the data channel output system type, using the HDFS Sink format to transfer the processed data to the big data platform storage system. The data refers to transaction data.

[0106] Fig. 9 FIG. 1 is a schematic diagram of the overall architecture of data synchronization according to an embodiment of the present invention. Fig. 9 As shown in the figure, in the institutional end system (i.e., the first system), the regulatory probe reuses the gateway application resources of the institution, captures the data packets of the transaction data of the first system from the network equipment (including the virtual network card), extracts the key content and pushes it to the institutional message middleware cluster, i.e., the institutional end Kafka cluster. The chain service divides the original data in the institutional message middleware cluster (i.e., the institutional end transaction data in the institutional end message middleware cluster) into batches. The institutional end transaction data after the batches are divided into batches is called batch data. The Hash of the batch data is stored on the chain, and the batch data is pushed to the new message middleware for persistent storage. The blockchain network and the message middleware synchronize the Hash and original data of the batch data to the central bank end system (i.e., the second system) respectively. In the central bank end system, the chain application extracts the Hash of the batch data and pushes it to the local message middleware cluster of the central bank, i.e., the Kafka cluster of the central bank end. The central bank's big data platform consumes the Hash and original data of the batch data in the central bank's message middleware cluster.

[0107] Fig.10 It is a schematic diagram of the full-link process of data synchronization according to an embodiment of the present invention.

[0108] The probe program is used to capture the transaction data of the institutional end system and extract the message. Specifically, all the message data of the specified address and protocol (i.e., the target message data) can be extracted. The messages of which addresses and protocols are to be extracted can be determined according to the needs, and the extracted data can be pushed to the institutional end system message middleware cluster, i.e., Kafka1. The chain service consumes the data in Kafka1 and divides the data into batches to obtain the institutional end transaction data and the corresponding batch information after the batch division, such as the batch number, and stores the institutional end transaction data after the batch division in Kafka1. The chain service also calculates the hash value of the institutional end transaction data after the batch division, and the obtained hash value is the Hash of the batch data. The Hash of the batch data is stored on the chain, and the institutional end blockchain node broadcasts the Hash of the batch data, and the Hash of the batch data is synchronized to the central bank end blockchain node. The chain application on the central bank end can pull the Hash of the batch data from the central bank end blockchain node. The chain application pushes the Hash of the batch data to the central bank end message middleware cluster, i.e., Kafka2. The big data platform can pull the hash of batch data and the original data of the corresponding batch from Kafka2, that is, the institutional transaction data of the corresponding batch. The big data platform can perform data analysis and other operations on the pulled institutional transaction data.

[0109] The probe program can push transaction data of the institutional system to Kafka1 in a single message. The message format, data volume and examples are shown in Table 1.

[0110] Table 1

[0111]

[0112]

[0113] The on-chain service stores the batched institutional transaction data in Kafka1 and pushes them in batch messages. The message format, data volume and examples are shown in Table 2.

[0114] Table 2

[0115]

[0116] The data format, data volume, and examples of the on-chain service storing the Hash of batch data on the chain are shown in Table 3.

[0117] Table 3

[0118]

[0119]

[0120] Fig.11FIG. 1 is a schematic diagram of the deployment of a probe program, a Kafka cluster, and an on-chain service according to an embodiment of the present invention. Fig.11 As shown in the figure, on the institutional side, the deployment topology involves three components: probe program, chain service and Kafka. The probe program is deployed at the gateway application layer of the institutional side system, and its deployment architecture follows the principle of institutional side gateway deployment. Currently, operating institutions mostly use cloud infrastructure to deploy applications, such as using virtual machines and containers as the underlying mechanism for application deployment and operation, so the probe program will run in the virtualized environment of the cloud platform and be deployed in a multi-instance manner.

[0121] The deployment of on-chain services should also be implemented in accordance with the institution’s application deployment principles, which are determined in advance by the institution based on business needs.

[0122] Kafka can be deployed in cluster mode when deployed on the institutional side. Because the network latency in the same city is small, multiple data centers in the same city can share a Kafka cluster. Cluster synchronization is not mandatory between Kafka clusters in different locations, as long as high availability and disaster recovery requirements are guaranteed. The specific plan is determined by the operating organization according to needs. The specific deployment plan must also consider the implementation of the DC / EP system architecture (two-tier operating system) of the organization. Fig.11 As shown, a typical two-site three-center architecture is used as an example to illustrate a feasible deployment solution. The institution adopts a two-site three-center architecture. The two IDCs (Internet Data Centers) in Beijing are "active-active" and process business traffic at the same time; the Shanghai IDC is an off-site disaster recovery data center that does not process business traffic and is only enabled when the Beijing IDC fails. When the transaction data of the institution-side system is pushed to the message middleware cluster of the institution-side system, the transaction data of the institution-side system is pushed to the currently surviving message middleware clusters of the institution-side system, such as the Kafka clusters of the two IDCs in Beijing.

[0123] Fig.12 FIG. 1 is a data flow diagram of an uplink service according to an embodiment of the present invention. Fig.12 As shown in the figure, the transaction data captured by the probe is directly sent to the chain service via HTTP, where the transaction data (i.e. Fig.12 The original data in the data can be sent in batches or one by one. Fig.12The data is sent to the chain service in batches of 100. The chain service will cache the data in memory, and then when it reaches a fixed number, such as 2,000, these 2,000 data will be sent to Kafka as a Kafka record. At the same time, the number of this batch is counted. If it reaches a certain number, such as 6,000, all the data in this batch will be hashed and then uploaded to the chain. Specifically, the batch number (batchNO), the number of original data in the batch (txCnt) and the hash are placed in the smart contract. The final data consumer of the transaction data captured by the probe is the big data platform on the central bank side. As a Consumer, the big data platform continuously consumes the transaction data synchronized to the central bank through Kafka.

[0124] Fig.13 Schematic diagram of blockchain architecture according to one embodiment of the present invention. Fig.13 As shown, the overall architecture of an embodiment of the present invention includes eight parts: probe, chain service, institutional Kafka, central bank Kafka, blockchain, chain reading service, query service, and big data. The institutional end is for example institution A and institution B. The probe is responsible for the message reorganization and screening of institutional transaction data; the chain service is responsible for allocating batch numbers of institutional transaction data, sending the original data and batch numbers to the Kafka queue, and hashing the data on the chain; institutional Kafka is responsible for transaction data storage and data synchronization; the central bank Kafka is responsible for temporarily storing the synchronization data from the institutional Kafka and the data from the chain reading service, and providing it to the big data platform for consumption; the blockchain is responsible for storing the hash value of transaction data; the chain reading service is responsible for obtaining the transaction data hash on the chain and pushing it to Kafka. Specifically, the batch transaction data hash and corresponding batch information can be obtained by querying the smart contract; the query service is a unified query entry for the data stored on the institutional end; the big data platform is responsible for analyzing the transaction data and issuing reports. The central bank end and the institutional end are based on MirrorMaker data synchronization. MirrorMaker is a tool in the Kafka suite that is used to synchronize records between clusters.

[0125] Fig.14 FIG. 1 is a schematic diagram of a message middleware cluster deployment according to an embodiment of the present invention. Fig.14As shown in the figure, each operating institution (taking institution A, institution B, and institution C as examples) deploys an independent Kafka cluster locally, and the transaction data captured by the probe of this institution is written into the locally deployed Kafka cluster. The central bank deploys a Kafka cluster locally, and the number of topics and partition copies is consistent with that of the operating institution. Multiple MirrorMaker processes are started on the central bank side. MirrorMaker pulls messages from the Kafka cluster of each operating institution and synchronizes them to the Kafka cluster on the central bank side to ensure high availability. Data transmission across data centers can be decoupled by introducing message middleware. For example, if there is an abnormality in the connection between the institution and the central bank, data can be synchronized from the Kafka cluster on the institution side after it is restored to normal.

[0126] Fig.15 FIG. 1 is a schematic diagram of the main modules of a data synchronization device according to an embodiment of the present invention. Fig.15 As shown, a data synchronization device 1500 of an embodiment of the present invention is arranged at the first system end, and the data synchronization device 1500 mainly includes: a transaction data capture module 1501, a chain evidence storage module 1502, and a transaction data synchronization module 1503.

[0127] The transaction data capture module 1501 is used to capture the transaction data of the first system using the data capture component, and push the transaction data of the first system to the message middleware cluster of the first system;

[0128] The on-chain evidence storage module 1502 is used to perform hash processing on the transaction data of the first system to obtain the on-chain evidence storage data of the transaction data of the first system, and store the on-chain evidence storage data of the transaction data of the first system in the blockchain network, so that the second system can obtain the on-chain evidence storage data of the transaction data of the first system from the blockchain network;

[0129] The transaction data synchronization module 1503 is used to synchronize the transaction data of the first system to the second system through the message middleware cluster of the first system after receiving a data synchronization request issued by the second system based on the on-chain evidence data of the transaction data of the first system.

[0130] In one embodiment, the transaction data capture module 1501 is further used to capture the transaction data of the first system using a probe program deployed at the gateway application layer of the first system.

[0131] The transaction data capture module 1501 can be specifically used to: copy traffic from the network card of the application of the first system to a physical machine, a virtual machine or a container, the copied traffic includes the transaction data of the first system, and the application is used to execute transactions in the first system to obtain the transaction data of the first system.

[0132] In another embodiment, the transaction data capture module 1501 is also used to: capture the transaction data of the first system using the probe program deployed in at least two probe servers of the first system, where at least two probe servers are set in the same Internet data center of the first system and use the same probe program configuration.

[0133] In another embodiment, the transaction data capture module 1501 is also used to: utilize a probe program in a physical machine, virtual machine or container where the application of the first system is located to capture the transaction data of the first system, and apply the first system to execute transactions to obtain the transaction data of the first system.

[0134] The transaction data capture module 1501 is also used to: when the probe program is deployed in a docker container, run the image file of the probe program in the container to capture the transaction data of the first system, and the image file includes a basic image made based on busybox of the docker container.

[0135] In one embodiment, the transaction data capture module 1501 is also used to: monitor the message data of the first system during transactions through a probe program, and extract target message data of the specified address and protocol from the message data, parse the target message data into transaction data in CSV format, and push the transaction data in CSV format to the message middleware cluster of the first system.

[0136] In one embodiment, the transaction data capture module 1501 includes a parsing submodule, which is used to: store the extracted target message data into a shared memory, read the target message data from the shared memory and reassemble the message to obtain a combined TCP data stream, parse the TCP data stream according to the HTTP protocol to obtain JSON format data, extract the transaction data of the preset business from the JSON format data, and rearrange the transaction data of the preset business into CSV format to obtain transaction data in CSV format.

[0137] The extracted target message data can be stored in multiple shared memories. The parsing submodule is also used to perform message reassembly, data parsing and arrangement operations through a multi-process approach.

[0138] In one embodiment, the transaction data capture module 1501 includes a push submodule, which is used to: divide the transaction data of the first system into batches, and push the obtained batch transaction data and corresponding batch information to the message middleware cluster of the first system.

[0139] The push submodule is also used to: send the transaction data of the first system one by one or in batches to the upchain service of the first system, and the upchain service determines the batch information of the transaction data of the first system, and saves the transaction data of the first system and the corresponding batch information to the cache. After the number of transaction data in the cache reaches a first number, the first number of transaction data and the corresponding batch information are pushed as a record to the message middleware cluster of the first system.

[0140] In one embodiment, the on-chain evidence storage module 1502 is specifically used to: perform hash processing on the obtained batch of transaction data to obtain a hash value of the obtained batch of transaction data; store the hash value of the obtained batch of transaction data and the corresponding batch information in the blockchain network, so that the second system can issue a data synchronization request based on the hash value and the corresponding batch information.

[0141] The message middleware cluster of the first system includes multiple;

[0142] The transaction data capture module 1501 is specifically used to push the transaction data of the first system to each currently surviving message middleware cluster of the first system.

[0143] Fig.16 FIG. 1 is a schematic diagram of the main modules of a data synchronization device according to another embodiment of the present invention. Fig.16 As shown, a data synchronization device 1600 according to an embodiment of the present invention is disposed at the second system end, and the data synchronization device 1600 mainly includes:

[0144] The on-chain evidence data acquisition module 1601 is used to obtain the on-chain evidence data of the transaction data of the first system from the blockchain network. The on-chain evidence data is obtained by the first system performing hash processing on the captured transaction data;

[0145] The to-be-synchronized data determination module 1602 is used to determine the transaction data of the first system to be synchronized to the second system according to the on-chain evidence data of the transaction data of the first system;

[0146] The data synchronization execution module 1603 is used to synchronize the transaction data of the first system to be synchronized to the second system through the message middleware cluster of the second system.

[0147] The data synchronization execution module 1603 is specifically used to distribute the transaction data of the first system to be synchronized to multiple data collection components for parsing through the message middleware cluster of the second system, and store the various types of data obtained by parsing into corresponding storage locations in the second system.

[0148] In addition, the specific implementation content of the data synchronization device in the embodiment of the present invention has been described in detail in the data synchronization method described above, so the repeated content will not be described again here.

[0149] Fig.17 An exemplary system architecture 1700 is shown to which the data synchronization method or data synchronization device according to the embodiment of the present invention can be applied.

[0150] like Fig.17 As shown, system architecture 1700 may include terminal devices 1701, 1702, 1703, a network 1704, and a server 1705. Network 1704 is used to provide a medium for communication links between terminal devices 1701, 1702, 1703 and server 1705. Network 1704 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0151] Users can use terminal devices 1701, 1702, and 1703 to interact with server 1705 through network 1704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 1701, 1702, and 1703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0152] The terminal devices 1701 , 1702 , and 1703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0153] The server 1705 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 1701, 1702, and 1703. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0154] It should be noted that the data synchronization method provided in the embodiment of the present invention is generally executed by the server 1705 , and accordingly, the data synchronization device is generally set in the server 1705 .

[0155] It should be understood that Fig.17 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0156] Reference below Fig.18 , which shows a schematic diagram of the structure of a computer system 1800 of a terminal device or server suitable for implementing an embodiment of the present application. Fig.18 The terminal device or server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0157] like Fig.18 As shown, the computer system 1800 includes a central processing unit (CPU) 1801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1802 or a program loaded from a storage part 1808 to a random access memory (RAM) 1803. In the RAM 1803, various programs and data required for the operation of the system 1800 are also stored. The CPU 1801, the ROM 1802, and the RAM 1803 are connected to each other via a bus 1804. An input / output (I / O) interface 1805 is also connected to the bus 1804.

[0158] The following components are connected to the I / O interface 1805: an input section 1806 including a keyboard, a mouse, etc.; an output section 1807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN card, a modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to the I / O interface 1805 as needed. A removable medium 1811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1810 as needed, so that a computer program read therefrom is installed into the storage section 1808 as needed.

[0159] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1809, and / or installed from the removable medium 1811. When the computer program is executed by the central processing unit (CPU) 1801, the above-mentioned functions defined in the system of the present application are executed.

[0160] It should be noted that the computer-readable medium shown in the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0161] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as follows: a processor includes a transaction data capture module, an on-chain evidence storage module, and a transaction data synchronization module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the transaction data capture module may also be described as "a module for capturing transaction data of the first system using a data capture component, and pushing the transaction data of the first system to the message middleware cluster of the first system."

[0163] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: using a data capture component to capture the transaction data of the first system, and pushing the transaction data of the first system to the message middleware cluster of the first system; performing hash processing on the transaction data of the first system to obtain the on-chain evidence data of the transaction data of the first system, and storing the on-chain evidence data of the transaction data of the first system in the blockchain network, so that the second system can obtain the on-chain evidence data of the transaction data of the first system from the blockchain network; after receiving the data synchronization request issued by the second system based on the on-chain evidence data of the transaction data of the first system, the transaction data of the first system is synchronized to the second system through the message middleware cluster of the first system.

[0164] According to the technical solution of the embodiment of the present invention, it is possible to actively and quasi-real-time obtain transaction data within the institution, avoid data loss, ensure that the captured data cannot be tampered with and is traceable, and provide a reliable data source for regulatory agencies to conduct data analysis. The probe is highly available and has strong disaster recovery and expansion capabilities.

[0165] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data synchronization method, characterized in that: include: Using the data capture component to capture the transaction data of the first system, and pushing the transaction data of the first system to the message middleware cluster of the first system, including: dividing the transaction data of the first system into batches, and pushing the obtained transaction data of the batches and corresponding batch information to the message middleware cluster of the first system; Performing hash processing on the transaction data of the first system to obtain on-chain evidence data of the transaction data of the first system, and storing the on-chain evidence data of the transaction data of the first system in a blockchain network, so that the second system can obtain the on-chain evidence data of the transaction data of the first system from the blockchain network, including: performing hash processing on the obtained batch of transaction data to obtain a hash value of the obtained batch of transaction data; storing the hash value of the obtained batch of transaction data and corresponding batch information in the blockchain network, so that the second system can issue a data synchronization request according to the hash value and the corresponding batch information; After receiving a data synchronization request from the second system based on the on-chain evidence data of the transaction data of the first system, the transaction data of the first system is synchronized to the message middleware cluster of the second system through the inter-cluster synchronization method between the message middleware cluster of the first system and the message middleware cluster of the second system; The method of using the data capture component to capture the transaction data of the first system and pushing the transaction data of the first system to the message middleware cluster of the first system also includes: monitoring the message data of the first system during transactions through a probe program, and extracting target message data of a specified address and protocol from the message data; storing the extracted target message data in a shared memory, reading the target message data from the shared memory and reorganizing the messages to obtain a combined TCP data stream; parsing the TCP data stream according to the HTTP protocol to obtain JSON format data; extracting the transaction data of a preset business from the JSON format data, and rearranging the transaction data of the preset business into CSV format to obtain transaction data in CSV format; and pushing the transaction data in CSV format to the message middleware cluster of the first system.

2. The method according to claim 1, characterized in that The method of using the data capture component to capture the transaction data of the first system includes: The transaction data of the first system is captured by using a probe program deployed at the gateway application layer of the first system.

3. The method according to claim 2, characterized in that The method of capturing transaction data of the first system by using a probe program deployed at the gateway application layer of the first system includes: The traffic is copied from the network card of the application of the first system to a physical machine, a virtual machine or a container, the copied traffic includes the transaction data of the first system, and the application is used to execute transactions with the first system to obtain the transaction data of the first system.

4. The method according to claim 1, characterized in that: The method of using the data capture component to capture the transaction data of the first system includes: The transaction data of the first system is captured by using a probe program deployed in at least two probe servers of the first system, wherein the at least two probe servers are arranged in the same Internet data center of the first system and adopt the same probe program configuration.

5. The method according to claim 1, characterized in that The method of using the data capture component to capture the transaction data of the first system includes: The transaction data of the first system is captured by using a probe program in a physical machine, a virtual machine or a container where an application of the first system is located. The application is used to execute transactions with the first system to obtain the transaction data of the first system.

6. The method according to claim 5, characterized in that In the case where the probe program is deployed in a docker container, an image file of the probe program is run in the container to capture transaction data of the first system, and the image file includes a basic image made based on busybox of the docker container.

7. The method according to claim 1, characterized in that The extracted target message data is stored in a plurality of shared memories, and the message reassembly, data parsing and arrangement operations are performed in a multi-process manner.

8. The method according to claim 1, characterized in that The dividing the transaction data of the first system into batches, and pushing the obtained transaction data of the batches and the corresponding batch information to the message middleware cluster of the first system includes: The transaction data of the first system are sent one by one or in batches to the uplink service of the first system, and the uplink service determines the batch information of the transaction data of the first system, and saves the transaction data of the first system and the corresponding batch information to the cache. After the amount of transaction data in the cache reaches a first amount, the first amount of transaction data and the corresponding batch information are pushed as a record to the message middleware cluster of the first system.

9. The method according to claim 1, characterized in that: The message middleware clusters of the first system include multiple ones; the step of pushing the transaction data of the first system to the message middleware cluster of the first system includes: The transaction data of the first system is pushed to each currently existing message middleware cluster of the first system.

10. A data synchronization method, characterized in that: include: Obtaining on-chain evidence data of the transaction data of the first system from the blockchain network, the on-chain evidence data comprising: a hash value generated by hashing the transaction data of the batch obtained by dividing the captured transaction data into batches by the first system, and corresponding batch information; Determine the transaction data of the first system to be synchronized to the second system according to the on-chain evidence data of the transaction data of the first system; Synchronize the transaction data of the first system to be synchronized to the message middleware cluster of the second system through the inter-cluster synchronization method of the message middleware cluster of the first system and the message middleware cluster of the second system; The transaction data of the first system is determined in the first system according to the following steps: monitoring the message data of the first system during transactions through a probe program, and extracting target message data of a specified address and protocol from the message data; storing the extracted target message data in a shared memory, reading the target message data from the shared memory and performing message reorganization to obtain a combined TCP data stream; parsing the TCP data stream according to the HTTP protocol to obtain JSON format data; extracting transaction data of a preset business from the JSON format data, and rearranging the transaction data of the preset business into CSV format to obtain transaction data in CSV format; pushing the transaction data in CSV format to the message middleware cluster of the first system.

11. The method according to claim 10, characterized in that The step of synchronizing the transaction data of the first system to be synchronized to the second system through the message middleware cluster of the second system includes: The transaction data of the first system to be synchronized is distributed to multiple data acquisition components for parsing through the message middleware cluster of the second system, and various types of data obtained from the parsing are stored in corresponding storage locations in the second system.

12. A data synchronization device, characterized in that: include: A transaction data capture module, used to capture the transaction data of the first system using a data capture component, and push the transaction data of the first system to a message middleware cluster of the first system; an on-chain evidence storage module, configured to perform hash processing on the transaction data of the first system to obtain on-chain evidence data of the transaction data of the first system, and store the on-chain evidence data of the transaction data of the first system in a blockchain network, so that the second system can obtain the on-chain evidence data of the transaction data of the first system from the blockchain network; A transaction data synchronization module, configured to synchronize the transaction data of the first system to the message middleware cluster of the second system through the inter-cluster synchronization method between the message middleware cluster of the first system and the message middleware cluster of the second system after receiving a data synchronization request issued by the second system based on the on-chain evidence data of the transaction data of the first system; The transaction data capture module includes a push submodule, which is used to: divide the transaction data of the first system into batches, and push the obtained batch transaction data and corresponding batch information to the message middleware cluster of the first system; The on-chain evidence storage module is further used to: perform hash processing on the obtained batch of transaction data to obtain a hash value of the obtained batch of transaction data; store the hash value of the obtained batch of transaction data and the corresponding batch information in the blockchain network, so that the second system can issue the data synchronization request according to the hash value and the corresponding batch information; The transaction data capture module is also used to: monitor the message data of the first system during the transaction through a probe program, and extract the target message data of the specified address and protocol from the message data, parse the target message data into transaction data in CSV format, and push the transaction data in CSV format to the message middleware cluster of the first system; The transaction data capture module includes a parsing submodule, which is used to: store the extracted target message data into a shared memory, read the target message data from the shared memory and reassemble the message to obtain a combined TCP data stream, parse the TCP data stream according to the HTTP protocol to obtain JSON format data, extract the transaction data of a preset business from the JSON format data, rearrange the transaction data of the preset business into the CSV format, and obtain the transaction data in the CSV format.

13. The device according to claim 12, characterized in that The transaction data capture module is also used for: The transaction data of the first system is captured by using a probe program deployed at the gateway application layer of the first system.

14. The device according to claim 13, characterized in that The transaction data capture module is also used for: The traffic is copied from the network card of the application of the first system to a physical machine, a virtual machine or a container, the copied traffic includes the transaction data of the first system, and the application is used to execute transactions with the first system to obtain the transaction data of the first system.

15. The device according to claim 12, characterized in that The transaction data capture module is also used for: The transaction data of the first system is captured by using a probe program deployed in at least two probe servers of the first system, wherein the at least two probe servers are arranged in the same Internet data center of the first system and adopt the same probe program configuration.

16. The device according to claim 12, characterized in that The transaction data capture module is also used for: The transaction data of the first system is captured by using a probe program in a physical machine, a virtual machine or a container where an application of the first system is located. The application is used to execute transactions with the first system to obtain the transaction data of the first system.

17. The device according to claim 16, characterized in that The transaction data capture module is also used to: when the probe program is deployed in a docker container, run the image file of the probe program in the container to capture the transaction data of the first system, and the image file includes a basic image made based on busybox of the docker container.

18. The device according to claim 12, characterized in that The extracted target message data is stored in a plurality of shared memories, and the parsing submodule is further used to: execute the message reassembly, the data parsing and the arrangement operations in a multi-process manner.

19. The device according to claim 12, characterized in that The push submodule is also used for: The transaction data of the first system are sent one by one or in batches to the uplink service of the first system, and the uplink service determines the batch information of the transaction data of the first system, and saves the transaction data of the first system and the corresponding batch information to the cache. After the amount of transaction data in the cache reaches a first amount, the first amount of transaction data and the corresponding batch information are pushed as a record to the message middleware cluster of the first system.

20. The device according to claim 12, characterized in that The message middleware cluster of the first system includes multiple ones; the transaction data capture module is also used for: The transaction data of the first system is pushed to each currently existing message middleware cluster of the first system.

21. A data synchronization device, characterized in that: include: An on-chain evidence data acquisition module is used to obtain on-chain evidence data of transaction data of the first system from the blockchain network, wherein the on-chain evidence data includes: a hash value generated by hashing the transaction data of a batch obtained by dividing the captured transaction data into batches by the first system, and corresponding batch information; A module for determining data to be synchronized, used to determine the transaction data of the first system to be synchronized to the second system according to the on-chain evidence data of the transaction data of the first system; A data synchronization execution module, used to synchronize the transaction data of the first system to be synchronized to the message middleware cluster of the second system through an inter-cluster synchronization method between the message middleware cluster of the first system and the message middleware cluster of the second system; The first system is used to: monitor the message data of the first system during transactions through a probe program, and extract target message data of a specified address and protocol from the message data; store the extracted target message data in a shared memory, read the target message data from the shared memory and reassemble the message to obtain a combined TCP data stream; parse the TCP data stream according to the HTTP protocol to obtain JSON format data; extract transaction data of a preset business from the JSON format data, and rearrange the transaction data of the preset business into CSV format to obtain transaction data in CSV format; and push the transaction data in CSV format to a message middleware cluster of the first system.

22. The device according to claim 21, characterized in that The data synchronization execution module is also used for: The transaction data of the first system to be synchronized is distributed to multiple data acquisition components for parsing through the message middleware cluster of the second system, and various types of data obtained from the parsing are stored in corresponding storage locations in the second system.

23. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 11.

24. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images