Configurable database real-time synchronization method and device, equipment and storage medium

Abstract

The invention relates to the technical field of information processing, in particular to a configurable database real-time synchronization method, device and equipment and a storage medium. A downstream database connected with a source database is registered, a preset data mapping template is called according to the source database and the downstream database, and a data mapping rule is configured in the data mapping template; the method comprises the following steps: capturing data change information of a source database by using a database change monitoring mechanism, writing the data change information into a Redis queue, extracting the data change information from the Redis queue in real time through a script, preprocessing the data change information, carrying out data verification on the data change preprocessing information, and when a verification result passes, sending the data change information to the source database; according to the method and the device, the intelligent degree is high, the resource and technical cost is reduced, data synchronization of different types of databases or different table structures can be supported according to mapping relation configuration, synchronization of partial fields is supported, and the flexibility of function implementation is improved.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a configurable database real-time synchronization method, device, equipment and storage medium. Background Art

[0002] At present, database synchronization generally adopts the master-slave synchronization method. Although this synchronization method has mature technical implementation, it has two problems: first, the maintenance cost is high. Database master-slave synchronization requires additional hardware and network resources to support data synchronization, and regular monitoring and maintenance are required. In addition, the configuration and optimization of master-slave synchronization also require professional technical support, which further increases the maintenance cost. Second, it is not flexible enough and cannot support data synchronization of different types of databases or different table structures. It also does not support the need to synchronize only some fields of a table. It has poor flexibility and can only be synchronized manually, with a low degree of intelligence. Summary of the invention

[0003] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a configurable database real-time synchronization method, device, equipment and storage medium that are highly intelligent, reduce resource and technical costs, can support data synchronization of different types of databases or different table structures according to mapping relationship configuration, and support partial field synchronization, thereby improving the flexibility of function implementation.

[0004] A first aspect of the present invention provides a configurable database real-time synchronization method, comprising: defining database connection information, and registering a downstream database connected to a source database according to the database connection information; calling a preset data mapping template according to the source database and the downstream database, and configuring data mapping rules in the data mapping template; using a database change monitoring mechanism to capture data change information of the source database, and writing the data change information into a Redis queue; extracting the data change information from the Redis queue in real time through a script, and preprocessing the data change information to obtain data change preprocessing information; performing data verification on the data change preprocessing information to obtain a verification result, and when the verification result passes, synchronizing and updating the downstream database according to the data change preprocessing information.

[0005] Optionally, in a first implementation method of the first aspect of the present invention, defining database connection information and registering a downstream database connected to a source database according to the database connection information includes: defining database connection information, wherein the database connection information includes a database type, a database host address, a port number, a database name, a user name, and a password; registering a downstream database connected to a source database according to the database connection information; and creating a connection pool, and using the connection pool to manage the connection between the downstream database and the source database.

[0006] Optionally, in a second implementation method of the first aspect of the present invention, calling a preset data mapping template according to the source database and the downstream database, and configuring data mapping rules in the data mapping template, includes: identifying downstream table structure information in the downstream database and source table structure information in the source database; comparing and analyzing the downstream table structure information and the source table structure information to obtain difference structure information; calling a preset data mapping template according to the difference structure information, and configuring table mapping relationships and field mapping relationships in the data mapping template.

[0007] Optionally, in a third implementation method of the first aspect of the present invention, the use of a database change monitoring mechanism to capture data change information of the source database and write the data change information into a Redis queue includes: creating a listener, and using the listener to monitor the data change information of the source database; when the listener monitors the data change information of the source database, obtaining the current data change information; and writing the data change information into a Redis queue.

[0008] Optionally, in a fourth implementation method of the first aspect of the present invention, the data change information is extracted from the Redis queue in real time through a script, and the data change information is pre-processed to obtain data change pre-processing information, including: obtaining the data change information from the Redis queue in real time through a script; performing data cleaning on the data change information to obtain data change cleaning information; filling missing values ​​in the data change cleaning information to obtain data change completion information; and formatting the data change completion information based on a preset target standard format to obtain data change pre-processing information.

[0009] Optionally, in a fifth implementation method of the first aspect of the present invention, data verification is performed on the data change preprocessing information to obtain a verification result, and when the verification result passes, the downstream database is synchronously updated according to the data change preprocessing information, including: format verification is performed on the data change preprocessing information to obtain a first verification result; when the first verification result passes, integrity verification is performed on the data change preprocessing information to obtain a second verification result; when the second verification result passes, business rule verification is performed on the data change preprocessing information to obtain a third verification result; when the third verification result passes, data consistency verification is performed on the data change preprocessing information to obtain a fourth verification result; when the fourth verification result passes, the data change preprocessing information is written into the downstream database.

[0010] Optionally, in a sixth implementation method of the first aspect of the present invention, the data change preprocessing information is subjected to data verification to obtain a verification result, and when the verification result passes, the downstream database is synchronously updated according to the data change preprocessing information, further comprising: generating a database update record, and obtaining an update timestamp from a timestamp service platform based on the update time; merging the database update record and the update timestamp to obtain merged information; encrypting the merged information to obtain encrypted information; and uploading the encrypted information to the blockchain.

[0011] The second aspect of the present invention provides a configurable database real-time synchronization device, including: a definition registration module, used to define database connection information, and register a downstream database connected to a source database according to the database connection information; a call configuration module, used to call a preset data mapping template according to the source database and the downstream database, and configure data mapping rules in the data mapping template; a capture and write module, used to use a database change monitoring mechanism to capture the data change information of the source database, and write the data change information into a Redis queue; an extraction and processing module, used to extract the data change information from the Redis queue in real time through a script, and pre-process the data change information to obtain data change pre-processing information; a verification and update module, used to perform data verification on the data change pre-processing information to obtain a verification result, and when the verification result passes, the downstream database is synchronously updated according to the data change pre-processing information.

[0012] Optionally, in a first implementation method of the second aspect of the present invention, the definition registration module includes: a definition unit, used to define database connection information, the database connection information including database type, database host address, port number, database name, user name and password; a registration unit, used to register a downstream database connected to a source database according to the database connection information; and a creation management unit, used to create a connection pool, and use the connection pool to manage the connection between the downstream database and the source database.

[0013] Optionally, in a second implementation of the second aspect of the present invention, the calling configuration module includes: an identification unit, used to identify downstream table structure information in the downstream database and source table structure information in the source database; an analysis unit, used to compare and analyze the downstream table structure information and the source table structure information to obtain difference structure information; and a calling configuration unit, used to call a preset data mapping template according to the difference structure information, and configure table mapping relationships and field mapping relationships in the data mapping template.

[0014] Optionally, in a third implementation method of the second aspect of the present invention, the capture and write module includes: creating a monitoring unit, used to create a listener, and using the listener to monitor the data change information of the source database; a first acquisition unit, used to obtain the current data change information when the listener monitors the data change information of the source database; and a first writing unit, used to write the data change information into a Redis queue.

[0015] Optionally, in a fourth implementation method of the second aspect of the present invention, the extraction and processing module includes: a second acquisition unit, used to obtain the data change information from the Redis queue in real time through a script; a cleaning unit, used to perform data cleaning on the data change information to obtain data change cleaning information; a filling unit, used to fill missing values ​​in the data change cleaning information to obtain data change completion information; a conversion unit, used to perform format conversion on the data change completion information based on a preset target standard format to obtain data change preprocessing information.

[0016] Optionally, in a fifth implementation of the second aspect of the present invention, the verification and update module includes: a first verification unit, used to perform format verification on the data change preprocessing information to obtain a first verification result; a second verification unit, used to perform integrity verification on the data change preprocessing information to obtain a second verification result when the first verification result passes; a third verification unit, used to perform business rule verification on the data change preprocessing information to obtain a third verification result when the second verification result passes; a fourth verification unit, used to perform data consistency verification on the data change preprocessing information to obtain a fourth verification result when the third verification result passes; and a second writing unit, used to write the data change preprocessing information into the downstream database when the fourth verification result passes.

[0017] Optionally, in a sixth implementation method of the second aspect of the present invention, it also includes: a generation and acquisition module, used to generate a database update record, and obtain an update timestamp from a timestamp service platform based on the update time; a merging module, used to merge the database update record and the update timestamp to obtain merged information; an encryption module, used to encrypt the merged information to obtain encrypted information; and an uploading module, used to upload the encrypted information to the blockchain.

[0018] A third aspect of the present invention provides a configurable database real-time synchronization device, the configurable database real-time synchronization device comprising: a memory and at least one processor, the memory storing instructions; at least one of the processors calling the instructions in the memory so that the configurable database real-time synchronization device executes each step of the configurable database real-time synchronization method described in any one of the above.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored thereon, and when the instructions are executed by a processor, the steps of any of the above-mentioned configurable database real-time synchronization methods are implemented.

[0020] In the technical solution of the present invention, by registering a downstream database connected to a source database, calling a preset data mapping template according to the source database and the downstream database, configuring data mapping rules in the data mapping template, using a database change monitoring mechanism to capture data change information of the source database, writing the data change information into a Redis queue, extracting the data change information from the Redis queue in real time through a script, preprocessing the data change information, and performing data verification on the data change preprocessing information. When the verification result passes, the downstream database is synchronously updated according to the data change preprocessing information. The invention has a high degree of intelligence, reduces resource and technical costs, can support data synchronization of different types of databases or different table structures according to the mapping relationship configuration, and supports synchronization of partial fields, thereby improving the flexibility of function implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A first flow chart of a configurable database real-time synchronization method provided by an embodiment of the present invention;

[0022] Figure 2 A second flow chart of a configurable database real-time synchronization method provided by an embodiment of the present invention;

[0023] Figure 3 A third flow chart of a configurable database real-time synchronization method provided by an embodiment of the present invention;

[0024] Figure 4 A fourth flow chart of a configurable database real-time synchronization method provided by an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of the structure of a configurable database real-time synchronization device provided by an embodiment of the present invention;

[0026] Figure 6 Another structural schematic diagram of a configurable database real-time synchronization device provided by an embodiment of the present invention;

[0027] Figure 7 A schematic diagram of the structure of a configurable database real-time synchronization device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention provides a configurable database real-time synchronization method, device, equipment and storage medium, which are highly intelligent, reduce resource and technical costs, can support different types of databases or different table structure data synchronization according to mapping relationship configuration, and support partial field synchronization, thereby improving the flexibility of function implementation.

[0029] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 An embodiment of a configurable database real-time synchronization method in an embodiment of the present invention includes:

[0031] 101. Define database connection information, and register a downstream database connected to the source database according to the database connection information;

[0032] In this embodiment, the database type, host address, port number, database name, user name and password are configured. Once the database connection information is defined, the next step is to use this information to establish a connection with the downstream database, use the database driver to establish a connection with the database, and initialize the database connection through the incoming configuration information (such as host address, port, user name, password, etc.). Create a database connection pool, which can maintain a certain number of database connections. The application obtains idle connections through the pool for use to avoid frequent establishment and disconnection of connections. When using the connection pool, there is no need to manually close the connection in the connection pool. The pool will automatically manage the life cycle of the connection, create a connection to the downstream database, obtain data and transmit it, and treat the connections of the source database and the downstream database as different database instances in the application to ensure that they can operate independently.

[0033] 102. Call a preset data mapping template according to the source database and the downstream database, and configure data mapping rules in the data mapping template;

[0034] In this embodiment, the data mapping template is a tool or configuration file for defining how to map the data structure in the source database to the data structure in the target database. It is a pre-designed template that contains the mapping relationship between the source field and the target field, as well as possible other conversion rules. The step of configuring the data mapping rules is to ensure that the source data can be correctly processed and adapted when converted to the target data. The data mapping rules are not just simple field matching, but also involve how to perform data type conversion, format conversion, calculation processing, etc. according to business logic or database requirements. In the mapping template, it is first necessary to specify the correspondence between the source field and the target field. If the field name in the target database is different from the field name in the source database, it is necessary to rename the field through the mapping template. Once the data mapping template and rules are configured, these templates can be called through the program and data conversion operations can be performed.

[0035] 103. Use the database change monitoring mechanism to capture data change information of the source database and write the data change information into the Redis queue;

[0036] In this embodiment, the database change monitoring mechanism monitors data change events (such as insertion, update, deletion, etc.) in the source database through specific technical means and captures these changes. Redis is an efficient memory storage system, which can be used as a message queue to process high-concurrency and high-throughput data. After writing the captured data change information into the Redis queue, the system can further asynchronously process, store or transmit this information to other systems.

[0037] 104. Extract data change information from the Redis queue in real time through a script, and pre-process the data change information to obtain data change pre-processing information;

[0038] In this embodiment, the original data change information is extracted from the Redis queue in real time through a script. Once the script successfully extracts the data change information from the Redis queue, the information needs to be preprocessed. The purpose of the preprocessing is to convert the original data change information into a form that is more suitable for subsequent processing, analysis or storage. This process may include data cleaning, data formatting, etc.

[0039] 105. Perform data verification on the data change preprocessing information to obtain a verification result. When the verification result passes, synchronize the downstream database according to the data change preprocessing information;

[0040] In this embodiment, the validity and integrity of the data change preprocessing information are checked to verify whether the data complies with the business rules and data structure requirements. When the data change preprocessing information is verified, the downstream database is synchronously updated according to the data change preprocessing information.

[0041] In an embodiment of the present invention, a downstream database connected to a source database is registered, a preset data mapping template is called according to the source database and the downstream database, data mapping rules are configured in the data mapping template, a database change monitoring mechanism is used to capture data change information of the source database, the data change information is written into a Redis queue, the data change information is extracted from the Redis queue in real time through a script, the data change information is preprocessed, and data verification is performed on the data change preprocessing information. When the verification result passes, the downstream database is synchronously updated according to the data change preprocessing information. The system has a high degree of intelligence, reduces resource and technical costs, can support data synchronization of different types of databases or different table structures according to the mapping relationship configuration, and supports synchronization of partial fields, thereby improving the flexibility of function implementation.

[0042] See also Figure 2 , a second embodiment of the configurable database real-time synchronization method in an embodiment of the present invention includes:

[0043] 201. Define database connection information, which includes database type, database host address, port number, database name, user name and password;

[0044] In this embodiment, the database type, host address, port number, database name, user name and password are configured. The database type determines the database management system used, and the database type determines the connection string and the specific connection method. The database host address refers to the IP address or host name of the server where the database is located. Different database management systems use different port numbers to listen to client requests. The database name refers to the specific database to be connected. The database user name is one of the credentials used for identity authentication. Through the user name, the system can confirm whether the operation has sufficient permissions. The database password is a credential paired with the user name, and the password is used to ensure that only authorized users can access and operate the database.

[0045] 202. Register a downstream database connected to the source database according to the database connection information;

[0046] In this embodiment, the database connection information of the source database and the downstream database is confirmed, the downstream database connection is configured in the system, it is ensured that the downstream database can receive data from the source database, and after registering the connection information of the downstream database, it is verified whether the connection is successful.

[0047] 203. Create a connection pool and use the connection pool to manage the connection between the downstream database and the source database;

[0048] In this embodiment, SQLAlchemy is used to create connection pools for the source database and the downstream database respectively, parameters are configured for the connection pools, and the connection pools are used to manage the connection between the downstream database and the source database.

[0049] 204. Identify downstream table structure information in the downstream database and source table structure information in the source database;

[0050] In this embodiment, the table structure refers to the fields, data types, primary keys, foreign keys, indexes, constraints and other information of the database table. When obtaining the source table structure information and downstream table structure information, SQL query statements can be used directly from the metadata of the database system.

[0051] 205. Compare and analyze the downstream table structure information and the source table structure information to obtain difference structure information;

[0052] In this embodiment, the downstream table structure information and the source table structure information are compared and analyzed. The core of the comparison process is to find the differences between the source table and the downstream table based on their structure information. The main dimensions of the comparison include field differences, constraint differences, and index differences. After the source table and the downstream table structures are compared, the differences are analyzed and recorded. The results of the comparison analysis are recorded as difference structure information, which can be a table that lists the detailed information of each difference item.

[0053] 206. Calling a preset data mapping template according to the difference structure information, and configuring a table mapping relationship and a field mapping relationship in the data mapping template;

[0054] In this embodiment, the role of the data mapping template is to map the data in the source table to the downstream table and ensure the consistency of information such as data types, field names, constraints, etc. during the migration process. The table mapping relationship defines the mapping rules between the source table and the target table. The field mapping relationship configuration is the core part of the data mapping template. The purpose is to ensure that the fields in the source table and the fields in the downstream table can be correctly matched. In the comparison of the source table and the downstream table structure, it may be found that the fields are missing or newly added. These differences need to be processed in the data mapping template. Once the data mapping template configuration is completed, the next step is to call these templates through tools or scripts to automate the data migration process.

[0055] In the embodiment of the present invention, connection information management, table structure comparison and data mapping are all performed through preset templates and automated processes, which greatly reduces manual intervention and improves efficiency and accuracy. Through the use of connection pools, multiple database connections can be effectively managed, connection overhead is reduced, and system performance and stability are improved. By comparing the differences in table structures between the source database and the downstream database, it helps to promptly discover and resolve structural inconsistencies and ensure correct data migration and synchronization. Through data mapping templates, the data mapping relationship between different databases is clear and customizable, which enhances the flexibility and scalability of the system and adapts to the needs of different databases.

[0056] See also Figure 3 A third embodiment of the configurable database real-time synchronization method in the embodiment of the present invention includes:

[0057] 301. Create a listener and use the listener to monitor the data change information of the source database;

[0058] In this embodiment, a trigger is created in the source database. When data is inserted, updated, or deleted, the trigger executes a predefined operation and uses a stream processing framework such as Kafka and Debezium to capture and transmit data change information.

[0059] 302. When the listener monitors the data change information of the source database, the current data change information is obtained;

[0060] In this embodiment, the listener captures data changes by defining trigger rules. For example, when data in a table is inserted, updated, or deleted, the trigger is automatically activated to perform specific operations. When the listener listens to data change information in the source database, it obtains the current data change information.

[0061] 303. Write data change information into the Redis queue;

[0062] In this embodiment, the process of writing captured data changes (such as insert, update, delete and other operations) into the Redis queue will follow the following steps. Before writing the data into the Redis queue, the data needs to be formatted. The formatted data will be pushed to the queue through the Redis command. After writing to the Redis queue, the data will be read from the queue and appropriate processing will be performed according to the data change type (insert, update, delete).

[0063] 304. Obtain data change information from the Redis queue in real time through scripts;

[0064] In this embodiment, first, the script needs to connect to Redis and use the List structure or Stream structure to obtain data changes from the Redis queue in real time. If the amount of data is large, the data changes can be processed in parallel through multi-threading or multi-process to improve processing efficiency.

[0065] 305. Perform data cleaning on the data change information to obtain data change cleaning information;

[0066] In this embodiment, duplicate data in the data change information is removed by checking the unique identifier of the data change information to obtain the data change cleaning information.

[0067] 306. Fill missing values ​​in the data change cleaning information to obtain data change completion information;

[0068] In this embodiment, when certain fields are missing in the data change information, you can choose to fill in a default value, fill in the mean or median of numeric data, fill in the previous value or current time of time data, delete the field, or skip the entire data change.

[0069] 307. Based on a preset target standard format, convert the data change completion information into a format to obtain data change preprocessing information;

[0070] In this embodiment, the target standard format is usually formulated based on business needs and data system requirements to ensure that data from different sources can be processed uniformly. The naming rules of field names and contents are often inconsistent between different data sources. The first step of format conversion is to map these fields and convert the original fields into fields in the target standard format. In addition to the field name and data type, the content of the field also needs to be formatted to meet the requirements of the standard format and obtain data change preprocessing information.

[0071] In the embodiment of the present invention, a listener is created to monitor the data changes of the source database in real time to ensure that any data changes can be responded to quickly, delays are avoided and the real-time nature of data synchronization is ensured. The data change information is written into the Redis queue. As a memory data storage engine, Redis has efficient read and write performance, which is helpful for real-time caching and transmission of data change information, ensuring low latency and high throughput of the system. The data change information is cleaned and missing values ​​are filled, which can effectively remove noise, erroneous data and complete data, ensuring that the downstream system can obtain accurate and complete data. This improves data quality and reduces the risk of subsequent errors caused by data problems. The data change completion information is converted according to a preset target standard format to ensure that the data remains consistent during subsequent use and meets system requirements. This process improves the availability and compatibility of the data and facilitates the processing and analysis of the downstream system. The data change information is obtained in real time through the Redis queue and script, and the process of data cleaning, filling and format conversion is coordinated to ensure that the data flow from the source database to the target system is efficient, accurate and in accordance with the target standard format, thereby enhancing the overall processing capability of the system.

[0072] See also Figure 4 The fourth embodiment of the configurable database real-time synchronization method in the embodiment of the present invention includes:

[0073] 401. Perform format verification on the data change preprocessing information to obtain a first verification result;

[0074] In this embodiment, field validation is the first step of format validation, and its main purpose is to check the match between the name of the original data field and the target standard format. Data type validation is mainly to ensure that the value of each field conforms to its expected data type. An important part of format validation is to check whether all required fields exist and are not empty. For numeric and date fields, it is necessary to further check whether their data is within a reasonable range. The last step of format validation is to ensure that the format of the field content conforms to expectations. After format validation, the first verification result is obtained.

[0075] 402. When the first verification result passes, integrity verification is performed on the data change preprocessing information to obtain a second verification result;

[0076] In this embodiment, when the first verification result passes, it is checked whether all key fields in the data are filled, and duplicate items in the data set are checked, and the second verification result is obtained after integrity verification.

[0077] 403. When the second verification result passes, business rule verification is performed on the data change preprocessing information to obtain a third verification result;

[0078] In this embodiment, when the second verification result passes, the fields are checked item by item to see if they comply with the business rules, the logical consistency between the fields is verified, and the consistency with other data sources is verified. When performing business rule verification, if some data violates the business rules, these errors should be recorded and error prompts should be generated to indicate the erroneous fields and causes. The third verification result is obtained after business rule verification.

[0079] 404. When the third verification result passes, data consistency verification is performed on the data change preprocessing information to obtain a fourth verification result;

[0080] In this embodiment, when the third verification result passes, the data change preprocessing information is subjected to cross-table data consistency verification, cross-system consistency verification, historical data consistency verification, and state consistency verification. After the data consistency verification, the fourth verification result is obtained. The goal of the data consistency verification is to ensure that there is no logical conflict or inconsistency between multiple dimensions of the data.

[0081] 405. When the fourth verification result passes, the data change preprocessing information is written into the downstream database;

[0082] In this embodiment, when the fourth verification result passes, the data change preprocessing information is written into the downstream database, and the writing operation includes insertion and deletion.

[0083] 406. Generate a database update record, and obtain an update timestamp from a timestamp service platform based on the update time;

[0084] In this embodiment, the main purpose of generating database update records is to track and record the modification behavior of the database to ensure the traceability of data changes. The update record includes the data status before and after the change, and records the specific field changes to facilitate backtracking and analysis. When a database update record is generated, a request will be initiated to the timestamp service platform, usually through an API call, to request the service platform to obtain the current standard timestamp.

[0085] 407. Merge the database update record and the update timestamp to obtain merged information;

[0086] In this embodiment, during the database update process, the database record only contains the changed data, and the timestamp is obtained independently. The purpose of the merge operation is to combine these two independent elements into a complete update information record to ensure that each change has a clear time stamp. By merging the update timestamp with the database update record, the user can accurately understand the specific time when each data change occurs. Specifically, a new data structure containing all key information is created. For example, the data contained in the database update record can be stored as an object or row record, and the timestamp will be added as part of the record. The timestamp needs to be converted to an appropriate format and ensure consistency with other record data to avoid data problems caused by time zone or format errors.

[0087] 408. Encrypt the combined information to obtain encrypted information;

[0088] In this embodiment, an asymmetric encryption algorithm is selected to generate a pair of keys, the keys including a public key and a private key. The public key is used to encrypt data, and the private key is used to decrypt. Information is encrypted by combining the public key to obtain encrypted information.

[0089] 409. Upload the encrypted information to the blockchain;

[0090] In this embodiment, a hash value is generated based on the encrypted information, and the hash value is packaged into a transaction. This transaction contains the identity information of the uploader. The transaction is submitted to the node in the blockchain network, and the node begins to verify the validity of the transaction. The verification content includes the transaction signature, the legitimacy of the transaction structure, whether the uploaded data conforms to the format, etc. The blockchain uses a consensus mechanism to ensure the validity of the transaction. After the consensus mechanism is passed, the transaction will be packaged into a block and added to the blockchain.

[0091] In the embodiment of the present invention, the accuracy, security and integrity of the data are ensured through multiple verifications and phased processing. First, errors and non-compliant data are gradually eliminated through layered verification to ensure that only data that meets the standards is processed and stored. Secondly, the data traceability and audit capabilities are enhanced by combining timestamps and database update records. Finally, encryption and blockchain upload ensure the security and non-tamperability of the data, and improve the transparency and credibility of the data. This process effectively improves the efficiency and security of data processing and ensures the traceability and reliability of data changes.

[0092] The above describes the configurable database real-time synchronization method in the embodiment of the present invention. The following describes the configurable database real-time synchronization device in the embodiment of the present invention. Figure 5 An embodiment of a configurable database real-time synchronization device in an embodiment of the present invention includes:

[0093] A definition registration module 501 is used to define database connection information and register a downstream database connected to a source database according to the database connection information;

[0094] A calling configuration module 502 is used to call a preset data mapping template according to a source database and a downstream database, and configure data mapping rules in the data mapping template;

[0095] The capture and write module 503 is used to capture the data change information of the source database using the database change monitoring mechanism and write the data change information into the Redis queue;

[0096] The extraction processing module 504 is used to extract data change information from the Redis queue in real time through a script, and pre-process the data change information to obtain data change pre-processing information;

[0097] The verification and update module 505 is used to perform data verification on the data change preprocessing information to obtain a verification result. When the verification result passes, the downstream database is synchronously updated according to the data change preprocessing information.

[0098] In this embodiment, by registering a downstream database connected to a source database, calling a preset data mapping template according to the source database and the downstream database, configuring data mapping rules in the data mapping template, using a database change monitoring mechanism to capture data change information of the source database, writing the data change information into a Redis queue, extracting the data change information from the Redis queue in real time through a script, preprocessing the data change information, and performing data verification on the data change preprocessing information. When the verification result passes, the downstream database is synchronously updated according to the data change preprocessing information. It has a high degree of intelligence, reduces resource and technical costs, can support data synchronization of different types of databases or different table structures according to the mapping relationship configuration, and supports synchronization of partial fields, thereby improving the flexibility of function implementation.

[0099] See also Figure 6 Another embodiment of the configurable database real-time synchronization device in the embodiment of the present invention includes:

[0100] A definition registration module 501 is used to define database connection information and register a downstream database connected to a source database according to the database connection information;

[0101] A calling configuration module 502 is used to call a preset data mapping template according to a source database and a downstream database, and configure data mapping rules in the data mapping template;

[0102] The capture and write module 503 is used to capture the data change information of the source database using the database change monitoring mechanism and write the data change information into the Redis queue;

[0103] The extraction processing module 504 is used to extract data change information from the Redis queue in real time through a script, and pre-process the data change information to obtain data change pre-processing information;

[0104] The verification and update module 505 is used to perform data verification on the data change preprocessing information to obtain a verification result. When the verification result passes, the downstream database is synchronously updated according to the data change preprocessing information;

[0105] In this embodiment, the registration module 501 is defined to include: a definition unit 5011, which is used to define database connection information, the database connection information includes database type, database host address, port number, database name, user name and password; a registration unit 5012, which is used to register a downstream database connected to a source database according to the database connection information; a creation management unit 5013, which is used to create a connection pool and use the connection pool to manage the connection between the downstream database and the source database.

[0106] In this embodiment, the calling configuration module 502 includes: an identification unit 5021, which is used to identify the downstream table structure information in the downstream database and the source table structure information in the source database; an analysis unit 5022, which is used to compare and analyze the downstream table structure information and the source table structure information to obtain difference structure information; a calling configuration unit 5023, which is used to call a preset data mapping template according to the difference structure information, and configure the table mapping relationship and the field mapping relationship in the data mapping template.

[0107] In this embodiment, the capture write module 503 includes: a creation monitoring unit 5031, used to create a listener, and use the listener to monitor the data change information of the source database; a first acquisition unit 5032, used to obtain the current data change information when the listener monitors the data change information of the source database; a first writing unit 5033, used to write the data change information into the Redis queue.

[0108] In this embodiment, the extraction processing module 504 includes: a second acquisition unit 5041, which is used to obtain data change information from the Redis queue in real time through a script; a cleaning unit 5042, which is used to clean the data change information to obtain data change cleaning information; a filling unit 5043, which is used to fill missing values ​​in the data change cleaning information to obtain data change completion information; a conversion unit 5044, which is used to convert the format of the data change completion information based on a preset target standard format to obtain data change preprocessing information.

[0109] In this embodiment, the verification and update module 505 includes: a first verification unit 5051, which is used to perform format verification on the data change preprocessing information to obtain a first verification result; a second verification unit 5052, which is used to perform integrity verification on the data change preprocessing information when the first verification result passes, and obtain a second verification result; a third verification unit 5053, which is used to perform business rule verification on the data change preprocessing information when the second verification result passes, and obtain a third verification result; a fourth verification unit 5054, which is used to perform data consistency verification on the data change preprocessing information when the third verification result passes, and obtain a fourth verification result; a second writing unit 5055, which is used to write the data change preprocessing information into the downstream database when the fourth verification result passes.

[0110] In this embodiment, it also includes: a generation and acquisition module 506, which is used to generate a database update record and obtain an update timestamp from a timestamp service platform based on the update time; a merging module 507, which is used to merge the database update record and the update timestamp to obtain merged information; an encryption module 508, which is used to encrypt the merged information to obtain encrypted information; and an uploading module 509, which is used to upload the encrypted information to the blockchain.

[0111] above Figure 5 and Figure 6 The configurable database real-time synchronization device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The configurable database real-time synchronization device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0112] Figure 7 6 is a schematic diagram of the structure of a configurable database real-time synchronization device provided by an embodiment of the present invention. The configurable database real-time synchronization device 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 610 (for example, one or more processors) and a memory 620, and one or more storage media 630 (for example, one or more mass storage devices) storing application programs 633 or data 632. Among them, the memory 620 and the storage medium 630 can be short-term storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the configurable database real-time synchronization device 600. Furthermore, the processor 610 may be configured to communicate with the storage medium 630, and execute a series of instruction operations in the storage medium 630 on the configurable database real-time synchronization device 600 to implement the steps of the configurable database real-time synchronization method provided by the above-mentioned various method embodiments.

[0113] The configurable database real-time synchronization device 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input and output interfaces 660, and / or one or more operating systems 631, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 7 The illustrated structure of the configurable real-time database synchronization device does not constitute a limitation on the configurable real-time database synchronization device, and may include more or less components than those illustrated, or may combine certain components, or may have different component arrangements.

[0114] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of a configurable database real-time synchronization method.

[0115] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0117] Finally, it should be noted that the above description is only a preferred example of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A configurable database real-time synchronization method, characterized in that: include: Defining database connection information, and registering a downstream database connected to a source database according to the database connection information; Calling a preset data mapping template according to the source database and the downstream database, and configuring data mapping rules in the data mapping template; Use the database change monitoring mechanism to capture the data change information of the source database and write the data change information into the Redis queue; Extracting the data change information from the Redis queue in real time through a script, and preprocessing the data change information to obtain data change preprocessing information; The data change preprocessing information is subjected to data verification to obtain a verification result, and when the verification result passes, the downstream database is synchronously updated according to the data change preprocessing information.

2. The configurable database real-time synchronization method according to claim 1, characterized in that: The defining of database connection information and registering a downstream database connected to a source database according to the database connection information includes: Define database connection information, which includes database type, database host address, port number, database name, user name and password; Register a downstream database connected to the source database according to the database connection information; A connection pool is created, and the connection pool is used to manage the connection between the downstream database and the source database.

3. The configurable database real-time synchronization method according to claim 1, characterized in that: The calling a preset data mapping template according to the source database and the downstream database, and configuring data mapping rules in the data mapping template, includes: Identifying downstream table structure information in the downstream database and source table structure information in the source database; Comparing and analyzing the downstream table structure information and the source table structure information to obtain difference structure information; A preset data mapping template is called according to the difference structure information, and a table mapping relationship and a field mapping relationship are configured in the data mapping template.

4. The configurable database real-time synchronization method according to claim 1, characterized in that: The method of using a database change monitoring mechanism to capture data change information of the source database and writing the data change information into a Redis queue includes: Creating a listener, and using the listener to monitor data change information of the source database; When the listener listens to the data change information of the source database, the current data change information is obtained; The data change information is written into the Redis queue.

5. The configurable database real-time synchronization method according to claim 1, characterized in that: The step of extracting the data change information from the Redis queue in real time through a script and preprocessing the data change information to obtain data change preprocessing information includes: Obtain the data change information from the Redis queue in real time through a script; Performing data cleaning on the data change information to obtain data change cleaning information; Filling missing values ​​in the data change cleaning information to obtain data change completion information; Based on a preset target standard format, the data change completion information is format converted to obtain data change preprocessing information.

6. The configurable database real-time synchronization method according to claim 1, characterized in that: The data change preprocessing information is subjected to data verification to obtain a verification result, and when the verification result passes, the downstream database is synchronously updated according to the data change preprocessing information, including: Performing format verification on the data change preprocessing information to obtain a first verification result; When the first verification result passes, integrity verification is performed on the data change preprocessing information to obtain a second verification result; When the second verification result passes, business rule verification is performed on the data change preprocessing information to obtain a third verification result; When the third verification result passes, data consistency verification is performed on the data change preprocessing information to obtain a fourth verification result; When the fourth verification result passes, the data change preprocessing information is written into the downstream database.

7. The configurable database real-time synchronization method according to claim 1, characterized in that: The data change preprocessing information is subjected to data verification to obtain a verification result, and when the verification result passes, the downstream database is synchronously updated according to the data change preprocessing information, further comprising: Generate a database update record, and obtain an update timestamp from a timestamp service platform based on the update time; Merging the database update record and the update timestamp to obtain merged information; Encrypting the combined information to obtain encrypted information; The encrypted information is uploaded to the blockchain.

8. A configurable database real-time synchronization device, characterized in that: include: A registration module is defined, which is used to define database connection information, and register a downstream database connected to a source database according to the database connection information; A calling configuration module is used to call a preset data mapping template according to the source database and the downstream database, and configure data mapping rules in the data mapping template; A capture and write module is used to capture data change information of the source database using a database change monitoring mechanism, and write the data change information into a Redis queue; An extraction processing module, used to extract the data change information from the Redis queue in real time through a script, and pre-process the data change information to obtain data change pre-processing information; The verification and update module is used to perform data verification on the data change preprocessing information to obtain a verification result. When the verification result passes, the downstream database is synchronously updated according to the data change preprocessing information.

9. A configurable database real-time synchronization device, characterized in that: The configurable database real-time synchronization device comprises: a memory and at least one processor, wherein instructions are stored in the memory; At least one of the processors calls the instructions in the memory to enable the configurable database real-time synchronization device to perform each step of the configurable database real-time synchronization method according to any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the configurable database real-time synchronization method as described in any one of claims 1 to 7 are implemented.

Images