Business information synchronization method and apparatus, computer device, and storage medium

Abstract

This application relates to a business information synchronization method, apparatus, computer device, and storage medium. The method includes: acquiring the logs of committed transactions from database slave nodes; the logs record committed transaction marker information and all database table change records during the real-time synchronization process from the database master node to the database slave nodes; decomposing the database slave node logs into multiple log fragments, extracting the database table change record information from each log fragment, and converting each database table change record information from the log fragments into a business message based on a pre-established mapping relationship between database tables and business fields; packaging the business messages corresponding to all database table change records corresponding to each transaction marker information into a standard business message array, and pushing the integrated business message array to multiple target systems in real time. This method can improve synchronization timeliness and enable target systems to perceive changes in the core system's table structure in real time without modifying the core system structure.

Description

Technical Field

[0001] This application relates to the field of data synchronization technology, and in particular to methods, apparatus, computer equipment and storage media for synchronizing business information. Background Technology

[0002] The core system handles numerous fundamental business processes and is one of the most basic source systems. With the expansion of its business service scope, the core system interacts with hundreds of related systems, handling a massive volume of business data daily. When related systems need to synchronously perceive parts of the core system's business for operational internal control or regulatory alerts, the existing synchronization solutions are mainly threefold: First, the core system records business information in a business register and synchronizes it to external systems in real time via transaction scheduling on the message service bus, but this increases the load on the core system and the service bus; second, a data extraction operation is deployed on the data exchange platform, directly connecting to the core system's database, extracting the business register on a T+1 basis and pushing it to the target system, which has less impact on the core system but results in a synchronization delay; third, the core system exports business information to files and transfers the files to external systems via FTP / TFTP, but this method has poor scalability due to increasing demands. All existing synchronization solutions impact the core system's structure or synchronization timeliness, and the synchronized business data lacks a unified standard, resulting in poor adaptability to the target system. Summary of the Invention

[0003] Based on this, it is necessary to provide a business information synchronization method, apparatus, computer equipment, and storage medium that does not affect the structure of the core system and can improve the synchronization timeliness and the synchronization adaptability of data between the core system and related systems, in order to address the above-mentioned technical problems.

[0004] Firstly, this application provides a method for synchronizing business information, the method comprising:

[0005] Retrieve the logs of committed transactions from the database slave nodes; the logs contain transaction markers and records of all database table changes during the real-time synchronization process from the database master node to the database slave nodes.

[0006] The database logs from the nodes are broken down into multiple log fragments. The database table change record information of the log fragments is extracted. Based on the pre-established mapping relationship between database tables and business fields, each database table change record information of the log fragment is converted into a business message.

[0007] The business messages corresponding to all database table change records associated with each transaction marker are packaged into a standard business message array, and the integrated business message array is pushed to multiple target systems in real time.

[0008] In one embodiment, the database slave node's log is broken down into multiple log fragments, and the database table change record information of the log fragments is extracted. Based on the pre-established database table and business field mapping relationship, each database table change record information of the log fragment is converted into a business message, including:

[0009] The database logs from the nodes are broken down into multiple log segments, and the log segments are marked using a dual marking method of transaction marking and sequence number marking. Log segments belonging to the same transaction have the same transaction marking information.

[0010] Extract the business fields from the log fragments, assign field values ​​to the changed business fields in the database table records, and use the field values ​​to identify the changes in the log fragments.

[0011] Based on the pre-established mapping relationship between database tables and business fields, determine the database table name corresponding to the business field;

[0012] Each database table change record in the log fragment is converted into a business message; the database table change record information includes the business field, change flag, database table name corresponding to the business field, transaction flag information to which the business field belongs, transaction commit time and serial number.

[0013] In one embodiment, the business fields for extracting log fragments include:

[0014] Read the signal file to obtain the transaction marker information of the log fragments and the largest sequence number marker corresponding to the transaction marker information; the signal file stores the transaction marker information and the largest sequence number marker corresponding to the transaction marker information.

[0015] The number of data extraction subprocesses is determined based on the largest sequence number flag corresponding to the transaction flag information, and each data extraction subprocess reads multiple log fragments;

[0016] Each data extraction subprocess sequentially reads the log fragment of the current transaction's tag information, and simultaneously loads the pre-established mapping relationship between the database table and the business field, extracting the business field of the database table from the log fragment based on the mapping relationship.

[0017] In one embodiment, the business messages corresponding to all database table change records corresponding to each transaction marker information are packaged into a standard business message array, and the integrated business message array is pushed to multiple target systems in real time, including:

[0018] Read the business messages corresponding to all database table change records for each transaction marker information, and sort them by serial number;

[0019] Based on the structural attributes of the database table corresponding to the current transaction marker information, determine whether the database table includes a primary key; if the database table does not include a primary key, determine the final state business message corresponding to all database table change record information corresponding to the current transaction marker information in the database table.

[0020] Extract the transaction flag information, transaction commit time, number of records, and array number from the final state business message as a common information header. Use the business fields and change identifiers in the final state business message as elements of the array to obtain a standard business message array.

[0021] In one embodiment, business messages corresponding to all database table change records corresponding to the same transaction marker information are read and sorted by serial number, including:

[0022] Read the result file to obtain all transaction flag information and the number of database table types corresponding to each transaction flag information; the result file stores all transaction flag information and the number of database table types corresponding to each transaction flag information.

[0023] The number of packaging subprocesses is determined based on the number of database table types corresponding to the current transaction marking information.

[0024] Each packaging subprocess sequentially reads the business message corresponding to the database table change record information corresponding to its current transaction mark information, and sorts them according to the serial number.

[0025] In one embodiment, if the database table includes a primary key, the primary key field value and change identifier of the business message are extracted, and business messages with non-unique primary key field values ​​are filtered to obtain business messages with unique primary key values.

[0026] Determine the change pattern corresponding to the database table change record information, match the change pattern with the preset change pattern, filter out the business messages corresponding to the continuous changes of the database table change record information, and take the business message corresponding to the last change of the database table change record information as the final state business message.

[0027] In one embodiment, if the number of records exceeds a preset value, the final state service messages are integrated into multiple service message arrays, and the data number is recorded.

[0028] Secondly, this application also provides a business information synchronization device. The device includes:

[0029] The acquisition module is used to acquire the logs of committed transactions from the database slave nodes. The logs contain the transaction marker information and all database table change records during the real-time synchronization process from the database master node to the database slave nodes.

[0030] The decomposition module is used to decompose the logs of the database slave node into multiple log fragments, extract the database table change record information of the log fragments, and convert each database table change record information of the log fragment into a business message according to the pre-established database table and business field mapping relationship.

[0031] The push module is used to package the business messages corresponding to all database table change records corresponding to each transaction mark information into a standard business message array, and push the integrated business message array to multiple target systems in real time.

[0032] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0033] Retrieve the logs of committed transactions from the database slave nodes; the logs contain the transaction markers and all table change records during the real-time synchronization process from the database master node to the database slave nodes.

[0034] The database logs from the nodes are broken down into multiple log fragments. The database table change record information of the log fragments is extracted. Based on the pre-established mapping relationship between database tables and business fields, each database table change record information of the log fragment is converted into a business message.

[0035] The business messages corresponding to all database table change records associated with each transaction marker are packaged into a standard business message array, and the integrated business message array is pushed to multiple target systems in real time.

[0036] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0037] Retrieve the logs of committed transactions from the database slave nodes; the logs contain the transaction markers and all table change records during the real-time synchronization process from the database master node to the database slave nodes.

[0038] The database logs from the nodes are broken down into multiple log fragments. The database table change record information of the log fragments is extracted. Based on the pre-established mapping relationship between database tables and business fields, each database table change record information of the log fragment is converted into a business message.

[0039] The business messages corresponding to all database table change records associated with each transaction marker are packaged into a standard business message array, and the integrated business message array is pushed to multiple target systems in real time.

[0040] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0041] Retrieve the logs of committed transactions from the database slave nodes; the logs contain the transaction markers and all table change records during the real-time synchronization process from the database master node to the database slave nodes.

[0042] The database logs from the nodes are broken down into multiple log fragments. The database table change record information of the log fragments is extracted. Based on the pre-established mapping relationship between database tables and business fields, each database table change record information of the log fragment is converted into a business message.

[0043] The business messages corresponding to all database table change records associated with each transaction marker are packaged into a standard business message array, and the integrated business message array is pushed to multiple target systems in real time.

[0044] The aforementioned business information synchronization method, device, computer equipment, and storage medium convert each database table change record in the log fragment into a business message based on a pre-established mapping relationship between database tables and business fields; package the business messages corresponding to all database table change records of a single database table into a standard business message array according to the target transaction tag information; and push the integrated business message array to multiple target systems in real time. In the above method, a master-slave replication relationship is established for the core system database, with the database slave node responsible for data synchronization with the target system. This approach balances the synchronization timeliness and system performance of heterogeneous systems. By clarifying the correspondence between the core system's business database tables and business fields, the data format dependencies between heterogeneous systems are decoupled, freeing them from the constraints of the database master node's table structure. The database master node records are integrated into standard business messages, allowing the target system to perceive the core system's business in real time. The standardized business message array enhances the adaptability of data synchronization between the core system and the target system. New target systems can directly consume the standardized business message array of the core system to achieve synchronization perception, without requiring additional modifications to the core system. This facilitates the interaction between the core system and multiple target systems and enhances the adaptability of data synchronization between the core system and surrounding systems. Attached Figure Description

[0045] Figure 1 This is an application environment diagram of a business information synchronization method in one embodiment;

[0046] Figure 2 This is a flowchart illustrating a business information synchronization method in one embodiment;

[0047] Figure 3This is a schematic diagram illustrating the process of converting each database table change record in a log fragment into a business message in one embodiment.

[0048] Figure 4 This is a schematic diagram of the extraction and packaging process in another embodiment;

[0049] Figure 5 This is a schematic diagram illustrating the process of packaging service messages into a standard array of service messages in one embodiment;

[0050] Figure 6 This is a detailed flowchart illustrating a business information synchronization method in one embodiment;

[0051] Figure 7 This is a structural block diagram of a service information synchronization device in one embodiment;

[0052] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] The business information synchronization method provided in this application embodiment can be applied to, for example, Figure 1In the application environment shown, terminal 102 includes a database master node 104, a database slave node 106, and multiple target systems 108. The database master node 104 uses a same-source database replication mechanism to synchronize the data of the database master node 104 to the database slave node 106 in real time, and then the database slave node 106 completes the data synchronization with the target system 108. The specific steps for database slave node 106 to complete data synchronization with target system 108 are as follows: Obtain the logs of committed transactions from database slave node 106; the logs record the marker information of committed transactions and all database table change records during the real-time synchronization process from database master node 104 to database slave node 106; decompose the logs of database slave node 106 into multiple log fragments, extract the database table change record information from the log fragments, and convert each database table change record information in the log fragment into a business message based on the pre-established database table and business field mapping relationship; respond to the perception request of the target system, package the business messages corresponding to all database table change record information of a single database table into a standard business message array according to the target transaction marker information, and push the integrated business message array to multiple target systems 108 in real time. The terminal 102 can be, but is not limited to, various servers, personal computers, laptops, smartphones, tablets, and portable wearable devices. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc.

[0055] There are three main existing heterogeneous system synchronization solutions: First, the core system records business information in a business register and synchronizes it to peripheral systems in real time via transaction scheduling on the message service bus. However, this increases the load on the core system and the service bus. Second, a data extraction operation is deployed on the data exchange platform, directly connecting to the core system's database. The business register is extracted and pushed to the target system on a T+1 basis, having minimal impact on the core system but resulting in synchronization delays. Third, the core system exports business information to files and transfers the files to peripheral systems via FTP / TFTP. However, this method has poor scalability as demand increases. All existing synchronization solutions impact the structure or synchronization timeliness of the core system, and the synchronized business data lacks a unified standard, resulting in poor adaptability to the target system.

[0056] To address the above problems, in one embodiment, such as Figure 2 As shown, a business information synchronization method is provided. This embodiment illustrates the method applied to terminal 102. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0057] Step 202: Obtain the log of committed transactions from the database slave node; the log contains the transaction marking information and all database table change records during the real-time synchronization process from the database master node to the database slave node.

[0058] The transaction tagging information is used to mark the target transaction, distinguishing it from other transactions. For example, the transaction tagging information could be a transaction number. A database table, also known as a database table, is a collection of two-dimensional arrays used to represent and store relationships between data objects. The database master node stores a large amount of basic business data for the core system. When related systems need to interact with the core system, the core system replicates data from the database master node to the database slave nodes. The database slave nodes can handle read requests; this method avoids data conflict issues.

[0059] Existing heterogeneous system synchronization technologies require modifications to the core system to ensure timely synchronization, inevitably impacting its performance and failing to balance synchronization timeliness with minimizing the impact on the core system. Therefore, this embodiment employs a same-source database replication mechanism to synchronize data from the master node to the slave nodes in real time. The slave nodes then synchronize data with the target system, thereby mitigating the impact of heterogeneous system data synchronization on the core system's master node.

[0060] During the real-time synchronization of data from the database master node to the database slave nodes, the synchronized data includes the database transaction marker information of the database master node, the names of different database tables storing database transaction-related data, and database table change information. The database slave node's log records important data generated during the synchronization process, such as the marker information of committed transactions and change records for all database tables. The marker information for the same database transaction is identical and distributed across different database tables. Table changes include three types: inserts, deletes, and updates. The change information records the changes to the database tables within a certain period, such as the order and timing of inserts, deletes, or updates within that timeframe.

[0061] Specifically, the database master node uses a replication mechanism of the same source database to synchronize the data of the database master node to the database slave node in real time. When the business information records of the database master node change, the change information is synchronized to the database slave node in real time, and the database table change record information of the database master node is synchronized to the database table change record information of the database slave node to obtain the log of the database slave node.

[0062] Step 204: Decompose the database logs from the node into multiple log fragments, extract the database table change record information from the log fragments, and convert each database table change record information from the log fragment into a business message based on the pre-established database table and business field mapping relationship.

[0063] The log fragments record the database table changes synchronized from the master node's logs to the slave nodes. These changes include the transaction markers, the names of the different tables storing transaction-related data, and the specific changes made to those tables. Each change corresponds to a business message. Transaction markers are identical within the same log fragment. For example, log A is broken down into fragments A1 to A4; these fragments share the same transaction markers but have different sequence numbers.

[0064] Because database synchronization leads to high coupling between the target system and the core system, and the target system cannot perceive changes in the core system's table structure in real time, it can easily cause production accidents. Therefore, to solve the above problems, this embodiment sets up a mapping relationship between database tables and business fields to reflect the field relationships in database tables corresponding to different attributes. When a business field is added, deleted, or modified, the database table to which the changed business field belongs can be determined according to the mapping relationship between database tables and business fields. This decouples the data format dependencies between heterogeneous systems, sorts out the correspondence between the core system's business database tables and business fields, freeing them from the constraints of the core system's database table structure, and converts the database table name and related database table change records, such as the changed business fields, into business messages. Thus, the target system can perceive the core system's business in real time.

[0065] Specifically, the binary logs of committed transactions from the database slave nodes are read in real time. These logs are then broken down into independent log segments based on the transaction's tagging information. Log segments containing business database tables are extracted and stored in chronological order of transaction commit time. Database table changes are recorded, and log segments are named using transaction tagging information plus a sequence number. Simultaneously, the transaction tagging information within each log segment, along with the maximum sequence number of that transaction tagging information, is written to a signal file to indicate the completion of log decomposition and sequence number naming. All business messages corresponding to each log segment are written to shared memory, and all transaction tagging information, along with the number of database table types corresponding to each transaction tagging information, is written to a results file.

[0066] Step 206: Package the business messages corresponding to all database table change records corresponding to each transaction mark information into a standard business message array, and push the integrated business message array to multiple target systems in real time.

[0067] One issue is that the lack of a unified standard for the business data synchronized by the core system leads to poor adaptability to the target system. Therefore, to solve the above problem, this embodiment standardizes the business message array, which facilitates the interaction between the core system and multiple target systems and enhances the adaptability of the core system to data synchronization with peripheral systems.

[0068] In the aforementioned business information synchronization method, a master-slave replication relationship is established for the core system database. The database slave node is responsible for synchronizing data with the target system, while also considering the synchronization timeliness and system performance of heterogeneous systems. By sorting out the correspondence between the core system's business database tables and business fields, the data format dependencies between heterogeneous systems are decoupled, freeing them from the constraints of the database master node's table structure. The database master node records are integrated into standard business messages, allowing the target system to perceive the core system's business in real time. The standardized business message array enhances the adaptability of data synchronization between the core system and the target system. New target systems can directly consume the standardized business message array of the core system to achieve synchronization perception, without requiring additional modifications to the core system. This facilitates the interaction needs between the core system and multiple target systems and enhances the adaptability of data synchronization between the core system and surrounding systems.

[0069] In one embodiment, such as Figure 3 As shown, the database logs from the nodes are broken down into multiple log fragments. Database and table change records are extracted from these fragments. Based on a pre-established mapping relationship between database tables and business fields, each database and table change record in the log fragment is converted into a business message, including:

[0070] Step 302: The database slave node logs are split into multiple log fragments, and the log fragments are marked using a dual marking method of transaction marking and sequence number marking. Log fragments belonging to the same transaction have the same transaction marking information.

[0071] The dual marking method, which combines transaction tagging and sequence number marking, assigns the same transaction tag to log segments belonging to the same transaction, and then marks the log segments with sequence numbers according to their commit time. For example, if log segments M1 to M4, which are committed sequentially, belong to the same transaction, then log segments M1 to M4 are marked with the transaction tag m, and log segments M1 to M4 are marked with sequence numbers 1 to 4 respectively. That is, the marking information for log segment M1 is m_1.

[0072] Specifically, the database slave node breaks down the database slave node's log into multiple log fragments, marks the same transaction tag for the same transaction in the log fragments, and marks the log fragments belonging to the same transaction again with sequence numbers according to the order of their commit time. The transaction tag information in the log fragment and the maximum sequence number of the tag information belonging to that transaction are written to the signal file to indicate that the log splitting and sequence numbering operation has been completed.

[0073] Step 304: Extract the business fields of the log fragments, assign field values ​​to the changed business fields in the database table records, and use the field values ​​to identify the changes in the log fragments.

[0074] Specifically, the changed values ​​of each field in each row of the database table are assigned according to the field name-field value format. For example, the field name's value before the change is 0, and the field name's value after the change is 1. The change of the field is determined by judging the change of the field value. If the field value changes, a change identifier is set for the log fragment, and the change identifier is recorded in the change record information of the database table.

[0075] In some embodiments, to improve the efficiency of converting database records into business messages, this embodiment dynamically configures a data extraction process and uses the data extraction process to extract business fields from log fragments, such as... Figure 4 As shown, it includes the following steps:

[0076] Step 1: Read the signal file to obtain the transaction marker information of the log fragment and the largest sequence number marker corresponding to the transaction marker information; the signal file stores the transaction marker information and the largest sequence number marker corresponding to the transaction marker information.

[0077] Step 2: Determine the number of data extraction subprocesses based on the largest sequence number corresponding to the transaction tag information. Each data extraction subprocess reads multiple log segments.

[0078] Among them, the number of extraction subprocesses is less than the largest sequence number flag corresponding to the transaction flag information.

[0079] Step 3: Each data extraction subprocess sequentially reads the log fragment of the current transaction's tag information, and simultaneously loads the pre-established database table and business field mapping relationship, extracting the business fields of the database table from the log fragment based on the mapping relationship.

[0080] Specifically, each data extraction subprocess sequentially reads the log fragment containing the current transaction's tag information, while simultaneously loading the pre-established mapping relationship between the database table and business fields. Based on the mapping relationship, it extracts the business fields, transaction tag information, transaction commit time, change identifier, database table name, and serial number from the log fragment.

[0081] Step 306: Determine the name of the database table corresponding to the business field based on the pre-established mapping relationship between database tables and business fields.

[0082] Specifically, after a field name changes, the database table name, transaction flag information, and transaction commit time corresponding to the business field are determined based on the pre-established mapping relationship between database tables and business fields. A serial number is then assigned to the database table change record information corresponding to the business field.

[0083] Step 308: Convert each database table change record in the log fragment into a business message; the database table change record information includes the business field, change flag, database table name corresponding to the business field, transaction flag information to which the business field belongs, transaction commit time and serial number.

[0084] Specifically, the database slave node determines the database table change record information corresponding to the log segment based on the pre-established mapping relationship between database tables and business fields, integrates the database table change record information into business messages, and business messages corresponding to database table change record information with the same transaction mark information can be associated through transaction mark information; writes the business messages corresponding to all log segments belonging to the same transaction mark information to shared memory, and writes the transaction mark information and the number of database table types with the same transaction mark information to the result file.

[0085] In this embodiment, a mapping relationship between database tables and business fields is established to reflect the field relationships in database tables corresponding to different attributes. When a business field is added, deleted, or modified, the database table to which the changed business field belongs can be determined according to the mapping relationship between database tables and business fields. This decouples the data format dependencies between heterogeneous systems, sorts out the correspondence between the core system's business database tables and business fields, freeing them from the constraints of the core system's database table structure, and converts the database table name and related database table change record information such as the changed business fields into business messages. Thus, the target system can perceive the core system's business in real time.

[0086] In one embodiment, a single transaction may involve changes to multiple records across multiple database tables, with each change corresponding to a business message. If the target system receives individual business messages, the common information fields within these messages become redundant. Without common information fields, the target system struggles to associate records belonging to the same transaction. Furthermore, business message transmission may experience loss or delays, which the target system may not detect, making it difficult to guarantee data consistency. Therefore, to address these issues, this embodiment packages all business messages corresponding to database table change records associated with the same transaction marker information into a standard business message array and pushes the integrated business message array to multiple target systems in real time. Specifically, as shown... Figure 4 As shown, the business messages corresponding to all database table change records associated with each transaction marker are packaged into a standard business message array, and the integrated business message array is pushed to multiple target systems in real time, including:

[0087] Step 502: Read the business messages corresponding to all database table change records corresponding to each transaction marker information, and sort them according to the serial number.

[0088] In this embodiment, the business messages corresponding to all database table change records for each transaction marker are stored in shared memory, while the transaction marker information and the number of database table types corresponding to each transaction marker are stored in the result file. To improve the conversion efficiency from database records to business messages, this embodiment dynamically configures the packaging process. Therefore, as... Figure 4 As shown, this step specifically includes the following steps:

[0089] Step 1: Read the result file to obtain the transaction flag information and the number of database table types corresponding to each transaction flag information; the result file stores all the transaction flag information and the number of database table types corresponding to each transaction flag information.

[0090] Step 2: Determine the number of packaging subprocesses based on the number of database table types corresponding to the current transaction marker information.

[0091] The number of packaging subprocesses is less than the number of database table types corresponding to the current transaction flag information. Each packaging subprocess processes multiple business messages corresponding to database table change records corresponding to the current transaction flag information in priority order.

[0092] Step 3: Each packaging subprocess sequentially reads the business message corresponding to the database table change record information corresponding to its current transaction mark information, and sorts them according to the serial number.

[0093] Step 504: Based on the structural attributes of the database table corresponding to the current transaction marker information, determine whether the database table includes a primary key; if the database table does not include a primary key, determine the final state business message corresponding to all database table change record information corresponding to the current transaction marker information in the database table.

[0094] In a database table, the primary key is a candidate key selected to uniquely identify a row. It primarily serves as a unique identifier. A primary key can consist of one field or multiple fields, referred to as a single-field primary key or a multi-field primary key, respectively. Whether a database table includes a primary key can be determined by loading its structural attributes.

[0095] When the same transaction marker information is stored in multiple database tables, changes to this information can affect multiple tables simultaneously, resulting in multiple business messages. Current heterogeneous system data synchronization involves capturing all change records from the database. However, multiple changes to tables with primary keys generate numerous intermediate records. Pushing these intermediate records presents several problems: First, pushing intermediate records through a distributed system struggles to guarantee the order and continuity of data transmission, especially in high-concurrency application scenarios where data synchronization accuracy is compromised. Second, the core system often needs to synchronize with multiple target systems, requiring all target systems to sort and filter the intermediate data, unnecessarily increasing their workload. Finally, target systems focus on the final state data of the core system, not the process data; therefore, pushing intermediate records reduces the data quality of the core system and increases the workload of the target systems.

[0096] To address the aforementioned issues, in some embodiments, if the database table does not include a primary key, a corresponding final state business message is generated based on all database table change record information corresponding to the current transaction marker information in the database table.

[0097] The changes to database tables include three forms: insert, delete, and update. For database tables that do not have primary keys, a single change record for the same database table within the same transaction corresponds to one final state business message. For example, if the same database table within the same transaction undergoes I insert operations, U update operations, and D delete operations, then the same database table within the same transaction will have I+U+D change records, and each change record corresponds to one final state business message.

[0098] In some embodiments, if the database table includes a primary key, the primary key field value and change identifier of the business message are extracted, and business messages with non-unique primary key field values ​​are filtered to obtain business messages with unique primary key values. Based on the business messages with unique primary key values, the change pattern corresponding to the database table change record information is determined, and the change pattern is matched with a preset change pattern. Business messages corresponding to continuous changes in the database table change record information are filtered out, and the business message corresponding to the last change in the database table change record information is taken as the final state business message.

[0099] For tables with primary keys, table changes must fall into at least four preset change patterns. The first pattern is: first, an insert operation is performed, adding a new business field to the table; then, at least zero update operations are performed on the new business field; finally, a delete operation is performed, deleting all the new business field. Because the newly added business field is deleted at the end, the number of records in the original table remains unchanged, making this an invalid change. This pattern is abbreviated as: 1 insert + 1 update + 1 delete = 0, where 0 represents an invalid change. The second pattern is: first, an insert operation is performed, adding a new business field to the table; then, at least zero update operations are performed on the new business field; finally, no delete operation is performed. This pattern retains the change records for both insert and update operations, therefore, the number of records in the original table changes, making this a valid change. This pattern is abbreviated as: 1 insert + 1 update + 0 delete = 1, where 1 represents a valid change. The third change pattern is as follows: First, no insert operation is performed. Then, at least zero update operations are performed on the original business field. If zero update operations are performed, a final delete operation is performed to delete the original business field. If at least one update operation is performed, a final delete operation is performed to delete the record information of the one update operation. In both cases, the number of records in the original database table changes, which is considered a valid change. This third change pattern is abbreviated as: 0 insert U update 1 delete = 1, where 1 represents a valid change. The fourth change pattern is as follows: First, no insert operation is performed. Then, at least zero update operations are performed on the original business field. Finally, no delete operation is performed. This change pattern retains the change record information of the update operations; therefore, the number of records in the original database table changes, which is considered a valid change. This fourth change pattern is abbreviated as: 0 insert U update 0 delete = 1, where 1 represents a valid change.

[0100] Specifically, if the database table includes a primary key, the primary key field value and change identifier of the business message are extracted, and business messages with non-unique primary key field values ​​are filtered to obtain business messages with unique primary key values. Based on the business messages with unique primary key values, the change pattern corresponding to the database table change record information is determined, and the change pattern is matched with the preset change pattern. If the change pattern is an invalid change, the business message corresponding to the database table change record information is not pushed. If the change pattern is a valid change, the business messages corresponding to consecutive changes in the database table change record information are filtered out, and the business message corresponding to the last change in the database table change record information is taken as the final state business message.

[0101] Step 506: Extract the transaction flag information, transaction commit time, number of records and array number of the final state business message as a common information header, and use the business fields and change identifiers in the final state business message as elements of the array to obtain a standard business message array.

[0102] If the number of records exceeds a preset value, the final-state business messages are integrated into multiple business message arrays, and the data numbers are recorded. Finally, the integrated message arrays are sent to the distributed messaging system, which pushes them to multiple target systems in near real-time. The target systems can associate the messages based on transaction marker information and array numbers, and verify whether the message arrays are lost by checking the number of records.

[0103] In this embodiment, for database tables with primary keys, business messages corresponding to consecutive changes in the database table change record information are filtered out. The business message corresponding to the last change in the database table change record information is taken as the final state business message. This can improve the accuracy of data synchronization between the database slave node and the target system, as well as the data quality of the core system, and reduce the workload of the target system. For database tables without primary keys, each database table change record information corresponds to a final state business message, avoiding the omission of synchronized data. Packing the business messages corresponding to the database table change record information of the same transaction into a business message array can shorten the effective length of the message and effectively ensure the data consistency of the target system.

[0104] In one embodiment, this embodiment also provides a business information synchronization method, such as... Figure 6 As shown, the specific steps include:

[0105] Step 602: Obtain the log of committed transactions from the database slave node; the log contains the transaction marking information and all database table change records during the real-time synchronization process from the database master node to the database slave node.

[0106] Step 604: Decompose the log of the database slave node into multiple log segments, and mark the log segments using a dual marking method of transaction mark and sequence number mark. At the same time, write the transaction mark information in the log segment and the maximum sequence number of the transaction mark information to the signal file.

[0107] Step 606: Read the signal file to obtain the transaction marker information of the log fragment and the largest sequence number marker corresponding to the transaction marker information.

[0108] Step 608: Determine the number of data extraction subprocesses based on the largest sequence number corresponding to the transaction tag information, and each data extraction subprocess reads multiple log segments.

[0109] Step 610: Each data extraction subprocess sequentially reads the log fragment of the current transaction tag information, and simultaneously loads the pre-established database table and business field mapping relationship, and extracts the business fields of the database table from the log fragment according to the mapping relationship.

[0110] Step 612: Determine the database table name corresponding to the business field based on the pre-established mapping relationship between database tables and business fields.

[0111] Step 614: Convert each database table change record in the log fragment into a business message; the database table change record information includes the business field, change flag, database table name corresponding to the business field, transaction flag information to which the business field belongs, transaction commit time and serial number.

[0112] Step 616: Read the result file to obtain the transaction flag information and the number of database table types corresponding to the transaction flag information; the result file stores all the transaction flag information and the number of database table types corresponding to each transaction flag information.

[0113] Step 618: Determine the number of packaging subprocesses based on the number of database table types corresponding to the current transaction tag information.

[0114] Step 620: Each packaging subprocess sequentially reads the business message corresponding to the database table change record information corresponding to its current transaction mark information, and sorts them according to the serial number.

[0115] Step 622: Determine whether the database table includes a primary key based on the structural attributes of the database table corresponding to the current transaction marker information; if the database table does not include a primary key, proceed to step 624; if the database table includes a primary key, proceed to step 626.

[0116] Step 624: Generate the corresponding final state business message based on all database table change record information corresponding to the current transaction mark information in the database table, and then execute step 628.

[0117] Step 626: Extract the primary key field value and change identifier of the business message, filter out business messages with non-unique primary key field values, and obtain business messages with unique primary key values; determine the change pattern corresponding to the database table change record information, match the change pattern with the preset change pattern, filter out business messages corresponding to continuous changes in the database table change record information, and take the business message corresponding to the last change in the database table change record information as the final state business message.

[0118] Step 628: Extract the transaction marker information, transaction commit time, number of records and array number of the final state business message as a common information header, and use the business fields and change identifiers in the final state business message as elements of the array to obtain a standard business message array.

[0119] In this embodiment, to address the real-time perception problem of heterogeneous systems, and considering both synchronization timeliness and system performance, data synchronization between systems is migrated to the slave database node of the core system. This ensures synchronization timeliness while avoiding impact on the performance of the core system's master node. Data format dependencies between heterogeneous systems are decoupled, and the mapping relationship between the core system's business database tables and business fields is streamlined, freeing them from the constraints of the core system's database table structure. Database records are integrated into standard business messages, enabling target systems to perceive the core system's business in real time. The data extraction and packaging processes are dynamically configured to improve the conversion efficiency from database records to business messages. A standard business message specification is established for the core system, enhancing the adaptability of data synchronization between the core system and surrounding systems. New target systems can directly consume the core system's standard messages to achieve synchronous perception, without requiring additional modifications to the core system.

[0120] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0121] Based on the same inventive concept, this application also provides a business information synchronization device for implementing the business information synchronization method described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more business information synchronization device embodiments provided below can be found in the limitations of the business information synchronization method described above, and will not be repeated here.

[0122] In one embodiment, such as Figure 7 As shown, a business information synchronization device is provided. The device includes:

[0123] The acquisition module 100 is used to acquire the logs of committed transactions of the database slave node; the logs record the transaction marking information and all database table change records during the real-time synchronization process from the database master node to the database slave node;

[0124] The disassembly module 200 is used to disassemble the logs of the database slave node into multiple log fragments, extract the database table change record information of the log fragments, and convert each database table change record information of the log fragment into a business message according to the pre-established database table and business field mapping relationship.

[0125] The push module 300 is used to package the business messages corresponding to all database table change records corresponding to each transaction mark information into a standard business message array, and push the integrated business message array to multiple target systems in real time.

[0126] In one embodiment, the disassembly module 200 is further configured to: disassemble the logs of the database slave node into multiple log fragments, and mark the log fragments using a dual marking method of transaction marking and sequence number marking, wherein log fragments belonging to the same transaction have the same transaction marking information;

[0127] Extract the business fields from the log fragments, assign field values ​​to the changed business fields in the database table records, and use the field values ​​to identify the changes in the log fragments.

[0128] Based on the pre-established mapping relationship between database tables and business fields, determine the database table name corresponding to the business field;

[0129] Each database table change record in the log fragment is converted into a business message; the database table change record information includes the business field, change flag, database table name corresponding to the business field, transaction flag information to which the business field belongs, transaction commit time and serial number.

[0130] In one embodiment, the disassembly module 200 is further configured to: read the signal file to obtain the transaction marker information of the log fragment and the largest sequence number marker corresponding to the transaction marker information; the signal file stores the transaction marker information and the largest sequence number marker corresponding to the transaction marker information;

[0131] The number of data extraction subprocesses is determined based on the largest sequence number flag corresponding to the transaction flag information, and each data extraction subprocess reads multiple log fragments;

[0132] Each data extraction subprocess sequentially reads the log fragment of the current transaction's tag information, and simultaneously loads the pre-established mapping relationship between the database table and the business field, extracting the business field of the database table from the log fragment based on the mapping relationship.

[0133] In one embodiment, the push module 300 is further configured to: read the business messages corresponding to all database table change record information corresponding to each transaction marker information, and sort them according to the serial number;

[0134] Based on the structural attributes of the database table corresponding to the current transaction marker information, determine whether the database table includes a primary key; if the database table does not include a primary key, determine the final state business message corresponding to all database table change record information corresponding to the current transaction marker information in the database table.

[0135] Extract the transaction flag information, transaction commit time, number of records, and array number from the final state business message as a common information header. Use the business fields and change identifiers in the final state business message as elements of the array to obtain a standard business message array.

[0136] In one embodiment, the push module 300 is further configured to: read the result file to obtain all transaction marker information and the number of database table types corresponding to each transaction marker information; the result file stores all transaction marker information and the number of database table types corresponding to each transaction marker information;

[0137] The number of packaging subprocesses is determined based on the number of database table types corresponding to the current transaction marking information.

[0138] Each packaging subprocess sequentially reads the business message corresponding to the database table change record information corresponding to its current transaction mark information, and sorts them according to the serial number.

[0139] In one embodiment, the push module 300 is further configured to: when the database table includes a primary key, extract the primary key field value and change identifier of the business message, filter out business messages with non-unique primary key field values, and obtain business messages with unique primary key values;

[0140] Determine the change pattern corresponding to the database table change record information, match the change pattern with the preset change pattern, filter out the business messages corresponding to the continuous changes of the database table change record information, and take the business message corresponding to the last change of the database table change record information as the final state business message.

[0141] In one embodiment, the push module 300 is further configured to: integrate the final state service messages into multiple service message arrays and record the data number when the number of records exceeds a preset value.

[0142] Each module in the aforementioned business information synchronization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0143] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a business information synchronization method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0144] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0145] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0146] Retrieve the logs of committed transactions from the database slave nodes; the logs contain the transaction markers and all table change records during the real-time synchronization process from the database master node to the database slave nodes.

[0147] The database logs from the nodes are broken down into multiple log fragments. The database table change record information of the log fragments is extracted. Based on the pre-established mapping relationship between database tables and business fields, each database table change record information of the log fragment is converted into a business message.

[0148] The business messages corresponding to all database table change records associated with each transaction marker are packaged into a standard business message array, and the integrated business message array is pushed to multiple target systems in real time.

[0149] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0150] The database logs from the nodes are broken down into multiple log segments, and the log segments are marked using a dual marking method of transaction marking and sequence number marking. Log segments belonging to the same transaction have the same transaction marking information.

[0151] Extract the business fields from the log fragments, assign field values ​​to the changed business fields in the database table records, and use the field values ​​to identify the changes in the log fragments.

[0152] Based on the pre-established mapping relationship between database tables and business fields, determine the database table name corresponding to the business field;

[0153] Each database table change record in the log fragment is converted into a business message; the database table change record information includes the business field, change flag, database table name corresponding to the business field, transaction flag information to which the business field belongs, transaction commit time and serial number.

[0154] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0155] Read the signal file to obtain the transaction marker information of the log fragments and the largest sequence number marker corresponding to the transaction marker information; the signal file stores the transaction marker information and the largest sequence number marker corresponding to the transaction marker information.

[0156] The number of data extraction subprocesses is determined based on the largest sequence number flag corresponding to the transaction flag information, and each data extraction subprocess reads multiple log fragments;

[0157] Each data extraction subprocess sequentially reads the log fragment of the current transaction's tag information, and simultaneously loads the pre-established mapping relationship between the database table and the business field, extracting the business field of the database table from the log fragment based on the mapping relationship.

[0158] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0159] Read the business messages corresponding to all database table change records for each transaction marker information, and sort them by serial number;

[0160] Based on the structural attributes of the database table corresponding to the current transaction marker information, determine whether the database table includes a primary key; if the database table does not include a primary key, determine the final state business message corresponding to all database table change record information corresponding to the current transaction marker information in the database table.

[0161] Extract the transaction flag information, transaction commit time, number of records, and array number from the final state business message as a common information header. Use the business fields and change identifiers in the final state business message as elements of the array to obtain a standard business message array.

[0162] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0163] Read the result file to obtain all transaction flag information and the number of database table types corresponding to each transaction flag information; the result file stores all transaction flag information and the number of database table types corresponding to each transaction flag information.

[0164] The number of packaging subprocesses is determined based on the number of database table types corresponding to the current transaction marking information.

[0165] Each packaging subprocess sequentially reads the business message corresponding to the database table change record information corresponding to its current transaction mark information, and sorts them according to the serial number.

[0166] In one embodiment, if the database table includes a primary key, the processor, when executing the computer program, also performs the following steps:

[0167] Extract the primary key field value and change identifier of the business message, filter out business messages with non-unique primary key field values, and obtain business messages with unique primary key values;

[0168] Determine the change pattern corresponding to the database table change record information, match the change pattern with the preset change pattern, filter out the business messages corresponding to the continuous changes of the database table change record information, and take the business message corresponding to the last change of the database table change record information as the final state business message.

[0169] In one embodiment, if the number of records exceeds a preset value, the processor, when executing the computer program, further implements the following steps: integrating the final state service messages into multiple service message arrays and recording the data numbers.

[0170] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0171] Retrieve the logs of committed transactions from the database slave nodes; the logs contain the transaction markers and all table change records during the real-time synchronization process from the database master node to the database slave nodes.

[0172] The database logs from the nodes are broken down into multiple log fragments. The database table change record information of the log fragments is extracted. Based on the pre-established mapping relationship between database tables and business fields, each database table change record information of the log fragment is converted into a business message.

[0173] The business messages corresponding to all database table change records associated with each transaction marker are packaged into a standard business message array, and the integrated business message array is pushed to multiple target systems in real time.

[0174] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0175] The database logs from the nodes are broken down into multiple log segments, and the log segments are marked using a dual marking method of transaction marking and sequence number marking. Log segments belonging to the same transaction have the same transaction marking information.

[0176] Extract the business fields from the log fragments, assign field values ​​to the changed business fields in the database table records, and use the field values ​​to identify the changes in the log fragments.

[0177] Based on the pre-established mapping relationship between database tables and business fields, determine the database table name corresponding to the business field;

[0178] Each database table change record in the log fragment is converted into a business message; the database table change record information includes the business field, change flag, database table name corresponding to the business field, transaction flag information to which the business field belongs, transaction commit time and serial number.

[0179] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0180] Read the signal file to obtain the transaction marker information of the log fragments and the largest sequence number marker corresponding to the transaction marker information; the signal file stores the transaction marker information and the largest sequence number marker corresponding to the transaction marker information.

[0181] The number of data extraction subprocesses is determined based on the largest sequence number flag corresponding to the transaction flag information, and each data extraction subprocess reads multiple log fragments;

[0182] Each data extraction subprocess sequentially reads the log fragment of the current transaction's tag information, and simultaneously loads the pre-established mapping relationship between the database table and the business field, extracting the business field of the database table from the log fragment based on the mapping relationship.

[0183] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0184] Read the business messages corresponding to all database table change records for each transaction marker information, and sort them by serial number;

[0185] Based on the structural attributes of the database table corresponding to the current transaction marker information, determine whether the database table includes a primary key; if the database table does not include a primary key, determine the final state business message corresponding to all database table change record information corresponding to the current transaction marker information in the database table.

[0186] Extract the transaction flag information, transaction commit time, number of records, and array number from the final state business message as a common information header. Use the business fields and change identifiers in the final state business message as elements of the array to obtain a standard business message array.

[0187] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0188] Read the result file to obtain all transaction flag information and the number of database table types corresponding to each transaction flag information; the result file stores all transaction flag information and the number of database table types corresponding to each transaction flag information.

[0189] The number of packaging subprocesses is determined based on the number of database table types corresponding to the current transaction marking information.

[0190] Each packaging subprocess sequentially reads the business message corresponding to the database table change record information corresponding to its current transaction mark information, and sorts them according to the serial number.

[0191] In one embodiment, if the database table includes a primary key, the computer program, when executed by the processor, also performs the following steps:

[0192] Extract the primary key field value and change identifier of the business message, filter out business messages with non-unique primary key field values, and obtain business messages with unique primary key values;

[0193] Determine the change pattern corresponding to the database table change record information, match the change pattern with the preset change pattern, filter out the business messages corresponding to the continuous changes of the database table change record information, and take the business message corresponding to the last change of the database table change record information as the final state business message.

[0194] In one embodiment, if the number of records exceeds a preset value, the computer program, when executed by the processor, further performs the following steps: integrating the final state service messages into multiple service message arrays and recording the data numbers.

[0195] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0196] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0197] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0198] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0199] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A service information synchronization method, characterized by, The method includes: Obtain the log of committed transactions from the database slave node; the log records the transaction marking information and all database table change records during the real-time synchronization process from the database master node to the database slave node; The database logs from the nodes are broken down into multiple log fragments. The database table change record information of the log fragments is extracted. Based on the pre-established mapping relationship between database tables and business fields, each database table change record information of the log fragment is converted into a business message. Package the business messages corresponding to all database table change records corresponding to each of the aforementioned transaction marker information into a standard business message array, and push the integrated business message array to multiple target systems in real time; The step of packaging the business messages corresponding to all database table change records corresponding to each of the transaction marker information into a standard business message array, and pushing the integrated business message array to multiple target systems in real time, includes: Read the business messages corresponding to all database table change records for each transaction marker information, and sort them by serial number; Based on the structural attributes of the database table corresponding to the current transaction marking information, determine whether the database table includes a primary key; if the database table does not include a primary key, then determine the final state business message corresponding to all database table change record information corresponding to the current transaction marking information in the database table. If the database table includes a primary key, then extract the primary key field value and change identifier of the business message, filter out business messages with non-unique primary key field values, and obtain business messages with unique primary key values. Determine the change pattern corresponding to the database table change record information, match the change pattern with the preset change pattern, if the change pattern is an invalid change, do not push the business message of the database table change record information; if the change pattern is a valid change, filter out the business messages corresponding to the continuous changes of the database table change record information, and take the business message corresponding to the last change of the database table change record information as the final state business message. Extract the transaction marker information, transaction commit time, number of records, and array number from the final state service message as a common information header, and use the service fields and change identifiers in the final state service message as elements of the array to obtain a standard service message array.

2. The method of claim 1, wherein, The process involves breaking down the database node's logs into multiple log fragments, extracting database table change records from each fragment, and converting each database table change record from the log fragment into a business message based on a pre-established mapping relationship between database tables and business fields. This includes: The database logs from the nodes are broken down into multiple log segments, and the log segments are marked using a dual marking method of transaction marking and sequence number marking. Log segments belonging to the same transaction have the same transaction marking information. Extract the business fields of the log fragments, assign field values ​​to the changed business fields in the database table records, and use the field values ​​to identify the change flag of the log fragments; Based on the pre-established mapping relationship between database tables and business fields, determine the database table name corresponding to the business field; Each database table change record in the log fragment is converted into a business message; the database table change record information includes the business field, the change flag, the database table name corresponding to the business field, the transaction flag information to which the business field belongs, the transaction commit time and serial number.

3. The method of claim 2, wherein, The business fields for extracting log fragments include: Read the signal file to obtain the transaction marker information of the log segment and the largest sequence number marker corresponding to the transaction marker information; the signal file stores the transaction marker information and the largest sequence number marker corresponding to the transaction marker information. The number of data extraction subprocesses is determined based on the largest sequence number flag corresponding to the transaction flag information, and each data extraction subprocess reads multiple log segments; Each of the data extraction subprocesses sequentially reads the log fragment of the transaction tag information to which it belongs, and simultaneously loads the pre-established mapping relationship between the database table and the business field, and extracts the business field of the database table from the log fragment according to the mapping relationship.

4. The method of claim 1, wherein, The step of reading all database table change record information corresponding to the same transaction marker information and sorting them by serial number includes: Read the result file to obtain all transaction flag information and the number of database table types corresponding to each transaction flag information; the result file stores all transaction flag information and the number of database table types corresponding to each transaction flag information. The number of packaging subprocesses is determined based on the number of database table types corresponding to the current transaction marking information. Each of the packaging subprocesses sequentially reads the business message corresponding to the database table change record information corresponding to its current transaction mark information, and sorts them according to the serial number.

5. The method of claim 1, wherein, If the number of records exceeds a preset value, the final state service messages are integrated into multiple service message arrays, and the data numbers are recorded.

6. A business information synchronization device, characterized in that, The device includes: The acquisition module is used to acquire the logs of committed transactions from the database slave nodes. The logs contain the transaction marker information and all database table change records during the real-time synchronization process from the database master node to the database slave nodes. The decomposition module is used to decompose the logs of the database slave node into multiple log fragments, extract the database table change record information of the log fragments, and convert each database table change record information of the log fragment into a business message according to the pre-established database table and business field mapping relationship. The push module is used to package the business messages corresponding to all database table change records corresponding to each transaction mark information into a standard business message array, and push the integrated business message array to multiple target systems in real time. The push module is used for: Read the business messages corresponding to all database table change records for each transaction marker information, and sort them by serial number; Based on the structural attributes of the database table corresponding to the current transaction marking information, determine whether the database table includes a primary key; if the database table does not include a primary key, then determine the final state business message corresponding to all database table change record information corresponding to the current transaction marking information in the database table. If the database table includes a primary key, then extract the primary key field value and change identifier of the business message, filter out business messages with non-unique primary key field values, and obtain business messages with unique primary key values. Determine the change pattern corresponding to the database table change record information, match the change pattern with the preset change pattern, if the change pattern is an invalid change, do not push the business message of the database table change record information; if the change pattern is a valid change, filter out the business messages corresponding to the continuous changes of the database table change record information, and take the business message corresponding to the last change of the database table change record information as the final state business message. Extract the transaction marker information, transaction commit time, number of records, and array number from the final state service message as a common information header, and use the service fields and change identifiers in the final state service message as elements of the array to obtain a standard service message array.

7. The apparatus according to claim 6, characterized in that, The disassembly module is used for: The database logs from the nodes are broken down into multiple log segments, and the log segments are marked using a dual marking method of transaction marking and sequence number marking. Log segments belonging to the same transaction have the same transaction marking information. Extract the business fields of the log fragments, assign field values ​​to the changed business fields in the database table records, and use the field values ​​to identify the change flag of the log fragments; Based on the pre-established mapping relationship between database tables and business fields, determine the database table name corresponding to the business field; Each database table change record in the log fragment is converted into a business message; the database table change record information includes the business field, the change flag, the database table name corresponding to the business field, the transaction flag information to which the business field belongs, the transaction commit time and serial number.

8. The apparatus according to claim 7, characterized in that, The disassembly module is used for: Read the signal file to obtain the transaction marker information of the log segment and the largest sequence number marker corresponding to the transaction marker information; the signal file stores the transaction marker information and the largest sequence number marker corresponding to the transaction marker information. The number of data extraction subprocesses is determined based on the largest sequence number flag corresponding to the transaction flag information, and each data extraction subprocess reads multiple log segments; Each of the data extraction subprocesses sequentially reads the log fragment of the transaction tag information to which it belongs, and simultaneously loads the pre-established mapping relationship between the database table and the business field, and extracts the business field of the database table from the log fragment according to the mapping relationship.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images