Message processing method and apparatus

Abstract

Embodiments of the present application provide a message processing method and device, the method comprising: receiving a to-be-processed message sent by a source database, the to-be-processed message being generated when a target data stored in the source database is operated to change the target data; parsing the to-be-processed message to obtain a timestamp and operation information on the target data, the timestamp being used to indicate a time when the target data is operated to change the target data; determining to-be-operated data in a target database according to the timestamp in the to-be-processed message and primary key information corresponding to the target data; and operating and processing the to-be-operated data according to the timestamp and the operation information, so as to synchronize data in the target database with data in the source database. Embodiments of the present application can solve the problem of data operation error in the target database caused by disorder of message sequence, so that data in the final target database is consistent with data in the source database.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data transmission, and in particular to a message processing method and device. BACKGROUND

[0002] With the rapid development of information technology and the rise of big data and artificial intelligence, data as intangible assets is increasingly valued by enterprises. In the face of the exponential growth of massive data, traditional single-machine storage systems have been unable to support business development due to limited storage and computing capacity. Therefore, distributed storage systems with multiple storage nodes have emerged. The distributed storage system includes a master node and a slave node. The master node can accept external request messages, process data by delivering the messages to slave nodes that meet the request conditions, and ensure the efficiency of the distributed storage system.

[0003] In related technologies, with the continuous development of various storage systems, more and more data needs to be shared and backed up between multiple database systems to fully and effectively utilize the data and ensure the reliability of the data. When the source database changes, the data of the target database also needs to be updated at the same time, so as to ensure the consistency of the data. Compared with the traditional file synchronization method, messages require less storage space and higher transmission efficiency, and have become the main way of synchronization between systems.

[0004] However, in the actual message transmission process, the order of the messages received by the target database may not be consistent with the order in which the messages are generated, resulting in message out-of-order problems, which causes the final data of the target database to be inconsistent with the data stored in the source database. Therefore, how to solve the problem of inconsistency between the final data of the target database and the data stored in the source database caused by message out-of-order during data transmission has become a problem to be solved. SUMMARY

[0005] Embodiments of the present application provide a message processing method and device to solve the problem of inconsistency between the final data of the target database and the data stored in the source database caused by message out-of-order in the prior art.

[0006] A first aspect of embodiments of the present application provides a message processing method, comprising:

[0007] receiving a to-be-processed message sent by a source database, the to-be-processed message being generated when a target data stored in the source database is operated to change the target data;

[0008] parsing the to-be-processed message to obtain a timestamp and operation information on the target data, the timestamp being used to indicate a time when the target data is operated to change the target data;

[0009] According to the time stamp in the to-be-processed message and the primary key information corresponding to the target data, determine to-be-operated data in a target database;

[0010] According to the time stamp and the operation information, perform operation processing on the to-be-operated data, so as to synchronize data in the target database with data in the source database.

[0011] Optionally, one primary key information corresponds to at least one data life cycle, and each data life cycle includes a first time stamp, a second time stamp, a data field value corresponding to the primary key information, a third time stamp corresponding to a last operation time of each field, and an indication value corresponding to the primary key information.

[0012] The first time stamp is a time stamp corresponding to a new creation of the data, and the second time stamp is a time stamp corresponding to an invalidation of the data or a maximum value of a current system time.

[0013] The indication value is used to indicate whether the primary key information is modified, if the indication value is empty, it indicates that the primary key information is not modified, and if the indication value is not empty, the indication value is modified primary key information.

[0014] Optionally, the determining to-be-operated data in a target database according to the time stamp in the to-be-processed message and the primary key information corresponding to the target data includes:

[0015] According to the time stamp in the to-be-processed message and the primary key information corresponding to the target data, determine a target life cycle corresponding to the primary key information to which the to-be-processed message belongs;

[0016] Determine a value of each field in the target life cycle or the primary key information as to-be-operated data.

[0017] Optionally, the operation information includes insertion operation information, the insertion operation information includes primary key information of to-be-inserted data and field values of the to-be-inserted data, and the operation processing on the to-be-operated data according to the time stamp and the operation information includes:

[0018] According to the primary key information of the to-be-inserted data and the field values of the to-be-inserted data, create new data;

[0019] Generate a first life cycle corresponding to the primary key information, a first time stamp and a third time stamp of the first life cycle are the time stamp in the to-be-processed message, a second time stamp of the first life cycle is a maximum value of a current system time, a data field value corresponding to the primary key information is the field value of the to-be-inserted data, and an indication value corresponding to the primary key information in the first life cycle is empty.

[0020] Optionally, the operation information includes data update operation information, the data update operation information is used for indicating modification of a data field value, and the operation processing of the to-be-operated data according to the timestamp and the operation information includes:

[0021] If the indication value corresponding to the primary key information in the target life cycle is empty, and the timestamp of the to-be-processed message is greater than the third timestamp corresponding to the last operation time of the field in the target life cycle, the data field value corresponding to the primary key information is modified according to the data update operation information;

[0022] If the indication value corresponding to the primary key information in the target life cycle is empty, and the timestamp of the to-be-processed message is less than the third timestamp corresponding to the last operation time of the field in the target life cycle, the data field value corresponding to the primary key information remains unchanged;

[0023] If the indication value corresponding to the primary key information in the target life cycle is not empty, the data storage position corresponding to the modified primary key information is determined according to the indication value, and the data corresponding to the modified primary key information is modified according to the data update operation information.

[0024] Optionally, the operation information includes deletion operation information, and the operation processing of the to-be-operated data according to the timestamp and the operation information includes:

[0025] The second timestamp in the target life cycle is set as the timestamp in the to-be-processed message, so that the to-be-operated data becomes invalid data.

[0026] Optionally, the operation information includes primary key update operation information, the primary key update operation information is used for indicating modification of primary key information, and the operation processing of the to-be-operated data according to the timestamp and the operation information includes:

[0027] The indication value corresponding to the primary key information in the target life cycle is modified according to the primary key update operation information, and the second timestamp in the target life cycle is set as the timestamp in the to-be-processed message;

[0028] Data corresponding to the modified primary key information is created, and a new data life cycle corresponding to the modified primary key information is generated.

[0029] The second aspect of the embodiment of the application provides a message synchronization processing device, which includes:

[0030] The receiving module is used for receiving a to-be-processed message sent by a source database, the to-be-processed message being generated when target data stored in the source database is operated to change the target data;

[0031] a parsing module, configured to parse the to-be-processed message to obtain a timestamp and operation information on the target data, the timestamp being used to represent a time when the target data is changed by the operation on the target data;

[0032] a determining module, configured to determine to-be-operated data in a target database according to the timestamp in the to-be-processed message and primary key information corresponding to the target data;

[0033] a synchronizing module, configured to perform operation processing on the to-be-operated data according to the timestamp and the operation information, so as to synchronize data in the target database with data in the source database.

[0034] A third aspect of the embodiment of the present application provides a computer device, comprising at least one processor and a memory;

[0035] The memory stores computer execution instructions;

[0036] The at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the message processing method in the first aspect of the embodiment of the present application.

[0037] A fourth aspect of the embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing computer execution instructions, when a processor executes the computer execution instructions, the message processing method in the first aspect of the embodiment of the present application is realized.

[0038] A fifth aspect of the embodiment of the present application provides a computer program product, comprising a computer program, when the computer program is executed by a processor, the message processing method in the first aspect of the embodiment of the present application is realized.

[0039] The embodiment of the present application provides a message processing method and device, wherein the method receives a to-be-processed message sent by a source database, the to-be-processed message is generated when a target data stored in the source database is operated to change the target data, since the to-be-processed message contains a timestamp of operating the target data to change the target data and specific operation information of the target data, therefore, after the target database analyzes the to-be-processed message to obtain the timestamp and the operation information, the target data to be operated can be determined according to the timestamp and primary key information corresponding to the target data, since the primary key information can be used to identify the data in the target database and the value of the timestamp is strictly increased with time, therefore, the actual order of the message generated by the source database, i.e. the actual order of the data change in the source database, can be accurately obtained through the timestamp and the primary key information, therefore, the data in the target database can be accurately synchronized with the data in the source database by operating and processing the to-be-processed data according to the timestamp and the operation information. The problem that the data in the target database is inconsistent with the data in the source database due to the operation error of the data in the target database caused by the disorder of the message received by the target database is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, hereinafter, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0041] Figure 1 is an application scenario diagram of the message processing method shown by an exemplary embodiment of the present application;

[0042] Figure 2 is a flowchart of the message processing method shown by an exemplary embodiment of the present application;

[0043] Figure 3 is a schematic diagram of a data life cycle shown by an exemplary embodiment of the present application;

[0044] Figure 4a is a schematic diagram of a data life cycle shown by another exemplary embodiment of the present application;

[0045] Figure 4b is a schematic diagram of a data life cycle shown by another exemplary embodiment of the present application;

[0046] Figure 5 is a structural schematic diagram of the message processing device shown by an exemplary embodiment of the present application;

[0047] Figure 6 Figure 1 is a structural schematic diagram of a computer device according to an example embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0049] The terms "first", "second", "third", "fourth" and the like (if any) in the description, claims and above drawings of the present application are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units need not be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0050] With the rapid development of information technology and the rise of big data and artificial intelligence, data as intangible assets is increasingly valued by enterprises. In the face of the exponential growth of massive data, traditional single-machine storage systems have been unable to support business development due to limited storage and computing capacity. Therefore, distributed storage systems with multiple storage nodes have emerged. The distributed storage system includes a master node and a slave node. The master node can accept external request messages, process data by delivering the messages to slave nodes that meet the request conditions, and ensure the efficiency of the distributed storage system.

[0051] With the continuous development of various storage systems, more and more data needs to be shared and backed up between multiple database systems to fully and effectively utilize the data and ensure the reliability of the data. When the source database changes, the data of the target database also needs to be updated at the same time, so as to ensure the consistency of the data. Compared with the traditional file synchronization mode, messages need less storage space and higher transmission efficiency, and have become the main mode of synchronization between systems. Taking the source database as Oracle as an example, when the data in the source database changes, the data replication component (Oracle Golden Gate, Ogg) can record the change message of the data, and send the message with a time stamp (SCN) to the distributed subscription message system for storage, and then the distributed subscription message system sends the message sent by the source database to the target database in batches, and the target database works asynchronously, and updates the corresponding data in the target database according to the received message. However, in the actual message transmission process, the order of the messages received by the target database may not be consistent with the order of the messages generated, which causes the problem of message disorder, such as the failure of the data replication component Ogg as the source of the message, which causes the previous message to be retransmitted, the transmission disorder of the intermediate system Kafka as the message storage, and the network jitter during the transmission of the message, etc. may cause the final data of the target database to be inconsistent with the data stored in the source database, so how to solve the problem that the final data of the target database is inconsistent with the data stored in the source database due to the message disorder during the transmission of the data has become a problem to be solved.

[0052] In view of the defect, the technical scheme of the present application mainly lies in that when the target data stored in the source database is operated, the target data is changed, the data replication component Ogg in the source database records the data change message with a timestamp as a to-be-processed message and sends it to the target database, when the target database receives the to-be-processed message sent by the source database, since the to-be-processed message contains the timestamp of the change of the target data caused by the operation on the target data and the specific operation information of the target data, therefore, after the target database parses the to-be-processed message to obtain the timestamp and the operation information, the target data to be operated in the target data can be determined according to the timestamp and the primary key information corresponding to the target data, since the primary key information can be used to identify the data in the target database and the value of the timestamp is strictly increasing with time, therefore, the actual order of the message generated by the source database, i.e. the actual order of the change of the data in the source database, can be accurately obtained through the timestamp and the primary key information, therefore, according to the timestamp and the operation information, the operation processing on the target data to be operated can accurately synchronize the data in the target database with the data in the source database. The problem that the data in the target database is inconsistent with the data in the source database due to the operation error of the data in the target database caused by the disorder of the order of the messages received by the target database is avoided.

[0053] Figure 1 is an application scenario diagram of the message processing method provided in an example embodiment of the present application.

[0054] As shown in Figure 1 , the basic architecture of the application scenario provided in the present embodiment mainly includes a source database 101, a distributed subscription message system Kafka 102 and a target database 103; wherein when the data stored in the source database is operated to cause the data to change, the data replication component Ogg in the source database records the data change message, the source database sends the data change message to the distributed subscription message system Kafka for storage, and then the distributed subscription message system Kafka batch-sends the data change message to the target database, and the target database operates and processes the corresponding data stored in the target database according to the received data change message.

[0055] Figure 2 is a flowchart of the message processing method provided in an example embodiment of the present application, and the execution subject of the method provided in the present embodiment can be the target database in the Figure 1 embodiment shown in the figure.

[0056] As shown in Figure 2 , the method provided in the present embodiment can include the following steps.

[0057] S201, receiving a to-be-processed message sent by a source database, the to-be-processed message being generated when a target data stored in the source database is operated to change the target data.

[0058] Specifically, when the data in the source database changes, a component Ogg in the source database records a message with a timestamp (denoted as SCN) and sends the message to a distributed subscription message system Kafka for storage, and then the distributed subscription message system Kafka sends the message to the target database.

[0059] S202, parsing the to-be-processed message to obtain a timestamp and operation information of the target data, the timestamp being used to indicate a time when the target data is operated to change the target data.

[0060] Specifically, when the target database receives the to-be-processed message with the timestamp, the message is first parsed to determine the time when the data in the source database changes and the specific operation on the data. The time when the data in the source database changes is marked by the timestamp SCN. The value of the timestamp SCN strictly increases with time, for example, for time i and time j, if i < j is true, then SCNi < SCNj is true. Therefore, the order in which the messages are actually generated, that is, the order in which the data in the source database changes, can be determined by judging the size of the timestamp SCN of the message.

[0061] In some embodiments, the operation information includes insertion operation information, data update operation information, deletion operation information, and primary key update operation information. The four specific operation modes of the data can be delivered in the form of a message.

[0062] S203, determining to-be-operated data in the target database according to the timestamp in the to-be-processed message and primary key information corresponding to the target data.

[0063] In some embodiments, determining to-be-operated data in the target database according to the timestamp in the to-be-processed message and the primary key information corresponding to the target data includes: determining a target life cycle corresponding to the primary key information to which the to-be-processed message belongs according to the timestamp in the to-be-processed message and the primary key information corresponding to the target data; and determining a value of each field in the target life cycle or the primary key information as to-be-operated data.

[0064] Specifically, according to the time stamp of the message and the primary key of the data changed in the source database contained in the message, the data to be modified in the target database and the life cycle thereof are determined. In a real problem, the primary key of the data often appears to be reused, and the most common one is the user's mobile phone number. When the user opens an account, he or she will choose a mobile phone number, at which time the database will create data with the mobile phone number as the primary key. When the user logs out, the database will mark the data as invalid. If another user also chooses the same mobile phone number when opening an account, the primary key will be reused. If the source database operates the data with the mobile phone number as the primary key and sends a message to the target database, since the messages to the target database will be out of order, the operation of updating the data of the first user will be executed on the data of the second user. In order to avoid this problem, it is necessary to clearly define the life cycle of the same primary key corresponding to different data. The life cycle of the data contains all the information contained in the entire period from the creation to the invalidation of the data.

[0065] For example, the same primary key information corresponds to at least one life cycle of data, and the structure of the at least one life cycle of data corresponding to the same primary key information is as shown in Figure 3 The first time stamp, the second time stamp, the data field value corresponding to the primary key information, the third time stamp corresponding to the last operation of each field, and the indication value corresponding to the primary key information are included. The first time stamp is the time stamp corresponding to the newly created data, and the second time stamp is the time stamp corresponding to the invalidation of the data or the maximum value of the current system time. The indication value is used to indicate whether the primary key information is modified. If the indication value is empty, it means that the primary key information is not modified. If the indication value is not empty, the indication value is the modified primary key information.

[0066] S204, according to the time stamp and the operation information, operating and processing the to-be-operated data, so that the data in the target database is synchronized with the data in the source database.

[0067] In this step, after the to-be-operated data in the target database is determined, the time stamp of the to-be-processed message and the primary key information can be used to perform corresponding modification operations on the data field corresponding to the primary key information in the life cycle to which the to-be-processed message in the target database belongs.

[0068] In this embodiment, a to-be-processed message sent by a source database is received, the to-be-processed message is generated when a target data stored in the source database is operated to change the target data. Since the to-be-processed message contains a timestamp of operating the target data to change the target data and specific operation information of the target data, after the target database parses the to-be-processed message to obtain the timestamp and the operation information, the target data to be operated can be determined according to the timestamp and primary key information corresponding to the target data. Since the primary key information can be used to identify the data in the target database and the value of the timestamp is strictly increasing with time, the actual order of the message generated by the source database, i.e., the actual order of the data change in the source database, can be accurately obtained through the timestamp and the primary key information. Therefore, according to the timestamp and the operation information, operating and processing the to-be-operated data can accurately synchronize the data in the target database with the data in the source database. The problem that the data in the target database is inconsistent with the data in the source database due to the operation error of the data in the target database caused by the disorder of the order of the messages received by the target database is avoided.

[0069] In order to better understand the present application, the message processing method provided by the present application will be described in detail below with specific examples.

[0070] For example, it is assumed that the data format stored in the source database is (primary key, field 1, field 2), and the data stored in the source database has been operated six times. The component Ogg in the source database records six messages as follows:

[0071] Message 1: SCN1+ create data (A, 1, 3);

[0072] Message 2: SCN3+ change the primary key of data (A, 1, 3) to B;

[0073] Message 3: SCN4+ create data (A, 7, 8);

[0074] Message 4: SCN2+ modify the second field of the primary key A to 5;

[0075] Message 5: SCN6+ modify the second field of the primary key A to 6;

[0076] Message 6: SCN5+ modify the second field of the primary key A to 7.

[0077] In some embodiments, the order of the messages actually generated is message 1, message 4, message 2, message 3, message 6, and message 5, as can be seen from the timestamp SCN values of the messages. However, due to the message reordering problem, the order in which the target database actually receives the messages is message 1 to message 6. If the existing method is used, the target database directly executes operations according to the received message order, and the final data stored in the database is (B, 1, 3) and (A, 7, 7). However, according to the order in which the messages are actually generated, the target database should store data (B, 1, 5) and (A, 7, 6) after executing the corresponding data operations according to the six messages. The message reordering causes the data stored in the target database to be inconsistent with the source database.

[0078] In a possible embodiment, the operation information includes insertion operation information, the insertion operation information includes primary key information of to-be-inserted data and field values of the to-be-inserted data, and the operation processing on the to-be-operated data according to the timestamp and the operation information includes: creating new data according to the primary key information of the to-be-inserted data and the field values of the to-be-inserted data; generating a first life cycle corresponding to the primary key information, wherein a first timestamp and a third timestamp of the first life cycle are the timestamps in the to-be-processed message, a second timestamp of the first life cycle is a maximum value of a current system time, data field values corresponding to the primary key information are the field values of the to-be-inserted data, and an indication value corresponding to the primary key information in the first life cycle is empty.

[0079] For example, when the target database receives message 1, the target database parses message 1 to obtain the timestamp SCN1 of message 1 and identify that the operation information corresponding to the message is an insertion operation on data. Then, the target database creates data with A as the primary key information, generates a first life cycle (denoted as LT1) corresponding to the primary key A, sets the generation time (that is, the first timestamp) of the data to SCN1, sets the data invalidation time (that is, the second timestamp) to the maximum value of the current system time, sets the data field values corresponding to the primary key to 1 and 3 respectively, sets the third timestamp values corresponding to each field to SCN1, and sets the indication value corresponding to the primary key information to NULL (that is, empty) because the primary key has not been modified.

[0080] In a possible implementation, the operation information includes primary key update operation information, the primary key update operation information is used to indicate modification of primary key information, and the operation processing of the to-be-operated data according to the timestamp and the operation information includes: modifying an indication value corresponding to the primary key information in the target life cycle according to the primary key update operation information, setting the second timestamp in the target life cycle as the timestamp in the to-be-processed message, creating data corresponding to the modified primary key information, and generating a new data life cycle corresponding to the modified primary key information.

[0081] For example, when the target database receives the message 2, the message 2 is parsed to obtain the timestamp SCN3 of the message, and it is identified that the message performs an operation of changing the data primary key. Since the data primary key is modified, it is indicated that the first life cycle of the data with the primary key A ends, and therefore the invalidation time (i.e., the second timestamp) of the data is changed to SCN3, and the indication value corresponding to the primary key information is set to the new primary key B. Subsequently, the target database creates data with the primary key B, generates the first life cycle LT1 of the primary key B, sets the first timestamp to SCN1, sets the second timestamp to the maximum value of the current system time, sets the data field values corresponding to the primary key information to 1 and 3 respectively, sets the third timestamp corresponding to each field to SCN1, and sets the indication value corresponding to the primary key information to NULL. Since the data primary key is changed from A to B, the distributed storage system allocates the data to other storage nodes, but can locate the data according to the indication value in the first life cycle of the primary key A.

[0082] In a possible case in the embodiment, when the target database receives the message 3, the message 3 is parsed to obtain the timestamp SCN4 of the message, and it is identified that the message performs an insertion operation on the data. Since the primary key of the inserted data is still A, the second life cycle LT2 with the primary key A is created, the first timestamp is set to SCN4, the second timestamp is set to the large value of the current system time, the data fields corresponding to the primary key information are set to 7 and 8 respectively, the third timestamp corresponding to each field is set to SCN4, and the indication value corresponding to the primary key information is set to NULL since the primary key is not modified.

[0083] In a possible implementation, the operation information includes data update operation information used to indicate modification of a data field value, and the operation on the data to be operated is performed according to the timestamp and the operation information, including: if the indication value corresponding to the primary key information in the target life cycle is empty, and the timestamp of the message to be processed is greater than the third timestamp corresponding to the last operation of the field in the target life cycle, modifying the data field value corresponding to the primary key information according to the data update operation information; if the indication value corresponding to the primary key information in the target life cycle is empty, and the timestamp of the message to be processed is less than the third timestamp corresponding to the last operation of the field in the target life cycle, the data field value corresponding to the primary key information remains unchanged; if the indication value corresponding to the primary key information in the target life cycle is not empty, determining the data storage location corresponding to the modified primary key information according to the indication value, and performing a modification operation on the data corresponding to the modified primary key information according to the data update operation information.

[0084] For example, when the target database receives the message 4, the message 4 is parsed to obtain the timestamp SCN2 of the message, and it is identified that the message performs an update operation on the second field of the data with the primary key A. It is judged that the life cycle to which the message 4 belongs, and since SCN1< SCN2< SCN3, the message 4 belongs to the first life cycle of the data with the primary key A. Then, it is checked that the indication value of the life cycle is not NULL, which indicates that the primary key of the data is modified, and the data field corresponding to the primary key B that needs to be updated is found through the indication value. Next, the monitoring module judges that the message 4 belongs to the life cycle of the primary key B, and since SCN1< SCN2< the maximum value of the current system time, the message 4 belongs to the first life cycle of the data corresponding to the primary key B. Since the message 4 is an update operation on the second field of the data, it is necessary to judge the size of the third timestamp corresponding to the second field and SCN2. Because SCN1< SCN2, it is indicated that the message is indeed the latest operation, and the field value corresponding to the second field is set to 5, and the corresponding third timestamp is modified to SCN2.

[0085] In one possible case of the embodiment, when the target database receives the message 5, the message 5 is parsed to obtain the time stamp SCN6 of the message, and it is identified that the message performs an update operation on the second field of the data with the primary key A. The monitoring module first judges the life cycle to which the message belongs. Since SCN4< SCN6< the maximum value of the current system time, the message 5 belongs to the second life cycle of the data with the primary key A. Then it is checked that the indication of the second life cycle is NULL, which means that the primary key of the data has not been modified. The monitoring module judges the size of the third time stamp corresponding to the second field and SCN6. Since SCN4< SCN6, it is identified that the message is the latest operation, and the value corresponding to the second field is set to 6, and the corresponding third time stamp is modified to SCN6.

[0086] In one possible case of the embodiment, when the target database receives the message 6, the message 6 is parsed to obtain the time stamp SCN5 of the message, and it is identified that the message performs an update operation on the second field of the data with the primary key A. First, it is judged that the message 6 belongs to the second life cycle of the data with the primary key A, since SCN4< SCN5< the maximum value of the current system time. Then it is checked that the indication of the life cycle is NULL, which means that the primary key of the data has not been modified. It is judged that the size of the third time stamp corresponding to the second field and SCN5. Since SCN5< SCN6, it is identified that the message is not the latest update, and the value corresponding to the second field and the time corresponding to the second field do not need to be modified, and remain unchanged.

[0087] In one possible embodiment, the operation information includes deletion operation information, and the operation processing on the data to be operated according to the time stamp and the operation information includes: setting the second time stamp in the target life cycle to the time stamp in the message to be processed, so that the data to be operated becomes invalid data.

[0088] Further, after the above processing of the message, the valid data stored in the target database can be obtained by checking the life cycle that has not ended, that is, if the second time stamp is the maximum value of the current system time, it is identified that the data is valid data.

[0089] Exemplarily, after the target database processes the above messages 1-6 according to the method provided in the present application, the data life cycle structure obtained is as shown in Figure 4a and Figure 4b as shown in Figure 4a and Figure 4bAs can be seen, the valid data in the target database is the data (A, 7, 6) in the second life cycle corresponding to the primary key A and the data (B, 1, 5) in the first life cycle corresponding to the primary key B.

[0090] In the embodiment, the problem of inconsistency between the target database and the source database caused by message disorder can be avoided by processing the message.

[0091] Figure 5 FIG. 1 is a structural schematic diagram of a message processing device according to an example embodiment of the present application.

[0092] As shown in FIG. 1, the device provided by the embodiment includes: Figure 5

[0093] The receiving module 501 is configured to receive a to-be-processed message sent by a source database, wherein the to-be-processed message is generated when target data stored in the source database is operated to change the target data;

[0094] The parsing module 502 is configured to parse the to-be-processed message to obtain a timestamp and operation information on the target data, wherein the timestamp is used to indicate a time when the target data is operated to change the target data;

[0095] The determining module 503 is configured to determine to-be-operated data in a target database according to the timestamp in the to-be-processed message and primary key information corresponding to the target data.

[0096] The synchronizing module 504 is configured to operate and process the to-be-operated data according to the timestamp and the operation information, so as to synchronize data in the target database with data in the source database.

[0097] Further, one primary key information corresponds to at least one life cycle of data, and each life cycle of data includes a first timestamp, a second timestamp, a data field value corresponding to the primary key information, a third timestamp corresponding to a last operation of each field, and an indication value corresponding to the primary key information.

[0098] The first timestamp is a timestamp corresponding to a time when the data is newly created, and the second timestamp is a timestamp corresponding to a time when the data is invalidated or a maximum value of a current system time.

[0099] The indication value is used to indicate whether the primary key information is modified. If the indication value is empty, it indicates that the primary key information is not modified. If the indication value is not empty, the indication value is the modified primary key information.

[0100] Further, the determining module is specifically configured to: ​

[0101] determining, according to the time stamp in the to-be-processed message and the primary key information corresponding to the target data, a target life cycle corresponding to the primary key information to which the to-be-processed message belongs;

[0102] determining, as to-be-operated data, the value of each field in the target life cycle or the primary key information.

[0103] Further, the operation information includes insertion operation information, the insertion operation information including primary key information of to-be-inserted data and field values of the to-be-inserted data, and the synchronization module is specifically configured to:

[0104] creating new data according to the primary key information of the to-be-inserted data and the field values of the to-be-inserted data;

[0105] generating a first life cycle corresponding to the primary key information, a first time stamp and a third time stamp of the first life cycle being the time stamp in the to-be-processed message, a second time stamp of the first life cycle being a maximum value of a current system time, a data field value corresponding to the primary key information being the field values of the to-be-inserted data, and an indication value corresponding to the primary key information in the first life cycle being empty.

[0106] Further, the operation information includes data update operation information, the data update operation information being used to indicate modification of data field values, and the synchronization module is specifically configured to:

[0107] if the indication value corresponding to the primary key information in the target life cycle is empty and the time stamp of the to-be-processed message is greater than the third time stamp corresponding to the last operation time of the field in the target life cycle, modifying the data field value corresponding to the primary key information according to the data update operation information;

[0108] if the indication value corresponding to the primary key information in the target life cycle is empty and the time stamp of the to-be-processed message is less than the third time stamp corresponding to the last operation time of the field in the target life cycle, keeping the data field value corresponding to the primary key information unchanged;

[0109] if the indication value corresponding to the primary key information in the target life cycle is not empty, determining a data storage position corresponding to the modified primary key information according to the indication value and performing a modification operation on the data corresponding to the modified primary key information according to the data update operation information.

[0110] Further, the operation information includes deletion operation information, and the synchronization module is specifically configured to:

[0111] setting the second time stamp in the target life cycle as the time stamp in the to-be-processed message, so that the to-be-operated data becomes invalid data.

[0112] Further, the operation information includes primary key update operation information, the primary key update operation information is used to indicate to modify primary key information, and the synchronization module is specifically used for:

[0113] According to the primary key update operation information, the indication value corresponding to the primary key information in the target life cycle is modified, and the second time stamp in the target life cycle is set as the time stamp in the to-be-processed message;

[0114] The data corresponding to the modified primary key information is created, and a new data life cycle corresponding to the modified primary key information is generated.

[0115] The specific function implementation of each module provided in the embodiment can refer to the detailed description in the above method embodiment.

[0116] Figure 6 The hardware structure schematic diagram of the computer device provided in the embodiment is shown in the figure. Figure 6 As shown in the figure, the computer device 60 provided in the embodiment includes at least one processor 601 and a memory 602. Wherein, the processor 601 and the memory 602 are connected through a bus 603.

[0117] In the specific implementation process, the at least one processor 601 executes the computer execution instruction stored in the memory 602, so that the at least one processor 501 executes the message processing method in the above method embodiment.

[0118] The specific implementation process of the processor 601 can refer to the above method embodiment, and the implementation principle and technical effect are similar, which will not be described here in detail.

[0119] In the above Figure 6 In the embodiment shown in the figure, it should be understood that the processor can be a central processing unit (English: Central Processing Unit, for short: CPU), and can also be other general-purpose processors, digital signal processors (English: Digital Signal Processor, for short: DSP), application specific integrated circuits (English: Application Specific Integrated Circuit, for short: ASIC) and the like. The general-purpose processor can be a microprocessor, or the processor can be any conventional processor. The steps of the method disclosed in the application can be directly embodied as hardware processor execution, or executed by hardware and software modules in the processor.

[0120] The memory can contain a high-speed RAM memory, and can also include a non-volatile storage NVM, for example, at least one disk memory.

[0121] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0122] Another embodiment of the present application provides a computer readable storage medium, wherein computer executable instructions are stored in the computer readable storage medium, and when a processor executes the computer executable instructions, a message processing method in the above method embodiments is implemented.

[0123] Another embodiment of the present application provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the message processing method in the first aspect of the embodiments of the present application.

[0124] The computer readable storage medium described above can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0125] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0126] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the foregoing storage medium includes ROM, RAM, magnetic disk or optical disk and various storage medium capable of storing program codes.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A message processing method characterized by, The method comprises the following steps: receiving a to-be-processed message sent by a source database, the to-be-processed message being generated when an operation on target data stored in the source database causes the target data to change; parsing the to-be-processed message to obtain a timestamp and operation information on the target data, the timestamp being used to indicate a time when the operation on the target data causes the target data to change, and the operation information comprising insertion operation information, data update operation information, deletion operation information and primary key update operation information; determining a target life cycle corresponding to the primary key information to which the to-be-processed message belongs according to the timestamp in the to-be-processed message and the primary key information corresponding to the target data; determining a to-be-operated data as a value of each field in the target life cycle or the primary key information, at least one data life cycle corresponding to one primary key information, each data life cycle comprising a first timestamp, a second timestamp, a data field value corresponding to the primary key information, a third timestamp corresponding to a last operation time of each field and an indication value corresponding to the primary key information; wherein the first timestamp is a timestamp corresponding to a time when the data is newly created, the second timestamp is a timestamp corresponding to a time when the data is invalidated or a maximum current system time; and the indication value is used to indicate whether the primary key information is modified, wherein if the indication value is empty, it indicates that the primary key information is not modified, and if the indication value is not empty, the indication value is the modified primary key information; performing operation processing on the to-be-operated data according to the timestamp and the operation information, so as to synchronize data in the target database with data in the source database.

2. The method of claim 1, wherein, The operation information comprises insertion operation information, the insertion operation information comprising primary key information of to-be-inserted data and field values of the to-be-inserted data, and the operation processing on the to-be-operated data according to the timestamp and the operation information comprises: creating new data according to the primary key information of the to-be-inserted data and the field values of the to-be-inserted data; generating a first life cycle corresponding to the primary key information, the first timestamp and the third timestamp of the first life cycle being the timestamp in the to-be-processed message, the second timestamp of the first life cycle being a maximum current system time, the data field value corresponding to the primary key information being the field values of the to-be-inserted data, and the indication value corresponding to the primary key information in the first life cycle being empty.

3. The method of claim 1, wherein, The operation information comprises data update operation information, the data update operation information being used to indicate modified data field values, and the operation processing on the to-be-operated data according to the timestamp and the operation information comprises: if the indication value corresponding to the primary key information in the target life cycle is empty and the timestamp of the to-be-processed message is greater than the third timestamp corresponding to the last operation time of the field in the target life cycle, modifying the data field value corresponding to the primary key information according to the data update operation information; If the indication value corresponding to the primary key information in the target life cycle is empty and the timestamp of the to-be-processed message is less than the third timestamp corresponding to the last operation of the field in the target life cycle, the data field value corresponding to the primary key information remains unchanged; If the indication value corresponding to the primary key information in the target life cycle is not empty, the data storage location corresponding to the modified primary key information is determined according to the indication value, and the data corresponding to the modified primary key information is modified according to the data update operation information.

4. The method of claim 1, wherein, The operation information includes deletion operation information, and the operation processing of the to-be-operated data according to the timestamp and the operation information includes: Setting the second timestamp in the target life cycle as the timestamp in the to-be-processed message, so that the to-be-operated data becomes invalid data.

5. The method of claim 1, wherein, The operation information includes primary key update operation information, the primary key update operation information is used to indicate that the primary key information is modified, and the operation processing of the to-be-operated data according to the timestamp and the operation information includes: According to the primary key update operation information, the indication value corresponding to the primary key information in the target life cycle is modified, and the second timestamp in the target life cycle is set as the timestamp in the to-be-processed message; Creating data corresponding to the modified primary key information and generating a new data life cycle corresponding to the modified primary key information.

6. A message synchronization processing apparatus characterized by comprising: It includes: A receiving module configured to receive a to-be-processed message sent by a source database, the to-be-processed message being generated when a target data stored in the source database is operated to change the target data; An analyzing module configured to analyze the to-be-processed message to obtain a timestamp and operation information of the target data, the timestamp being used to indicate a time when the target data is operated to change the target data, and the operation information including insertion operation information, data update operation information, deletion operation information and primary key update operation information; A determining module configured to determine a target life cycle corresponding to primary key information of the to-be-processed message according to the timestamp in the to-be-processed message and the primary key information of the target data, and determine to-be-operated data as a value of each field in the target life cycle or the primary key information, at least one data life cycle corresponding to one primary key information, each data life cycle including a first timestamp, a second timestamp, a data field value corresponding to the primary key information, a third timestamp corresponding to the last operation of each field and an indication value corresponding to the primary key information; wherein the first timestamp is a timestamp corresponding to the creation of the data, the second timestamp is a timestamp corresponding to the invalidation of the data or a maximum current system time; the indication value is used to indicate whether the primary key information is modified, if the indication value is empty, it indicates that the primary key information is not modified, and if the indication value is not empty, the indication value is a modified primary key information; A synchronization module is configured to operate the data to be operated according to the timestamp and the operation information, so as to synchronize the data in the target database with the data in the source database.

7. A computer device, comprising: The method comprises: at least one processor and a memory; the memory stores computer-executed instructions; the at least one processor executes the computer-executed instructions stored in the memory, so that the at least one processor executes the message processing method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executed instructions, and when the processor executes the computer-executed instructions, the message processing method according to any one of claims 1-5 is realized.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the message processing method according to any one of claims 1-5.

Images