Data operating system and method based on message queue driving

Abstract

The invention relates to the technical field of internet, in particular to a data operation system and method based on message queue driving, in the system, a data operation module is used for receiving a request for performing data operation on databases and sequentially operating a master database and a slave database; and performing process judgment of the master database and the slave database and data compensation operation judgment for the slave database. According to the data operation module and the compensation operation module, the functions of data submission, data rollback and the like of a transactional-free database master-slave cluster are achieved, the transactional-free database is helped to achieve a database transaction management mechanism, and therefore consistency and integrity of master-slave database data are guaranteed. According to the method, the message queue is used, on one hand, the orderliness of data operation is guaranteed through the orderliness of the queue, concurrent operation can be executed in sequence, and data chaos caused by out-of-order operation is avoided; and on the other hand, external requests are cached by using the message queue, the system can process the requests one by one during business peak, and the concurrent pressure is reduced.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a data operating system and method based on message queue drive. Background Art

[0002] Non-transactional databases do not use a transaction management mechanism when operating data, do not have data commit and rollback operations, and do not use transaction logs to record data operation content. Therefore, they cannot guarantee the atomicity, consistency, isolation, and durability (ACID) of data operations. HBase, MongoDB, etc. are all non-transactional databases.

[0003] In order to ensure the high availability of the software system, the software system generally adopts a master-slave database cluster, and operates data on both the master and slave databases at the same time, and ensures the data consistency of the master and slave databases. However, if an exception occurs during data operation, such as network congestion, abnormal power outage, etc., since non-transactional databases have no transaction management mechanism, it cannot guarantee that the master and slave databases can successfully operate data at the same time, which may cause inconsistent data in the two databases and cannot be restored to the state before the operation. Summary of the invention

[0004] The purpose of the present invention is to provide a data operation system and method based on message queue drive to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a data operation system based on message queue drive, the system comprises: a data operation module, a message queue construction module and a compensation operation module, The data operation module is used to receive a request for performing data operation on a database, operate the master database and the slave database in sequence, and perform process determination on the master database and the slave database and data compensation operation determination on the slave database; When the data operation module performs the data compensation operation, the message queue construction module receives the data compensation operation message sent by the data operation module and generates a data compensation message queue; stores the generated data compensation message queue and feeds back the information in the data compensation message queue to the compensation operation module; The compensation operation module is based on the data compensation message queue, performs compensation operations on the slave database after querying the data records in the master database; and rolls back to the data records of the master database after the slave database compensation fails, ensuring that the master database and the slave database are restored to the state before the corresponding data compensation operation is executed.

[0006] The present invention makes data compensation operation judgment and generates various data compensation operation messages to ensure the consistency of data in the master database and the slave database; and the execution of the data compensation operation message is realized by constructing a data compensation message queue, considering that when there are multiple data inconsistencies between the master database and the slave database and corresponding different data compensation operation messages are generated respectively, operation confusion may occur, resulting in errors in the data compensation operation behavior for the slave database, further aggravating the inconsistency between the master database and the slave database; and the construction of the data compensation message queue utilizes the orderliness of the data queue to ensure the orderliness of the data operation, so that concurrent operations can be executed sequentially, avoiding data confusion caused by disorderly operations.

[0007] Furthermore, the data operation module includes an operation request receiving unit, an operation result feedback unit, an operation process determination unit and a compensation operation management unit. The operation request receiving unit is used to receive data operation requests for the master database and the slave database, and the data operation requests include deleting data records and adding data records; The operation result feedback unit is used to feedback the operation result based on the received data operation request for the master database and the slave database; The operation process determination unit is used to perform process determination on the master database and the slave database; The compensation operation management unit determines the data compensation operation to be performed on the slave database according to the feedback result of the operation result feedback unit.

[0008] Further, the operation flow determination unit performs operation flow determination on the master database during the operation on the data record of the master database. If the operation on the master database fails, the operation failure record is fed back and the operation flow is determined to be over; if the operation on the master database is successful, the data records of the slave database are operated in turn, and the operation flow determination on the slave database is performed; if the operation on the slave database is successful, the operation success record is fed back and the flow is determined to be over; if the data operation module fails to operate the data record of the slave database, a data compensation operation message is sent to the data compensation message queue; The compensation operation management unit receives the feedback result of the operation result feedback unit, summarizes the data compensation operation message sent to the data compensation message queue after the operation result feedback unit fails to record the data from the database, and saves it at the end of the corresponding data compensation message queue in sequence.

[0009] Furthermore, the message queue construction module includes a data compensation message queue construction unit, a data compensation message queue dynamic update unit and a compensation information storage feedback unit. The data compensation message queue construction unit receives the data compensation operation message sent by the data operation module when the data operation module performs the data compensation operation, and generates the data compensation message queue; When generating the data compensation message queue, the data compensation message queue dynamic update unit generates the data compensation message queue in real time according to the data compensation operation information sent by the data operation module received by the data compensation message queue construction unit and the new data compensation operation information corresponding to the failure of the corresponding data compensation operation information operation; The compensation information storage and feedback unit stores the generated data compensation message queue and feeds back the information in the data compensation message queue to the compensation operation module.

[0010] Further, the data compensation message queue dynamic update unit deletes the executed element after the element in the data compensation message queue corresponding to the current time is executed; when the data compensation message operation corresponding to the executed element fails, generates a new data compensation message corresponding to the corresponding operation failure, and sequentially adds it to the end of the data compensation message queue corresponding to the current time, so as to obtain the data compensation message queue updated at the current time; Bind the original data compensation message of failed operation with the corresponding new data compensation message, extract the compensation operation object corresponding to the original data compensation message of failed operation, record it as the historical reference operation object, and exclude the corresponding historical reference operation object when constructing the corresponding new data compensation message; In the process of constructing the data compensation message, the frequency ratio of each compensation operation object corresponding to the data compensation message generated when executing the corresponding operation request on the master database and the slave database in the historical data is obtained, and the compensation operation object with the largest frequency ratio is used as the compensation operation object for constructing the data compensation message.

[0011] The selection of the compensation operation object in the present invention may also be adjusted manually, and the compensation operation object after the manual adjustment is not affected by the compensation operation object obtained by the frequency ratio method.

[0012] Furthermore, the compensation operation module consumes data compensation operation messages from the data compensation message queue, including deletion operations and addition operations. If a deletion operation is performed, the slave database record is directly deleted according to the data record primary key rowKey; if a addition operation is performed, the corresponding data record is queried from the master database according to the data record primary key rowKey, and the queried corresponding data record is inserted into the slave database; In the process of consuming data compensation operation messages from the data compensation message queue, if the compensation operation succeeds from the database, the compensation operation module will feedback the successful operation record and the process ends; if the compensation operation module fails to operate from the database, the number of retries corresponding to the primary key rowKey of the data record will be increased by 1. If the number of retries is less than the preset threshold, the slave database will be operated again until the number of retries is greater than the preset threshold; During the process of the compensation operation module making a rollback judgment on the main database, the main database data is rolled back and restored to the state before the operation: if it is a new operation, the main database record is deleted according to the data record primary key rowKey; if it is a deletion operation, the data record is queried in the slave database according to the data record primary key rowKey, and then the record is added to the main database; after the compensation operation module restores the main database data, the operation failure record is fed back and the process ends; if the compensation operation module fails to restore the main database data, the data record primary key rowKey is recorded in the exception log file, the operation failure record is fed back and the process ends; the system operation and maintenance personnel manually process the main database record according to the data record primary key rowKey saved in the exception log file to ensure data consistency between the main database and the slave database.

[0013] A data operation method based on message queue drive, comprising: S1. Send a data operation request to the data operation module. The operation request includes deleting data records and adding data records. S2. The data operation module operates the data records of the master database and determines the operation flow of the master database. If the operation of the master database fails, the operation failure record is fed back and the operation flow is determined to be ended; if the operation of the master database succeeds, the data records of the slave database are operated in turn and the operation flow of the slave database is determined; S3. If the operation from the database is successful, the successful operation record is fed back and the process is determined to be finished; if the data operation module fails to operate the data record from the database, a data compensation operation message is sent to the data compensation message queue and is sequentially saved at the end of the corresponding data compensation message queue; S4, the compensation operation module consumes the data compensation operation message from the data compensation message queue; S5. Obtain the result of executing the data compensation message queue in S4, make a rollback determination on the main database, and feed back the obtained execution result of the data compensation message queue and the rollback determination result on the main database.

[0014] Further, the S4 includes: The data compensation operation messages consumed from the data compensation message queue include deletion operations and addition operations. If a delete operation is performed, the slave database record is directly deleted based on the data record primary key rowKey; If a new operation is performed, the corresponding data record is queried from the master database based on the data record primary key rowKey, and the queried corresponding data record is inserted into the slave database.

[0015] Further, the result of the compensation operation module consuming the data compensation operation message from the data compensation message queue in S4 includes: S41, if the compensation operation is successful from the database, the compensation operation module will feedback the successful operation record, and the process ends; S42, if the operation fails from the database, the compensation operation module will increase the number of retries corresponding to the data record primary key rowKey by 1. If the number of retries is less than the preset threshold, the slave database will be operated again. If successful, it will return to step S41. If failed, it will repeat step S42 until the number of retries is greater than the preset threshold.

[0016] Further, the method for performing rollback determination on the primary database in S5 includes: S51. Roll back the data in the master database and restore it to the state before the operation: if it is a new operation, delete the master database record according to the data record primary key rowKey; if it is a delete operation, query the data record in the slave database according to the data record primary key rowKey, and then add the record to the master database; after the compensation operation module restores the master database data, feedback the operation failure record, and the process ends; S52: If the compensation operation module fails to restore the data in the main database, the primary key rowKey of the data record is recorded in the abnormal log file, and the operation failure record is fed back, and the process ends; S53. System operation and maintenance personnel manually process the master database records according to the primary key rowKey of the data record saved in the abnormal log file to ensure data consistency between the master database and the slave database.

[0017] Compared with the prior art, the beneficial effects achieved by the present invention are: the data operation module and the compensation operation module of the present invention realize the functions of data submission, data rollback, exception log recording, etc. of the master-slave cluster of the transaction-free database, and help the transaction-free database to realize the database transaction management mechanism, thereby ensuring the consistency and integrity of the master-slave database data.

[0018] The present invention uses a message queue. On the one hand, the orderliness of the queue is used to ensure the orderliness of data operations, so that concurrent operations can be executed sequentially and data confusion caused by disordered operations can be avoided. On the other hand, the message queue is used to cache external requests. During business peaks, the system can process requests one by one, reducing concurrency pressure and ensuring operational stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 It is a flow chart of a data operation method based on message queue drive of the present invention; Figure 2 It is a schematic diagram of the operation flow of a data operating system driven by a message queue according to the present invention; Figure 3 It is a structural schematic diagram of a data operating system driven by a message queue according to the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] See also Figure 1 The present invention provides a technical solution: a data operation method based on message queue drive, comprising: S1. Send a data operation request to the data operation module. The operation request includes deleting data records and adding data records. S2. The data operation module operates the data records of the master database and determines the operation flow of the master database. If the operation of the master database fails, the operation failure record is fed back and the operation flow is determined to be ended; if the operation of the master database succeeds, the data records of the slave database are operated in turn and the operation flow of the slave database is determined; S3. If the operation from the database is successful, the successful operation record is fed back and the process is determined to be finished; if the data operation module fails to operate the data record from the database, a data compensation operation message is sent to the data compensation message queue and is sequentially saved at the end of the corresponding data compensation message queue; S4, the compensation operation module consumes the data compensation operation message from the data compensation message queue; The S4 includes: The data compensation operation messages consumed from the data compensation message queue include deletion operations and addition operations. If a delete operation is performed, the slave database record is directly deleted based on the data record primary key rowKey; If a new operation is performed, the corresponding data record is queried from the master database based on the data record primary key rowKey, and the queried corresponding data record is inserted into the slave database.

[0022] In this embodiment, when the compensation operation module consumes data compensation operation messages from the data compensation message queue, a distinction is made between deletion operations and new addition operations. This is reflected in the code by recording the code judgment condition for the deletion operation as "operate=2" and the code judgment condition for the new addition operation as "operate=1".

[0023] The result of the compensation operation module consuming the data compensation operation message from the data compensation message queue in S4 includes: S41, if the compensation operation is successful from the database, the compensation operation module will feedback the successful operation record, and the process ends; S42, if the operation fails from the database, the compensation operation module will increase the number of retries corresponding to the data record primary key rowKey by 1. If the number of retries is less than the preset threshold, the slave database will be operated again. If successful, it will return to step S41. If failed, it will repeat step S42 until the number of retries is greater than the preset threshold.

[0024] In this embodiment, the compensation operation module consumes data compensation operation messages from the data compensation message queue in an iterative process. The iterative judgment condition is that if the compensation operation module fails to operate from the database and the number of retries is less than a preset threshold; and the judgment condition for the end of the iterative operation is that if the compensation operation module succeeds from the database or the number of retries is greater than a preset threshold, in this embodiment, the number of retries corresponds to the preset threshold of 3.

[0025] S5, obtaining the result of executing the data compensation message queue in S4, making a rollback determination on the primary database, and feeding back the obtained execution result of the data compensation message queue and the rollback determination result on the primary database; The method for determining rollback of the primary database in S5 includes: S51. Roll back the data in the master database and restore it to the state before the operation: if it is a new operation, delete the master database record according to the data record primary key rowKey; if it is a delete operation, query the data record in the slave database according to the data record primary key rowKey, and then add the record to the master database; after the compensation operation module restores the master database data, feedback the operation failure record, and the process ends; S52: If the compensation operation module fails to restore the data in the main database, the primary key rowKey of the data record is recorded in the abnormal log file, and the operation failure record is fed back, and the process ends; S53. System operation and maintenance personnel manually process the master database records according to the primary key rowKey of the data record saved in the abnormal log file to ensure data consistency between the master database and the slave database.

[0026] like Figure 2 and Figure 3 As shown, a data operation system based on message queue drive, the system includes: a data operation module, a message queue construction module and a compensation operation module, The data operation module includes an operation request receiving unit, an operation result feedback unit, an operation process determination unit and a compensation operation management unit. The operation request receiving unit is used to receive data operation requests for the master database and the slave database, and the data operation requests include deleting data records and adding data records; In this embodiment, the operation request body is in Json format; The operation result feedback unit is used to feedback the operation result based on the received data operation request for the master database and the slave database; The operation process determination unit is used to perform process determination on the master database and the slave database; The operation flow determination unit performs operation flow determination on the master database during the operation on the data record of the master database. If the operation on the master database fails, the operation failure record is fed back and the operation flow is determined to be over. If the operation on the master database succeeds, the data records of the slave database are operated in sequence to perform operation flow determination on the slave database. If the operation on the slave database succeeds, the operation success record is fed back and the flow is determined to be over. If the data operation module fails to operate the data record of the slave database, a data compensation operation message is sent to the data compensation message queue. The compensation operation management unit receives the feedback result of the operation result feedback unit, summarizes the data compensation operation message sent to the data compensation message queue after the operation result feedback unit fails to record the data from the database, and saves it at the end of the corresponding data compensation message queue in sequence; In this embodiment, the message body of the data compensation message in the data compensation message queue is in Json format; The compensation operation management unit determines the data compensation operation to be performed on the slave database according to the feedback result of the operation result feedback unit.

[0027] The message queue construction module includes a data compensation message queue construction unit, a data compensation message queue dynamic update unit and a compensation information storage feedback unit. The data compensation message queue construction unit receives the data compensation operation message sent by the data operation module when the data operation module performs the data compensation operation, and generates the data compensation message queue; When generating the data compensation message queue, the data compensation message queue dynamic update unit generates the data compensation message queue in real time according to the data compensation operation information sent by the data operation module received by the data compensation message queue construction unit and the new data compensation operation information corresponding to the failure of the corresponding data compensation operation information operation; The data compensation message queue dynamic update unit deletes the executed elements after the elements in the data compensation message queue corresponding to the current time are executed; when the data compensation message operation corresponding to the executed elements fails, generates a new data compensation message corresponding to the corresponding operation failure, and sequentially adds it to the end of the data compensation message queue corresponding to the current time, so as to obtain the data compensation message queue updated at the current time; Bind the original data compensation message of failed operation with the corresponding new data compensation message, extract the compensation operation object corresponding to the original data compensation message of failed operation, record it as the historical reference operation object, and exclude the corresponding historical reference operation object when constructing the corresponding new data compensation message; In the process of constructing the data compensation message, the frequency ratio of each compensation operation object corresponding to the data compensation message generated when executing the corresponding operation request for the master database and the slave database in the historical data is obtained, and the compensation operation object with the largest frequency ratio is used as the compensation operation object for constructing the data compensation message; the selection of the compensation operation object can also be adjusted manually; The compensation information storage and feedback unit stores the generated data compensation message queue and feeds back the information in the data compensation message queue to the compensation operation module.

[0028] The compensation operation module performs compensation operation on the slave database after querying the data record in the master database based on the data compensation message queue; and rolls back to the data record of the master database after the compensation of the slave database fails, so as to ensure that the master database and the slave database are restored to the state before the execution of the corresponding data compensation operation; The compensation operation module consumes data compensation operation messages from the data compensation message queue, including deletion operations and addition operations. If a deletion operation is performed, the slave database record is directly deleted according to the data record primary key rowKey; if a addition operation is performed, the corresponding data record is queried from the master database according to the data record primary key rowKey, and the queried corresponding data record is inserted into the slave database; In the process of consuming data compensation operation messages from the data compensation message queue, if the compensation operation succeeds from the database, the compensation operation module will feedback the successful operation record and the process ends; if the compensation operation module fails to operate from the database, the number of retries corresponding to the primary key rowKey of the data record will be increased by 1. If the number of retries is less than the preset threshold, the slave database will be operated again until the number of retries is greater than the preset threshold; During the process of the compensation operation module making a rollback judgment on the main database, the main database data is rolled back and restored to the state before the operation: if it is a new operation, the main database record is deleted according to the data record primary key rowKey; if it is a deletion operation, the data record is queried in the slave database according to the data record primary key rowKey, and then the record is added to the main database; after the compensation operation module restores the main database data, the operation failure record is fed back and the process ends; if the compensation operation module fails to restore the main database data, the data record primary key rowKey is recorded in the exception log file, the operation failure record is fed back and the process ends; the system operation and maintenance personnel manually process the main database record according to the data record primary key rowKey saved in the exception log file to ensure data consistency between the main database and the slave database.

[0029] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0030] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data operating system based on message queue drive, characterized in that: The system includes: a data operation module, a message queue construction module and a compensation operation module. The data operation module is used to receive a request for performing data operation on a database, operate the master database and the slave database in sequence, and perform process determination on the master database and the slave database and data compensation operation determination on the slave database; When the data operation module performs the data compensation operation, the message queue construction module receives the data compensation operation message sent by the data operation module and generates a data compensation message queue; stores the generated data compensation message queue and feeds back the information in the data compensation message queue to the compensation operation module; The compensation operation module is based on the data compensation message queue, performs compensation operations on the slave database after querying the data records in the master database; and rolls back to the data records of the master database after the slave database compensation fails, ensuring that the master database and the slave database are restored to the state before the corresponding data compensation operation is executed.

2. A data operating system based on message queue drive according to claim 1, characterized in that: The data operation module includes an operation request receiving unit, an operation result feedback unit, an operation process determination unit and a compensation operation management unit. The operation request receiving unit is used to receive data operation requests for the master database and the slave database, and the data operation requests include deleting data records and adding data records; The operation result feedback unit is used to feedback the operation result based on the received data operation request for the master database and the slave database; The operation process determination unit is used to perform process determination on the master database and the slave database; The compensation operation management unit determines the data compensation operation to be performed on the slave database according to the feedback result of the operation result feedback unit.

3. A data operating system based on message queue drive according to claim 2, characterized in that: The operation process determination unit performs operation process determination on the main database during the operation on the data record of the main database. If the operation on the main database fails, the operation failure record is fed back and the operation process is determined to be ended. If the operation on the master database is successful, the data records of the slave database are operated in sequence, and the operation flow judgment on the slave database is executed; If the operation from the database is successful, the successful operation record will be fed back and the process will be determined to be complete; If the data operation module fails to operate the data record from the database, a data compensation operation message is sent to the data compensation message queue; The compensation operation management unit receives the feedback result of the operation result feedback unit, summarizes the data compensation operation message sent to the data compensation message queue after the operation result feedback unit fails to record the data from the database, and saves it at the end of the corresponding data compensation message queue in sequence.

4. The data operating system based on message queue drive according to claim 1, characterized in that: The message queue construction module includes a data compensation message queue construction unit, a data compensation message queue dynamic update unit and a compensation information storage feedback unit. The data compensation message queue construction unit receives the data compensation operation message sent by the data operation module when the data operation module performs the data compensation operation, and generates the data compensation message queue; When generating the data compensation message queue, the data compensation message queue dynamic update unit generates the data compensation message queue in real time according to the data compensation operation information sent by the data operation module received by the data compensation message queue construction unit and the new data compensation operation information corresponding to the failure of the corresponding data compensation operation information operation; The compensation information storage and feedback unit stores the generated data compensation message queue and feeds back the information in the data compensation message queue to the compensation operation module.

5. A data operating system based on message queue drive according to claim 4, characterized in that: The data compensation message queue dynamic update unit deletes the executed elements after the elements in the data compensation message queue corresponding to the current time are executed; when the data compensation message operation corresponding to the executed elements fails, generates a new data compensation message corresponding to the corresponding operation failure, and sequentially adds it to the end of the data compensation message queue corresponding to the current time, so as to obtain the data compensation message queue updated at the current time; Bind the original data compensation message of failed operation with the corresponding new data compensation message, extract the compensation operation object corresponding to the original data compensation message of failed operation, record it as the historical reference operation object, and exclude the corresponding historical reference operation object when constructing the corresponding new data compensation message; In the process of constructing the data compensation message, the frequency ratio of each compensation operation object corresponding to the data compensation message generated when executing the corresponding operation request on the master database and the slave database in the historical data is obtained, and the compensation operation object with the largest frequency ratio is used as the compensation operation object for constructing the data compensation message.

6. The data operation system based on message queue drive according to claim 1, characterized in that: The compensation operation module consumes data compensation operation messages from the data compensation message queue, including deletion operations and addition operations. If a deletion operation is performed, the slave database record is directly deleted according to the data record primary key rowKey; if a addition operation is performed, the corresponding data record is queried from the master database according to the data record primary key rowKey, and the queried corresponding data record is inserted into the slave database; In the process of the compensation operation module consuming the data compensation operation message from the data compensation message queue, if the compensation operation is successful from the database, the compensation operation module will feedback the operation success record and the process ends; If the compensation operation module fails to operate the slave database, the number of retries corresponding to the data record primary key rowKey will be increased by 1. If the number of retries is less than the preset threshold, the slave database will be operated again until the number of retries is greater than the preset threshold; When the compensation operation module makes a rollback judgment on the main database, the main database data is rolled back and restored to the state before the operation: if it is a new operation, the main database record is deleted according to the data record primary key rowKey; If it is a delete operation, the data record is queried in the slave database according to the data record primary key rowKey, and then a new record is added to the master database; After the compensation operation module recovers the main database data, it will provide feedback on the operation failure record and the process ends; If the compensation operation module fails to restore the data of the main database, the primary key rowKey of the data record will be recorded in the exception log file, the operation failure record will be fed back, and the process will end; System operation and maintenance personnel manually process the master database records based on the primary key rowKey of the data record saved in the exception log file to ensure data consistency between the master and slave databases.

7. A data operation method based on message queue drive, characterized in that: include: S1. Send a data operation request to the data operation module. The operation request includes deleting data records and adding data records. S2. The data operation module operates the data records of the main database and determines the operation flow of the main database. If the operation of the main database fails, the operation failure record is fed back and the operation flow is determined to be ended; If the operation of the master database is successful, the data records of the slave database are operated in sequence to determine the operation process of the slave database; S3. If the operation from the database is successful, the successful operation record is fed back and the process is determined to be finished; if the data operation module fails to operate the data record from the database, a data compensation operation message is sent to the data compensation message queue and is sequentially saved at the end of the corresponding data compensation message queue; S4, the compensation operation module consumes the data compensation operation message from the data compensation message queue; S5. Obtain the result of executing the data compensation message queue in S4, make a rollback determination on the main database, and feed back the obtained execution result of the data compensation message queue and the rollback determination result on the main database.

8. The data operation method based on message queue drive according to claim 7 is characterized in that: The S4 includes: The data compensation operation messages consumed from the data compensation message queue include deletion operations and addition operations. If a delete operation is performed, the slave database record is directly deleted based on the data record primary key rowKey; If a new operation is performed, the corresponding data record is queried from the master database based on the data record primary key rowKey, and the queried corresponding data record is inserted into the slave database.

9. The data operation method based on message queue drive according to claim 7, characterized in that: The result of the compensation operation module consuming the data compensation operation message from the data compensation message queue in S4 includes: S41, if the compensation operation is successful from the database, the compensation operation module will feedback the successful operation record, and the process ends; S42, if the operation fails from the database, the compensation operation module will increase the number of retries corresponding to the data record primary key rowKey by 1. If the number of retries is less than the preset threshold, the slave database will be operated again. If successful, it will return to step S41. If failed, it will repeat step S42 until the number of retries is greater than the preset threshold.

10. The data operation method based on message queue drive according to claim 7, characterized in that: The method for determining rollback of the primary database in S5 includes: S51. Roll back the data in the master database and restore it to the state before the operation: if it is a new operation, delete the master database record according to the data record primary key rowKey; if it is a delete operation, query the data record in the slave database according to the data record primary key rowKey, and then add the record to the master database; after the compensation operation module restores the master database data, feedback the operation failure record, and the process ends; S52: If the compensation operation module fails to restore the data in the main database, the primary key rowKey of the data record is recorded in the abnormal log file, and the operation failure record is fed back, and the process ends; S53. System operation and maintenance personnel manually process the master database records according to the primary key rowKey of the data record saved in the abnormal log file to ensure data consistency between the master database and the slave database.

Images