Data processing method, device and system, storage medium and product

Abstract

The invention discloses a data processing method, device and system, a storage medium and a product. The method comprises the following steps: in response to a first instruction for opening a transaction, opening a transaction related to an execution operation of the relational database; wherein the transaction has an annotation used for identifying a transaction management method or starting execution operation based on a transaction template; monitoring a message event, and if it is determined that the message event belongs to a transaction and has a retry call annotation, packaging a message entity corresponding to the message event and storing the message entity to a retry message table; and after determining that a second instruction for indicating to submit the transaction is received, executing a write-in operation of the non-relational database based on the retry message table. After the execution operation of the relational database is successfully executed, the write-in operation of the non-relational database is executed based on the retry message table, so that the data consistency between the relational database and the non-relational database is ensured, and the stability and reliability of a database application system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to a data processing method, device, system, storage medium and product. BACKGROUND

[0002] In related technologies, there are relational databases and non-relational databases. The relational database organizes data in a table form, establishes a correlation through a primary key and a foreign key, and supports strict ACID transaction characteristics, namely Atomicity, Consistency, Isolation and Durability transaction characteristics. The non-relational database (NoSQL) adopts a flexible storage mode such as a key-value pair, a document and a column family, and pays more attention to scalability and high performance. Based on the combination of the above two databases, advantages can be complemented in processing business logic. For example, in a white box control plane, data is first written into a relational database MySQL, and then synchronized to a non-relational database Etcd. Then, a bottom agent listens to the data of Etcd and consumes it. The data integrity of the database can be guaranteed based on the transaction operation of the relational database, and the scalability and high performance of the non-relational database can meet the bottom listening demand.

[0003] However, when the above two databases are used in a database application system at the same time, there is often a problem of inconsistent data between the relational database and the non-relational database. For example, if the Etcd operation fails due to network problems or other faults, although the transaction indicating the execution of the business operation has been committed in the MySQL, data inconsistency will also occur. SUMMARY

[0004] Therefore, the embodiments of the present application provide a data processing method, device, system, storage medium and product, aiming to guarantee the data consistency between the relational database and the non-relational database in a database application system, and improve the stability and reliability of the database application system.

[0005] The technical solution of the embodiments of the present application is as follows: The embodiments of the present application provide a data processing method applied to a database application system having a relational database and a non-relational database. The method comprises the following steps: In response to a first instruction of starting a transaction, a transaction related to an execution operation of the relational database is started. The transaction has an annotation for identifying a transaction management method or starts the execution operation based on a transaction template. A message event is listened to, and it is determined that the message event belongs to the transaction and has a retry call annotation. Then, a message entity corresponding to the message event is encapsulated, and the message entity is saved to a retry message table. determining that a second instruction indicating to commit the transaction is received, performing the write operation of the non-relational database based on the retry message table.

[0006] In the above solution, before the listening message event, the method further comprises: receiving indication information for generating a message event in performing an execution operation of the relational database, the indication information being used for generating a message event associated with an execution operation of the non-relational database and adding the retry call annotation in the message event.

[0007] In the above solution, the encapsulating the message entity corresponding to the message event comprises: if it is determined that the thread for generating the message event is the thread of the transaction based on the aspect of aspect oriented programming (AOP), then encapsulating the message entity corresponding to the message event based on the method name, the parameters and the class of the message event; wherein the method name represents the operation identifier of the message event, the parameters are used to provide necessary data input for the method execution of the message event, and the class is used to define the data structure of the method execution of the message event.

[0008] In the above solution, after determining that the second instruction indicating to commit the transaction is received, performing the write operation of the non-relational database based on the retry message table comprises: determining that a second instruction indicating to commit the transaction is received, performing the write operation of the non-relational database related to the transaction; determining whether the write operation of the non-relational database is successful, if yes, deleting the message entity corresponding to the write operation in the retry message table; if not, performing the write operation of the non-relational database based on the message entity in the retry message table and the first timer.

[0009] In the above solution, performing the write operation of the non-relational database based on the message entity in the retry message table and the first timer comprises: determining the interval duration for repeatedly performing the write operation of the non-relational database and the set number of times for repetition based on the obtained configuration information; The first timer triggers the non-relational database to perform a write operation based on the message entity in the retry message table based on the interval duration, and determines whether the write operation of the non-relational database is successful. If not, the process continues to return to the first timer to trigger the non-relational database to perform a write operation based on the message entity in the retry message table based on the interval duration until it is determined that the write operation is successful and the message entity corresponding to the write operation in the retry message table is deleted, or the number of repeated executions is greater than or equal to the set number of times.

[0010] In the above solution, the method further includes: The message entities corresponding to the number of repetitions greater than or equal to the set number are saved in the abnormal message table; The data in the abnormal message table is sent to the monitoring platform based on the second timer.

[0011] In the above solution, before the first timer triggers the non-relational database to perform a write operation based on the message entity in the retry message table based on the interval duration, the method further includes: Extracting the message entities in the retry message table into multiple retry queues based on a shard scanning method; wherein the message entities in each retry queue are prioritized based on the time of submission; The first timer triggers the non-relational database to perform a write operation based on the message entity in the retry message table based on the interval duration, including: Based on the priorities of the message entities in the retry queue, the first timer triggers the non-relational database to sequentially execute write operations on the corresponding message entities based on the interval duration.

[0012] The present application also provides a data processing device for use in a database application system having a relational database and a non-relational database. The device includes: a starting module, configured to start a transaction related to an execution operation of the relational database in response to a first instruction for starting a transaction; wherein the transaction has an annotation for identifying a transaction management method or starts the execution operation based on a transaction template; A monitoring module, configured to monitor a message event and, if the message event belongs to the transaction and has a retry call annotation, encapsulate a message entity corresponding to the message event and save the message entity to a retry message table; The processing module is configured to determine that after receiving a second instruction instructing to commit the transaction, a write operation of the non-relational database is executed based on the retry message table.

[0013] The embodiment of the present application further provides a database application system having a relational database and a non-relational database, the database application system comprising: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor is used to run the computer program, and the steps of the method of the embodiment of the present application are performed.

[0014] The embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method of the embodiment of the present application.

[0015] The embodiment of the present application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the steps of the method of the embodiment of the present application.

[0016] The technical scheme provided by the embodiment of the present application responds to a first instruction for starting a transaction, and starts a transaction related to an execution operation of a relational database; wherein the transaction has an annotation for identifying a transaction management method or starts the execution operation based on a transaction template; a message event is listened to, and it is determined that the message event belongs to the transaction and has a retry call annotation, so that a message entity corresponding to the message event is encapsulated and the message entity is saved to a retry message table; it is determined that a second instruction indicating that the transaction is submitted is received, and a write operation of a non-relational database is executed based on the retry message table. In this way, by listening to the message event after starting the transaction and before submitting the transaction, and encapsulating the message entity based on the message event belonging to the transaction and having the retry call annotation and saving the message entity to the retry message table, the write operation of the non-relational database is executed based on the retry message table after the transaction is submitted, the write operation of the non-relational database can be executed based on the retry message table after the execution operation of the relational database is successfully executed, the data consistency between the relational database and the non-relational database is ensured, the stability and reliability of the database application system are improved, the user's trust in data reliability is enhanced, and thus the user satisfaction is improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 It is a flowchart of the data processing method of the embodiment of the present application; Figure 2 It is a framework diagram of the database application system of the application embodiment; Figure 3 It is a flowchart of the data processing method of the application embodiment; Figure 4 It is another flowchart of the data processing method of the application embodiment; Figure 5 It is a structural diagram of the data processing device of the embodiment of the present application; Figure 6 This is a structural diagram of the database application system of the embodiment of the present application. DETAILED DESCRIPTION

[0018] The present application will be described in further detail below with reference to the accompanying drawings and embodiments.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application.

[0020] In the related art, when both relational and non-relational databases are used in a database application system, data inconsistency between the relational and non-relational databases often occurs, as follows: After a relational database transaction is successfully submitted, if the write to the non-relational database fails, data inconsistency may occur, affecting data reliability and integrity, leading to data consistency issues. In addition, after a non-relational database write fails, manual intervention or complex error handling mechanisms are required to restore consistency, which increases system complexity and maintenance costs, leading to error recovery complexity issues. Thirdly, the delay between the relational database transaction and the non-relational database write, especially under high load, may cause system performance to degrade if the transaction is started first and then the data is written to the non-relational database. This affects the application's responsiveness and leads to transaction delay issues.

[0021] Based on this, in various embodiments of the present application, by introducing a retry message table, the final data consistency between the relational database and the non-relational database is achieved based on the retry message table, thereby improving the stability and reliability of the database application system.

[0022] The embodiment of the present application provides a data processing method, which is applied to a database application system having a relational database and a non-relational database, such as Figure 1 As shown, the method includes: Step 101: In response to a first instruction to start a transaction, start a transaction related to an execution operation of the relational database; wherein the transaction has an annotation for identifying a transaction management method or starts the execution operation based on a transaction template.

[0023] Here, the programmable gateway control plane or service application can open a transaction for the database application system, for example, a transaction can be opened through the transaction management mechanism of the Spring framework, that is, a first instruction for opening a transaction is sent to the database application system, the transaction has an annotation for identifying a transaction management method or is started to perform an operation based on a transaction template, so that the relational database performs the operation atomically, that is, all succeed or all fail. For example, under the Spring framework, the method that needs transaction management can be identified by the annotation @Transactional, or the business can be operated by the TransactionTemplate, thereby providing data consistency guarantee.

[0024] Step 102, listen to the message event, and determine that the message event belongs to the transaction and has a retry call annotation, then encapsulate the message entity corresponding to the message event and save the message entity to the retry message table.

[0025] Here, the database application system can set an event listener, for example, EtcdListener, use the annotation @TransactionalEventListener to ensure that the message event is listened to before the transaction is committed, if the message event has a retry call annotation, for example, the message event uses the annotation @RetryInvoke, then encapsulate the message entity corresponding to the message event and save the message entity to the retry message table, thereby guaranteeing the data consistency between the non-relational database and the relational database.

[0026] Step 103, after determining that a second instruction indicating to commit the transaction is received, performing the write operation of the non-relational database based on the retry message table.

[0027] It can be understood that after the transaction is committed, the write operation of the non-relational database based on the retry message table can be performed after the execution operation of the relational database is successfully executed, thereby ensuring the data consistency between the relational database and the non-relational database, improving the stability and reliability of the database application system, enhancing the user's trust in data reliability, and thus improving user satisfaction.

[0028] It should be noted that the introduction of the retry message table can realize an automatic error detection and recovery mechanism, reduce the need for manual intervention, simplify the maintenance process, and improve the stability and reliability of the system; in addition, the write operation of the non-relational database based on the retry message table can not only avoid the transaction delay caused by writing data into the non-relational database in the transaction in related technologies, but also enhance the guarantee of data consistency between the relational database and the non-relational database, thereby improving the stability and reliability of the database application system.

[0029] Exemplarily, before monitoring the message event, the method further includes: Instruction information for generating a message event in executing the execution operation of the relational database is received, wherein the instruction information is used to generate a message event associated with the execution operation of the non-relational database and add the retry call annotation to the message event.

[0030] Here, the programmable gateway control plane or business application can publish the indication information by defining an event class (such as EtcdPutEvent). Specifically, ApplicationEventPublisher can be used to publish the indication information. Its purpose is to notify the system that subsequent operations are to be performed, thereby preparing for event-driven processing logic.

[0031] Exemplarily, the message entity encapsulating the message event includes: Based on the AOP aspect, it is determined that the thread generating the message event is the thread of the transaction, and then based on the method name, parameters and class of the message event, the message entity corresponding to the message event is encapsulated; wherein, the method name represents the operation identifier of the message event, the parameters are used to provide necessary data input for the method execution of the message event, and the class is used to define the data structure of the method execution of the message event.

[0032] In the Spring framework, AOP is used to encapsulate the common behavior of multiple classes, encapsulating logic that is not business-related but is commonly called by business modules, reducing system duplication and lowering the coupling between modules. Here, after determining that a message event has the @RetryInvoke annotation, based on the AOP aspect, the thread generating the message event is determined to be the transaction thread. The message entity of the message event (such as RetryInvokeDTO) is then encapsulated. This message entity can include the aforementioned method name, parameters, and class information.

[0033] For example, the encapsulated message entity can be saved in a retry message table. For example, a hash code can be calculated based on the business type and business unique identifier and stored in the retry message table of a relational database. This facilitates subsequent write operations to a non-relational database based on the retry message table, ensuring data consistency between the relational and non-relational databases. Storing the message entity in the retry data table of a relational database can use a data access object to interact with the relational database to ensure that the message record is persisted in the database. For example, the data access object is RetryInvokeRecordDao.

[0034] Exemplarily, after determining that the second instruction indicating to commit the transaction is received, performing the write operation of the non-relational database related to the transaction based on the retry message table comprises: determining that the second instruction indicating to commit the transaction is received, performing the write operation of the non-relational database related to the transaction; judging whether the write operation of the non-relational database is successful, if yes, deleting the message entity corresponding to the write operation in the retry message table, if no, performing the write operation of the non-relational database based on the message entity in the retry message table according to the first timer.

[0035] Here, the database application system can perform the write operation of the non-relational database related to the transaction after the transaction is committed, i.e., after determining that the second instruction indicating to commit the transaction is received, for example, a callback method is registered using TransactionSynchronizationManager, and the callback method is responsible for performing the write operation of the non-relational database. The design mechanism ensures that only after the relational data operation is successful, the interaction with the non-relational database will be attempted, reducing the risk of data inconsistency.

[0036] It can be understood that if the write operation of the non-relational database is successfully performed, the corresponding message entity in the retry message table is deleted to avoid redundant data. Exemplarily, the deletion can be realized by calling a deletion method, for example, RetryInvokeRecordDao.deleteById(). If the write operation of the non-relational database fails, the write operation of the non-relational database is performed on the message entity in the retry message table based on the retry mechanism of the first timer.

[0037] It should be noted that based on the retry mechanism of the first timer, the final data consistency between the relational database and the non-relational database can be finally realized. The final data consistency is applicable to the case where the result data is not required to be consistent at all times, and the intermediate state is allowed, but after a period of time, the data of each node can reach a state of final consistency. Based on the retry mechanism, the transaction delay caused by the operation between the relational database and the non-relational database can be effectively reduced, the overall performance of the system can be improved, and the support for the application can be ensured.

[0038] Exemplarily, the write operation of the non-relational database based on the message entity in the retry message table according to the first timer comprises: determining the interval length of repeatedly performing the write operation of the non-relational database and the set number of repetitions based on the obtained configuration information; The first timer triggers the non-relational database to perform a write operation based on the message entity in the retry message table based on the interval duration, and determines whether the write operation of the non-relational database is successful. If not, the process continues to return to the first timer to trigger the non-relational database to perform a write operation based on the message entity in the retry message table based on the interval duration until it is determined that the write operation is successful and the message entity corresponding to the write operation in the retry message table is deleted, or the number of repeated executions is greater than or equal to the set number of times.

[0039] Here, determining the interval duration and the set number of repetitions for repeatedly executing the write operation of the non-relational database based on the obtained configuration information can be understood as determining the configuration parameters of the retry mechanism of the first timer. The obtained configuration information can be obtained from a configuration center or from loading information such as startup files or program parameters, and this embodiment of the application does not limit this.

[0040] It is understandable that if the current write operation of the non-relational database fails, the first timer will update the number of repeated executions and the time of the next repeated execution based on the interval duration.

[0041] Exemplarily, the method further includes: The message entities corresponding to the number of repetitions greater than or equal to the set number are saved in the abnormal message table; The data in the abnormal message table is sent to the monitoring platform based on the second timer.

[0042] Here, once the number of repeated executions is greater than or equal to the set number, it is determined that the number of retries exceeds the preset maximum value, the status of the message entity is updated to failure (FAIL), and the failed message entity is transferred to the exception message table. This means that the operation requires manual intervention. This mechanism ensures that the system can detect and handle faults in a timely manner, avoiding long-term data inconsistency.

[0043] For example, a second timer can be used to periodically scan the data in the exception message table for failed message entities. These entities are then sent to the monitoring platform via middleware (such as Kafka). This solution supports different messaging middleware and monitoring platforms through the SPI (Service Provider Interface) mechanism. This allows the team to promptly understand system status and quickly respond to issues.

[0044] Exemplarily, before the first timer triggers the non-relational database to perform a write operation based on the message entity in the retry message table based on the interval duration, the method further includes: extracting the message entities in the retry message table into a plurality of retry queues based on a sharding scan, wherein the message entities in each retry queue are determined priority based on a time of submission; The first timer triggers the non-relational database to perform a write operation based on the message entities in the retry message table based on the interval duration, including: Based on the priority of the message entities in the retry queue, the first timer triggers the non-relational database to sequentially perform a write operation of the corresponding message entities based on the interval duration.

[0045] Here, the sharding scan refers to dividing the message entities in the retry message table into a plurality of slices based on the unique identifier of each message entity, for example, the aforementioned hashCode, and each slice corresponds to a retry queue. In this way, the retry write operation can be performed based on the retry queue after slicing, which can improve the throughput of the system. In addition, the retry queue uses a priority queue, and the priority is determined according to the time of submission. This step can ensure the order of the retry message as much as possible, that is, even if the system is down, the timing task will reload the message entities in the retry message table to the memory queue and queue them according to the write time of the message entities.

[0046] The application will be further described in detail in conjunction with the application examples.

[0047] In the application examples, the relational database MySQL and the non-relational database Etcd are taken as examples for illustrative description. The framework of the database application system is shown in Figure 2 Figure 3 The flowchart of the data processing method of the database application system is shown, and the data processing method includes: Figure 2 Figure 3 Step 301, load configuration.

[0048] Here, the system can load the configuration from the startup file or the program parameter, for example, reading the startup file can be reading the configuration file (such as config.json or config.yml) according to the predefined path, or extracting the configuration items from the command line parameters, and then merging the configuration file in the startup file with the configuration items in the command line parameters to obtain the merged configuration file, wherein the priority of the command line parameters is higher.

[0049] ​​​After loading the configuration, the system can parse the configuration file. For example, the corresponding parsing library (such as Jackson or SnakeYAML) can be used to parse the configuration file content into an object, and then the SPI mechanism can be used to select a suitable configuration center (such as Zookeeper, Consul, etcd, database, etc.) based on the parsed object. Then, the configuration information (for example, the configuration parameters of the retry mechanism) is obtained from the configuration center and saved in the configuration context for subsequent use.

[0050] Step 302: Start a transaction.

[0051] Here, business applications can use the Spring Framework's transaction management mechanism to initiate a transaction. This involves sending a transaction-initiating instruction to the database application system. This transaction can include annotations identifying transaction management methods or initiate operations based on transaction templates. This ensures that when executing operations on the relational database, all changes are atomic, meaning either all succeed or all fail. For example, in the Spring Framework, you can use the @Transactional annotation to identify methods requiring transaction management or use TransactionTemplate operations to ensure data consistency.

[0052] Step 303: Publish a message event.

[0053] Here, before committing the transaction, a message event is created and published. For example, this can be achieved by defining an event class (such as EtcdPutEvent) and using ApplicationEventPublisher to publish the message event. This step notifies the system of the upcoming operation, thus preparing for event-driven processing logic.

[0054] Step 304 , determine whether the monitored message event is annotated with @RetryInvoke , if so, execute step 305 , if not, execute step 308 .

[0055] Here, the system can set up an event listener, such as EtcdListener, and use the @TransactionalEventListener annotation to ensure that the message event is listened to before the transaction is committed. If the message event has a retry call annotation, for example, the message event uses the @RetryInvoke annotation, then execute step 305 to ensure data consistency between the non-relational database and the relational data; otherwise, execute step 308.

[0056] Step 305: encapsulate the message entity.

[0057] Here, AOP aspects (such as RetryInvokeAspect) are used to determine whether the current thread is in a transaction. If so, a message entity (such as RetryInvokeDTO) is encapsulated, including information such as the method name, parameters, and class. The retry time and number of attempts are dynamically calculated based on the configuration obtained from the configuration center. Furthermore, a hash code is calculated based on the business type and unique business ID and stored in a relational database. This step primarily requires slicing calculations during scheduled task table scans to improve system throughput. Storing retry messages is key to achieving eventual consistency, ensuring that failed operations can be tracked and retried.

[0058] Step 306: Save the message entity to the database.

[0059] Here, the encapsulated message entity is saved to the database retry message table so that the subsequent retry mechanism can be used. This usually involves a data access object (such as RetryInvokeRecordDao) to interact with the database and ensure that the message record is persisted in the database.

[0060] Step 307: Commit the local transaction.

[0061] Here, after receiving a transaction submission from a business application, the system can use TransactionSynchronizationManager to register a callback method. This method is responsible for executing the write operation to the non-relational database Etcd. This design ensures that interaction with Etcd is attempted only after the database operation is successful, reducing the risk of data inconsistency.

[0062] Step 308: Transfer the message to the memory queue.

[0063] Here, after the message entity that needs to be retried is saved to the database, the message entity is submitted to the retry queue. The retry queue uses a priority queue and determines the priority based on the submission time. This step can ensure the order of the retried message entities as much as possible. Even after the system crashes, the scheduled task will reload the message entities in the retry message table into the memory queue and queue them for processing according to the write time of the message entity.

[0064] Here, for messages without consistency requirements (that is, message events without @RetryInvoke annotation), they are directly written to the memory queue. If Etcd is not successfully written, there is no need to execute the subsequent retry mechanism.

[0065] Step 309 , determine whether the execution of Etcd is successful, if not, execute step 310 , if so, execute step 311 .

[0066] Step 310, update the number of retries and time.

[0067] Here, if the Etcd write fails, the exception is captured and the number of retries and the next retry time are updated. For example, the next retry time can be calculated according to the retry policy (i.e. the configuration parameters of the retry mechanism) configured from the configuration center or the database.

[0068] Step 311, delete the database message record.

[0069] Here, if the Etcd write is successful, the related message entity in the database is immediately deleted to avoid redundant data. This can be achieved by calling the delete method, such as RetryInvokeRecordDao.deleteById().

[0070] Step 312, shard scanning the whole database waiting for execution data.

[0071] In order to process the unsuccessful message records, the system sets a timing task (i.e. the first timer) to execute once every interval (e.g. 5 seconds), scanning the waiting records. This part can be implemented by elastic hob to periodically scan the shard records in order to improve the throughput and processing speed of the system, and then call the retry method (such as RetryInvokeService.retry()), so as to continuously attempt the operation until success. If the maximum number of retries is exceeded and the operation still fails, the failed record is transferred to the exception message table.

[0072] Step 313, determine whether the number of retries exceeds the limit, if not, execute step 314, if yes, execute step 315.

[0073] Step 314, continue retrying, i.e. return to step 308.

[0074] Step 315, update the state to FAIL.

[0075] Here, once the number of retries exceeds the preset maximum value, the state of the message entity is updated to FAIL, and step 316 is executed, i.e. the failed message entity is transferred to the exception message table.

[0076] Step 316, transfer data to the exception message table.

[0077] Here, the message entity with the state of FAIL is removed from the retry message table and transferred to the exception message table as an exception message.

[0078] Step 317, save the exception message to the exception message table.

[0079] Step 318, scan all data in the exception message table.

[0080] Here, the data in the abnormal message table may be scanned periodically based on the second timer, that is, failed message entities may be scanned.

[0081] Step 319: Send successfully and update the monitoring status.

[0082] Here, the failed message entity is sent to the monitoring platform through middleware (such as Kafka), and the monitoring status is updated to facilitate manual processing, achieve problem tracing and rapid response when problems occur.

[0083] Figure 4 Another flow chart of the data processing method of this application embodiment is shown, wherein the database application system includes: a MySQL database, an event system, an AOP aspect, an Etcd cache module, and a timer. The data processing method includes the following steps: Step 401: Start a transaction.

[0084] Here, the business application can start a transaction through the transaction management mechanism of the Spring framework. For example, in the Spring framework, you can use the @Transactional annotation to identify the method that requires transaction management or use the TransactionTemplate to operate the business, thereby providing data consistency guarantees.

[0085] Step 402: Publish a message event.

[0086] Here, before committing a transaction, the business application creates and publishes a message event to the event system. For example, this can be achieved by defining an event class (such as EtcdPutEvent) and using ApplicationEventPublisher to publish the message event.

[0087] Step 403: monitor message events.

[0088] Here, the event system can set an event listener, such as EtcdListener, and use the @TransactionalEventListener annotation to ensure that the message event is listened to before the transaction is committed. If the message event has a retry call annotation, for example, the message event uses the @RetryInvoke annotation, step 404 is executed.

[0089] Step 404: encapsulate the message entity.

[0090] Here, the current thread is determined whether in the transaction based on the AOP aspect. If in the transaction, the message entity (such as RetryInvokeDTO) is encapsulated, including the method name, parameter and class information, and the retry time and number can be dynamically calculated according to the configuration obtained from the configuration center.

[0091] Step 405, save the message entity.

[0092] Here, the encapsulated message entity is saved to the MySQL data library retry message table, so that the subsequent retry mechanism can be used.

[0093] Step 406, commit the transaction.

[0094] Here, the business application can commit the transaction to the MySQL database after determining that the execution operation of the transaction is successfully executed.

[0095] Step 407, write data.

[0096] Here, after the transaction is committed, the Etcd write operation can be performed, if the write is successful, step 408 is executed, if the write fails, steps 409 to 411 are executed, until the retry number is greater than or equal to the set number, the state of the message entity is updated to FAIL, and transferred to the exception message table.

[0097] Step 408, delete the record.

[0098] Here, if the Etcd is successfully written, the related message entity in the MySQL database is immediately deleted to avoid redundant data.

[0099] Step 409, update the retry information.

[0100] Here, if the Etcd write fails, the exception is captured and the retry number and next retry time are updated, for example, the next retry time can be calculated according to the retry strategy (i.e. the configuration parameters of the retry mechanism) configured from the configuration center or the database.

[0101] Step 410, scan the records waiting for retry regularly.

[0102] Here, the timer scans the records waiting for execution based on the set interval, for example, elastichob can be used to realize regular scanning of shard records.

[0103] Step 411, continue to retry.

[0104] Here, the retry method (such as RetryInvokeService.retry()) can be called to continuously attempt the operation until success, if the maximum number of retries fails, the failed record is transferred to the exception message table.

[0105] Step 412: Periodically scan failure records.

[0106] Here, the data in the abnormal message table is scanned periodically based on a timer, that is, the failed message entities are scanned.

[0107] Step 413: Send monitoring information.

[0108] Here, the failed message entity is sent to the event system through middleware (such as Kafka), and the monitoring status is updated to facilitate manual processing, achieve problem tracing and rapid response when problems occur.

[0109] In order to implement the method of the embodiment of the present application, the embodiment of the present application also provides a data processing device, which is applied to a database application system having a relational database and a non-relational database. The data processing device corresponds to the above-mentioned data processing method, and each step in the above-mentioned data processing method embodiment is also fully applicable to the embodiment of the present data processing device.

[0110] like Figure 5 As shown, the data processing device includes: a startup module 501, a monitoring module 502 and a processing module 503. The startup module 501 is used to respond to a first instruction to start a transaction, start a transaction related to the execution operation of the relational database; wherein the transaction has an annotation for identifying a transaction management method or starts the execution operation based on a transaction template; the monitoring module 502 is used to monitor message events, and when it is determined that the message event belongs to the transaction and has a retry call annotation, encapsulate the message entity corresponding to the message event and save the message entity to a retry message table; the processing module 503 is used to determine that after receiving a second instruction instructing to commit the transaction, the write operation of the non-relational database is performed based on the retry message table.

[0111] In some embodiments, before monitoring the message event, the processing module 503 is further configured to: Instruction information for generating a message event in executing the execution operation of the relational database is received, wherein the instruction information is used to generate a message event associated with the execution operation of the non-relational database and add the retry call annotation to the message event.

[0112] In some embodiments, the monitoring module 502 encapsulates the message entity corresponding to the message event, including: If the thread that generates the message event is determined to be the thread of the transaction based on the AOP aspect, a message entity corresponding to the message event is encapsulated based on a method name, parameters, and a class of the message event, wherein the method name represents an operation identifier of the message event, the parameters are used to provide necessary data input for method execution of the message event, and the class is used to define a data structure of the method execution of the message event.

[0113] In some embodiments, the processing module 503 is specifically configured to: perform a write operation of the non-relational database related to the transaction after determining that the second instruction indicating that the transaction is submitted is received; determine whether the write operation of the non-relational database is successful, and if yes, delete the message entity corresponding to the write operation in the retry message table; and if no, perform the write operation of the non-relational database on the message entity in the retry message table based on the first timer.

[0114] In some embodiments, the processing module 503 performs the write operation of the non-relational database on the message entity in the retry message table based on the first timer, including: determining, based on the obtained configuration information, an interval duration for repeatedly performing the write operation of the non-relational database and a set number of times of repetition; the first timer triggers the non-relational database to perform the write operation based on the message entity in the retry message table based on the interval duration, and determines whether the write operation of the non-relational database is successful, and if no, continues to return the first timer triggering the non-relational database to perform the write operation based on the message entity in the retry message table based on the interval duration, until it is determined that the write operation is successful and the message entity corresponding to the write operation in the retry message table is deleted, or the number of times of repetition is greater than or equal to the set number of times.

[0115] In some embodiments, the processing module 503 is further configured to: save the message entity corresponding to the number of times of repetition being greater than or equal to the set number of times to an exception message table; send data in the exception message table to a monitoring platform based on a second timer.

[0116] In some embodiments, before the first timer triggers the non-relational database to perform the write operation based on the message entity in the retry message table based on the interval duration, the processing module 503 is further configured to: extract the message entity in the retry message table to a plurality of retry queues in a manner of shard scanning, wherein the message entity in each retry queue is determined to have a priority based on a time of submission; The first timer triggers the non-relational database to perform a write operation based on a message entity in the retry message table based on the interval duration, including: Based on the priority of the message entity in the retry queue, the first timer triggers the non-relational database to sequentially perform a write operation of the corresponding message entity based on the interval duration.

[0117] In actual application, the starting module 501, the monitoring module 502 and the processing module 503 can be implemented by a processor in the database application system. Of course, the processor needs to run a computer program in the memory to realize its function.

[0118] It should be noted that the data processing apparatus provided in the above embodiments is only exemplified by the division of the above program modules during data processing. In actual application, the above processing can be completed by different program modules according to needs, that is, the internal structure of the apparatus is divided into different program modules to complete all or part of the above-described processing. In addition, the data processing apparatus and the data processing method provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0119] Based on the hardware implementation of the above program modules, and in order to realize the method of the embodiments of the present application, the embodiments of the present application also provide a database application system. Figure 6 Only the exemplary structure of the database application system is shown, not all structures, and the required Figure 6 structure or all structures can be implemented.

[0120] As Figure 6 shown, the database application system 600 provided by the embodiments of the present application includes at least one processor 601, a memory 602, a user interface 603 and at least one network interface 604. The various components in the database application system 600 are coupled together through a bus system 605. It can be understood that the bus system 605 is used to realize the connection communication between the components. The bus system 605 includes not only a data bus, but also a power bus, a control bus and a status signal bus. However, in order to clearly illustrate, all kinds of buses are marked as bus system 605 in Figure 6 .

[0121] Among them, the user interface 603 can include a display, a keyboard, a mouse, a trackball, a click wheel, a key, a button, a touchpad or a touch screen, etc.

[0122] The memory 602 in the embodiments of the present application is used to store various types of data to support the operation of the database application system. Examples of these data include: any computer program used to operate on the database application system.

[0123] The data processing methods disclosed in the embodiments of this application can be applied to or implemented by processor 601. Processor 601 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the data processing method can be completed by hardware integrated logic circuits or software instructions in processor 601. Processor 601 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc. Processor 601 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium located in memory 602. Processor 601 reads information from memory 602 and, in conjunction with its hardware, completes the steps of the data processing methods provided in the embodiments of this application.

[0124] In an exemplary embodiment, the database application system can be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the aforementioned method.

[0125] It is understood that memory 602 can be volatile memory or non-volatile memory, or can include both volatile and non-volatile memory. Non-volatile memory can include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); magnetic surface memory can include magnetic disk or tape memory. Volatile memory can include random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory described in the embodiments of the present application is intended to include, but not limited to, these and any other suitable types of memory.

[0126] In the example embodiments, the embodiments of the present application further provide a computer storage medium, which can be specifically a computer readable storage medium, for example, a memory 602 storing a computer program, which can be executed by the processor 601 of the database application system to complete the steps described in the embodiments of the present application. The computer readable storage medium can be a ROM, a PROM, an EPROM, an EEPROM, a Flash Memory, a magnetic surface memory, an optical disc, or a CD-ROM memory, etc.

[0127] In the example embodiments, the embodiments of the present application further provide a computer program product, which includes a computer program, which can be executed by the processor 601 of the database application system 600 to complete the steps described in the embodiments of the present application.

[0128] It should be noted that "first", "second", etc. are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0129] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0130] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. A data processing method, characterized in that: Applied to a database application system having a relational database and a non-relational database, the method includes: In response to a first instruction to start a transaction, start a transaction related to an execution operation of the relational database; wherein the transaction has an annotation for identifying a transaction management method or starts the execution operation based on a transaction template; Monitor a message event, and determine that the message event belongs to the transaction and has a retry call annotation, then encapsulate a message entity corresponding to the message event and save the message entity to a retry message table; After determining that a second instruction instructing to commit the transaction is received, a write operation of the non-relational database is performed based on the retry message table.

2. The method according to claim 1, characterized in that Before monitoring the message event, the method further includes: Instruction information for generating a message event in executing the execution operation of the relational database is received, wherein the instruction information is used to generate a message event associated with the execution operation of the non-relational database and add the retry call annotation to the message event.

3. The method according to claim 1, characterized in that The message entity corresponding to the encapsulated message event includes: Based on the aspect-oriented programming (AOP) aspect, it is determined that the thread generating the message event is the thread of the transaction, and then based on the method name, parameters and class of the message event, the message entity corresponding to the message event is encapsulated; wherein, the method name represents the operation identifier of the message event, the parameters are used to provide necessary data input for the method execution of the message event, and the class is used to define the data structure of the method execution of the message event.

4. The method according to claim 1, wherein After determining that the second instruction instructing to commit the transaction is received, performing the write operation on the non-relational database based on the retry message table includes: After determining that a second instruction instructing to commit the transaction is received, executing a write operation on the non-relational database related to the transaction; Determine whether the write operation of the non-relational database is successful. If so, delete the message entity corresponding to the write operation in the retry message table; if not, execute the write operation of the non-relational database on the message entity in the retry message table based on the first timer.

5. The method according to claim 4, characterized in that The performing the write operation of the non-relational database on the message entity in the retry message table based on the first timer includes: Determining, based on the acquired configuration information, an interval duration for repeatedly executing the write operation of the non-relational database and a set number of repeated executions; The first timer triggers the non-relational database to perform a write operation based on the message entity in the retry message table based on the interval duration, and determines whether the write operation of the non-relational database is successful. If not, the process continues to return to the first timer to trigger the non-relational database to perform a write operation based on the message entity in the retry message table based on the interval duration until it is determined that the write operation is successful and the message entity corresponding to the write operation in the retry message table is deleted, or the number of repeated executions is greater than or equal to the set number of times.

6. The method according to claim 5, characterized in that The method further comprises: The message entities corresponding to the number of repetitions greater than or equal to the set number are saved in the abnormal message table; The data in the abnormal message table is sent to the monitoring platform based on the second timer.

7. The method according to claim 5, characterized in that Before the first timer triggers the non-relational database to perform a write operation based on the message entity in the retry message table based on the interval duration, the method further includes: Extracting the message entities in the retry message table into multiple retry queues based on a shard scanning method; wherein the message entities in each retry queue are prioritized based on the time of submission; The first timer triggers the non-relational database to perform a write operation based on the message entity in the retry message table based on the interval duration, including: Based on the priorities of the message entities in the retry queue, the first timer triggers the non-relational database to sequentially execute write operations on the corresponding message entities based on the interval duration.

8. A data processing device, characterized in that: Applicable to a database application system having a relational database and a non-relational database, the device comprises: a starting module, configured to start a transaction related to an execution operation of the relational database in response to a first instruction for starting a transaction; wherein the transaction has an annotation for identifying a transaction management method or starts the execution operation based on a transaction template; A monitoring module, configured to monitor a message event and, if the message event belongs to the transaction and has a retry call annotation, encapsulate a message entity corresponding to the message event and save the message entity to a retry message table; The processing module is configured to determine that after receiving a second instruction instructing to commit the transaction, a write operation of the non-relational database is executed based on the retry message table.

9. A database application system, characterized in that: The database application system has a relational database and a non-relational database, and includes: a processor and a memory for storing a computer program that can be run on the processor, wherein: The processor is configured to execute the steps of the method according to any one of claims 1 to 7 when running a computer program.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 7.

Images