Database data processing method and device, program product, equipment and storage medium

Abstract

The invention provides a data processing method and device for a database, a computer program product, equipment and a storage medium, related embodiments can be applied to various scenes such as database migration and full data backup, and the method comprises the following steps: analyzing binary log information to obtain a to-be-processed transaction; detecting an event distribution state of the to-be-processed transaction to obtain a distribution state detection result; determining a thread matched with the to-be-processed transaction according to the distribution state detection result; executing the to-be-processed transaction through the thread to obtain an execution result of the to-be-processed transaction; and sending the execution result of the to-be-processed transaction to the target database, thereby not only controlling the transaction execution process according to the relevance between the transactions and ensuring the ordered execution between the associated transactions, but also controlling a plurality of threads to perform parallel execution on the transactions and improving the transaction execution efficiency. And the time for waiting for transaction execution is shortened.

Description

Technical Field

[0001] The present application relates to database processing technology, and in particular to a database data processing method, device, computer program product, electronic device and storage medium. Background Art

[0002] In database processing technology, data replication (or data synchronization) between databases mainly relies on the playback of binary logs. Transactions in binary logs are generated by the source database, and there is a sequence between related transactions. If the order of transaction execution is disrupted, the migrated data will be disordered, or data migration cannot be achieved due to data errors. Therefore, in order to ensure the accuracy of data replication between databases, related technologies can only control the serial execution of transactions, resulting in low efficiency and long waiting time for transaction execution. Therefore, a database data processing method is needed to ensure the orderly execution of related transactions on the basis of parallel execution of transactions and ensure the accuracy of transaction execution results. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a database data processing method, apparatus, computer program product, device and storage medium, which can not only control the transaction execution process according to the correlation between transactions to ensure the orderly execution of related transactions, but also control multiple threads to execute transactions in parallel, thereby improving the efficiency of transaction execution and reducing the time waiting for transaction execution to complete.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a method for processing database data, the method comprising:

[0006] Get the binary log information of the source database;

[0007] Parsing the binary log information to obtain pending transactions;

[0008] Detecting the event distribution status of the pending transaction to obtain a distribution status detection result;

[0009] Determining a thread matching the transaction to be processed according to the distribution status detection result;

[0010] Executing the pending transaction through the thread to obtain an execution result of the pending transaction;

[0011] The execution result of the pending transaction is sent to the target database.

[0012] The present application also provides a database data processing device, the device comprising:

[0013] Information transmission module, used to obtain binary log information of the source database;

[0014] A status detection module is used to parse the binary log information to obtain pending transactions;

[0015] The state detection module is further used to detect the event distribution state of the pending transaction to obtain a distribution state detection result;

[0016] An information processing module, used for determining a thread matching the transaction to be processed according to the distribution status detection result;

[0017] The information processing module is further used to execute the pending transaction through the thread to obtain the execution result of the pending transaction;

[0018] The information transmission module is also used to send the execution result of the pending transaction to the target database.

[0019] In some embodiments, the information processing module is further used to determine a first thread matching the to-be-processed transaction according to a load of each thread in a thread queue when the distribution status detection result indicates that the start event is distributed;

[0020] The information processing module is further configured to detect a position semaphore corresponding to the data manipulation language event to obtain a state of the position semaphore when the distribution state detection result indicates that the data manipulation language event is distributed, and

[0021] The information processing module is further configured to determine, in the thread queue, a second thread matching the transaction to be processed when the state of the position semaphore is a completion state.

[0022] In some embodiments, the information processing module is further configured to change the status of the position semaphores of all events in the pending transaction to a completion status when the distribution status detection result indicates that the end event has been distributed.

[0023] In some embodiments, the information processing module is further used to determine a data manipulation language event and an end event associated with the start event;

[0024] The information processing module is further used to obtain the load of each thread in the thread queue;

[0025] The information processing module is further used to sort the load of each thread in the thread queue, and determine the thread with the smallest load according to the sorting result to serve as the first thread.

[0026] In some embodiments, the information processing module is further used to send the data operation language event and the end event associated with the start event in the pending transaction to the event waiting execution queue of the first thread;

[0027] The information processing module is further used to configure a position semaphore for the data operation language event in the event waiting execution queue, and obtain the position semaphore of the data operation language event of the pending transaction;

[0028] The information processing module is further configured to obtain a position signal of a data operation language event in the associated transaction when it is determined that the to-be-processed transaction has data correlation with the associated transaction;

[0029] The information processing module is further used to generate a transaction waiting chain according to the execution order of the pending transaction and the associated transaction, using the position semaphore of the data operation language event of the pending transaction and the position semaphore of the data operation language event in the associated transaction;

[0030] The information processing module is further used to control the parallel execution of the data operation language event in the pending transaction and the data operation language event of the associated transaction through the transaction waiting chain.

[0031] In some embodiments, the information processing module is further used to adjust the state of the position semaphore corresponding to each event in the waiting execution queue to a completion state when each event in the waiting execution queue is completed by the first thread.

[0032] In some embodiments, the information processing module is further used to parse the data operation language event to obtain a parsing result of the data operation language event;

[0033] The information processing module is further used to filter the analysis results according to the identifier of the data correlation keyword to obtain the correlation keyword;

[0034] The information processing module is further used to detect the position semaphore corresponding to the data operation language event according to the correlation keyword to obtain a target position semaphore, wherein the target position semaphore is used to represent the state of the data operation language event in the transaction associated with the to-be-processed transaction;

[0035] The information processing module is further used to detect the state of the target position signal quantity and determine the state of the target position signal quantity.

[0036] In some embodiments, the information processing module is further used to perform real-time detection on the state of the target position signal quantity to obtain a real-time detection result;

[0037] The information processing module is further configured to select a thread with the smallest load in the thread queue when the real-time detection result indicates that the state of the target position signal quantity changes from a pending state to a completed state;

[0038] The information processing module is further configured to use the thread with the smallest load as a second thread matching the transaction to be processed.

[0039] In some embodiments, the information processing module is further used to search for at least one target database corresponding to the source database in the target database list;

[0040] The information processing module is further used to determine the cloud server network corresponding to the at least one target database according to the identification information of the data of the at least one target database;

[0041] The information processing module is further configured to obtain the execution result of the transaction to be processed when the status of all position semaphores of the transaction to be processed is a completion status;

[0042] The information transmission module is further used to store the execution result of the pending transaction in the cloud server network;

[0043] The information transmission module is further used to obtain the execution result of the pending transaction stored in the cloud server network when data loss occurs in any target database of the at least one target database;

[0044] The information transmission module is also used to send the execution result of the pending transaction to the corresponding target database.

[0045] The present application also provides an electronic device, the electronic device comprising:

[0046] A memory for storing executable instructions;

[0047] The processor is used to implement the above-mentioned database data processing method when running the executable instructions stored in the memory.

[0048] The embodiment of the present application also provides a computer-readable storage medium storing executable instructions, which implement the above-mentioned database data processing method when executed by a processor.

[0049] The embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the above-mentioned database data processing method.

[0050] The embodiments of the present application have the following beneficial effects:

[0051] 1) The embodiment of the present application obtains binary log information of the source database; parses the binary log information to obtain pending transactions; detects the event distribution status of the pending transaction to obtain a distribution status detection result; and determines the thread that matches the pending transaction according to the distribution status detection result; thereby, it is possible to configure a suitable thread for the pending transaction according to the distribution status detection result, and achieve load balancing of the threads that execute the transaction. In addition, the transaction execution process can be controlled according to the correlation between transactions, ensuring the orderly execution of related transactions, and controlling multiple threads to execute transactions in parallel, thereby improving the efficiency of transaction execution and reducing the time waiting for transaction execution to complete.

[0052] 2) Execute the pending transaction through the thread to obtain the execution result of the pending transaction; send the execution result of the pending transaction to the target database. In this way, the data in the source database is migrated to the target database using the execution result of the pending transaction, ensuring that the data after the data migration is not disordered and affects the normal use of the target database. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of an application scenario of the database data processing method provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of an implementation flow of a method for processing data in a database provided in an embodiment of the present application;

[0055] Figure 3 This is a schematic diagram of the composition of binary log information in an embodiment of the present application;

[0056] Figure 4 This is a diagram illustrating a scenario of database migration in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of an implementation flow of a method for processing data in a database provided in an embodiment of the present application;

[0058] Figure 6 This is a schematic diagram of thread processing when the pending transactions have no correlation in an embodiment of the present application;

[0059] Figure 7 This is a schematic diagram of thread processing when there is correlation between the transactions to be processed in an embodiment of the present application;

[0060] Figure 8 This is a schematic diagram of thread processing when there is correlation between the transactions to be processed in an embodiment of the present application;

[0061] Fig. 9It is a structural diagram of an electronic device provided in an embodiment of the present application for executing the database data processing method provided in the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0063] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0064] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0065] 1) Terminals, including but not limited to: ordinary terminals and dedicated terminals, wherein ordinary terminals maintain a long connection or a short connection with the sending channel, and dedicated terminals maintain a long connection with the sending channel.

[0066] 2) Client, which is a carrier for implementing specific functions in the terminal. For example, a mobile client (APP) is a carrier for specific functions in a mobile terminal, such as the function of executing report making or the function of displaying reports.

[0067] 3) Logical log: A log that records logical changes to data in a database, generally used to synchronize data between databases.

[0068] 4) Database (DB): In short, it can be regarded as an electronic file cabinet - a place to store electronic files, where users can add, query, update, delete, etc. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application.

[0069] 5) Database Management System (DBMS) is a computer software system designed for managing databases. It generally has basic functions such as storage, interception, security, backup, migration, and repair. Database migration systems can be classified according to the database model they support, such as relational, XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters, mobile phones; or according to the query language used, such as SQL (Structured Query Language), XQuery; or according to performance impulse focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMS can cross categories, for example, supporting multiple query languages ​​at the same time.

[0070] 6) Server cluster refers to the collection of at least two servers to perform the same service together, which looks like only one server to the client. The server cluster can use multiple computers for parallel computing to obtain a very high computing speed, and can also use multiple computers for backup to achieve data migration in the source database, so that the entire system can still operate normally even if any machine is broken. In the data processing method of the database provided in this application, it can be applied to cloud server usage scenarios and distributed server usage scenarios to achieve flexible migration of source database data in different usage scenarios and ensure the accuracy of data migration. For example, a cloud server (CVM, CloudVirtual Machine) is a simple, efficient, safe and reliable computing service with elastically scalable processing power. Its management method is simpler and more efficient than that of a traditional single physical server. The target object can quickly create or release any number of cloud servers for the target object's business process to use without purchasing hardware in advance, and store the data of the target object of the cloud server. The data and program of the target object in the distributed server usage environment may not be located on one server, but may be distributed to multiple servers. Similarly, the data processing method of the database provided in this application can be used to control the execution process of parallel transactions to achieve migration of data stored in the cloud server.

[0071] 7)Binlog: MySQL binary log, mainly used for replication between MySQL master and slave servers.

[0072] 8) Data Definition Language (DDL): It is the language in the Structured Query Language (SQL) language set that is responsible for data structure definition and database object definition.

[0073] 9) Data Manipulation Language (DML) is a set of instructions in the SQL language set that is responsible for performing data access tasks on database objects.

[0074] 10) Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing. It is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application, which can form a resource pool and be used on demand, flexible and convenient. Cloud computing technology will become an important support. The background services of technical network systems require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0075] 11) Cloud Security refers to the general term for security software, hardware, target objects, organizations, and security cloud platforms based on cloud computing business model applications. Cloud security integrates emerging technologies and concepts such as parallel processing, grid computing, and unknown virus behavior judgment. Through a large number of clients in a network, it detects abnormal software behavior in the network, obtains the latest information on Trojans and malicious programs on the Internet, and sends it to the Unreal Engine server for automatic analysis and processing, and then distributes virus and Trojan solutions to each client. In order to ensure the security of data stored in the cloud server, it is often necessary to migrate the data in the source database to the cloud server when the cloud security detection passes, so as to achieve data backup or replication.

[0076] Before describing the database data processing method provided by the present application, the defects in the related art are described first. The inventors found in their research that when migrating data stored in the source database, the control of the transaction execution process mainly includes the following methods:

[0077] 1) In data migration, transactions are executed serially in the order of source database records. The disadvantage of this method is that although the order of data migration can be guaranteed, transactions cannot be executed in parallel in the target database, resulting in slow data migration and long waiting time for users.

[0078] 2) In data migration, transactions without data correlation are controlled to be executed in parallel, and transactions with data correlation are executed in series. The defects of this method are as follows: a) Due to the complexity of the data in the source database, the complexity of judging correlation is high, and the data correlation is often only partially correlated, and the unrelated parts cannot be executed in parallel. Related transactions are queued for execution in the same thread, which often leads to unbalanced loads between different threads, resulting in a decrease in the parallel processing capability of the thread for transactions. b) Due to the complexity of the data in the source database, the judgment of transaction correlation often requires complete transaction information to be judged, which results in a large cache overhead to cache the entire transaction for correlation judgment for transactions with a large number of events, and high requirements for the hardware equipment of data migration. In extreme cases, it may not even be cached, which increases the hardware equipment cost of users. c) Due to the different data types of the source database, a transaction may be correlated with multiple previous transactions, but these related transactions may be distributed to different threads for execution, resulting in the inability to distribute the transaction in advance. It must wait for all previous transactions to be executed before continuing, resulting in a decrease in the parallel capability of the transaction, slow data migration, and long waiting time for users.

[0079] In order to solve the above-mentioned defects, the present application provides a database data processing method, which can control the transaction execution process according to the correlation between transactions to ensure the orderly execution of related transactions, and can control multiple threads to execute transactions in parallel, thereby improving the efficiency of transaction execution and reducing the time waiting for transaction execution to complete, so as to adapt to the needs of parallel transaction processing during different types of data migration.

[0080] Figure 1 This is a schematic diagram of an application scenario of the data processing method of the database provided in the embodiment of the present application, see Figure 1, with the continuous development of computer technology, cloud servers can provide secure and reliable elastic computing services, and can also provide different instance types to meet the specific usage scenarios of target objects. Electronic device 100 can include two different types of terminals or electronic devices, namely terminal 10-1 and terminal 10-2, and the target object can be selected according to the usage requirements of different types of data migration usage scenarios in the database. The terminal (including terminal 10-1 and terminal 10-2) is provided with a corresponding client capable of running the data processing function of the database. When implementing the data processing method of the database provided by the present application, when the binary log information that needs to be migrated is encrypted data, the terminal (including terminal 10-1 and terminal 10-2) needs to first obtain the corresponding key from the cloud server 200 to decrypt the obtained binary log information. After obtaining the plaintext binary log information, it is converted into a pending transaction to be executed by the corresponding thread. According to the result of the transaction execution, the data in the source database in terminal 10-1 is migrated to the target database of terminal 10-2. The above migration process of migrating the data of the source database in the terminal 10 - 1 to the target database in the terminal 10 - 2 through the cloud server 200 depends on the binary log (Binlog) of the source database.

[0081] Binary log (Binlog) is a binary format file used to record the SQL statement information of the user to update the database. For example, SQL statements for changing database tables and changing content will be recorded in Binlog, but queries on database tables and other contents will not be recorded. By default, Binlog logs are in binary format and cannot be viewed using commands of text viewing tools. Instead, they can be parsed and viewed using corresponding tools.

[0082] Binlog can be used in cross-terminal data migration. For example, when data is written to the database, the updated SQL statements are also written to the corresponding Binlog file. When backing up the database, you can only back up all the data for a period of time. However, if the database server fails after the backup, you can migrate the data based on the update statements recorded in the Binlog. Binlog is also used for data backup of master-slave databases.

[0083] See also Figure 2 , Figure 2A schematic diagram of an implementation flow of a data processing method for a database provided in an embodiment of the present application, wherein the embodiment of the present application can be implemented in combination with cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to realize data calculation, storage, processing, and sharing. It can also be understood as a general term for network technology, information technology, integration technology, management platform technology, and application technology based on cloud computing business model applications. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites, and more portal websites, so cloud technology needs to be supported by cloud computing.

[0084] It should be noted that cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called the "cloud". From the user's point of view, the resources in the "cloud" can be infinitely expanded, and can be obtained at any time, used on demand, expanded at any time, and paid for by use. As a basic capability provider of cloud computing, a cloud computing resource pool platform will be established, referred to as a cloud platform, generally referred to as Infrastructure as a Service (IaaS Infrastructure as a Service), and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (which can be virtualized machines, including operating systems), storage devices and network devices. Different operation permissions are granted to the target objects that execute transactions through cloud servers.

[0085] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network to work together through application software or application interfaces to provide data storage and business access functions to the outside world. In the related art, the storage method of the storage system is: create a logical volume. When creating a logical volume, a physical storage space is allocated to each logical volume. The physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, the data is stored on the file system. The file system divides the data into many parts, each of which is an object. The object not only contains data but also contains additional information such as data identification (ID IDentity). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object. Therefore, when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object. The process of the storage system allocating physical storage space to a logical volume is as follows: based on the estimated capacity of the objects stored in the logical volume (this estimate often has a large margin relative to the capacity of the actual objects to be stored) and the grouping of independent redundant disk arrays (RAID Redundant Array of Independent Disks), the physical storage space is divided into stripes in advance. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0086] Understandably, Figure 2 The execution body of the steps shown can be implemented independently by a terminal of a data processing device running a database, or can be implemented by a server of the data processing device running a database. When performing data migration, the server of the data processing device running a database can be used as a data migration module for users to deploy and use in the data migration system. Figure 2 The processing steps shown in the figure are explained by taking the server as the execution subject as an example. Figure 2 The steps shown are explained.

[0087] Step 201: Obtain binary log information of the source database.

[0088] refer to Figure 3 , Figure 3 The following is a schematic diagram of the composition of binary log information in an embodiment of the present application. Figure 3As shown in the figure, binary log information can be represented in the form of binlog event stream. According to the characteristics of binlog event stream, events of the same transaction appear in order with transactions as the boundary. After one transaction ends, another transaction begins. During the duration of the entire data migration task, the binlog event stream obtained by the data migration module does not stop. In the binlog event stream, a pending transaction uses the begin event as the boundary of the transaction start, followed by a series of data manipulation language events (DML events), and then an event is the end event (end event) that marks the end of the transaction. After a transaction ends, another transaction will begin, and there is no event overlap between transactions.

[0089] Step 202: Parse the binary log information to obtain pending transactions.

[0090] In some embodiments, by using the pending transactions obtained by binary log information to execute the database data processing method provided in the present application, data migration, full database backup and database repair can be achieved. In the data migration scenario, the source of the binary log information is the target database; the execution result of the pending transaction is sent to the target database, so that the data in the source database can be migrated to the target database.

[0091] In some embodiments, in a full backup scenario of a database, the source of binary log information may be a cloud server or a source database. When a full backup of the database is performed, the execution results of pending transactions may be sent to a data backup server in a cloud server cluster, and full backup of data in the cloud server or source database may be achieved in the data backup server, so as to quickly obtain corresponding data from the data backup server.

[0092] In some embodiments, in a full database backup scenario and a database repair scenario, the source of the binary log information may be a cloud server or a source database. When repairing the database, the execution results of pending transactions may be sent to the target database, so as to achieve targeted repair of damaged data in the target database or comprehensive repair of data in the target database.

[0093] Step 203: Detect the event distribution status of the transaction to be processed to obtain a distribution status detection result.

[0094] In some embodiments, the types of events of the pending transaction include three types, namely: a start event, at least one data manipulation language event (DML event), and an end event, wherein the data manipulation language event may refer to statements such as INSERT, UPDATE, and DELETE that modify data in a table or view of a database. For example, the trigger condition of a DML trigger is set to a predetermined operation such as data insertion, data update, and data deletion performed on a database table. When a user performs a data insertion (INSERT), data update (UPDATE), or data deletion (DELETE) operation, the DML trigger is activated, and the trigger operation is performed when the trigger condition is met, that is, the current value of the sequence of the database is obtained.

[0095] Step 204: Determine a thread that matches the transaction to be processed according to the distribution status detection result.

[0096] In some embodiments, determining a thread matching a pending transaction according to a distribution status detection result may be implemented in the following manner:

[0097] When the distribution status detection result represents a distribution start event, the first thread matching the transaction to be processed is determined according to the load of each thread in the thread queue; since the current server is in a distribution start event, it means that a transaction has started to be executed. At this time, it is necessary to configure a corresponding thread for the transaction to be executed, and the configured thread executes the start event, at least one data operation language event, and the end event in the transaction in sequence until the transaction is executed.

[0098] In some embodiments, determining the first thread matching the to-be-processed transaction according to the load of each thread in the thread queue may be implemented in the following manner:

[0099] Determine a data manipulation language event and an end event associated with the start event; obtain the load of each thread in the thread queue; sort the load of each thread in the thread queue, and determine the thread with the smallest load as the first thread according to the sorting result. For example: there are three threads (thread 1, thread 2, and thread 3) in the thread queue that can execute pending transaction a. When it is determined that the thread with the smallest load is thread 1, the start event, at least one data manipulation language event, and the end event of transaction a are all distributed to thread 1, and thread 1 executes the start event, at least one data manipulation language event, and the end event of transaction a in sequence.

[0100] In some embodiments, when the number of threads with the smallest load is greater than or equal to 2, the thread with the first identification order among the threads with the smallest load is selected to execute the corresponding pending transaction according to the identification order of the threads. For example, there are three threads (thread 1, thread 2 and thread 3) in the thread queue that can execute pending transaction a. When the load of thread 1 is equal to the load of thread 2 and the load of thread 1 is less than the load of thread 3, thread 1 with the first identification order among the threads with the smallest load is selected, and the start event, at least one data operation language event and the end event of transaction a are executed in sequence through thread 1.

[0101] In some embodiments, when executing pending transactions through the first thread, the pending transactions may need to be executed according to the execution results of associated transactions, for example, the data operation language event a12 of pending transaction a needs the execution result of associated transaction b, so the orderly execution between associated transactions can ensure that the migrated data is not disordered. Therefore, the execution order between multiple transactions needs to be accurately controlled.

[0102] In some embodiments, in order to ensure the orderly execution of associated transactions, the data manipulation language event and the end event associated with the start event in the pending transaction can be first sent to the event waiting execution queue of the first thread; secondly, a position semaphore is configured for the data manipulation language event in the event waiting execution queue to obtain the position semaphore of the data manipulation language event of the pending transaction; then, when it is determined that there is data correlation between the pending transaction and the associated transaction, the position semaphore of the data manipulation language event in the associated transaction is obtained; according to the execution order of the pending transaction and the associated transaction, a transaction waiting chain is generated using the position semaphore of the data manipulation language event of the pending transaction and the position semaphore of the data manipulation language event in the associated transaction; finally, through the transaction waiting chain, the parallel execution of the data manipulation language event in the pending transaction and the data manipulation language event of the associated transaction is controlled.

[0103] In some embodiments, taking the migration of game data as an example, with the explosive growth of game data, it is increasingly difficult for traditional relational databases to meet the requirements of high concurrent reading and writing, efficient storage and access of massive data, high scalability and high availability. Therefore, it is necessary to migrate the game data in the source database to the target database. Due to the fragmentation of game time, frequent interactions between players, large data volumes and many operational activities, the read requests, write requests and delete requests for game data in the source database are not only numerous but also instantaneous. Therefore, the execution of the data operation language event a12 (prop query for all game characters of game users) of the pending transaction a (game user prop migration) requires the execution result of the associated transaction b (query on changes in the number of game user props before the migration time 9:00); wherein the associated transaction b is executed by thread 2. After determining the matching thread 1 of the pending transaction a, the data manipulation language event a11, data manipulation language event a12 and end event a2 associated with the start event a0 are sent to the waiting execution queue of thread 1, and the position semaphore signala12 is configured for the data manipulation language event a12; then the position semaphore signalb12 is configured for the data manipulation language event b12 in the associated transaction b (the type of props obtained by the game user in all operation activities), and according to the execution order ba of the pending transaction and the associated transaction, the position semaphore signala12 of the data manipulation language event of the pending transaction a and the position semaphore signalb12 of the data manipulation language event in the associated transaction b are used to generate the transaction waiting chain signalb12-signala12, and the parallel execution of the data manipulation language event in the pending transaction a and the data manipulation language event in the associated transaction is controlled through the transaction waiting chain signalb12-signala12. The parallel execution of thread 1 and thread 2 can be expressed as follows:

[0104] Thread 1: start event a0, data manipulation language event a11, data manipulation language event a12, end event a2;

[0105] Thread 2: start event b0, data manipulation language event b11, data manipulation language event b12, end event b2.

[0106] Among them, the start event a0 and data operation language event a11 of thread 1 can be executed in parallel with the start event b0 and data operation language event b11 of thread 2. After the data operation language event b12 of thread 2 is executed, the execution result of the data operation language event b12 is sent to thread 1 according to the transaction waiting chain signalb12-signala12, so that the data operation language event a12 continues to be executed, so that thread 1 no longer needs to wait for thread 2 to execute a complete transaction b before making a correlation judgment, which effectively saves the waiting time for parallel processing of transactions.

[0107] In some embodiments, when the first thread executes the completion event to wait for each event in the execution queue, the state of the position semaphore corresponding to each event in the execution queue is adjusted to the completion state. Thus, by setting the position semaphores of all events in the transaction to the completion state, the subsequent waiting events can be awakened in time to enter the execution state, thereby avoiding the increase of the idle time of the thread. Since the time taken for the thread to execute the start event and the end event in the pending transaction is relatively short, when configuring the position semaphore, the position semaphore may not be configured for the start event and the end event. Therefore, in the embodiment of the present application, the state of the position semaphore corresponding to each event in the execution queue is adjusted to the completion state. In actual use, only the state of the position semaphore of the data operation language event in the execution queue can be adjusted.

[0108] In some embodiments, determining the thread that matches the pending transaction based on the distribution status detection result can be achieved in the following manner: when the distribution status detection result represents the distribution of a data operation language event, detecting the position semaphore corresponding to the data operation language event to obtain the state of the position semaphore, and when the state of the position semaphore is a completed state, determining a second thread that matches the pending transaction in the thread queue. Since the distribution of the data operation language event indicates that when the current transaction is executed through the thread, the execution results of the corresponding events in other transactions that have data associations with the current transaction are required as the conditions for the execution of the current transaction, therefore, in order to ensure the sequential execution of transactions, the execution order of transactions needs to be accurately controlled through the corresponding position semaphores.

[0109] In some embodiments, when the distribution state detection result indicates that a data manipulation language event is distributed, the position semaphore corresponding to the data manipulation language event is detected to obtain the state of the position semaphore, which can be implemented in the following manner:

[0110] Parse the data operation language event to obtain the parsing result of the data operation language event; filter the parsing result according to the identifier of the data correlation keyword to obtain the correlation keyword. According to the correlation keyword, detect the position semaphore corresponding to the data operation language event to obtain the target position semaphore, wherein the target position semaphore is used to characterize the state of the data operation language event in the transaction associated with the pending transaction; detect the state of the target position semaphore to determine the state of the target position semaphore. Among them, in the full database backup of financial data, due to the large amount of financial data and complex data types, in order to improve the writing efficiency, the full database backup can be performed in table-level concurrency or row-level concurrency. For example, the source database of Bank A has a user set with deposits higher than 100,000 yuan and the corresponding credit scores, refer to Table 1.

[0111] Table 1

[0112]

[0113] When performing a full backup of the source database of Bank A, a row-level concurrent method can be used to perform a full backup of the credit scores of users with deposits greater than 100,000 yuan. The corresponding data operation language events of transaction c of the full backup are c11 (find users with deposits greater than 100,000 yuan) and c12 (credit scores of users with deposits greater than 100,000 yuan). The data operation language event c11 is parsed, and after obtaining the parsing result of the data operation language event, the parsing result is filtered according to the identifier of the data correlation keyword, and the correlation keyword is "user ID card ID". The correlation keyword "user ID card ID" corresponds to the data operation language event d1 (user ID card ID legitimacy check) in the associated transaction d. The state of the target position semaphore corresponding to the data operation language event d1 is detected to determine the state of the target position semaphore.

[0114] In some embodiments, determining the second thread matching the pending transaction in the thread queue may be implemented in the following manner:

[0115] The state of the target position semaphore is detected in real time to obtain a real-time detection result; when the real-time detection result indicates that the state of the target position semaphore has changed from a pending state to a completed state, the thread with the smallest load is selected in the thread queue; the thread with the smallest load is used as the second thread matching the pending transaction. In combination with the above-mentioned embodiment, when the credit scores of users with deposits of more than 100,000 are fully backed up in a row-level concurrent manner, if the real-time detection result indicates that the state of the target position semaphore has changed from a pending state to a completed state, it means that the data operation language event d1 (user ID card ID legitimacy detection) has been executed (for example, ID card ID 37003 in Table 1 is illegal, and the other ID cards are legal). At this time, the thread with the smallest load can be selected in the thread queue; the thread with the smallest load is used as the second thread matching the pending transaction c. At this time, the load of all threads in the thread queue can be minimized, saving the hardware resource consumption of the server.

[0116] In some embodiments, due to the large number of transactions to be processed for the full backup of financial data, combined with the above embodiments, if all threads in the thread queue are fully loaded, the thread with the longest full-load running time can be used as the second thread matching the pending transaction c to save the waiting time of the pending transaction c.

[0117] In some embodiments, when the distribution state detection result indicates a distribution end event, the state of the position semaphores of all events in the pending transaction is changed to a completion state. Thus, by setting the position semaphores of all events in the transaction to a completion state, subsequent waiting events can be awakened in time to enter an execution state, thereby avoiding an increase in the idle time of threads.

[0118] Step 205: Execute the pending transaction through the thread to obtain the execution result of the pending transaction.

[0119] In some embodiments, the execution results of the pending transactions can be sent to the target database to migrate the data in the source database to the target database; the execution results of the pending transactions can be sent to the data backup server in the cloud server cluster to back up the full amount of data in the cloud server or the source database to the data backup server, so as to quickly obtain the corresponding data from the data backup server; the execution results of the pending transactions can be sent to the target database to perform targeted repair of damaged data in the target database or perform comprehensive repair of the data in the target database.

[0120] Step 206: Send the execution result of the pending transaction to the target database.

[0121] In some embodiments, since the target database may cause data loss due to unstable storage, in order to solve this defect, at least one target database corresponding to the source database can be searched in the target database list; according to the identification information of the data of at least one target database, the cloud server network corresponding to at least one target database can be determined; when the status of all position semaphores of the pending transaction is completed, the execution result of the pending transaction is obtained; the execution result of the pending transaction is saved in the cloud server network; when any target database in at least one target database loses data, the execution result of the pending transaction saved in the cloud server network is obtained; and the execution result of the pending transaction is sent to the corresponding target database. For example: through the target database list, the target database corresponding to the source database can be found to be database 1, and the cloud server network corresponding to database 1 is cloud server network 1. The execution result of the pending transaction is saved in cloud server network 1. When data loss occurs in database 1, the execution result of the pending transaction saved in cloud server network 1 is obtained, which can achieve rapid repair of target database 1 and avoid repeated parsing of binary log information, thereby improving the security of target database 1.

[0122] In order to better illustrate the working process of the data processing method for the database provided by the present application, the following takes database migration as an example to illustrate the working process of the data processing method for the database provided by the present application. Figure 4 , Figure 4 Schematic diagram of the scenario of database migration in the embodiment of the present application, such as Figure 4 The database migration system shown in the figure consists of three parts: source database, data migration server and target database. The data migration server of the database migration system synchronizes the binary log information of the source database in real time through the binlog dump interface of the source database, and converts it into SQL statements after parsing, and then applies it to the target database in parallel to achieve database migration.

[0123] Figure 4 In the database migration process shown, when the binary log information is a data record deletion operation type, since the binary log information of the data record deletion operation contains the unique identifier of the deleted data record, the unique identifier of the data record to be deleted can be parsed (for example, it can be the values ​​of all fields in the primary key of the data table); through the unique identifier of the data record, the data record is queried in the database system at the destination end; finally, the data record is deleted in the target database to synchronize the data.

[0124] When the binary log information is of the data record update operation type, since the binary log information of the data record update operation contains the unique identifier of the updated data record, the unique identifier of the data record to be updated can be parsed (for example, it can be the values ​​of all fields in the primary key of the data table), and then the data record is queried in the database system at the destination end through the unique identifier of the data record; then, the values ​​of the fields to be modified contained in the binary log information are used to update the fields to be updated in the target database.

[0125] When the target database system completes data synchronization, the cloud server network corresponding to the target database system can be determined based on the identification information of the target database system's data; the update results of the target database system's data are saved in the cloud server network; when data loss occurs in the target database system, the update results are sent to the target database system through the cloud server network, thereby enhancing the data security of the target database system and avoiding user losses caused by data loss in the target database system.

[0126] See also Figure 5 , Figure 5 This is a schematic diagram of an implementation flow of the data processing method of the database provided in the embodiment of the present application. It can be understood that: Figure 5 The execution body of the steps shown can be implemented independently by a terminal of a data processing device running a database, or can be implemented by a server of the data processing device running a database. When performing data migration, the server of the data processing device running a database can be used as a data migration module for users to deploy and use in the data migration system. Figure 5The processing steps shown in the figure are explained by taking the server as the execution subject as an example. Figure 5 The steps shown are explained.

[0127] Step 501: Obtain pending transactions, which at least include: a start event, a data operation language event, and an end event.

[0128] In some embodiments, according to the characteristics of the binlog event stream (i.e., the event stream of binary log information), the events of the same transaction appear in an overall order with the transaction as the boundary, and another transaction starts after one transaction ends. During the duration of the entire data migration task, the Binlog event stream obtained by the data migration module does not stop. In the binlog event stream, a pending transaction uses a begin event as the boundary of the transaction start, followed by a series of data manipulation language events (DML events), and then an event is an end event (end event) indicating the end of the transaction. Another transaction will start only after a transaction ends, and there is no event intersection between transactions.

[0129] Step 502: Detect the event distribution status of the transaction to be processed and obtain a distribution status detection result.

[0130] In some embodiments, the distribution status detection result includes: a start event, at least one data manipulation language event (DML event), and an end event, wherein the DML event in database migration includes INSERT, UPDATE, and DELETE statements that modify data in a table or view of the database.

[0131] Step 503: When the distribution state detection result indicates a distribution start event, a first thread matching the transaction to be processed is determined according to the load of each thread in the thread queue, and the transaction to be processed is executed by the first thread.

[0132] In some embodiments, if a transaction starts distributing a start event, in order to avoid the problem of thread load skew, the data operation language event and the end event associated with the start event can be first determined; then the load of each thread in the thread queue is obtained; the load of each thread in the thread queue is sorted, and the thread with the smallest load is determined as the first thread according to the sorting result; the data operation language event and the end event associated with the start event are sent to the event waiting execution queue of the first thread; each event in the event waiting execution queue is executed by the first thread until the pending transaction is executed and completed by the first thread. For example: there are three threads (thread 1, thread 2 and thread 3) in the thread queue that can execute pending transaction a. When it is determined that the thread with the smallest load is thread 1, the start event, data operation language event and end event of transaction a are all distributed to thread 1.

[0133] In some embodiments, the data migration module can configure a position semaphore for each data operation language event of transaction a, which is used to generate a wait chain between transactions. By using the position semaphores of events of different transactions to generate the wait chain, the execution order of related transactions can be controlled. There is no need to wait for a complete transaction to make a correlation judgment, thereby overcoming the defects of related technologies that the data after migration is disordered, or data migration cannot be achieved due to data errors.

[0134] In some embodiments, when the first thread executes each event in the waiting execution queue, the state of the position semaphore corresponding to each event in the waiting execution queue is adjusted to the completion state, so that the subsequent waiting execution transactions can be awakened to enter the execution state.

[0135] Step 504: When the distribution status detection result indicates that a data manipulation language event is distributed, the position semaphore corresponding to the data manipulation language event is detected to obtain the status of the position semaphore.

[0136] Step 505: When the state of the position semaphore is the completion state, a second thread matching the transaction to be processed is determined in the thread queue, and the transaction to be processed is executed by the second thread.

[0137] In some embodiments, for DML events, after calculating the data correlation keyword, the position semaphore obtained after the last distribution of the keyword is obtained. Because the event and the related events in the previous transaction have a timing relationship when executed, it is necessary to wait for the position semaphore of the DML event to change to the completion state during actual execution, and the corresponding event in the associated transaction is awakened and executed, thereby ensuring the sequential execution of transactions and avoiding confusion in the migrated data, or the inability to migrate data due to data errors.

[0138] Step 506: When the distribution status detection result indicates a distribution end event, the status of the position semaphores of all events in the pending transaction is adjusted to achieve continuous execution of the associated transaction associated with the pending transaction through the corresponding thread.

[0139] Through the database data processing method shown in the above embodiment, it can be achieved that: 1. Only related transactions among multiple transactions to be processed need to wait for the change of corresponding position semaphores to achieve control of the transaction execution order, and transactions without associated relationships can be executed in parallel; 2. Multiple transactions to be processed with associated relationships only need to wait for the execution of corresponding events in transactions with associated relationships when they reach a specific position (i.e., the position semaphore of the waiting chain), and irrelevant parts of transactions with associated relationships (i.e., data operation language events before the position semaphore of the waiting chain) can be executed in advance; therefore, the parallel execution efficiency of all transactions is higher.

[0140] In order to better illustrate the data processing method of the database of the present application, the parallel processing of related transactions is described below by taking three threads performing parallel processing of transactions as an example.

[0141] refer to Figure 6 , Figure 6 Schematic diagram of thread processing when there is no correlation between pending transactions in the embodiment of the present application. Figure 6 As shown, thread 1 executes transaction TRX1, and thread 2 executes transaction TRX2. When thread 1 executes transaction TRX1, it starts from the start event of transaction TRX1, then executes all data operation language events of transaction TRX1 in sequence, and finally executes the end event of transaction TRX1. When thread 2 executes transaction TRX2, it starts from the start event of transaction TRX2, then executes all data operation language events of transaction TRX2 in sequence, and finally executes the end event of transaction TRX2. The execution process of transaction TRX1 and transaction TRX2 by thread 1 and thread 2 can be parallel.

[0142] refer to Figure 7 , Figure 7 Schematic diagram of thread processing when there is correlation between the transactions to be processed in the embodiment of the present application, such as Figure 7As shown, thread 1 executes transaction TRX1, thread 2 executes transaction TRX2, and thread 3 executes transaction TRX3; transaction TRX2 is correlated with transaction TRX1, and transaction TRX3 is not correlated with transaction TRX1 and transaction TRX2. Since transaction TRX2 is located after transaction TRX1 in the binary log information, the data manipulation language event (DMLEvent) with correlation in transaction TRX2 needs to wait for the data manipulation language event (DML Event) of transaction TRX1 to be executed before it can continue, that is, when the position semaphore of the data manipulation language event of transaction TRX1 reaches the completion state, thereby achieving precise control of the transaction execution order.

[0143] refer to Figure 8 , Figure 8 Schematic diagram of thread processing when there is correlation between the transactions to be processed in the embodiment of the present application, such as Figure 8 As shown, thread 1 executes transaction TRX1, thread 2 executes transaction TRX2, and thread 3 executes transaction TRX3; transaction TRX2 is related to transaction TRX1, and transaction TRX3 is related to transaction TRX2. When the transaction is executed, the state of the corresponding position semaphore is used to form a waiting chain according to the order of the transactions in the database binary log information, such as Figure 8 As shown, the corresponding data manipulation language event (DML Event3) in transaction TRX3 waits for the execution of the data manipulation language event (DMLEvent2) in transaction TRX2, and the corresponding data manipulation language event (DML Event2) in transaction TRX2 waits for the execution of the data manipulation language event (DML Event1) in transaction TRX1. However, the events before the waiting point in the waiting chain in transactions TRX1, TRX2, and TRX3 (i.e., the events before DML Event3, DML Event2, and DML Event1) can still be executed in parallel, thereby controlling the transaction execution process according to the correlation between transactions, ensuring the orderly execution of related transactions, and controlling multiple threads to execute transactions in parallel, thereby improving the efficiency of transaction execution and reducing the time waiting for transaction execution to complete.

[0144] The present application detects the event distribution state of the pending transaction to obtain the distribution state detection result; when the distribution state detection result represents the distribution start event, the first thread matching the pending transaction is determined according to the load of each thread in the thread queue, and the pending transaction is executed by the first thread; when the distribution state detection result represents the distribution data operation language event, the position semaphore corresponding to the data operation language event is detected to obtain the state of the position semaphore; when the state of the position semaphore is the completion state, the second thread matching the pending transaction is determined in the thread queue, and the pending transaction is executed by the second thread; when the distribution state detection result represents the distribution end event, the state of the position semaphore of all events in the pending transaction is adjusted to realize the continuous execution of the associated transaction associated with the pending transaction through the corresponding thread. In this way, the process of transaction execution can be controlled according to the correlation between transactions to ensure the orderly execution between associated transactions, and multiple threads can be controlled to execute transactions in parallel, thereby improving the efficiency of transaction execution and reducing the time waiting for transaction execution to be completed.

[0145] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application for executing the database data processing method provided in the present application, Fig. 9 The electronic device 100 shown includes: each processor 410, a memory 450, each network interface 420 and a target object interface 430. The various components in the electronic device 100 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Fig. 9 Various buses are labeled as bus system 440 .

[0146] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0147] The target object interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and one or more visual display screens. The target object interface 430 also includes one or more input devices 432, including target object interface components that facilitate target object input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0148] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0149] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0150] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0151] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0152] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0153] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with a target object interface 430 (e.g., a target object interface for operating peripheral devices and displaying content and information);

[0154] The input processing module 454 is used to detect one or more target object inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0155] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Fig. 9The data processing device 455 of the database stored in the memory 450 is shown, which can be software in the form of programs and plug-ins, including the following software modules: information transmission module 4081, state detection module 4082, information processing module 4083, these modules are logical, so according to the functions implemented, they can be arbitrarily combined or further split. The functions of each module will be explained below.

[0156] Next, the functions of the various software modules in the database data processing device 455 are described, wherein:

[0157] The information transmission module 4081 is used to obtain binary log information of the source database;

[0158] The state detection module 4082 is used to parse the binary log information to obtain the pending transactions;

[0159] The state detection module 4082 is further used to detect the event distribution state of the transaction to be processed and obtain the distribution state detection result;

[0160] The information processing module 4083 is used to determine the thread matching the transaction to be processed according to the distribution status detection result;

[0161] The information processing module 4083 is further used to execute the pending transaction through the thread and obtain the execution result of the pending transaction;

[0162] The information transmission module 4081 is also used to send the execution result of the pending transaction to the target database.

[0163] In some embodiments, the information processing module 4083 is further used to determine a first thread matching the transaction to be processed according to the load of each thread in the thread queue when the distribution state detection result indicates a distribution start event;

[0164] The information processing module 4083 is further configured to detect the position signal corresponding to the data operation language event when the distribution state detection result indicates that the data operation language event is distributed, and obtain the state of the position signal; and

[0165] The information processing module 4083 is further configured to determine, in the thread queue, a second thread that matches the transaction to be processed when the state of the position semaphore is a completion state.

[0166] In some embodiments, the information processing module 4083 is further used to change the status of the position semaphores of all events in the pending transaction to a completion status when the distribution status detection result indicates a distribution end event.

[0167] In some embodiments, the information processing module 4083 is further used to determine a data manipulation language event and an end event associated with the start event;

[0168] The information processing module 4083 is also used to obtain the load of each thread in the thread queue;

[0169] The information processing module 4083 is further used to sort the load of each thread in the thread queue, and determine the thread with the smallest load as the first thread according to the sorting result.

[0170] In some embodiments, the information processing module 4083 is further used to send the data operation language event and the end event associated with the start event in the pending transaction to the event waiting execution queue of the first thread;

[0171] The information processing module 4083 is further used to configure a position semaphore for the data operation language event in the event waiting execution queue, and obtain the position semaphore of the data operation language event of the transaction to be processed;

[0172] The information processing module 4083 is further used to obtain the position signal of the data operation language event in the associated transaction when it is determined that the to-be-processed transaction and the associated transaction have data correlation;

[0173] The information processing module 4083 is further used to generate a transaction waiting chain according to the execution order of the pending transaction and the associated transaction, using the position semaphore of the data operation language event of the pending transaction and the position semaphore of the data operation language event in the associated transaction;

[0174] The information processing module 4083 is also used to control the parallel execution of the data operation language events in the pending transaction and the data operation language events of the associated transaction through the transaction waiting chain.

[0175] In some embodiments, the information processing module 4083 is further used to adjust the state of the position semaphore corresponding to each event in the waiting execution queue to a completion state when each event in the waiting execution queue is executed by the first thread to complete the event.

[0176] In some embodiments, the information processing module 4083 is further used to parse the data operation language event to obtain the parsing result of the data operation language event;

[0177] The information processing module 4083 is further used to filter the parsing results according to the identifier of the data correlation keyword to obtain the correlation keyword;

[0178] The information processing module 4083 is further used to detect the position semaphore corresponding to the data operation language event according to the correlation keyword to obtain the target position semaphore, wherein the target position semaphore is used to represent the state of the data operation language event in the transaction associated with the to-be-processed transaction;

[0179] The information processing module 4083 is also used to detect the state of the target position signal quantity and determine the state of the target position signal quantity.

[0180] In some embodiments, the information processing module 4083 is also used to perform real-time detection on the state of the target position signal quantity to obtain real-time detection results;

[0181] The information processing module 4083 is further used to select a thread with the smallest load in the thread queue when the real-time detection result indicates that the state of the target position signal quantity changes from a pending state to a completed state;

[0182] The information processing module 4083 is further configured to use the thread with the smallest load as the second thread matching the transaction to be processed.

[0183] In some embodiments, the information processing module 4083 is further used to search the target database list for at least one target database corresponding to the source database;

[0184] The information processing module 4083 is further used to determine the cloud server network corresponding to the at least one target database according to the identification information of the data of the at least one target database;

[0185] The information processing module 4083 is further used to obtain the execution result of the pending transaction when the status of all position semaphores of the pending transaction is in the completion status;

[0186] The information transmission module 4081 is also used to store the execution results of the pending transactions in the cloud server network;

[0187] The information transmission module 4081 is further used to obtain the execution result of the pending transaction stored in the cloud server network when data loss occurs in any target database of at least one target database;

[0188] The information transmission module 4081 is also used to send the execution result of the pending transaction to the corresponding target database.

[0189] An embodiment of the present application further provides an electronic device, which includes: a memory for storing executable instructions; and a processor for implementing the above-mentioned database data processing method when running the executable instructions stored in the memory.

[0190] The embodiment of the present application also provides a computer-readable storage medium storing executable instructions, which implement the above-mentioned database data processing method when executed by a processor.

[0191] according to Fig. 9 The electronic device shown, in one aspect of the present application, the present application also provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the different embodiments and combinations of the embodiments provided in the various optional implementations of the database data processing method provided in the embodiments of the present application.

[0192] The above are merely embodiments of the present application and are not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for processing data in a database, characterized in that: The method comprises: Get the binary log information of the source database; Parsing the binary log information to obtain pending transactions; Detecting the event distribution status of the pending transaction to obtain a distribution status detection result; Determining a thread matching the transaction to be processed according to the distribution status detection result; Executing the pending transaction through the thread to obtain an execution result of the pending transaction; The execution result of the pending transaction is sent to the target database.

2. The method according to claim 1, characterized in that The types of the pending transaction include: a start event, a data manipulation language event, and an end event. The determining, according to the distribution state detection result, a thread matching the pending transaction includes: When the distribution state detection result indicates that the start event is distributed, determining a first thread matching the transaction to be processed according to a load of each thread in a thread queue; When the distribution state detection result indicates that the data manipulation language event is distributed, a position semaphore corresponding to the data manipulation language event is detected to obtain a state of the position semaphore, and When the state of the position semaphore is a completion state, a second thread matching the to-be-processed transaction is determined in the thread queue.

3. The method according to claim 2, characterized in that The method further comprises: When the distribution status detection result indicates that the end event is distributed, the status of the position semaphores of all events in the pending transaction is changed to a completion status.

4. The method according to claim 2, characterized in that: When the distribution state detection result indicates that the start event is distributed, determining a first thread matching the to-be-processed transaction according to the load of each thread in the thread queue includes: determining a data manipulation language event and an end event associated with the start event; Obtaining the load of each thread in the thread queue; The load of each thread in the thread queue is sorted, and a thread with the smallest load is determined according to the sorting result to serve as the first thread.

5. The method according to claim 4, characterized in that After determining the first thread matching the transaction to be processed, the method further includes: Sending a data manipulation language event and an end event associated with the start event in the pending transaction to an event waiting execution queue of the first thread; configuring a position semaphore for the data manipulation language event in the event waiting execution queue, and obtaining the position semaphore of the data manipulation language event of the pending transaction; When it is determined that the to-be-processed transaction and the associated transaction have data correlation, obtaining a position semaphore of a data operation language event in the associated transaction; Generate a transaction waiting chain by using the position semaphore of the data operation language event of the transaction to be processed and the position semaphore of the data operation language event in the associated transaction according to the execution order of the transaction to be processed and the associated transaction; The transaction waiting chain is used to control the parallel execution of the data operation language event in the pending transaction and the data operation language event in the associated transaction.

6. The method according to claim 5, characterized in that The method further comprises: When each event in the event waiting execution queue is completed through the first thread, the state of the position semaphore corresponding to each event in the waiting execution queue is adjusted to a completion state.

7. The method according to claim 2, characterized in that When the distribution state detection result indicates that the data operation language event is distributed, detecting the position semaphore corresponding to the data operation language event to obtain the state of the position semaphore includes: Parsing the data operation language event to obtain a parsing result of the data operation language event; The analysis results are screened according to the identifiers of the data correlation keywords to obtain the correlation keywords; According to the correlation keyword, a position semaphore corresponding to the data manipulation language event is detected to obtain a target position semaphore, wherein the target position semaphore is used to represent a state of the data manipulation language event in the transaction associated with the to-be-processed transaction; The state of the target position signal is detected to determine the state of the target position signal.

8. The method according to claim 7, characterized in that When the state of the position semaphore is a completion state, determining a second thread matching the pending transaction in the thread queue comprises: Performing real-time detection on the state of the target position signal quantity to obtain a real-time detection result; When the real-time detection result indicates that the state of the target position semaphore changes from a pending state to a completed state, selecting a thread with the smallest load in the thread queue; The thread with the smallest load is used as the second thread matching the transaction to be processed.

9. The method according to claim 1, characterized in that: When the to-be-processed transaction originates from a source database in a data migration environment, the method further includes: Searching for at least one target database corresponding to the source database in the target database list; Determining a cloud server network corresponding to the at least one target database according to identification information of the data of the at least one target database; When the states of all position semaphores of the pending transaction are in the completed state, obtaining the execution result of the pending transaction; The execution result of the pending transaction is stored in the cloud server network; When data loss occurs in any of the at least one target database, obtaining the execution result of the pending transaction stored in the cloud server network; The execution result of the pending transaction is sent to the corresponding target database.

10. A data processing device for a database, characterized in that: The device comprises: Information transmission module, used to obtain binary log information of the source database; A status detection module is used to parse the binary log information to obtain pending transactions; The state detection module is further used to detect the event distribution state of the pending transaction to obtain a distribution state detection result; An information processing module, used for determining a thread matching the transaction to be processed according to the distribution status detection result; The information processing module is further used to execute the pending transaction through the thread to obtain the execution result of the pending transaction; The information transmission module is also used to send the execution result of the pending transaction to the target database.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the database data processing method according to any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor is used to implement the database data processing method according to any one of claims 1 to 9 when running the executable instructions stored in the memory.

13. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by a processor, the database data processing method according to any one of claims 1 to 9 is implemented.

Images