Data processing method, device, program product, equipment and storage medium of database

Abstract

The application provides a data processing method and device of a database, a computer program product, equipment and a storage medium. The related embodiments can be applied to various scenarios such as database migration and full data backup. The method comprises: analyzing binary log information to obtain a to-be-processed transaction; detecting an event distribution state of the to-be-processed transaction to obtain a distribution state detection result; determining a thread matched with the to-be-processed transaction according to the distribution state detection result; executing the to-be-processed transaction through the thread to obtain an execution result of the to-be-processed transaction. Thus, the process of transaction execution can be controlled according to the correlation between transactions to ensure the ordered execution between associated transactions, and multiple threads can be controlled to execute transactions in parallel to improve the efficiency of transaction execution and reduce the time for waiting for the completion of transaction execution.

Description

Technical Field

[0001] This application relates to database processing technology, and more particularly to database data processing methods, apparatus, computer program products, electronic devices and storage media. Background Technology

[0002] In database processing technology, data replication (or data synchronization) between databases primarily relies on the replay of binary logs. Transactions in the binary log are generated by the source database, and related transactions are sequential. Disrupting this sequential execution can lead to data corruption after migration, or even prevent data migration altogether due to data errors. Therefore, to ensure the accuracy of data replication between databases, existing technologies can only control the sequential execution of transactions, resulting in low efficiency and long waiting times. Thus, a database data processing method is needed that ensures the ordered execution of related transactions while maintaining parallel execution, thereby guaranteeing the accuracy of the transaction results. Summary of the Invention

[0003] In view of this, embodiments of this application provide a database data processing method, apparatus, computer program product, device, and storage medium, which can control the execution process of transactions based on the correlation between transactions to ensure the orderly execution of related transactions, and can also control multiple threads to execute transactions in parallel, thereby improving the efficiency of transaction execution and reducing the time spent waiting for transactions to complete.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a data processing method for a database, the method comprising:

[0006] Obtain binary log information from the source database;

[0007] The binary log information is parsed to obtain the transactions to be processed;

[0008] The event distribution status of the pending transaction is detected to obtain the distribution status detection result;

[0009] Based on the distribution status detection results, determine the thread that matches the pending transaction;

[0010] The pending transaction is executed by the thread to obtain the execution result of the pending transaction;

[0011] Send the execution result of the pending transaction to the target database.

[0012] This application embodiment also provides a database data processing apparatus, the apparatus comprising:

[0013] The information transmission module is used to obtain binary log information from the source database;

[0014] The status detection module is used to parse the binary log information to obtain the transactions to be processed;

[0015] The status detection module is also used to detect the event distribution status of the transaction to be processed and obtain the distribution status detection result.

[0016] The information processing module is used to determine the thread that matches the pending transaction based on the distribution status detection result;

[0017] The information processing module is also used to execute the pending transaction through the thread and obtain the execution result of the pending transaction;

[0018] The information transmission module is also used to send the execution result of the pending transaction to the target database.

[0019] In some embodiments, the information processing module is further configured to determine, based on the load of each thread in the thread queue, a first thread that matches the pending transaction when the distribution status detection result indicates that the distribution of the start event has occurred;

[0020] The information processing module is further configured to, when the distribution status detection result indicates that the data manipulation language event has been distributed, detect the location semaphore corresponding to the data manipulation language event, and obtain the status of the location semaphore.

[0021] The information processing module is further configured to determine a second thread in the thread queue that matches the pending transaction when the position semaphore is in a completed state.

[0022] In some embodiments, the information processing module is further configured to set the status of the position semaphores of all events in the pending transaction to a completed state when the distribution status detection result indicates that the distribution of the end event has been completed.

[0023] In some embodiments, the information processing module is further configured to determine a data manipulation language event and an end event associated with the start event;

[0024] The information processing module is also used to obtain the load of each thread in the thread queue;

[0025] The information processing module is further configured to sort the load of each thread in the thread queue, and determine the thread with the smallest load based on the sorting result, so as to be the first thread.

[0026] In some embodiments, the information processing module is further configured to send the data manipulation language event and the end event associated with the start event in the pending transaction to the event waiting execution queue of the first thread;

[0027] The information processing module is also used to configure a position semaphore for the data manipulation language events in the event waiting execution queue, and obtain the position semaphore of the data manipulation language event of the transaction to be processed;

[0028] The information processing module is further configured to, when it is determined that there is a data correlation between the transaction to be processed and the associated transaction, obtain the location semaphore of the data manipulation language event in the associated transaction;

[0029] The information processing module is further configured to generate a transaction wait chain based on the execution order of the pending transaction and the associated transaction, using the position semaphore of the data manipulation language event of the pending transaction and the position semaphore of the data manipulation language event in the associated transaction;

[0030] The information processing module is also used to control the parallel execution of data manipulation language events in the pending transaction and data manipulation language events in the associated transaction through the transaction wait chain.

[0031] In some embodiments, the information processing module is further configured to adjust the state of the position semaphore corresponding to each event in the waiting execution queue to a completed state when each event in the waiting execution queue is completed by the first thread.

[0032] In some embodiments, the information processing module is further configured to parse the data manipulation language event to obtain the parsing result of the data manipulation language event;

[0033] The information processing module is also used to filter the parsing results based on the identifiers of data relevance keywords to obtain relevance keywords;

[0034] The information processing module is further configured to detect the location semaphore corresponding to the data manipulation language event based on the relevant keywords, and obtain the target location semaphore, wherein the target location semaphore is used to characterize the state of the data manipulation language event in the transaction associated with the transaction to be processed;

[0035] The information processing module is also used to detect the state of the target location signal and determine the state of the target location signal.

[0036] In some embodiments, the information processing module is further configured to perform real-time detection of the state of the target location signal and obtain a real-time detection result;

[0037] The information processing module is also used to select the thread with the least load in the thread queue when the real-time detection result indicates that the state of the target location signal has changed from a pending state to a completed state.

[0038] The information processing module is also used to select the thread with the lowest load as the second thread to be matched with the transaction to be processed.

[0039] In some embodiments, the information processing module is further configured to search for at least one target database corresponding to the source database in the target database list;

[0040] The information processing module is further configured to determine the cloud server network corresponding to the at least one target database based on the identification information of the data in the at least one target database.

[0041] The information processing module is also used to obtain the execution result of the transaction when all the position semaphores of the transaction to be processed are in the completed state.

[0042] The information transmission module is also used to store the execution result of the transaction to be processed in the cloud server network;

[0043] The information transmission module is also used to obtain the execution result of the pending transaction stored in the cloud server network when data loss occurs in any of the at least one target database.

[0044] The information transmission module is also used to send the execution result of the pending transaction to the corresponding target database.

[0045] This application also provides an electronic device, the electronic device comprising:

[0046] Memory, used to store executable instructions;

[0047] The processor, when executing executable instructions stored in the memory, implements the data processing method of the database described above.

[0048] This application also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the data processing method of the database described above.

[0049] This application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the data processing method of the database described above.

[0050] The embodiments of this application have the following beneficial effects:

[0051] 1) This embodiment of the application obtains binary log information from the source database; parses the binary log information to obtain transactions to be processed; detects the event distribution status of the transactions to be processed to obtain distribution status detection results; and determines the thread that matches the transactions to be processed based on the distribution status detection results. Thus, it is possible to configure appropriate threads for the transactions to be processed based on the distribution status detection results, achieving load balancing of the threads executing the transactions. Furthermore, it can control the transaction execution process based on the correlation between transactions, ensuring the orderly execution of related transactions, controlling multiple threads to execute transactions in parallel, improving transaction execution efficiency, and reducing the time spent waiting for transactions to complete.

[0052] 2) The pending transaction is executed through the thread to obtain the execution result of the pending transaction; the execution result of the pending transaction is then sent to the target database. This achieves the migration of data from the source database to the target database using the execution result of the pending transaction, ensuring that the migrated data is not corrupted and does not affect the normal use of the target database. Attached Figure Description

[0053] Figure 1 This is a schematic diagram illustrating an application scenario of the database data processing method provided in the embodiments of this application;

[0054] Figure 2 A schematic diagram illustrating an implementation flow of the database data processing method provided in this application embodiment;

[0055] Figure 3 This is a schematic diagram illustrating the composition of binary log information in an embodiment of this application;

[0056] Figure 4 This is a schematic diagram of a database migration scenario in an embodiment of this application;

[0057] Figure 5 A schematic diagram illustrating an implementation flow of the database data processing method provided in this application embodiment;

[0058] Figure 6 This is a schematic diagram illustrating thread processing when the transactions to be processed are unrelated in an embodiment of this application.

[0059] Figure 7 This is a schematic diagram illustrating thread processing when the transactions to be processed are related, as shown in the embodiments of this application.

[0060] Figure 8 This is a schematic diagram illustrating thread processing when the transactions to be processed are related, as shown in the embodiments of this application.

[0061] Figure 9This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application for executing the data processing method of the database provided in this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0064] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0065] 1) Terminals, including but not limited to: ordinary terminals and dedicated terminals, wherein ordinary terminals maintain a long connection or a short connection with the transmission channel, and dedicated terminals maintain a long connection with the transmission channel.

[0066] 2) Client: The carrier that implements specific functions in the terminal. For example, a mobile client (APP) is a carrier of specific functions in a mobile terminal, such as performing report creation or report display functions.

[0067] 3) Logical log: In a database, a log that records logical modifications to data, generally used for synchronizing data between databases.

[0068] 4) Database (db): Simply put, it can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, which can be shared by multiple users, has the lowest possible redundancy, and is independent of application programs.

[0069] 5) A Database Management System (DBMS) is a computer software system designed for managing databases. It generally has basic functions such as storage, retrieval, security, backup, migration, and repair. DBMSs can be categorized based on the database model they support, such as relational or XML (Extensible Markup Language); or based on the type of computer they support, such as server clusters or mobile devices; or based on the query language used, such as SQL (Structured Query Language) or XQuery; or based on performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, simultaneously supporting multiple query languages.

[0070] 6) Server clustering refers to bringing together at least two servers to provide the same service, appearing to the client as if there were only one server. Server clusters can utilize multiple computers for parallel computing to achieve high computing speeds, and can also use multiple computers for backup to migrate data from the source database, ensuring the entire system continues to operate even if one machine fails. The database data processing method provided in this application can be applied to cloud server and distributed server scenarios, enabling flexible migration of source database data in different scenarios and ensuring the accuracy of data migration. For example, a cloud server (CVM, Cloud Virtual Machine) is a simple, efficient, secure, reliable, and elastically scalable computing service. Its management is simpler and more efficient than a traditional single physical server. The target object does not need to purchase hardware in advance; it can quickly create or release any number of cloud servers for its business processes and store its data. In a distributed server environment, the target object's data and programs may not reside on a single server but are distributed across multiple servers. Similarly, the database data processing method provided in this application can control the execution process of parallel transactions to migrate data stored in the cloud server.

[0071] 7) Binlog: MySQL's binary log, mainly used for replication between MySQL master and slave servers.

[0072] 8) Data Definition Language (DDL): It is a language in the language set of Structured Query Language (SQL) that is responsible for defining data structures and database objects.

[0073] 9) Data Manipulation Language (DML): This is a set of instructions within the SQL language set responsible for performing data access operations on database objects.

[0074] 10) Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. It encompasses network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on cloud computing business models. These technologies can form resource pools, allowing for flexible and convenient on-demand use. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring data to be transmitted to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0075] 11) Cloud security refers to the collective term for security software, hardware, target objects, organizations, and security cloud platforms based on cloud computing business models. Cloud security integrates emerging technologies and concepts such as parallel processing, grid computing, and unknown virus behavior detection. It uses a large network of clients to detect anomalies in software behavior on the network, obtains the latest information on Trojans and malware on the internet, and sends it to the Unreal Engine server for automatic analysis and processing. Solutions for viruses and Trojans are then distributed to each client. To ensure the security of data stored on the cloud server, it is often necessary to migrate data from the source database to the cloud server when cloud security checks are passed, achieving data backup or replication.

[0076] Before explaining the data processing method of the database provided in this application, the deficiencies in the related technology are first explained. The inventors found in their research that when migrating data stored in the source database, the control of the transaction execution process mainly includes the following methods:

[0077] 1) In data migration, transactions are executed serially in the order of records in the source database. The drawback of this approach is that although it can guarantee the order of data migration, it cannot execute transactions in parallel in the target database, resulting in slow data migration speed and long waiting time for users.

[0078] 2) In data migration, controlling transactions without data correlation to execute in parallel and transactions with data correlation to execute serially has the following drawbacks: a) Due to the complexity of the source database, determining correlation is highly complex, and data correlation is often only partial, with unrelated parts unable to be executed in parallel. Related transactions are queued in the same thread, often leading to unbalanced load across threads and reduced parallel processing capabilities. b) Due to the complexity of the source database, determining transaction correlation often requires complete transaction information. This necessitates significant caching overhead for transactions with a large number of events, placing high demands on data migration hardware. In extreme cases, caching may be impossible, increasing hardware costs for users. c) Due to different data types in the source database, a transaction may be correlated with multiple preceding transactions. However, these correlated transactions may be distributed to different threads, preventing the transaction from being pre-distributed and requiring waiting for all preceding transactions to complete. This reduces parallel processing capabilities, slows data migration, and increases user waiting time.

[0079] To address the aforementioned shortcomings, this application provides a database data processing method that can control the execution process of transactions based on the relationships between transactions, ensuring the orderly execution of related transactions, and can also control multiple threads to execute transactions in parallel, thereby improving the efficiency of transaction execution and reducing the time spent waiting for transactions to complete, thus adapting to the needs of parallel transaction processing during different types of data migration.

[0080] Figure 1 This is a schematic diagram illustrating an application scenario of the database data processing method provided in the embodiments of this application. See also: Figure 1With the continuous development of computer technology, cloud servers can provide secure and reliable elastic computing services, and can also provide different instance types to meet the specific use cases of the target object. Electronic device 100 may include two different types of terminals or electronic devices, namely terminal 10-1 and terminal 10-2, which the target object can select according to the usage requirements of different types of data migration use cases in the database. The terminals (including terminal 10-1 and terminal 10-2) are equipped with corresponding clients capable of running database data processing functions. When implementing the database data processing method provided in this application, when the binary log information to be migrated is encrypted data, the terminals (including terminal 10-1 and terminal 10-2) need to first obtain the corresponding key from the cloud server 200 in order to decrypt the obtained binary log information. After obtaining the plaintext binary log information, it is converted into a transaction to be processed and executed by the corresponding thread. Based on the result of the transaction execution, the data in the source database in terminal 10-1 is migrated to the target database in terminal 10-2. The migration process described above, which uses cloud server 200 to migrate data from the source database in terminal 10-1 to the target database in terminal 10-2, relies on the binary log of the source database.

[0081] The binary log (Binlog) is a binary-formatted file used to record SQL statements used by users to update the database. For example, SQL statements that modify database tables and their contents are recorded in the Binlog, but queries on tables and other database contents are not recorded. By default, the Binlog is in binary format and cannot be viewed using text viewing tools; it requires a dedicated tool for parsing and viewing.

[0082] Binlog can be used for cross-terminal data migration. For example, when data is written to the database, the updated SQL statements are also written to the corresponding Binlog file. When backing up a database, you can perform a full backup of only a specific period of data. However, if a database server failure is discovered after the backup, data migration can be performed based on the update statements recorded in the Binlog. Binlog is also used for data backup in master-slave databases.

[0083] See Figure 2 , Figure 2This is a schematic diagram illustrating an implementation flow of a database data processing method provided in this application embodiment. This application embodiment can be implemented using cloud technology. Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. It can also be understood as a general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on cloud computing business models. The backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites; therefore, cloud technology needs cloud computing as its support.

[0084] It's important to note that cloud computing is a computing model that distributes computing tasks across a resource pool comprised of numerous computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, resources in the "cloud" appear infinitely scalable, readily available, and can be used on demand, expanded at any time, and paid for based on usage. As the foundational providers of cloud computing capabilities, they establish cloud resource pool platforms, often referred to as cloud platforms or Infrastructure as a Service (IaaS). These platforms deploy various types of virtual resources within the resource pool for external customers to choose from. The cloud resource pool primarily includes: computing devices (which can be virtualized machines containing operating systems), storage devices, and network devices. Different operational permissions are assigned to the target objects performing transactions through the cloud server.

[0085] Cloud storage is a new concept that extends and develops from cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to aggregate a large number of storage devices of various types (storage devices are also called storage nodes) in a network to work together through application software or application interfaces to provide data storage and business access functions. In related technologies, the storage method of the storage system is as follows: create logical volumes. When creating a logical volume, allocate physical storage space for each logical volume. This physical storage space may be a disk of a storage device or several storage devices. When a client stores data on a logical volume, that is, stores the data on the file system, the file system divides the data into many parts, each part is an object. The object contains not only data but also additional information such as data identifier (ID IDentity). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object. Therefore, when a client requests to access data, the file system can allow the client to access the data based on the storage location information of each object. The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of the Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0086] Understandably, Figure 2 The steps shown can be performed by a terminal with a database-running data processing device, or by a server with a database-running data processing device. When performing data migration, the server with the database-running data processing device can be used as a data migration module by the user in the data migration system. Figure 2 The processing steps shown are illustrated using the server as the execution subject as an example. The following section will focus on... Figure 2 The steps shown are explained.

[0087] Step 201: Obtain the binary log information of the source database.

[0088] refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the composition of binary log information in an embodiment of this application, such as... Figure 3As shown, binary log information can be represented in the form of a binlog event stream. Based on the characteristics of the binlog event stream, events within the same transaction appear sequentially, with the transaction as the boundary. One transaction ends and another begins; the binlog event stream obtained by the data migration module continues throughout the entire data migration task. In the binlog event stream, a transaction to be processed begins with a start event, followed by a series of Data Manipulation Language (DML) events, and then an end event indicating the end of the transaction. Another transaction only begins after the previous one ends; there is no event overlap between transactions.

[0089] Step 202: Parse the binary log information to obtain the transactions to be processed.

[0090] In some embodiments, when the data processing method of the database provided in this application is executed using the pending transaction obtained from the binary log information, data migration, full backup of the database, and database repair can be achieved. In the data migration scenario, the source of the binary log information is the target database; the execution result of the pending transaction is sent to the target database, which can realize the migration of data from the source database to the target database.

[0091] In some embodiments, in a full database backup scenario, the source of binary log information can be a cloud server or a source database. During a full database backup, the execution result of the pending transaction can be sent to the data backup server in the cloud server cluster. This enables a full backup of data from the cloud server or source database to the data backup server, facilitating the rapid retrieval of the corresponding data from the data backup server.

[0092] In some embodiments, in the full backup and repair scenarios of the database, the source of binary log information can be a cloud server or a source database. When repairing the database, the execution result of the pending transaction can be sent to the target database, which can achieve targeted repair of the damaged data in the target database or comprehensive repair of the data in the target database.

[0093] Step 203: Detect the event distribution status of the transaction to be processed and obtain the distribution status detection result.

[0094] In some embodiments, the types of events for a pending transaction include three types: a start event, at least one Data Manipulation Language (DML) event, and an end event. A DML event can refer to statements such as INSERT, UPDATE, and DELETE that modify data in a database table or view. For example, the trigger condition for a DML trigger can be set to a predetermined operation such as data insertion, data update, or data deletion performed on a database table. When a user performs an INSERT, UPDATE, or DELETE operation, the DML trigger is activated, and the triggering operation that meets the trigger condition is executed, i.e., retrieving the current value of the sequence in the database.

[0095] Step 204: Based on the distribution status detection results, determine the thread that matches the pending transaction.

[0096] In some embodiments, determining the thread that matches the transaction to be processed based on the distribution status detection result can be achieved in the following ways:

[0097] When the distribution status detection result indicates a distribution start event, the first thread matching the pending transaction is determined based on the load of each thread in the thread queue. Since the server is currently in the distribution start event state, it means that a transaction has begun to be executed. At this time, it is necessary to configure the corresponding thread for the transaction that has started to be executed. The configured thread executes the start event, at least one data manipulation language event, and the end event in the transaction in sequence until the transaction is completed.

[0098] In some embodiments, determining the first thread to be processed based on the load of each thread in the thread queue can be achieved in the following way:

[0099] The process involves determining the data manipulation language (DML) events and end events associated with the start event; obtaining the load of each thread in the thread queue; sorting the loads of each thread in the thread queue; and determining the thread with the lowest load as the first thread based on the sorting result. For example, if there are three threads (thread 1, thread 2, and thread 3) in the thread queue that can execute transaction a, and thread 1 is determined to have the lowest load, then the start event, at least one DML event, and the end event of transaction a are all assigned to thread 1, which then executes these events sequentially.

[0100] In some embodiments, when the number of threads with the lowest load is greater than or equal to 2, the thread with the lowest load and the first thread in the identifier order is selected to execute the corresponding pending transaction according to the thread identifier order. For example, if there are three threads (thread 1, thread 2 and thread 3) in the thread queue that can execute pending transaction a, when the load of thread 1 is equal to the load of thread 2, and the load of thread 1 is less than the load of thread 3, thread 1 with the lowest load and the first thread in the identifier order is selected, and thread 1 executes the start event, at least one data manipulation language event and the end event of transaction a in sequence.

[0101] In some embodiments, when a pending transaction is executed by the first thread, the execution order of the pending transaction may depend on the execution results of related transactions. For example, the data manipulation language event a12 of pending transaction a may require the execution result of related transaction b. Ensuring the orderly execution of related transactions is crucial to prevent data corruption after migration. Therefore, precise control of the execution order among multiple transactions is necessary.

[0102] In some embodiments, to ensure the orderly execution of related transactions, the data manipulation language (DML) events and end events associated with the start event in the pending transaction can first be sent to the event waiting execution queue of the first thread. Next, position semaphores are configured for the DML events in the event waiting execution queue to obtain the position semaphores of the DML events in the pending transaction. Then, when it is determined that there is data correlation between the pending transaction and related transactions, the position semaphores of the DML events in the related transaction are obtained. Based on the execution order of the pending transaction and related transactions, a transaction wait chain is generated using the position semaphores of the DML events in the pending transaction and the position semaphores of the DML events in the related transaction. Finally, the parallel execution of the DML events in the pending transaction and the DML events in the related transaction is controlled through the transaction wait chain.

[0103] In some embodiments, taking the migration of game data as an example, with the explosive growth of game data, traditional relational databases are increasingly unable to meet the requirements of high-concurrency read and write, high-efficiency storage and access of massive data, high scalability and high availability. Therefore, it is necessary to migrate game data from the source database to the target database. Due to the fragmented nature of game time, the large amount of data and the numerous operational activities between players, the read, write and delete requests of game data in the source database are not only numerous but also instantaneous. Therefore, the execution of the data manipulation language event a12 (item query of all game characters of the game user) of the pending transaction a (game user item migration) requires the execution result of the associated transaction b (query of changes in the number of game user items before the migration time 9:00); where the associated transaction b is executed by thread 2. After determining the matching thread 1 for the pending transaction a, the data manipulation language events a11, a12, and a2 associated with the start event a0 are sent to the waiting execution queue of thread 1. A position semaphore signalala12 is configured for data manipulation language event a12. Then, a position semaphore signalb12 is configured for data manipulation language event b12 in the associated transaction b (the type of item obtained by the game user in all operational activities). Based on the execution order ba of the pending transaction and the associated transaction, a transaction wait chain signalb12-signala12 is generated using the position semaphore signalala12 of the data manipulation language event in the pending transaction a and the position semaphore signalb12 of the data manipulation language event in the associated transaction b. Through the transaction wait chain signalb12-signala12, the parallel execution of the data manipulation language events in the pending transaction a and the associated transaction is controlled. The parallel execution of thread 1 and thread 2 can be represented as follows:

[0104] Thread 1: Start event a0, Data Manipulation Language event a11, Data Manipulation Language event a12, End event a2;

[0105] Thread 2: Start event b0, Data Manipulation Language (DML) event b11, Data Manipulation Language (DML) event b12, End event b2.

[0106] In this scenario, the start event a0 and data manipulation language (DML) event a11 of thread 1 can be executed in parallel with the start event b0 and DML event b11 of thread 2. After the execution of the DML event b12 of thread 2, the execution result of the DML event b12 is sent to thread 1 according to the transaction wait chain signalb12-signalala12, so that the DML event a12 can continue to execute. This means that thread 1 no longer needs to wait for thread 2 to execute a complete transaction b before it can make a correlation judgment, effectively saving the waiting time for parallel transaction processing.

[0107] In some embodiments, when each event in the waiting execution queue is completed by the first thread, the state of the position semaphore corresponding to each event in the waiting execution queue is adjusted to the completed state. Therefore, by setting the position semaphores of all events in a transaction to the completed state, subsequent waiting events can be promptly woken up to enter the execution state, avoiding increased thread idle time. Since the execution time of the start and end events in the pending transaction is short, position semaphores do not need to be configured for the start and end events. Therefore, in practical use, adjusting the state of the position semaphore corresponding to each event in the waiting execution queue to the completed state in this embodiment can be done only for the position semaphores of the data manipulation language events in the waiting execution queue.

[0108] In some embodiments, determining the thread matching the pending transaction based on the dispatch status detection result can be achieved as follows: when the dispatch status detection result indicates the dispatch of a Data Operation Language (DOL) event, the position semaphore corresponding to the DOL event is detected to obtain the state of the position semaphore; and when the position semaphore state is complete, a second thread matching the pending transaction is determined in the thread queue. Since the dispatch of a DOL event indicates that the current transaction, when executed by a thread, requires the execution results of corresponding events in other transactions that have data association with the current transaction as a condition for the execution of the current transaction, it is necessary to accurately control the execution order of transactions through the corresponding position semaphores to ensure the sequential execution of transactions.

[0109] In some embodiments, when the distribution status detection result represents a distribution data manipulation language event, the location semaphore corresponding to the data manipulation language event is detected to obtain the status of the location semaphore. This can be achieved in the following ways:

[0110] Data Manipulation Language (DML) events are parsed to obtain the parsing results. The parsing results are then filtered based on data relevance keywords to obtain relevance keywords. Based on these relevance keywords, the location semaphores corresponding to the DML events are detected to obtain the target location semaphore. The target location semaphore represents the state of the DML events within the transaction associated with the transaction to be processed. The state of the target location semaphore is then detected to determine its status. In full backups of financial data databases, due to the large volume and complex data types, table-level or row-level concurrency can be used to improve write efficiency. For example, Bank A's source database contains a set of users with deposits exceeding 100,000 yuan and their corresponding credit scores (see Table 1).

[0111] Table 1

[0112]

[0113] When performing a full backup of Bank A's source database, a bank-level concurrent approach can be used to perform a full backup of the credit scores of users with deposits exceeding 100,000. The corresponding data manipulation language (DML) events for the full backup transaction c are c11 (find users with deposits exceeding 100,000) and c12 (credit scores of users with deposits exceeding 100,000). After parsing the DML event c11 and obtaining the parsing results, the parsing results are filtered based on the identifier of the data relevance keywords, resulting in the relevance keyword "user ID card". The relevance keyword "user ID card" corresponds to the DML event d1 (user ID card validity check) in the associated transaction d. The state of the target location semaphore corresponding to the DML event d1 is checked to determine the state of the target location semaphore.

[0114] In some embodiments, determining a second thread in the thread queue that matches the pending transaction can be achieved in the following ways:

[0115] The state of the target location semaphore is monitored in real time to obtain real-time monitoring results. When the real-time monitoring result indicates that the state of the target location semaphore has changed from a pending state to a completed state, the thread with the least load is selected from the thread queue. The thread with the least load is then selected as the second thread to be matched with the pending transaction. In conjunction with the above embodiment, when performing a full backup of the credit scores of users with deposits exceeding 100,000 using a row-level concurrent approach, if the real-time monitoring result indicates that the state of the target location semaphore has changed from a pending state to a completed state, it means that the data manipulation language event d1 (user ID card validity check) has been completed (e.g., ID card ID 37003 is invalid, while the other ID cards are valid in Table 1). At this time, the thread with the least load can be selected from the thread queue and selected as the second thread to be matched with the pending transaction c. This ensures that the load of all threads in the thread queue is minimized, saving server hardware resources.

[0116] In some embodiments, since the full backup of financial data involves a large number of transactions, as shown in the above embodiments, if all threads in the thread queue are at full load, the thread with the longest full load running time can be used as the second thread to match the transaction c to be processed, so as to save the waiting time of the transaction c.

[0117] In some embodiments, when the distribution status detection result indicates a distribution completion event, the status of the position semaphores of all events in the pending transaction is set to the completion state. Therefore, by setting the position semaphores of all events in the transaction to the completion state, subsequent waiting events can be promptly woken up to enter the execution state, avoiding an increase in thread idle time.

[0118] Step 205: Execute the pending transaction through a thread to obtain the execution result of the pending transaction.

[0119] In some embodiments, the execution result of the pending transaction can be sent to the target database to migrate data from the source database to the target database; the execution result of the pending transaction can be sent to the data backup server in the cloud server cluster to perform a full backup of the data from the cloud server or the source database to the data backup server, so as to quickly retrieve the corresponding data from the data backup server; the execution result of the pending transaction can be sent to the target database to perform targeted repair of the damaged data in the target database or to perform a comprehensive repair of the data in the target database.

[0120] Step 206: Send the execution results of the pending transactions to the target database.

[0121] In some embodiments, since the target database may suffer data loss due to storage instability, to address this deficiency, at least one target database corresponding to the source database can be found in the target database list; based on the identification information of the data in the at least one target database, the cloud server network corresponding to the at least one target database can be determined; when the status of all position semaphores of the pending transaction is in the completed state, the execution result of the pending transaction is obtained; the execution result of the pending transaction is stored in the cloud server network; when any of the at least one target database experiences data loss, the execution result of the pending transaction stored in the cloud server network is obtained; and the execution result of the pending transaction is sent to the corresponding target database. For example, through the target database list, the target database corresponding to the source database can be found as database 1, and the cloud server network corresponding to database 1 can be identified as cloud server network 1. The execution result of the pending transaction is stored in cloud server network 1. When database 1 experiences data loss, the execution result of the pending transaction stored in cloud server network 1 is obtained, which can achieve rapid repair of target database 1, avoid repeated parsing of binary log information, and thus improve the security of target database 1.

[0122] To better illustrate the working process of the database data processing method provided in this application, the following uses database migration as an example to explain the working process of the database data processing method provided in this application. (Refer to...) Figure 4 , Figure 4 This is a schematic diagram illustrating a database migration scenario in an embodiment of this application, such as... Figure 4 The database migration system shown consists of three parts: the source database, the data migration server, and the target database. The data migration server synchronizes the binary log information of the source database in real time through the source database's binlog dump interface. After parsing and converting this information into SQL statements, it applies them in parallel to the target database, thus achieving database migration.

[0123] Figure 4 In the database migration process shown, when the binary log information is a data record deletion operation type, since the binary log information of the data record deletion operation contains the unique identifier of the data record to be deleted, the unique identifier of the data record to be deleted can be parsed out (for example, it can be the value of all fields in the primary key of the data table); through the unique identifier of the data record, the data record can be queried in the database system at the destination end; finally, the data record is deleted in the target database to synchronize the data.

[0124] When the binary log information is of the data record update operation type, since the binary log information of the data record update operation contains the unique identifier of the updated data record, the unique identifier of the data record to be updated can be parsed out (for example, it can be the value of all fields in the primary key of the data table). Then, the data record can be retrieved in the target database system by using the unique identifier of the data record. Then, using the value of the field to be modified contained in the binary log information, the update operation is performed on the field to be updated in the target database.

[0125] Once the target database system completes data synchronization, the corresponding cloud server network can be determined based on the data identification information of the target database system. The update results of the target database system's data are then stored in the cloud server network. In the event of data loss in the target database system, the update results are sent to the target database system through the cloud server network. This enhances the data security of the target database system and prevents users from suffering losses due to data loss in the target database system.

[0126] See Figure 5 , Figure 5 This is a schematic diagram illustrating an implementation flow of a database data processing method provided in an embodiment of this application. It can be understood that... Figure 5 The steps shown can be performed by a terminal with a database-running data processing device, or by a server with a database-running data processing device. When performing data migration, the server with the database-running data processing device can be used as a data migration module by the user in the data migration system. Figure 5The processing steps shown are illustrated using the server as the execution subject as an example. The following section will focus on... Figure 5 The steps shown are explained.

[0127] Step 501: Obtain the pending transaction. The pending transaction includes at least: start event, data manipulation language event, and end event.

[0128] In some embodiments, based on the characteristics of the binlog event stream (i.e., the event stream of binary log information), events of the same transaction appear sequentially with the transaction as the boundary. One transaction ends and another begins. Throughout the entire data migration task, the binlog event stream obtained by the data migration module does not stop. In the binlog event stream, a transaction to be processed begins with a start event, followed by a series of Data Manipulation Language (DML) events, and then an end event indicating the end of the transaction. Another transaction only begins after one transaction ends; there is no event overlap between transactions.

[0129] Step 502: Detect the event distribution status of the transaction to be processed and obtain the distribution status detection result.

[0130] In some embodiments, the distribution status detection result includes: a begin event, at least one data manipulation language (DML) event, and an end event, wherein the DML event in the database migration includes INSERT, UPDATE, and DELETE statements that modify data in a table or view of the database.

[0131] Step 503: When the distribution status detection result represents the distribution start event, determine the first thread that matches the pending transaction based on the load of each thread in the thread queue, and execute the pending transaction through the first thread.

[0132] In some embodiments, when a transaction begins distributing a start event, to avoid thread load skew, the data manipulation language (DML) event and end event associated with the start event can be determined first. Then, the load of each thread in the thread queue is obtained; the loads of each thread in the thread queue are sorted, and the thread with the lowest load is determined as the first thread based on the sorting result; the DML event and end event associated with the start event are sent to the event waiting execution queue of the first thread; each event in the event waiting execution queue is executed by the first thread until the pending transaction is completed by the first thread. For example, if there are three threads (thread 1, thread 2, and thread 3) in the thread queue that can execute pending transaction a, and thread 1 is determined to have the lowest load, then the start event, DML event, and end event of transaction a are all distributed to thread 1.

[0133] In some embodiments, the data migration module can configure a position semaphore for each data manipulation language event of transaction a to generate a wait chain between transactions. By using the position semaphores of events from different transactions to generate a wait chain, the execution order of related transactions can be controlled. It is no longer necessary to wait for a complete transaction to make a correlation judgment, thereby overcoming the defects in related technologies where the migrated data is disordered or cannot be migrated due to data errors.

[0134] In some embodiments, when each event in the waiting execution queue is completed by the first thread, the state of the position semaphore corresponding to each event in the waiting execution queue is adjusted to the completed state, thereby waking up subsequent waiting transactions to enter the execution state.

[0135] Step 504: When the distribution status detection result represents a distribution data operation language event, the position semaphore corresponding to the data operation language event is detected to obtain the status of the position semaphore.

[0136] Step 505: When the position semaphore is in the completed state, determine the second thread that matches the pending transaction in the thread queue, and execute the pending transaction through the second thread.

[0137] In some embodiments, for a DML event, after calculating the data relevance keyword, the position semaphore obtained after the last distribution of the keyword is obtained. Because the event and the related events in the preceding transaction have a temporal relationship during execution, it is necessary to wait for the position semaphore of the DML event to change to the completed state during actual execution. Then, the corresponding events in the related transactions are awakened and executed, thereby ensuring the order of transaction execution and avoiding data corruption after migration, or failure to achieve data migration due to data errors.

[0138] Step 506: When the distribution status detection result indicates a distribution end event, adjust the status of the position semaphores of all events in the pending transaction to enable the continuous execution of associated transactions related to the pending transaction through the corresponding threads.

[0139] The database data processing method described in the above embodiments can achieve the following: 1. Among multiple transactions to be processed, only related transactions need to wait for changes in the corresponding position semaphores to control the execution order of transactions, while transactions without any relationship can be executed in parallel; 2. Within multiple transactions to be processed that have a relationship, only when they reach a specific position (i.e., when waiting for the position semaphore of the waiting chain) do they need to wait for the execution of the corresponding events in the related transactions, while the unrelated parts of the related transactions (i.e., data manipulation language events before the position semaphore of the waiting chain) can be executed in advance; therefore, the parallel execution efficiency of all transactions is higher.

[0140] To better illustrate the data processing method of the database in this application, the parallel processing process of related transactions will be explained below using the example of three threads processing transactions in parallel.

[0141] refer to Figure 6 , Figure 6 This is a schematic diagram illustrating thread processing when the transactions to be processed are unrelated, as shown in the embodiments of this application. Figure 6 As shown, thread 1 executes transaction TRX1, and thread 2 executes transaction TRX2. When thread 1 executes transaction TRX1, it starts execution from the start event of transaction TRX1, then executes all data manipulation language events of transaction TRX1 sequentially, and finally executes the end event of transaction TRX1. When thread 2 executes transaction TRX2, it starts execution from the start event of transaction TRX2, then executes all data manipulation language events of transaction TRX2 sequentially, and finally executes the end event of transaction TRX2. The execution processes of transactions TRX1 and TRX2 by threads 1 and 2 can be parallelized.

[0142] refer to Figure 7 , Figure 7 This is a schematic diagram illustrating thread processing when the transactions to be processed are related, as shown in the embodiments of this application. Figure 7As shown, thread 1 executes transaction TRX1, thread 2 executes transaction TRX2, and thread 3 executes transaction TRX3. Transaction TRX2 is related to transaction TRX1, while transaction TRX3 is not related to either transaction TRX1 or transaction TRX2. Since transaction TRX2 is located after transaction TRX1 in the binary log information, the related Data Manipulation Language (DML) events in transaction TRX2 must wait for the DML events in transaction TRX1 to complete before proceeding. That is, when the position semaphore of the DML events in transaction TRX1 reaches the completion state, precise control over the execution order of transactions is achieved.

[0143] refer to Figure 8 , Figure 8 This is a schematic diagram illustrating thread processing when the transactions to be processed are related, as shown in the embodiments of this application. Figure 8 As shown, thread 1 executes transaction TRX1, thread 2 executes transaction TRX2, and thread 3 executes transaction TRX3. Transaction TRX2 is related to transaction TRX1, and transaction TRX3 is related to transaction TRX2. During transaction execution, a wait chain is formed using the states of the corresponding position semaphores according to the order of the transactions in the database binary log information, such as... Figure 8 As shown, the corresponding Data Manipulation Language (DML) event (DML Event3) in transaction TRX3 waits for the completion of the DML event (DML Event2) in transaction TRX2, and the corresponding DML event (DML Event2) in transaction TRX2 waits for the completion of the DML event (DML Event1) in transaction TRX1. However, events in transactions TRX1, TRX2, and TRX3 before the waiting point in the waiting chain (i.e., events before DML Event3, DML Event2, and DML Event1) can still be executed in parallel. This allows for control over the transaction execution process based on the relationships between transactions, ensuring the orderly execution of related transactions, and also enables multiple threads to execute transactions in parallel, improving transaction execution efficiency and reducing the time spent waiting for transactions to complete.

[0144] This application detects the event dispatch status of transactions to be processed, obtaining a dispatch status detection result. When the dispatch status detection result indicates a dispatch start event, a first thread matching the transaction to be processed is determined based on the load of each thread in the thread queue, and the transaction to be processed is executed by the first thread. When the dispatch status detection result indicates a dispatch data manipulation language event, the position semaphore corresponding to the data manipulation language event is detected to obtain the status of the position semaphore. When the position semaphore status is complete, a second thread matching the transaction to be processed is determined in the thread queue, and the transaction to be processed is executed by the second thread. When the dispatch status detection result indicates a dispatch end event, the status of the position semaphores of all events in the transaction to be processed is adjusted to achieve continuous execution of related transactions associated with the transaction to be processed through corresponding threads. Therefore, it can control the transaction execution process based on the correlation between transactions, ensuring the orderly execution of related transactions, and can also control multiple threads to execute transactions in parallel, improving the efficiency of transaction execution and reducing the time waiting for transaction completion.

[0145] See Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device for executing the data processing method of the database provided in this application, according to an embodiment of the present application. Figure 9 The illustrated electronic device 100 includes: each processor 410, memory 450, each network interface 420, and target object interface 430. The various components in the electronic device 100 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 9 The general labeled all buses as Bus System 440.

[0146] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0147] The target object interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and one or more visual displays. The target object interface 430 also includes one or more input devices 432, including target object interface components that facilitate input to the target object, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0148] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0149] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0150] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0151] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0152] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0153] Presentation module 453 is configured to enable the presentation of information (e.g., the target object interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the target object interface 430.

[0154] The input processing module 454 is used to detect and translate one or more target object inputs or interactions from one or more input devices 432.

[0155] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 9A data processing device 455, which can be software in the form of programs and plug-ins, is shown. It includes the following software modules: an information transmission module 4081, a status detection module 4082, and an information processing module 4083. These modules are logically connected and can therefore be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0156] The functions of each software module in the database data processing device 455 will be described below.

[0157] The information transmission module 4081 is used to obtain binary log information from the source database;

[0158] The status detection module 4082 is used to parse the binary log information to obtain the transactions to be processed;

[0159] The status detection module 4082 is also used to detect the event distribution status of the transaction to be processed and obtain the distribution status detection result.

[0160] The information processing module 4083 is used to determine the thread that matches the transaction to be processed based on the distribution status detection result;

[0161] The information processing module 4083 is also used to execute the pending transaction through a thread and obtain the execution result of the pending transaction;

[0162] The information transmission module 4081 is also used to send the execution results of the pending transactions to the target database.

[0163] In some embodiments, the information processing module 4083 is further configured to determine the first thread that matches the transaction to be processed based on the load of each thread in the thread queue when the distribution status detection result represents a distribution start event;

[0164] The information processing module 4083 is further configured to, when the distribution status detection result represents a distribution data manipulation language event, detect the position semaphore corresponding to the data manipulation language event, and obtain the status of the position semaphore, and

[0165] The information processing module 4083 is also used to determine a second thread in the thread queue that matches the transaction to be processed when the position semaphore is in a completed state.

[0166] In some embodiments, the information processing module 4083 is further configured to set the status of the position semaphores of all events in the pending transaction to a completed state when the distribution status detection result indicates a distribution end event.

[0167] In some embodiments, the information processing module 4083 is further configured to determine a data manipulation language event and an end event associated with the start event;

[0168] The information processing module 4083 is also used to obtain the load of each thread in the thread queue;

[0169] The information processing module 4083 is also used to sort the load of each thread in the thread queue and determine the thread with the smallest load as the first thread based on the sorting result.

[0170] In some embodiments, the information processing module 4083 is further configured to send the data manipulation language event and the end event associated with the start event in the transaction to be processed to the event waiting execution queue of the first thread;

[0171] The information processing module 4083 is also used to configure position semaphores for data manipulation language events in the event waiting execution queue, and obtain the position semaphores of the data manipulation language events of the transaction to be processed;

[0172] The information processing module 4083 is also used to obtain the location semaphore of the data manipulation language event in the related transaction when it is determined that there is a data correlation between the transaction to be processed and the related transaction;

[0173] The information processing module 4083 is also used to generate a transaction wait chain based on the execution order of the pending transaction and related transactions, using the position semaphore of the data manipulation language event in the pending transaction and the position semaphore of the data manipulation language event in the related transaction.

[0174] The information processing module 4083 is also used to control the parallel execution of data manipulation language events in the pending transaction and data manipulation language events in related transactions through the transaction wait chain.

[0175] In some embodiments, the information processing module 4083 is further configured to adjust the state of the position semaphore corresponding to each event in the waiting execution queue to the completed state when each event in the waiting execution queue is executed by the first thread.

[0176] In some embodiments, the information processing module 4083 is further configured to parse the data manipulation language event to obtain the parsing result of the data manipulation language event;

[0177] The information processing module 4083 is also used to filter the parsing results based on the identifiers of data relevance keywords to obtain relevance keywords;

[0178] The information processing module 4083 is also used to detect the location semaphore corresponding to the data manipulation language event based on the relevance keywords, and obtain the target location semaphore, wherein the target location semaphore is used to characterize the state of the data manipulation language event in the transaction associated with the transaction to be processed;

[0179] The information processing module 4083 is also used to detect the state of the target position signal and determine the state of the target position signal.

[0180] In some embodiments, the information processing module 4083 is further configured to perform real-time detection of the state of the target location signal and obtain a real-time detection result;

[0181] The information processing module 4083 is also used to select the thread with the least load in the thread queue when the state of the target position signal represented by the real-time detection result changes from the pending state to the completed state.

[0182] The information processing module 4083 is also used to select the thread with the least load as the second thread to be matched with the transaction to be processed.

[0183] In some embodiments, the information processing module 4083 is further configured to search for at least one target database corresponding to the source database in the target database list;

[0184] The information processing module 4083 is also used to determine the cloud server network corresponding to at least one target database based on the identification information of the data in at least one target database.

[0185] The information processing module 4083 is also used to obtain the execution result of the transaction when all the position semaphores of the transaction to be processed are in the completed state.

[0186] The information transmission module 4081 is also used to store the execution result of the transaction to be processed in the cloud server network;

[0187] The information transmission module 4081 is also used to obtain the execution result of the pending transaction stored in the cloud server network when data loss occurs in any of the at least one target database.

[0188] The information transmission module 4081 is also used to send the execution results of the pending transactions to the corresponding target database.

[0189] This application also provides an electronic device, which includes: a memory for storing executable instructions; and a processor for implementing the above-described database data processing method when running the executable instructions stored in the memory.

[0190] This application also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the data processing method of the database described above.

[0191] according to Figure 9 The electronic device shown, in one aspect of this application, also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform various optional implementations of the database data processing method provided in the embodiments of this application, including different embodiments and combinations of embodiments.

[0192] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data processing method of a database, characterized by, The method includes: Obtain binary log information from the source database; The binary log information is parsed to obtain the transactions to be processed, wherein the types of the transactions to be processed include: start event, data manipulation language event, and end event; The event distribution status of the pending transaction is detected to obtain the distribution status detection result; Based on the distribution status detection result, a thread matching the pending transaction is determined; wherein, when the distribution status detection result indicates that the start event is being distributed, the first thread matching the pending transaction is determined based on the load of each thread in the thread queue. The data manipulation language (DML) events and end events associated with the start event in the pending transaction are sent to the event waiting execution queue of the first thread; position semaphores are configured for the DML events in the event waiting execution queue to obtain the position semaphores of the DML events in the pending transaction; when it is determined that there is data correlation between the pending transaction and the related transaction, the position semaphores of the DML events in the related transaction are obtained; according to the execution order of the pending transaction and the related transaction, a transaction wait chain is generated using the position semaphores of the DML events in the pending transaction and the position semaphores of the DML events in the related transaction; the parallel execution of the DML events in the pending transaction and the DML events in the related transaction is controlled through the transaction wait chain. The pending transaction is executed by the thread to obtain the execution result of the pending transaction; Send the execution result of the pending transaction to the target database.

2. The method according to claim 1, characterized in that, The method further includes: When the distribution state detection result indicates that the data manipulation language event is being distributed, the location semaphore corresponding to the data manipulation language event is detected to obtain the state of the location semaphore, and When the position semaphore is in a completed state, a second thread matching the pending transaction is determined in the thread queue.

3. The method according to claim 2, characterized in that, The method further includes: When the distribution status detection result indicates that the distribution of the end event has occurred, the status of the position semaphores of all events in the pending transaction is set to the completion status.

4. The method according to claim 2, characterized in that, When the distribution status detection result indicates that the start event is being distributed, the first thread matching the pending transaction is determined based on the load of each thread in the thread queue, including: Determine the data manipulation language event and end event associated with the start event; Obtain the load of each thread in the thread queue; The load of each thread in the thread queue is sorted, and the thread with the lowest load is determined based on the sorting result and selected as the first thread.

5. The method according to claim 1, characterized in that, The method further includes: When each event in the event waiting queue is completed by the first thread, the state of the position semaphore corresponding to each event in the waiting queue is adjusted to the completed state.

6. The method according to claim 2, characterized in that, When the distribution state detection result indicates that the data manipulation language event is being distributed, the location semaphore corresponding to the data manipulation language event is detected to obtain the state of the location semaphore, including: The data manipulation language event is parsed to obtain the parsing result of the data manipulation language event; The parsing results are filtered based on the identifiers of data relevance keywords to obtain relevance keywords; Based on the relevant keywords, the location semaphore corresponding to the data manipulation language event is detected to obtain the target location semaphore, wherein the target location semaphore is used to characterize the state of the data manipulation language event in the transaction associated with the transaction to be processed; The state of the target location signal is detected to determine the state of the target location signal.

7. The method according to claim 6, characterized in that, When the position semaphore is in a completed state, determining a second thread in the thread queue that matches the pending transaction includes: The state of the target location signal is detected in real time to obtain real-time detection results; When the real-time detection result indicates that the state of the target location semaphore has changed from a pending state to a completed state, the thread with the least load is selected from the thread queue. The thread with the lowest load is selected as the second thread to be matched with the pending transaction.

8. The method according to claim 1, characterized in that, When the transaction to be processed originates from a source database in a data migration environment, the method further includes: Search the target database list for at least one target database that corresponds to the source database; Based on the identification information of the data in the at least one target database, determine the cloud server network corresponding to the at least one target database; When all the position semaphores of the pending transaction are in the completed state, the execution result of the pending transaction is obtained; The execution result of the pending transaction is stored in the cloud server network; When data is lost in any of the at least one target database, the execution result of the pending transaction stored in the cloud server network is obtained; The execution result of the pending transaction is sent to the corresponding target database.

9. A data processing apparatus for a database, characterized in that, The device includes: The information transmission module is used to obtain binary log information from the source database; The status detection module is used to parse the binary log information to obtain the transactions to be processed, wherein the types of the transactions to be processed include: start event, data manipulation language event and end event; The status detection module is also used to detect the event distribution status of the transaction to be processed and obtain the distribution status detection result. An information processing module is used to determine a thread matching the pending transaction based on the distribution status detection result; wherein, when the distribution status detection result indicates that the start event is being distributed, a first thread matching the pending transaction is determined based on the load of each thread in the thread queue; the data manipulation language events and end events associated with the start event in the pending transaction are sent to the event waiting execution queue of the first thread; position semaphores are configured for the data manipulation language events in the event waiting execution queue to obtain the position semaphores of the data manipulation language events of the pending transaction; when it is determined that there is data correlation between the pending transaction and related transactions, the position semaphores of the data manipulation language events in the related transactions are obtained; a transaction waiting chain is generated using the position semaphores of the data manipulation language events of the pending transaction and the related transactions according to the execution order of the pending transaction and the related transactions; and the parallel execution of the data manipulation language events of the pending transaction and the related transactions is controlled through the transaction waiting chain. The information processing module is also used to execute the pending transaction through the thread and obtain the execution result of the pending transaction; The information transmission module is also used to send the execution result of the pending transaction to the target database.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the data processing method of the database according to any one of claims 1 to 8.

11. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the data processing method of the database according to any one of claims 1 to 8.

12. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the data processing method of the database according to any one of claims 1 to 8.

Images