Data processing method, system, electronic device, storage medium and computer program product

Abstract

The application discloses a data processing method and system, electronic equipment, storage medium and computer program product, and relates to the technical field of databases. The method comprises the following steps: in response to a synchronization trigger signal sent by a production end node, reading a composite log record stored by the production end node in a target storage space, wherein the composite log record is generated by the production end node based on transaction operation information when a row storage data change occurs in a target transaction in a target database, and the composite log record is used to represent physical redo information and change logical semantics of the row storage data change; and performing transaction playback based on the composite log record and transaction time sequence information corresponding to the target transaction, so as to update column storage data in a consumption end node, wherein the column storage data is used to respond to an analysis query of the target database. The application solves the technical problem of low data synchronization efficiency in the related art.

Description

Technical Field

[0001] This application relates to the field of database technology, and more specifically, to a data processing method, system, electronic device, storage medium, and computer program product. Background Technology

[0002] In Hybrid Transactional Analytical Processing (HTAP) scenarios, users require database systems to efficiently handle high-concurrency Online Transaction Processing (OLTP) and also to perform complex online analytical processing (OLAP) on large amounts of data. Therefore, synchronizing row-store data changes during transaction processing to column-store data to meet the real-time requirements of analytical queries has become one of the technical directions for improving database system performance.

[0003] In existing technologies, in order to support transaction processing and analytical queries simultaneously, two independent transactional databases and analytical databases are usually deployed, and data synchronization is performed through an Extract-Transform-Load (ETL) process. However, this approach has low data synchronization efficiency and poor data timeliness.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a data processing method, system, electronic device, storage medium, and computer program product to at least solve the technical problem of low data synchronization efficiency in related technologies.

[0006] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: in response to a synchronization trigger signal issued by a production node, reading a composite log record stored by the production node in a target storage space, wherein the composite log record is generated by the production node based on transaction operation information when row-level data changes occur in a target transaction in the target database, and the composite log record is used to characterize the physical redo information and change logic semantics of the row-level data change; performing transaction replay based on the composite log record and the transaction timing information corresponding to the target transaction to update column-level data in the consumer node, wherein the column-level data is used to respond to analysis queries in the target database.

[0007] According to another aspect of the embodiments of this application, a data processing method is provided, comprising: in response to row-level data changes triggered by a target transaction in a target database, generating a composite log record based on transaction operation information, wherein the composite log record is used to characterize the physical redo information and change logic semantics of the row-level data changes; storing the composite log record in a target storage space; sending a synchronization trigger signal, wherein the synchronization trigger signal is used to notify a consumer node to read the composite log record in the target storage space; and performing transaction replay based on the composite log record and the transaction timing information corresponding to the target transaction to update column-level data in the consumer node, wherein the column-level data is used to respond to analysis queries in the target database.

[0008] According to another aspect of the embodiments of this application, a task processing method is provided, applied to a consumer node of a target database. The task processing method includes: receiving an analysis query task initiated to the target database; processing the analysis query task using target columnar storage data to obtain analysis processing results, wherein the target columnar storage data includes columnar storage data that is updated in real time in response to row storage data changes according to any of the above data processing methods, and the row storage data changes are triggered by a transaction processing task initiated to the producer node of the target database.

[0009] According to another aspect of the embodiments of this application, a task processing method is provided, applied to a producer node of a target database. The task processing method includes: receiving a transaction processing task initiated to the target database; processing the transaction processing task using target row storage data to obtain a transaction processing result; and updating column storage data in the consumer node of the target database according to any one of the above data processing methods in response to row storage data changes triggered during the processing of the transaction processing task, wherein the column storage data is used to respond to analysis queries of the target database.

[0010] According to another aspect of the embodiments of this application, a data processing system is also provided, including: a producer node configured to generate composite log records based on transaction operation information in response to row-level data changes triggered by a target transaction in a target database, wherein the composite log records are used to characterize the physical redo information and change logic semantics of the row-level data changes; a target storage space configured to store the composite log records; and a consumer node configured to read the composite log records in the target storage space in response to a synchronization trigger signal issued by the producer node, and to perform transaction replay based on the composite log records and the transaction timing information corresponding to the target transaction to update the column-level data in the consumer node, wherein the column-level data is used to respond to analysis queries in the target database.

[0011] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the method of any one of the above when it runs.

[0012] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, the device where the storage medium is located executes any of the above methods.

[0013] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the method according to any one of the above.

[0014] In this embodiment, the composite log record read by the consumer node includes physical redo information and change logic semantics of row-store data changes, eliminating the need for additional logic semantic conversion during the parsing phase. The consumer node performs transaction replay based on the transaction sequence information corresponding to the target transaction, updating the column-store data sequentially. Based on this, the composite log record pre-associates physical redo information with change logic semantics, avoiding semantic extraction overhead at the consumer end and reducing computational losses during data synchronization. The ordered replay based on transaction sequence information ensures that the update order of column-store data matches the transaction commit order, avoiding retries or waiting caused by out-of-order replay, thereby further improving the data synchronization efficiency from row-store data changes to column-store data updates. Therefore, this embodiment achieves the goal of efficient and low-latency synchronization of row-store data changes to column-store data through composite log records and transaction sequence information, thereby improving the technical effect of column-store data synchronization efficiency and solving the technical problem of low data synchronization efficiency in related technologies.

[0015] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 This is a hardware structure block diagram of a computing device for implementing the methods provided in the embodiments of this application;

[0018] Figure 2 This is a schematic diagram of the system structure in an application scenario;

[0019] Figure 3This is a flowchart of a data processing method according to an embodiment of this application;

[0020] Figure 4 This is a flowchart of another data processing method according to an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of an optional scheme for real-time synchronization and parallel playback of columnar data based on physical log logic, according to an embodiment of this application.

[0022] Figure 6 This is a schematic diagram of an optional column-store index transaction replay pipeline based on a slot bucket mechanism according to an embodiment of this application;

[0023] Figure 7 This is a schematic diagram of an optional optimized processing flow for a data restructuring task according to an embodiment of this application;

[0024] Figure 8 This is a schematic diagram of an optional dynamic parsing thread switching process according to an embodiment of this application;

[0025] Figure 9 This is a schematic diagram of an optional dynamic parsing thread switching state machine according to an embodiment of this application;

[0026] Figure 10 This is a flowchart of a task processing method according to an embodiment of this application;

[0027] Figure 11 This is a flowchart of another task processing method according to an embodiment of this application;

[0028] Figure 12 This is a structural block diagram of a data processing apparatus according to an embodiment of this application;

[0029] Figure 13 This is a structural block diagram of another data processing apparatus according to an embodiment of this application;

[0030] Figure 14 This is a structural block diagram of a task processing device according to an embodiment of this application;

[0031] Figure 15 This is a structural block diagram of another task processing apparatus according to an embodiment of this application;

[0032] Figure 16 This is a schematic diagram of the structure of a data processing system according to an embodiment of this application;

[0033] Figure 17 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0037] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows.

[0038] In-Memory Column Index (IMCI): A columnar storage engine component deployed on read-only nodes of a database to accelerate analytical queries.

[0039] Redo Log: A physical log generated by the storage engine when data modification operations are performed, used for crash recovery and data replication.

[0040] Read-Write Node (RW node): Also known as the producer node or database master node, it is used to handle write transactions and data modification operations.

[0041] Read-Only Node (RO Node): Also known as a consumer node, it generates columnar storage data by consuming physical redo logs. RO nodes are also used to respond to analytical queries based on the columnar storage data.

[0042] Data compaction refers to the process of physically reorganizing columnar data to improve query efficiency and storage space utilization.

[0043] Out-of-place update: refers to an update strategy that does not directly modify the original data location, but expresses update semantics through a combination of deletion and insertion operations.

[0044] Multi-Version Concurrency Control (MVCC): A mechanism that enables concurrent read and write operations to proceed without blocking each other by maintaining multiple versions of the data.

[0045] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 This is a hardware structure block diagram of a computing device used to implement data processing methods. For example... Figure 1 As shown, the computing device 10 may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processors 102 may include, but are not limited to, microprocessors (MCUs) or field-programmable gate arrays (FPGAs), etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, the computing device 10 may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a computer bus), a network interface, a cursor control device (such as a mouse, touchpad, etc.), a keyboard, a power supply, and / or a camera.

[0046] Those skilled in the art will understand that Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, the arithmetic unit 10 may also include a ratio Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0047] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element of the computing device 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0048] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computing device 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0049] The transmission device 106 is used to connect to a network via a network interface to receive or send data. Specific examples of the network described above may include wired and / or wireless networks provided by the communication provider of the computing device 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module for wireless communication with the Internet.

[0050] like Figure 1 The display shown can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computing device 10.

[0051] Figure 1 The hardware structure block diagram shown can serve not only as an exemplary block diagram of the aforementioned computing device 10, but also as an exemplary block diagram of the aforementioned server. In one optional embodiment, Figure 2 It is a kind of use Figure 1The diagram shows a system architecture for an application scenario where the computing device acts as a transmitter or receiver. Figure 2 As shown, the computing device 10 can be connected to one or more servers (e.g., security servers, resource servers, game servers, etc.) via a data network connection or electronically. In an alternative embodiment, the computing device 10 can be any mobile computing device, etc. The data network connection can be a local area network (LAN) connection, a wide area network (WAN) connection, an Internet connection, or other types of data network connection. The computing device 10 can execute network services that connect to a server (e.g., a security server) or a group of servers 20. The network server is a network-based user service, such as social networks, cloud resources, email, online payments, or other online applications.

[0052] Under the aforementioned operating environment, this application provides the following: Figure 3 The data processing method shown. Figure 3 This is a flowchart of a data processing method according to an embodiment of this application. For example... Figure 3 As shown, the method may include the following steps S301 to S303.

[0053] Step S301: In response to row storage data changes triggered by the target transaction in the target database, a composite log record is generated based on the transaction operation information. The composite log record is used to characterize the physical redo information and change logic semantics of the row storage data changes.

[0054] The execution entity of the data processing method provided in this application embodiment can be a producer node deployed in a database system. This producer node is also called a read-write node or database master node. This producer node can be used to process write transactions in the target database and perform data modification operations.

[0055] The target database described above can be a database system that supports Hybrid Transactional and Analytical Processing (HTAP). This target database contains both row-store and column-store data. Row-store data can be deployed on the producer nodes of the target database. Column-store data can be deployed on the consumer nodes (which can be read-only nodes). By synchronizing changes to the row-store data to the column-store data in real time, the target database can simultaneously meet the performance requirements of transaction processing and analytical queries within the same system.

[0056] Row-store data refers to a data organization format where data is stored contiguously in storage media, organized according to rows. In database systems, master table data can be stored using row-store. When modifying master table data, row-store storage engines (such as transactional storage engines) arrange the values ​​of each field in the same row adjacently in physical storage. In row-store data, write operations (such as insert, update, and delete) are completed within a single or a few data blocks, making row-store data suitable for high-concurrency online transaction processing (OLTP) scenarios.

[0057] Column-oriented data refers to a data organization format where data is stored contiguously in storage media, organized by columns. In column-oriented data, all rows of data within the same column are physically adjacent, while data from different columns are stored in their respective data blocks (such as row groups or data packets). Based on column-oriented data, analytical queries only need to read the relevant columns, eliminating the need to scan entire rows, thus reducing input / output volume. Furthermore, the data type within the same column is consistent, facilitating compression encoding and vectorized computation. Therefore, column-oriented data is suitable for online analytical processing (OLAP) scenarios.

[0058] The target transaction can be any database transaction in the target database that modifies data. The transaction operation corresponding to the target transaction can include one or more of the following: insert, delete, and update operations. The target transaction can be a complete original transaction. When the amount of data in the original transaction is greater than a preset value, the target transaction can also be one of the multiple sub-transactions obtained by splitting the original transaction. In a streaming processing scenario, the target transaction can be multiple original transactions or multiple sub-transactions obtained by splitting original transactions.

[0059] The aforementioned row storage data changes refer to the modifications made to row storage data by the target transaction. These changes can include adding, deleting, and updating row data. In particular, updating row data can be expressed as a combination of deleting and adding row data.

[0060] The transaction operation information described above describes the operation content of the target transaction. This transaction operation information may include the identification information of the changed row (such as the primary key value), the specific content (such as the data values ​​of each column in the changed row), and the operation type. Based on this transaction operation information, physical redo information and change logical semantics can be determined.

[0061] The aforementioned composite log record can be log data that simultaneously carries physical redo information and change logic semantics. The physical redo information characterizes the modifications made to row-level data at the physical storage level. The change logic semantics characterizes the change semantics of the row-level data at the logical level. By using composite log records, the physical redo information and change logic semantics are associated in the same log file. Therefore, when a consumer node consumes this composite log record, it is not necessary to deduce the logic semantics by parsing the physical redo information.

[0062] For example, in an application scenario, when row storage data changes, in addition to generating regular physical redo logs, the production node (such as the RW node) can also determine the change logical semantics through transaction operation information and generate logical semantic tags. Furthermore, the logical semantic tags are associated with the log positions corresponding to the physical redo logs, and the logical semantic tags are merged or associated with the physical redo logs for storage to obtain composite log records.

[0063] Step S302: Store the composite log records to the target storage space.

[0064] The target storage space mentioned above can be a persistent storage space independent of the producer nodes and consumer nodes. This target storage space can be written to by producer nodes and read from by consumer nodes; that is, it can be a shared storage space. For example, it can be a distributed file system, cloud storage service, or network-attached storage.

[0065] When storing composite log records, producer nodes can append these records to the target storage space. Specifically, the newly generated composite log record is written to the end of the existing log data in the target storage space, thus ensuring that the writing order of the composite log records is consistent with the commit order of the target transactions. In this way, the producer node stores composite log records sequentially according to the commit order of the target transactions.

[0066] The target storage space described above can contain multiple log partitions. Each log partition is used to store composite log records generated within a certain period of time. The production node can determine the target log partition to be written to from multiple log partitions based on time or other log metadata.

[0067] For example, in an application scenario, the producer node encapsulates composite log records into log entries, and then appends these log entries to the end of the currently active log file in the shared storage space. This log entry may contain a log header and a log body. The log header records the length of the composite log record, the transaction identifier, and the commit sequence number. The log body records physical redo information and change logic semantics.

[0068] Step S303: Send a synchronization trigger signal, wherein the synchronization trigger signal is used to notify the consumer node to read the composite log record in the target storage space, and to perform transaction replay based on the transaction timing information corresponding to the composite log record and the target transaction, so as to update the columnar storage data in the consumer node, wherein the columnar storage data is used to respond to the analysis query of the target database.

[0069] In this embodiment, after completing the storage of composite log records, the producer node sends a synchronization trigger signal to the consumer node. This synchronization trigger signal informs the consumer node that new composite log records are available for reading in the target storage space, and the consumer node should respond to the signal by performing read and playback operations. The specific implementation of the synchronization trigger signal can be a signal message actively pushed by the producer node, such as a log position notification, log sequence number heartbeat, or a notification message on the network connection; it can also be a metadata change indication obtained by the consumer node from shared storage through a polling mechanism; or it can be a communication method agreed upon between the producer and consumer nodes that can transmit information.

[0070] In one optional implementation, the aforementioned synchronization trigger signal can be a LogPosition Notify, a signal message sent from the producer node to the consumer node to inform the consumer node that a new composite log record has been stored in the target storage space and is available for reading. This synchronization trigger signal may include location indication information to specify the storage location of the composite log record in the target storage space. The aforementioned synchronization trigger signal can be sent through the network connection channel between the producer node and the consumer node.

[0071] After receiving the synchronization trigger signal, the consumer node reads the corresponding composite log record from the target storage space based on the location indication information carried in the synchronization trigger signal, and then parses the physical redo information and change logic semantics from the composite log record.

[0072] The consumer node also obtains the transaction sequence information corresponding to the target transaction. This transaction sequence information is used to identify the commit order between different transactions, thereby determining the order of transaction replay. For example, this transaction sequence information can be a transaction commit sequence number, which is a globally incrementing sequence number generated by the producer node or the global sequence number allocator, and this transaction commit sequence number increments globally according to the transaction commit order. Another example is that this transaction sequence information can also be a timestamp (such as a time value generated by a global clock or distributed clock protocol), a logical clock, etc.

[0073] The consumer node determines the transaction replay order based on the transaction commit number, and applies the physical redo information and change logic semantics to the columnar storage data according to the transaction replay order, thereby updating the columnar storage data. The updated columnar storage data is then visible to the analysis and query processes to be processed.

[0074] The aforementioned consumer nodes are used to maintain columnar data and process analytical queries. When an analytical query arrives at the target database, it can be routed to the consumer node for execution. The consumer node responds to the analytical query using the updated columnar data and returns the query results.

[0075] Through the above steps S301 to S303, this embodiment of the application generates a composite log record containing physical redo information and change logic semantics, stores the composite log record in the target storage space, and sends a synchronization trigger signal to trigger the consumer node to perform transaction replay based on transaction timing information. This achieves the purpose of updating the row storage data of the producer node to the column storage data of the consumer node in a low-latency manner, thereby improving the technical effect of column storage data synchronization efficiency and solving the technical problem of low data synchronization efficiency in related technologies.

[0076] As an optional embodiment, the change logic semantics include change type; in step S301, a composite log record is generated based on transaction operation information, including the following method steps: Step S311, based on transaction operation information, determine the physical redo information and change type of at least one changed row, wherein the physical redo information includes the change primary key and row change content corresponding to the changed row, and the change type includes at least addition change and deletion change; Step S312, associate the change type with the log position corresponding to the physical redo information to generate a composite log record, wherein the composite log record includes the row record corresponding to the changed row.

[0077] In this embodiment, the target transaction can trigger a change to at least one row record in the row storage data; that is, the changed row can be a row record in the row storage data that has undergone data modification. Physical redo information describes the modifications made to the changed row at the physical storage level, and may include a changed primary key and row change content. The changed primary key is the primary key value that uniquely identifies a row of data, and the row change content includes the specific data values ​​of each column in the changed row. The change type indicates the semantic type of the row storage data change at the logical level, such as an add change corresponding to an add operation and / or a delete change corresponding to a delete operation. When the transaction operation corresponding to the row storage data change is an update operation, the change type can be expressed as a combination of delete and add changes, thereby implementing an out-of-place update strategy.

[0078] In the process of determining the physical redo information and change type of at least one changed row based on transaction operation information, the production node can extract the primary key value of each changed row as the change primary key from the transaction operation information, extract the data before and after the change of each field of the row to obtain the row change content, and determine whether the change type of the changed row is an insert change or a delete change based on the operation statement (such as an INSERT statement or a DELETE statement). For example, when receiving an INSERT statement, the production node determines that the change type of the row record involved in the statement is an insert change, and records the complete data before and after the change of the row as the row change content.

[0079] The aforementioned log positions can be identifiers of physical redo information within the log stream, such as log sequence numbers or file offsets. Production nodes can associate change types with the log positions of the physical redo information corresponding to those change rows, thereby generating composite log records. This association can be achieved by writing change type tags into the metadata fields of the physical redo log entries, or by recording the correspondence between log positions and change types in a separate mapping table. The composite log record can contain a row record corresponding to the change row, which may include the change primary key, the row change content, and the associated change type information.

[0080] In the application scenario, after determining the physical redo information and change type for each changed line, the producer node can write the physical redo information to the log buffer, obtain the log position corresponding to the physical redo information, and then bind the change type as additional information to the log position to obtain a complete composite log record. This composite log record is appended to the target storage space for subsequent reading by the consumer node.

[0081] Through the above optional embodiments, this application explicitly records the change type when generating composite log records on the production node and associates the change type with the log position of the physical redo information, enabling the composite log records to carry clear logical semantics. After reading the composite log record, the consumer node can determine the change type without performing semantic deduction on the physical redo information, thereby reducing the computational overhead of the consumer node and further improving the efficiency of columnar data synchronization.

[0082] As an optional embodiment, the target storage space is a persistent storage space in the target database that is independent of the producer node and the consumer node. In step S302, the composite log records are stored in the target storage space, including the following method steps: Step S321, the composite log records are written to the log partition of the target storage space in an append write manner, wherein the writing order of the composite log records in the log partition corresponds to the commit order of the target transaction.

[0083] In this embodiment, the target storage space can be a persistent storage space independent of the producer and consumer nodes, such as a distributed file system, cloud storage service, or network-attached storage. This target storage space can be written to by producer nodes and read from by consumer nodes. A log partition can be a logical unit within the target storage space used to organize log data by time or sequence number; the target storage space can contain multiple log partitions.

[0084] Append-only writing refers to writing newly generated composite log records to the end of existing log data in the target storage space without overwriting or modifying existing log data. Using append-only writing ensures that the physical storage order of composite log records is consistent with their generation order on the production nodes.

[0085] In this application scenario, after generating a composite log record, the producer node can determine the target log partition to be written to based on the current time or log size. The producer node obtains the current end position of the write operation for that log partition and then writes the composite log record to that position. Since the producer node generates composite log records sequentially according to the commit order of the target transaction, the write order of each composite log record in the log partition corresponds to the commit order of the target transaction. The consumer node reads the composite log records sequentially according to this write order, thus replaying the transactions in the order they were committed.

[0086] Through the above optional embodiments, this application uses an append-only writing method to write composite log records to an independent shared storage space and ensures that the writing order is consistent with the transaction commit order. This avoids complex random write overhead and enables consumer nodes to read logs in the correct transaction order, thereby improving the orderliness and reliability of columnar data synchronization.

[0087] As an optional implementation, transaction sequence information includes the transaction sequence number. The composite log record generated by the producer node carries the transaction sequence number corresponding to the target transaction and the changed primary key of at least one row record. This allows the consumer node to write the primary key index based on the transaction sequence number and the changed primary key during transaction replay, thereby obtaining the primary key index data. This primary key index data is used to determine the column storage location corresponding to the row record. The producer node also provides row change content through the composite log record, enabling the consumer node to write the row change content into the column storage data based on the primary key index data.

[0088] As an optional implementation, in the composite log records generated by the producer node, at least one row record is organized according to the transaction sequence number and the changed primary key. This allows the consumer node to distribute the row records to at least one primary key index processing process. The primary key index processing process sequentially processes multiple row records corresponding to the same changed primary key in multiple transactions according to the transaction sequence number. Within the primary key index processing process, the consumer node allocates column storage locations for the row records and writes the changed primary key and column storage location corresponding to the row record into the primary key index data.

[0089] As an optional implementation, the composite log records provided by the production node include row change content, so that the consumer node can convert the row change content of the row record into columnar data based on the columnar structure corresponding to the columnar data, and write the columnar data into the data unit corresponding to the columnar storage location in the columnar data according to the primary key index data, thereby updating the columnar storage data.

[0090] As an optional implementation, the change type corresponding to the row-level data changes generated by the production node is used to determine the mask type of the mask marker associated with the data unit in the column-level data. The marker value of this mask marker is used to characterize whether the data unit is visible to the analysis query.

[0091] As an optional implementation, the transaction sequence number corresponding to the target transaction generated by the producer node is used to support the consumer node in performing the following operations after reading the composite log record: creating a transaction status identifier in the slot bucket structure for the target transaction and configuring the transaction status identifier to the initialization state; configuring the transaction status identifier to the index commit state after completing the primary key index write; configuring the transaction status identifier to the data commit state after completing the row change content write to the column storage data, updating the globally visible sequence number queue of the target database according to the transaction sequence number of the target transaction, and updating the mask mark of the column storage data so that the data unit corresponding to the target transaction is visible to the analysis query; configuring the transaction status identifier to the global commit state after completing the update of the mask mark, and releasing the slot resources in the slot bucket structure corresponding to the transaction status identifier; configuring the transaction status identifier to the reclaimed state after completing the release of the slot resources.

[0092] As an optional implementation, the producer node can receive and process the original transaction when generating composite log records. When the data volume of the original transaction exceeds a preset value, the producer node can split the original transaction into multiple sub-transactions and generate corresponding composite log records for each sub-transaction as an independent target transaction.

[0093] As an optional implementation, the column storage location information corresponding to the row changes in the composite log record generated by the producer node is used to support the consumer node in recording the rollback log of the target transaction after writing the row changes to the column storage data. This rollback log includes the change type, the changed primary key, and the column storage location. When the target transaction needs to be rolled back, the consumer node performs a reverse change operation on the primary key index data and on the column storage data based on this rollback log, obtaining the rolled-back primary key index data and column storage data.

[0094] Under the aforementioned operating environment, this application provides the following: Figure 4 The data processing method shown. Figure 4 This is a flowchart of a data processing method according to an embodiment of this application. For example... Figure 4 As shown, the method may include the following steps S401 to S402.

[0095] Step S401: In response to the synchronization trigger signal sent by the production node, read the composite log record stored by the production node in the target storage space. The composite log record is generated by the production node based on the transaction operation information when the target transaction in the target database undergoes row storage data changes. The composite log record is used to characterize the physical redo information and change logic semantics of the row storage data changes.

[0096] The execution entity of the data processing method provided in this application embodiment can be a consumer node deployed in a database system. This consumer node can be a read-only (RO) node, used to maintain columnar data and process analytical queries.

[0097] The aforementioned producer node, also known as a read-write node or database master node, is used to process write transactions in the target database and perform data modification operations. When row-level data changes occur in the target database during a target transaction, the producer node generates a composite log record based on the transaction operation information, then stores the composite log record in the target storage space, and sends a synchronization trigger signal to the consumer node.

[0098] The aforementioned synchronization trigger signal can be a signal message sent from the producer node to the consumer node. This synchronization trigger signal informs the consumer node that a new composite log record has been stored in the target storage space and is available for reading. The synchronization trigger signal may include location indication information to specify the storage location of the composite log record in the target storage space. This storage location may be represented by a log sequence number, file offset, timestamp, etc.

[0099] The target storage space can be a persistent storage space independent of the producer and consumer nodes. This target storage space can be written to by producer nodes and read from by consumer nodes. This target storage space can also be a shared storage space. For example, it can be a distributed file system, cloud storage service, or network-attached storage.

[0100] The aforementioned composite log record can be log data that simultaneously carries physical redo information and change logic semantics. The physical redo information characterizes the modifications made to row-level data at the physical storage level. The change logic semantics characterizes the change semantics of row-level data at the logical level. Through composite log records, the physical redo information and change logic semantics are associated in the same log file.

[0101] For example, in an application scenario, the consumer node receives a synchronization trigger signal sent by the producer node via a network connection. This network connection can be a dedicated control channel established between the producer and consumer nodes. The consumer node parses the location indication information from the synchronization trigger signal and then reads the corresponding composite log record from the target storage space based on the location indication information. Specifically, the consumer node can read one or more composite log records sequentially according to the writing order.

[0102] Step S402: Transaction replay is performed based on the composite log record and the transaction time sequence information corresponding to the target transaction to update the columnar storage data in the consumer node. The columnar storage data is used to respond to the analysis query of the target database.

[0103] The aforementioned transaction replay refers to the process by which the consumer node reapplies the row-level data changes described in the composite log record to the column-level data. This transaction replay process ensures that the updated column-level data reflects the changed state of the row-level data in the producer node.

[0104] The aforementioned columnar storage data refers to a data organization format where data columns are stored contiguously in a storage medium. This columnar storage data is deployed on consumer nodes to respond to analytical queries from the target database. When an analytical query arrives at the target database, it can be routed to the consumer node for execution. The consumer node responds to the analytical query using the updated columnar storage data and returns the query results.

[0105] During transaction replay, the consumer node parses physical redo information and change logic semantics from the read composite log records. Furthermore, the consumer node can obtain the transaction sequence information corresponding to the target transaction. This transaction sequence information can be a globally incrementing transaction commit number, used to identify the transaction's commit order. Further, the consumer node determines the transaction replay order based on this transaction commit number and executes columnar data updates based on the physical redo information and change logic semantics according to this replay order.

[0106] Specifically, the consumer node determines the type of operation to be performed on the columnar storage data based on the parsed change logic semantics. This operation type can include insert, delete, or update operations. The consumer node can also locate the target data in the primary key index of the columnar storage data based on the changed row identifier information (such as the primary key value) in the physical redo information; and write the changed content into the basic storage unit (such as data block or row group) of the columnar storage data according to the parsed operation type.

[0107] Through the above steps S401 to S402, the embodiments of this application read the composite log record in response to the synchronization trigger signal issued by the production node, and perform transaction replay based on the composite log record and the transaction timing information corresponding to the target transaction to update the columnar storage data. This achieves the purpose of updating the row storage data of the production node to the columnar storage data of the consumer node in a low-latency manner, thereby improving the technical effect of columnar storage data synchronization efficiency and solving the technical problem of low data synchronization efficiency in related technologies.

[0108] The following will further explain other optional embodiments of the data processing method in the embodiments of this application.

[0109] As an optional implementation, the change logic semantics include change type. The composite log record is generated by the producer node based on transaction operation information. When generating the composite log record, the producer node determines the physical redo information and change type of at least one changed row based on the transaction operation information. The physical redo information includes the change primary key and row change content corresponding to the changed row. The change type includes at least addition changes and deletion changes. The change type is associated with the log position corresponding to the physical redo information to generate the composite log record. This composite log record includes the row record corresponding to the changed row. Accordingly, after the consumer node reads the composite log record from the target storage space, it can parse the physical redo information and change type of the changed row from the composite log record.

[0110] As an optional implementation, the target storage space is a persistent storage space in the target database that is independent of the producer and consumer nodes. In this target storage space, composite log records are written to the log partition using an append-only method. The write order of the composite log records in this log partition corresponds to the commit order of the target transaction. The consumer node reads the composite log records sequentially from the log partition of the target storage space according to this write order.

[0111] As an optional embodiment, the transaction sequence information includes the transaction sequence number. In step S402, transaction replay is performed based on the composite log record and the transaction sequence information to update the column storage data, including the following method steps: Step S421, primary key index is written based on the transaction sequence number and the change primary key of at least one row record in the composite log record to obtain primary key index data, wherein the primary key index data is used to determine the column storage location corresponding to the row record; Step S422, the row change content of at least one row record is written into the column storage data according to the primary key index data.

[0112] In this embodiment, the transaction sequence number can be a globally incrementing sequence number used to identify the commit order of the target transaction. The primary key index data can be data in a non-clustered index (NCI) structure, used to maintain the mapping relationship between the changed primary key of a row record and its column storage location. The column storage location can be the storage address of the row record in the column storage data, for example, the row identifier, the data block number to which it belongs, and the offset within the block.

[0113] In the application scenario, the consumer node parses the changed primary key of each row record from the composite log records and obtains the transaction sequence number of the target transaction to which that row record belongs. Further, based on the order of the transaction sequence number and combined with the hash value of the changed primary key, the rows belonging to different primary key ranges are distributed to different primary key index processing processes (NCI workers). This ensures that modifications to the same changed primary key are strictly executed in transaction sequence order, while modifications to different changed primary keys can be performed in parallel. Within each primary key index processing process, the consumer node allocates a corresponding column storage location for the row record and then writes the mapping relationship between the changed primary key of that row record and the allocated column storage location into the primary key index data.

[0114] After the primary key index is written, the consumer node can determine the column storage location corresponding to each row record based on the mapping relationship in the primary key index data. Then, the consumer node transforms the row changes (e.g., the specific data values ​​of each column in the row record) according to the columnar structure of the column storage data, breaking down a row of data into multiple column values ​​and writing them to the corresponding column's storage area. Further, the transformed columnar data is written to the data unit corresponding to that column storage location, thus completing the column storage data update.

[0115] Through the above optional embodiments, this application improves the parallelism of replay by including the primary key index writing stage and the columnar data writing stage in the transaction replay process, and by using the transaction sequence number and the changed primary key to achieve fine-grained row-level parallelism, thereby improving the efficiency of columnar data synchronization while ensuring the correctness of the transaction order.

[0116] As an optional embodiment, in step S421, primary key index writing is performed based on the transaction sequence number and the changed primary key of at least one row record to obtain primary key index data, including the following method steps: Step S423, according to the transaction sequence number and the changed primary key, at least one row record is distributed to at least one primary key index processing process, wherein the primary key index processing process is used to process multiple row records corresponding to the same changed primary key in multiple transactions in sequence according to the transaction sequence number; Step S424, in at least one primary key index processing process, column storage locations are allocated for the row records, and the changed primary key and column storage locations corresponding to the row records are written into the primary key index data.

[0117] In this embodiment, the primary key index processing process (NCI worker) can be a background worker thread in the consumer node used to maintain primary key index data. A consumer node can be configured with one or more primary key index processing processes to achieve parallel processing. Row records with the same primary key change may appear sequentially in multiple transactions, such as updating the same row of data multiple times. These row records need to be processed in the order of transaction sequence numbers to ensure that the index state is consistent with the transaction commit order.

[0118] In this application scenario, the consumer node parses multiple row records from the composite log records. Each row record contains the changed primary key and the transaction number of the transaction to which it belongs. The consumer node distributes the row records to the corresponding primary key index processing processes based on the hash value of the changed primary key. For row records with the same changed primary key (which may come from different transactions), these row records are processed in ascending order of transaction number to ensure that modifications to the same changed primary key do not cause conflicts.

[0119] In each primary key index processing process, when processing a row record, the primary key index processing process allocates a physical storage location (e.g., Row ID) for that row record in the columnar storage data, and then writes the mapping relationship between the row record's changed primary key and the allocated columnar storage location into the primary key index data. Subsequent columnar storage data writing stages can then find the corresponding location and write the data based on this mapping relationship.

[0120] Through the above optional embodiments, this application performs fine-grained distribution according to transaction sequence number and change primary key, which makes modifications to the same primary key strictly serialized and modifications to different primary keys completely parallel, thereby enhancing the parallelism of index writing while ensuring transaction consistency and further improving the efficiency of columnar data synchronization.

[0121] As an optional embodiment, in step S422, the row change content is written into the columnar storage data according to the primary key index data, including the following method steps: Step S425, based on the columnar structure corresponding to the columnar storage data, the row change content of the row record is converted into columnar data; Step S426, based on the primary key index data, the columnar data is written into the data unit corresponding to the columnar storage location in the columnar storage data to update the columnar storage data.

[0122] In this embodiment, the columnar structure can be an organization form in which columnar data is stored continuously in units of data columns. Row change content can be the specific data values ​​of each column in a row record. Columnar data can be multiple data values ​​organized by columns, obtained by decomposing the row change content. Columnar storage location can be the storage address of the row record in the columnar data, for example, the row identifier (Row ID), the data block number to which it belongs, and the offset within the block. The aforementioned data unit can be a storage unit for columnar data, also called a row group or data block (Pack), used to store the actual columnar data.

[0123] In the application scenario, after the consumer node completes the primary key index writing, it can obtain the column storage location corresponding to each row record from the primary key index data. Then, based on the columnar structure of the column storage data, the consumer node transforms the row changes (such as the values ​​of each field) of the row record. Specifically, it extracts the values ​​of different columns in the same row record and organizes them into columnar data.

[0124] The consumer node further locates the corresponding data unit in the columnar storage based on the columnar storage location recorded in the primary key index data. The consumer node then writes the transformed columnar data to the specific location corresponding to the columnar storage location within that data unit. After writing is complete, the changed data content of that row is updated and persisted to the columnar storage data, making it available for subsequent analysis and querying.

[0125] Through the above optional embodiments, this application converts row-level changes into columnar data and writes the columnar data into the corresponding data units according to the columnar storage location in the primary key index data. This achieves the correct conversion and storage of row-level data into columnar data, ensuring that the columnar data can be organized according to the columnar structure. This facilitates columnar scanning and vectorized calculation for subsequent analysis and query, and further improves the efficiency and availability of columnar data synchronization.

[0126] In one exemplary application scenario, such as Figure 5 The example shown is a scheme for real-time synchronization and parallel playback of columnar data based on physical log logic, such as... Figure 5 As shown, the system architecture of this scheme includes a read-write master node (i.e., the producer node), read-only nodes (i.e., the consumer node), and shared storage space. When a client transaction is received, the database engine of the read-write master node generates a column-store index logical log. This column-store index logical log is a composite log record containing physical redo information and change logic semantics (such as insert, delete, update, etc.). In addition, while generating the index logical log, the database engine performs synchronous write redo, that is, writes the column-store index log to the shared storage space in the order of its generation.

[0127] This shared storage space is shared by read-write master nodes and read-only nodes. It is used to store redo log files, which sequentially store multiple data entries of the column-stored index log, potentially corresponding to different change types. After the read-write master node writes the column-stored index logical log to the shared storage space, it can send a log position notification to the read-only nodes.

[0128] Upon receiving a log position notification, the read-only node reads the corresponding data entry from the redo log file in the shared storage space. The redo distribution component within the read-only node parses the read columnar index log into a logical log, organizes it according to transaction sequence information to obtain a columnar index logical log sequence, and delivers it to the columnar index transaction replay pipeline. This transaction replay pipeline first performs a primary key index write to obtain the primary key index data. Then, based on the primary key index data, it converts the row changes into columnar data and writes them to the corresponding data cells in the columnar index, thus completing the columnar index update. The updated columnar index is then used to respond to subsequent analytical queries initiated by the client.

[0129] pass Figure 5 The proposed solution, when applied to a specific implementation scenario, utilizes a column-store index logical log containing physical redo information and change logic semantics, combined with the parallel processing of the transaction replay pipeline, to achieve efficient and low-latency synchronization of row-store data changes to column-store data.

[0130] As an optional embodiment, in the data processing method, data units in the columnar data are associated with mask markers. The marker value of the mask marker is used to characterize whether the data unit is visible to the analysis query. The mask type of the mask marker is determined by the change type corresponding to the data unit.

[0131] In this embodiment, the mask marker can be a visibility identifier associated with a data unit, such as an insert mask or a delete mask. The marker value can be a Boolean value or a bitmap, used to indicate whether the data in the data unit is visible to the query under the current Multi-Version Concurrency Control (MVCC) snapshot. The mask type can include insert mask type and delete mask type, respectively corresponding to the change logic semantics (i.e., adding a change or deleting a change) carried by the data unit.

[0132] In application scenarios, when a consumer node writes data to a columnar storage data unit based on the change type (such as insert or delete) in the composite log record, it sets a corresponding mask flag for that data unit. For example, if the change type is insert, the consumer node can configure the associated mask flag as an insert mask, with its initial value set to X1, indicating an invisible state. After the target transaction is globally committed, the flag value is updated to X2, indicating a visible state. If the change type is delete, the consumer node can configure the associated mask flag as a delete mask, with its value set to X3, indicating a deleted state, thus making the data unit invisible to analysis queries. The specific values ​​of X1, X2, and X3 can be flexibly set according to the application scenario requirements. Through mask flags, the system can flexibly control data visibility without actually moving or deleting physical data, thereby supporting efficient concurrent read / write and transaction isolation.

[0133] Through the above optional embodiments, this application associates data units of columnar data with mask tags determined based on change type, enabling consumer nodes to control data visibility in a lightweight tagging manner during transaction replay without physical deletion or data movement, thereby improving the efficiency of columnar data updates and ensuring the correctness of the multi-version concurrency control mechanism.

[0134] As an optional embodiment, the data processing method further includes the following steps: Step S431, after reading the composite log record, creating a transaction status identifier in the slot bucket structure for the target transaction, and configuring the transaction status identifier to the initialization state; Step S432, in response to completing the primary key index writing, configuring the transaction status identifier to the index commit state; Step S433, in response to completing the row change content writing to the column storage data, configuring the transaction status identifier to the data commit state, updating the globally visible sequence number queue of the target database according to the transaction sequence number of the target transaction, and updating the mask mark of the column storage data so that the data unit corresponding to the target transaction is visible to the analysis query; Step S434, in response to completing the update of the mask mark, configuring the transaction status identifier to the global commit state, and releasing the slot resources corresponding to the transaction status identifier in the slot bucket structure; Step S435, in response to completing the release of the slot resources, configuring the transaction status identifier to the reclaimed state.

[0135] In this embodiment, the slot bucket structure can be a pipeline unit for managing the lifecycle of transaction replay. This slot bucket structure can include multiple slots, each containing a fixed number of buckets, and each bucket carrying the status information of a transaction. The transaction status identifier can be a marker indicating the stage of the target transaction in the replay pipeline. The transaction status identifier may correspond to the following transaction states: Initialized, NCI Committed, Pack Committed, Global Committed, and Recycled. The globally visible sequence number queue can be a data structure that maintains the sequence number of globally committed transactions, used to control the visibility of data under Multi-Version Concurrency Control (MVCC).

[0136] In the application scenario, after reading the composite log record, the consumer node allocates a slot in the slot bucket structure for the target transaction and creates a corresponding transaction status identifier. The transaction status identifier is configured to the initial state, indicating that the target transaction has entered the transaction replay pipeline but has not yet started processing. When the consumer node completes the primary key index write for the target transaction, it updates the transaction status identifier to the index commit state, indicating that the index mapping is ready. When the consumer node further completes the writing of the row changes of the target transaction into the columnar storage data, it updates the transaction status identifier to the data commit state. Simultaneously, it updates the globally visible sequence number queue according to the transaction sequence number and updates the mask marker of the corresponding data unit (e.g., changing the insertion mask from X1 to X2), making the columnar storage data of the target transaction visible for subsequent analysis and queries. Afterward, the consumer node updates the transaction status identifier to the global commit state and releases the slot resources (such as memory and slot number) occupied by the target transaction. Further, the consumer node configures the transaction status identifier to the reclaimed state, indicating that the lifecycle of the target transaction in the transaction replay pipeline has ended, and the relevant resources can be reused.

[0137] It should be noted that an analysis component (Analyzer) can also be configured during the transaction replay process described above. This analysis component can collect time-consuming event tracking data for each stage (such as the initialization stage, index commit stage, data commit stage, global commit stage, and resource reclamation stage). This time-consuming event tracking data can be used to monitor performance bottlenecks in the replay pipeline in real time, helping to pinpoint the specific points where delays occur, thereby providing a basis for system optimization.

[0138] In one exemplary application scenario, embodiments of this application also provide, for example... Figure 6 The diagram illustrates a column-oriented index transaction replay pipeline based on a slot bucket mechanism. Figure 6 As shown, this pipeline manages the entire lifecycle of a transaction from entering the replay system to committing and reclaiming resources, including a slot pool, a primary key index dispatcher, a primary key index processing process, a data writing process, a sequence number advancement component, a slot reclamation component, and an analysis component.

[0139] First, the input transaction stream enters the slot pool sequentially according to the transaction commit number, and is further allocated to the corresponding slot. Specifically, each slot contains a fixed number of buckets, and each bucket is used to carry the status information of a target transaction. When a read-only node reads the composite log record, it obtains an empty slot from the slot pool, creates a transaction status identifier for the target transaction, and configures the status identifier to the initial state.

[0140] Subsequently, the transaction enters the primary key index dispatcher. The primary key index dispatcher distributes the row records to the corresponding primary key index processing processes (i.e., the primary key index processing processes) based on the hash value of the changed primary key in the row records within the transaction. The primary key index processing processes process the received row records sequentially according to the transaction commit number, allocating column storage locations for each row record and writing the mapping relationship between the changed primary key and the column storage location into the primary key index data. Further, the transaction's status is updated from initialization to index commit, indicating that the index mapping is ready. At this point, the transaction is placed in the transaction queue (which includes transactions that have completed primary key index distribution), awaiting entry into the data writing phase.

[0141] During the data access phase, the data writing process retrieves transactions from the transaction queue, converts row changes into columnar data based on the columnar storage location recorded in the primary key index data, and writes the changes to the corresponding data cells in the columnar storage data. After the write is complete, the transaction status is updated from index committed to data committed.

[0142] Furthermore, the sequence number advancement component maintains a globally visible sequence number queue according to the transaction commit sequence numbers. When a transaction enters the data commit state, the sequence number advancement component updates the globally visible sequence number queue based on the transaction's sequence number and updates the mask flag of the corresponding data cell in the columnar storage data (e.g., changing the insertion mask from invisible to visible), so that the transaction's data can be seen in subsequent analysis and queries. At this time, the transaction's status is updated to the globally committed state.

[0143] Furthermore, the analysis component can collect time-consuming event tracking data for each stage of the pipeline (such as the initialization stage, index commit stage, data commit stage, global commit stage, and resource reclamation stage). This data is used to monitor and replay performance bottlenecks in the pipeline in real time, helping to pinpoint the specific points where latency occurs and providing a basis for system optimization. Based on the analysis results and global commit status, slot reclamation can be further performed.

[0144] The slot reclamation component is used to decompose transactions that have completed a global commit, release the slot resources they occupy, and return the slots to the slot pool. The released slot resources can be reused by new transactions entering the pipeline subsequently. The transaction's status is updated to "reclaimed," indicating that the transaction's lifecycle in the replay pipeline has ended.

[0145] Through the above optional embodiments, this application uses a slot bucket structure and a multi-state transition mechanism to accurately manage and monitor each stage of transaction replay, ensuring that transactions are committed and visibility switched in the correct order, while timely reclaiming resources, thereby improving the stability of columnar data synchronization and resource utilization efficiency.

[0146] As an optional embodiment, the data processing method further includes the following steps: Step S441, in response to the data reorganization task acting on the columnar storage data, clearing the row records to be distributed to the primary key index processing process, and distributing the data reorganization task to the primary key index processing process according to a preset batch granularity; Step S442, in the primary key index processing process, modifying the columnar storage data to be reorganized according to the data reorganization task to obtain the reorganized columnar storage data, and batch inserting the reorganized mapping data corresponding to the data reorganization task into the primary key index data to obtain the reorganized primary key index data.

[0147] In this embodiment, the data compaction task can be a process of physically reorganizing columnar data to improve query efficiency and storage space utilization. The data compaction task may involve reorganizing old columnar data blocks into new data blocks. During this process, the physical storage location (i.e., columnar storage location) of data rows within a data block may change, therefore the mapping relationship between the primary key and the columnar storage location maintained in the primary key index data needs to be updated synchronously. The preset batch granularity can be based on the size of the data to be processed in the data compaction task, dividing the task into multiple batches, each batch containing a certain number of row records.

[0148] In application scenarios, when a data reorganization task enters the transaction replay pipeline, the consumer node can trigger a dispatch barrier mechanism. First, it forcibly drains all backlogged preceding primary key index modification tasks (i.e., row record distribution tasks corresponding to regular data modification operations), ensuring that all index updates for regular transactions have been processed before the data reorganization task executes. Then, instead of fine-grained distribution based on the primary key hash, the consumer node directly splits the data reorganization task into multiple batches according to a preset batch granularity, and distributes each batch as a whole to an idle primary key index processing process.

[0149] In the primary key index processing process, the process can modify the columnar storage data to be reorganized based on the content of the data reorganization task (e.g., migrating data from old data blocks to new data blocks), generating reorganized columnar storage data. Simultaneously, because the pre-processing clear operation ensures no interference from concurrent regular data modification tasks, the process does not need to read the primary key index on disk to check for conflicts. Instead, it can directly append the new mapping relationship corresponding to the data reorganization task (i.e., the correspondence between the changed primary key and the new columnar storage location) to the primary key index data in a batch insert manner, thus obtaining the reorganized primary key index data. This batch insert can also be called a blind bulk insert. The above process requires no disk I / O operations and can achieve pure memory operation.

[0150] Through the above optional embodiments, this application creates an execution window for data reorganization tasks without interference from parallel conventional modifications by introducing a distribution barrier mechanism and batch granular distribution. This allows the primary key index processing process to skip disk conflict checks and directly perform blind write batch insertion, thereby avoiding the data reorganization operation from blocking the input and output of the transaction replay link and improving the overall efficiency of columnar data synchronization.

[0151] As an optional embodiment, the data processing method further includes the following steps: Step S451, in response to the completion of a batch of reorganized mapping data insertion in the primary key index processing process, the entry delay of the transaction to be entered is obtained; Step S452, in response to the entry delay being greater than a first threshold, the primary key index processing process is controlled to interrupt the execution of the data reorganization task.

[0152] In this embodiment, the entry delay can refer to the waiting time experienced by subsequent transactions (such as regular data modification transactions) waiting to enter the replay pipeline from their generation to the current moment. The first threshold can be a preset time upper limit value, used to determine whether the execution of the data reorganization task has caused excessive delay in the replay of regular transactions. Interrupting execution can refer to pausing the currently ongoing data reorganization task and prioritizing the processing of backlogged regular transactions.

[0153] In application scenarios, when the primary key index processing process handles data reorganization tasks, it checks the current system load status after each batch of reorganized mapping data insertion is completed. Specifically, the primary key index processing process can obtain the entry latency of the next transaction waiting to enter the pipeline (e.g., the difference between the transaction's commit timestamp and the current time). If the entry latency is less than or equal to a first threshold, it indicates that the execution of the data reorganization task has not yet significantly affected regular transactions, and the primary key index processing process can continue processing the next batch of data reorganization tasks. If the entry latency is greater than the first threshold, it indicates that the execution of the data reorganization task has caused regular transactions to wait for too long. In this case, the consumer node can control the primary key index processing process to actively interrupt the current data reorganization task, freeing up processing resources for subsequent regular transactions, thereby ensuring the timeliness of regular data modification replay. The interrupted data reorganization task can be resumed when the system is idle later.

[0154] Furthermore, embodiments of this application can also employ a dynamic context pool prefetching mechanism, which preloads old data blocks to be reorganized into the context pool in the background. These old data blocks can serve as prefetched data. The size of the context pool can be dynamically scaled based on the interruption and success of the data reorganization task. For example, when popping a prefetched context from the context pool fails (i.e., there is no available prefetched data in the context pool), the capacity of the context pool can be increased; when the data reorganization task is interrupted and there is remaining available prefetched data in the context pool, the capacity of the context pool can be decreased. Through this dynamic adjustment, a balance can be achieved between memory usage and transaction replay latency.

[0155] Through the above optional embodiments, this application introduces a dynamic interruption control mechanism based on entry latency, which can achieve flow control based on time thresholds between data reorganization tasks and regular transaction replay, ensuring that data reorganization tasks do not excessively block the replay link of regular data modifications, thereby maintaining the stability of overall synchronization latency while ensuring the efficiency of columnar data organization.

[0156] In one exemplary application scenario, embodiments of this application also provide, for example... Figure 7 This diagram illustrates an optimized processing flow for a data compaction task. Figure 7 As shown, the system first starts a dynamic context pool processing process in the background. This process is responsible for preloading the old row group data to be reorganized into the context pool. The size of the context pool can be dynamically expanded or contracted based on the interruption and success of the data reorganization task.

[0157] Then, the data reorganization task is distributed according to a preset batch granularity (e.g., 128 rows per batch). The old row group data for each batch is sent to the primary key index processing process. The primary key index processing process does not perform conflict detection; it directly appends the new mapping (primary key to new column storage location) to the primary key index data using a blind bulk insert method, while simultaneously generating the reorganized new row group data. After completing one batch, the system can enter the next batch loop to continue processing subsequent batches.

[0158] The interrupt mechanism continuously monitors the system status during data reorganization. Specifically, after migrating each batch of data blocks, the primary key index processing process checks the entry timemark of subsequent transactions, i.e., the difference between the time the transaction entered the replay pipeline and the current time. If this difference (i.e., the entry delay) exceeds a preset time threshold, the current data reorganization task is interrupted, and the backlog of regular transactions is processed first. The interrupted data reorganization task can be resumed when the system is idle later.

[0159] pass Figure 7 The optimized processing flow shown in this application embodiment utilizes dynamic context pool prefetching and interrupt control based on entry time to effectively reduce the impact on the replay delay of regular transactions while ensuring data reorganization efficiency, thus achieving a dynamic balance between data reorganization and regular writing.

[0160] As an optional embodiment, in the data processing method, the target transaction is the original transaction that enters the target database, or the target transaction includes multiple sub-transactions obtained by splitting the original transaction.

[0161] In this embodiment, the original transaction can refer to a complete database transaction initiated and submitted directly to the target database by an application or user. This original transaction may contain one or more data modification operations (e.g., insert, delete, update). A sub-transaction can refer to several smaller transaction units obtained by splitting the original transaction according to certain rules (e.g., by data size or by the number of rows operated). Each sub-transaction can independently enter the replay pipeline for processing. The splitting of the original transaction can be performed by the producer node before or during the generation of composite log records.

[0162] In application scenarios, when the amount of data in the original transaction is small, the producer node can directly process the original transaction as a target transaction, generate the corresponding composite log record, and the consumer node will replay it according to the normal process.

[0163] When the original transaction contains a large amount of data, treating it as a single target transaction may lead to Out of Memory (OOM) errors on the consumer node or long-tail latency during the transaction commit phase. To address this, the producer node can split the original transaction into multiple smaller sub-transactions before generating composite log records. Each sub-transaction contains a portion of the original transaction's operations (e.g., 1000 rows per sub-transaction). The producer node generates a corresponding composite log record for each sub-transaction and stores and sends these records sequentially according to the processing order of the sub-transactions. Upon receiving these composite log records, the consumer node can perform transaction replay processing, thus enabling streaming processing of the large original transaction.

[0164] It's important to note that after the original transaction is split into multiple sub-transactions, these sub-transactions are treated as independent target transactions in the transaction replay pipeline. Consumer nodes can update the columnar data sequentially according to the commit order of the sub-transactions, and the overall columnar data update effect can be approximated or equivalent to the overall commit of the original transaction. This splitting approach can effectively reduce the impact of large transactions on system memory and latency without compromising transaction consistency.

[0165] Through the above optional embodiments, this application allows the target transaction to be the original transaction or multiple sub-transactions obtained by splitting the original transaction, which can flexibly cope with transaction scenarios with different data volumes, avoid memory overflow or long-tail latency during large transaction processing, and thus improve the stability and scalability of columnar data synchronization.

[0166] As an optional embodiment, the data processing method further includes the following steps: Step S461, after writing the row change content into the column storage data, record the rollback log of the target transaction, wherein the rollback log includes the change type, the changed primary key, and the column storage location; Step S462, in response to the rollback trigger event corresponding to the target transaction, perform a change reversal operation on the primary key index data according to the rollback log to obtain the rollback primary key index data, and perform a change reversal operation on the column storage data according to the rollback log to obtain the rollback column storage data.

[0167] In this embodiment, the Undo Log can be a data structure used to record the history of transaction modifications, enabling reverse recovery when the target transaction needs to be rolled back. The Undo Log may include the change type (such as an insert or delete change), the change primary key (i.e., the primary key value of the modified row), and the column storage location (such as row identifier, data block number, and offset within the block). A reverse change operation can refer to the operation opposite to the original change operation. For example, if the original operation is an insert operation corresponding to an insert change, its reverse operation is a delete operation; if the original operation is a delete, the reverse operation is an insert.

[0168] In the application scenario, after successfully writing the row changes of the target transaction into the columnar storage data, the consumer node can record a reversal log corresponding to that row change in memory. This reversal log stores the change type, the change primary key, and the columnar storage location allocated during the write. When the target transaction needs to be rolled back, the consumer node can read all the reversal logs corresponding to the target transaction and perform a reverse change operation based on the content of each reversal log.

[0169] For example, in a scenario where a target transaction (referred to as a large transaction) with a data volume exceeding a preset threshold is processed using a streaming pre-commit method, each sub-transaction records a corresponding undo log after writing data to column-oriented storage. When a large transaction needs to be rolled back, the consumer node can use these undo logs to perform a pure memory rollback operation with no input / output (I / O). Specifically, for row records with a deletion change type, the rollback operation is equivalent to an insertion. The consumer node can re-insert the mapping relationship from the primary key (PK) to the column-oriented storage location (Row ID) (i.e., a data entry in the primary key index data) into the primary key index data. Since the column-oriented storage location (such as the row identifier Row ID) has already been recorded in the undo log, the append process does not require reading the disk. For row records with an addition change type, the rollback operation is equivalent to a deletion. The consumer node can directly mark the corresponding delete mask in the column-oriented storage data, making the data unit invisible to subsequent queries. During the rollback process described above, all operations are completed in memory without triggering disk I / O operations. This avoids the overhead of reading disk logs or redoing data during the rollback process, thus improving the processing efficiency of columnar data synchronization when facing large transaction rollbacks.

[0170] Through the above optional embodiments, this application achieves lightweight, input / output-free rollback capability for transactions by recording a rollback log containing the change type, change primary key, and column storage location, and by using the rollback log to perform reverse change operations. This reduces costs while improving the processing efficiency of column storage data synchronization when facing transaction rollback.

[0171] As an optional embodiment, the data processing method further includes the following steps: Step S471, obtaining the parsing delay of the composite log record; Step S472, in response to the parsing delay being less than a second threshold, after reading the composite log record at the consumer node, reusing the regular parsing thread of the consumer node to parse at least one line record in the composite log record; Step S473, in response to the parsing delay being greater than or equal to the second threshold, starting an independent parsing thread at the consumer node; Step S474, within a preset transaction process interval after starting the independent parsing thread, monitoring the parsing status of the regular parsing thread; Step S475, in response to the parsing status remaining unchanged within the preset transaction process interval, skipping the reading of the composite log record, using the independent parsing thread to parse at least one line record in the composite log record, and reading at least one line record.

[0172] In this embodiment, parsing latency refers to the time elapsed from when a composite log record is written to the target storage space to when the consumer node completes parsing, representing the backlog in the log parsing queue of the consumer node. A regular parsing thread can be a thread in the consumer node originally used for parsing physical logs (e.g., redo logs), which also serves the parsing of columnar data logs. An independent parsing thread can be a separate thread specifically started for parsing columnar data logs, physically isolated from the regular parsing thread.

[0173] The aforementioned second threshold can be a preset upper limit for latency, used to determine whether to start an independent parsing thread. The preset transaction process interval can refer to a specific range of transaction sequence numbers or a time window, used to observe the status changes of the regular parsing thread. The parsing status can include information such as whether the regular parsing thread is currently active, whether it is at a log progression point, and whether it is blocked.

[0174] In application scenarios, consumer nodes can periodically or in real-time obtain the parsing latency of composite log records. Figure 8 This is a schematic diagram of an optional dynamic parsing thread switching process according to an embodiment of this application. For example... Figure 8 As shown, the consumer node first determines whether the parsing delay is less than the second threshold. If the parsing delay is less than the second threshold, it indicates that the current log consumption speed is normal, and the consumer node can continue to reuse the regular parsing thread to parse the row records in the composite log. At this time, the system is in a normal coupled state with low overhead. If the parsing delay is greater than or equal to the second threshold, it indicates that log consumption is backlogged. In this case, the consumer node can start an independent parsing thread (e.g., the IMCI Parser thread) for parsing the logs of columnar data. This independent parsing thread can be started in the background in advance to pre-parse the logs to catch up with the consumption progress.

[0175] Furthermore, within a pre-defined transaction process interval after the independent parsing thread is started (e.g., the interval from the current log position to a certain end position), the consumer node can continuously monitor whether the parsing state of the regular parsing thread remains unchanged. If the parsing state of the regular parsing thread remains unchanged within this interval (e.g., always idle or without state switching), it indicates that the regular parsing thread is no longer blocked by other modules, and the system can enter the point of no return, completing the thread switch. Afterward, the independent parsing thread continues to be used for parsing, and the system enters a decoupled state. If the parsing state of the regular parsing thread changes, the thread switch is aborted, and the regular parsing thread is reused. The above switching process ensures the determinism of the system state under the point of no return mechanism; that is, switching is only allowed when the state of the regular parsing thread does not change abnormally. When the backlog of transactions is cleared, the system can switch back to using the regular parsing thread for parsing, returning to the normal coupled state.

[0176] Figure 9 This is a schematic diagram of the state machine for dynamically resolving thread switching provided in an embodiment of this application. For example... Figure 9 As shown, the state machine includes the following states: the regular parsing thread is running (coupled state), the independent parsing thread is starting, the regular parsing thread is waiting to stop, the regular parsing thread has stopped, the independent parsing thread is running (decoupled state), and the independent parsing thread has finished starting. This state machine is used to describe the complete process of the log parsing thread in the consumer node switching between the coupled and decoupled states.

[0177] Specifically, firstly, the "running regular parsing thread" (coupled state) indicates that the consumer node is currently using the regular parsing thread to parse the row records in the composite log. When the parsing delay of the composite log record is detected to reach or exceed the second threshold, the consumer node triggers a switchover process, transitioning to the state of starting an independent parsing thread. In this state, the consumer node starts an independent parsing thread, which pre-parses the composite log records to catch up with the consumption progress. When the independent parsing thread successfully reaches the start of the preset transaction process interval (e.g., log position APPLY_BEGIN) and the state of the regular parsing thread is successfully set, the system transitions to the state where the regular parsing thread has stopped.

[0178] Once the regular parsing thread enters the stopped state, it ceases distributing logs. At this point, the switching component starts an independent parsing thread. The regular parsing thread stops distributing logs, and the independent parsing thread officially takes over the log parsing task. The system then enters the independent parsing thread started and completed state, and subsequently enters the independent parsing thread running state (decoupled state).

[0179] Based on the independent parsing thread being in a started and completed state, when the switching component sets its state to the stopped state, the system transitions to a waiting state for the regular parsing thread to stop. In this state, the system waits for the regular parsing thread to complete its cleanup work. When coordination is successful at the end of the preset transaction process interval (e.g., the log position APPLY_END), the independent parsing thread stops parsing, the regular parsing thread resumes log distribution, and the system returns to the running state of the regular parsing thread (coupled state).

[0180] In addition, such as Figure 9 The state machine shown also incorporates a timeout rollback mechanism. If, while in the state of starting an independent parsing thread or waiting for the regular parsing thread to stop, the wait for the regular parsing thread to time out, the system will perform a rollback operation, returning to the state of starting the independent parsing thread to retry the switch.

[0181] When the system is in the independent parsing thread running (decoupled state), if the switching component confirms that the independent parsing thread has stopped and the regular parsing thread is running, the system can jump to the regular parsing thread running (coupled state), thus achieving a quick path to fall back from the decoupled state to the coupled state.

[0182] When the system completes the startup of the independent parsing thread, if the switching component detects that there is no need to continue the switching process, such as when the backlog of transactions has been cleared, the regular parsing thread has become available again, or the switching conditions are no longer met, the system can roll back to the state where the regular parsing thread has stopped, thereby terminating the current switching process and waiting for the subsequent re-triggering of the switching or directly resuming the distribution of the regular parsing thread.

[0183] Through such Figure 9 The state machine shown in this embodiment enables dynamic and seamless switching between the regular parsing thread and the independent parsing thread, ensuring smooth decoupling when the columnar log consumption backlog is accumulated. It also provides a timeout rollback mechanism to ensure the determinism and reliability of the system state.

[0184] Through the above optional embodiments, this application can dynamically switch between regular parsing threads and independent parsing threads to reuse existing threads to reduce overhead when log consumption is normal, and smoothly switch to independent parsing threads when consumption is backlogged, thereby avoiding the blocking of other critical operations (such as data definition language synchronization or transaction cleanup) by columnar log parsing, thus improving the reliability of columnar data synchronization while ensuring system stability.

[0185] The data processing method provided in this application can be widely applied to various mixed transaction and analysis processing scenarios, such as e-commerce transaction systems, financial payment platforms, IoT data aggregation and analysis, and real-time monitoring and reporting systems. For example, in an e-commerce transaction system, online transaction operations such as user order placement, payment, and returns trigger the production node to generate composite log records. The consumer node synchronizes transaction details to the columnar storage index within milliseconds through a parallel playback pipeline. The operations backend can then refresh analytical views such as sales details, sales rankings, and inventory warnings in real time, achieving second-level collaboration between transaction processing and analysis queries. In a financial risk control scenario, after payment records are written to the production node, this application adopts a large transaction streaming pre-commit and no-input-output rollback mechanism to synchronize transaction data to the columnar storage engine in real time. The risk control model can perform multi-dimensional aggregation queries based on real-time columnar storage data, completing risk assessment upon transaction completion and improving risk response speed. In the IoT device data aggregation and analysis platform, the time-series data reported by the devices is split by the production-end nodes and synchronized to columnar storage in the form of sub-transactions. The consumer-end nodes utilize slot-bucket pipelines and data reorganization blind write optimization to maintain stable query latency while performing large-volume writes, meeting the real-time requirements of device status monitoring, anomaly alarms, and trend prediction. Overall, the embodiments of this application can provide efficient, stable, and scalable columnar storage data synchronization capabilities for various systems that need to simultaneously support high-concurrency online transaction processing and low-latency online analysis processing, reducing system architecture complexity and improving the real-time performance of service decisions.

[0186] In the aforementioned operating environment, this application also provides, as follows: Figure 10 The method shown is a task processing method applied to the production node of the target database. Figure 10 This is a flowchart of a task processing method according to an embodiment of this application, such as... Figure 10 As shown, the task processing method includes: step S1001, receiving a transaction processing task initiated to the target database; step S1002, processing the transaction processing task using the target row storage data to obtain the transaction processing result; step S1003, in response to row storage data changes triggered during the processing of the transaction processing task, updating the column storage data in the consumer node of the target database according to any of the above data processing methods, wherein the column storage data is used to respond to the analysis query of the target database.

[0187] The aforementioned producer nodes can be read-write master nodes, used to receive and process transaction processing tasks initiated by clients. For example, such transaction processing tasks may include online transaction processing (OLTP) requests. Producer nodes execute transaction operations using locally maintained row-level data and return the transaction processing results. During transaction execution, if changes such as insertion, deletion, or update of row-level data occur, the producer nodes synchronize these changes to shared storage in real-time in log form according to the aforementioned data processing methods, and notify consumer nodes to consume and replay them. In this way, transaction processing and analytical workloads are physically isolated between the producer and consumer ends. The producer can complete transaction commits without waiting for column-level updates, thus avoiding the impact of analytical queries on transaction processing performance.

[0188] Through the above steps S1001 to S1003, this embodiment of the application completes the transaction processing using row storage data after receiving the transaction processing task at the production node, and updates the column storage data in the consumer node according to the aforementioned data processing method in response to changes in row storage data triggered during the transaction processing. This improves the task processing efficiency by synchronizing row storage changes to column storage data in real time without blocking transaction processing.

[0189] It should be noted that the preferred embodiments of steps S1001 to S1003 described above can be found in the foregoing description, and will not be repeated here.

[0190] In the aforementioned operating environment, this application also provides, as follows: Figure 11 Another task processing method shown is applied to the consumer node of the target database. Figure 11 This is a flowchart of another task processing method according to an embodiment of this application, such as... Figure 11 As shown, the task processing method includes: step S1101, receiving an analysis query task initiated to the target database; step S1102, processing the analysis query task using the target columnar storage data to obtain the analysis processing result, wherein the target columnar storage data includes columnar storage data that is updated in real time in response to changes in row storage data according to any of the above data processing methods, and the changes in row storage data are triggered by a transaction processing task initiated to the production node of the target database.

[0191] The aforementioned consumer nodes can be read-only nodes, used to receive and process analytical query tasks initiated by clients (such as OLAP requests). Each consumer node maintains local columnar storage data, which is updated in real-time from changes in the row-level storage data of the production nodes using the aforementioned data processing methods (e.g., through composite logging, transaction replay pipelines, etc.). When an analytical query task arrives, the consumer node directly uses its local columnar storage data for scanning, aggregation, and other computational operations, without needing to access the row-level storage data of the production nodes or perform additional data extraction and transformation. Because the columnar storage data is organized in a columnar structure and is kept in low-latency synchronization with the row-level storage data through the aforementioned parallel replay pipeline, analytical queries can achieve high data freshness and query performance.

[0192] Through the above steps S1101 to S1102, after receiving the analysis and query task through the consumer node, this embodiment of the application directly uses the columnar storage data that has been synchronized and updated in real time for processing. This can provide low-latency, high-throughput data services for analysis and query without affecting transaction processing, thereby improving task processing efficiency.

[0193] It should be noted that the preferred embodiments of steps S1101 to S1102 described above can be found in the foregoing description, and will not be repeated here.

[0194] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0195] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM), random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0196] According to an embodiment of this application, an apparatus embodiment for implementing the above-described data processing method is also provided. Figure 12 This is a schematic diagram of the structure of a data processing apparatus according to an embodiment of this application, such as... Figure 12 As shown, the device includes: a generation module 1201, used to generate composite log records based on transaction operation information in response to row-level data changes triggered by a target transaction in the target database, wherein the composite log records are used to characterize the physical redo information and change logic semantics of the row-level data changes; a storage module 1202, used to store the composite log records in the target storage space; and a sending module 1203, used to send a synchronization trigger signal, wherein the synchronization trigger signal is used to notify the consumer node to read the composite log records in the target storage space, and to perform transaction replay based on the composite log records and the transaction timing information corresponding to the target transaction, so as to update the column-level data in the consumer node, wherein the column-level data is used to respond to the analysis query of the target database.

[0197] Optionally, the change logic semantics include change type; the above-mentioned generation module 1201 is further configured to: determine the physical redo information and change type of at least one changed row based on transaction operation information, wherein the physical redo information includes the change primary key and row change content corresponding to the changed row, and the change type includes at least addition change and deletion change; associate the change type with the log position corresponding to the physical redo information to generate a composite log record, wherein the composite log record includes the row record corresponding to the changed row.

[0198] Optionally, the target storage space is a persistent storage space in the target database that is independent of the producer node and the consumer node. The storage module 1202 is further used to: write composite log records to the log partition of the target storage space in an append-only manner, wherein the writing order of composite log records in the log partition corresponds to the commit order of the target transaction.

[0199] It should be noted that the above-mentioned generation module 1201, storage module 1202 and sending module 1203 correspond to steps S301 to S303 in the embodiment. The three modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the foregoing embodiment.

[0200] According to an embodiment of this application, another embodiment of an apparatus for implementing the above-described data processing method is also provided. Figure 13 This is a schematic diagram of another data processing apparatus according to an embodiment of this application, such as... Figure 13As shown, the device includes: a reading module 1301, used to read composite log records stored in the target storage space by the production node in response to a synchronization trigger signal issued by the production node. The composite log records are generated by the production node based on transaction operation information when row-level data changes occur in the target transaction in the target database. The composite log records are used to characterize the physical redo information and change logic semantics of the row-level data changes. A playback module 1302 is used to perform transaction playback based on the composite log records and the transaction timing information corresponding to the target transaction to update the column-level data in the consumer node. The column-level data is used to respond to analysis queries in the target database.

[0201] Optionally, the transaction sequence information includes the transaction sequence number. The replay module 1302 is further used to: write the primary key index based on the transaction sequence number and the change primary key of at least one row record in the composite log record to obtain primary key index data, wherein the primary key index data is used to determine the column storage location corresponding to the row record; and write the row change content of at least one row record into the column storage data according to the primary key index data.

[0202] Optionally, the playback module 1302 is further configured to: distribute at least one row record to at least one primary key index processing process according to the transaction sequence number and the changed primary key, wherein the primary key index processing process is configured to process multiple row records corresponding to the same changed primary key in multiple transactions in sequence according to the transaction sequence number; in at least one primary key index processing process, allocate column storage locations for the row records, and write the changed primary key and column storage locations corresponding to the row records into the primary key index data.

[0203] Optionally, the playback module 1302 is further configured to: convert the row change content of the row record into columnar data based on the columnar structure corresponding to the columnar data; and write the columnar data into the data unit corresponding to the columnar location in the columnar data according to the primary key index data, so as to update the columnar data.

[0204] Optionally, in the above data processing device, the data units in the columnar data are associated with mask markers. The marker value of the mask marker is used to characterize whether the data unit is visible to the analysis query. The mask type of the mask marker is determined by the change type corresponding to the data unit.

[0205] Optionally, in addition to all the modules mentioned above, the data processing device also includes a status management module (not shown in the figure), used for: after reading the composite log record, creating a transaction status identifier in the slot bucket structure for the target transaction, and configuring the transaction status identifier to the initialization state; in response to completing the primary key index write, configuring the transaction status identifier to the index commit state; in response to completing the row change content write to the column storage data, configuring the transaction status identifier to the data commit state, updating the globally visible sequence number queue of the target database according to the transaction sequence number of the target transaction, and updating the mask mark of the column storage data so that the data unit corresponding to the target transaction is visible to the analysis query; in response to completing the update of the mask mark, configuring the transaction status identifier to the global commit state, and releasing the slot resources in the slot bucket structure corresponding to the transaction status identifier; in response to completing the release of the slot resources, configuring the transaction status identifier to the reclaimed state.

[0206] Optionally, in addition to all the modules mentioned above, the data processing device also includes a reorganization module (not shown in the figure), which is used to: respond to a data reorganization task applied to columnar data, clear the row records to be distributed to the primary key index processing process, and distribute the data reorganization task to the primary key index processing process according to a preset batch granularity; in the primary key index processing process, modify the columnar data to be reorganized according to the data reorganization task to obtain the reorganized columnar data, and batch insert the reorganized mapping data corresponding to the data reorganization task into the primary key index data to obtain the reorganized primary key index data.

[0207] Optionally, in addition to all the modules mentioned above, the data processing device also includes an interrupt module (not shown in the figure), which is used to: obtain the entry delay of the transaction to be entered in response to the completion of a batch of reorganized mapping data insertion in the primary key index processing process; and control the primary key index processing process to interrupt the execution of the data reorganization task in response to the entry delay being greater than a first threshold.

[0208] Optionally, in the above data processing apparatus, the target transaction is the original transaction that enters the target database, or the target transaction includes multiple sub-transactions obtained by splitting the original transaction.

[0209] Optionally, in addition to all the modules mentioned above, the data processing device also includes a rollback module (not shown in the figure), which is used to: write the row change content into the column storage data, record the rollback log of the target transaction, wherein the rollback log includes the change type, the changed primary key, and the column storage location; in response to the rollback trigger event corresponding to the target transaction, perform a change reversal operation on the primary key index data according to the rollback log to obtain the rollback primary key index data, and perform a change reversal operation on the column storage data according to the rollback log to obtain the rollback column storage data.

[0210] Optionally, in addition to all the modules mentioned above, the data processing device also includes a switching module (not shown in the figure), used for: obtaining the parsing delay of the composite log record; in response to the parsing delay being less than a second threshold, after reading the composite log record, reusing the regular parsing thread of the consumer node to parse at least one line record in the composite log record; in response to the parsing delay being greater than or equal to the second threshold, starting an independent parsing thread of the consumer node; monitoring the parsing status of the regular parsing thread within a preset transaction process interval after starting the independent parsing thread; in response to the parsing status remaining unchanged within the preset transaction process interval, skipping the reading of the composite log record, using the independent parsing thread to parse at least one line record in the composite log record, and reading at least one line record.

[0211] It should be noted that the above-mentioned reading module 1301 and playback module 1302 correspond to steps S401 to S402 in the embodiment. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the foregoing embodiment.

[0212] According to an embodiment of this application, an apparatus embodiment for implementing the above-described task processing method is also provided, which is applied to a consumer node of a target database. Figure 14 This is a schematic diagram of the structure of a task processing device according to an embodiment of this application, such as... Figure 14 As shown, the device includes: a receiving module 1401 for receiving an analysis query task initiated against a target database; and a processing module 1402 for processing the analysis query task using the target columnar storage data to obtain the analysis processing result. The target columnar storage data includes columnar storage data that is updated in real time in response to changes in row storage data according to any of the above data processing methods. The changes in row storage data are triggered by a transaction processing task initiated against the production node of the target database.

[0213] It should be noted that the receiving module 1401 and processing module 1402 mentioned above correspond to steps S1101 to S1102 in the embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the foregoing embodiments.

[0214] According to an embodiment of this application, another apparatus embodiment for implementing the above-described task processing method is also provided, which is applied to the production node of the target database. Figure 15 This is a schematic diagram of another task processing device according to an embodiment of this application, such as... Figure 15As shown, the device includes: a receiving module 1501 for receiving a transaction processing task initiated to a target database; a processing module 1502 for processing the transaction processing task using the target row storage data to obtain the transaction processing result; and an updating module 1503 for updating the column storage data in the consumer node of the target database according to any of the above data processing methods in response to row storage data changes triggered during the processing of the transaction processing task, wherein the column storage data is used to respond to the analysis query of the target database.

[0215] It should be noted that the receiving module 1501, processing module 1502 and updating module 1503 mentioned above correspond to steps S1001 to S1003 in the embodiments. The three modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the foregoing embodiments.

[0216] It should be noted that the above-mentioned modules or units may be hardware or software components stored in memory and processed by one or more processors. The above-mentioned modules may also be part of a device and run in a computer terminal.

[0217] It should be noted that the preferred implementation methods of this embodiment can be found in the relevant descriptions of the foregoing embodiments, and will not be repeated here. The preferred implementation methods involved in the above embodiments of this application are the same as the solutions, application scenarios, and implementation processes provided in the above embodiments, but are not limited to the solutions provided in the above embodiments.

[0218] Figure 16 This is a schematic diagram of the structure of a data processing system according to an embodiment of this application, such as... Figure 16 As shown, the data processing system includes: a producer node, configured to generate composite log records based on transaction operation information in response to row-level data changes triggered by a target transaction in the target database; the composite log records are used to characterize the physical redo information and change logic semantics of the row-level data changes; a target storage space, configured to store the composite log records; and a consumer node, configured to read the composite log records in the target storage space in response to a synchronization trigger signal issued by the producer node, and to perform transaction replay based on the composite log records and the transaction timing information corresponding to the target transaction to update the column-level data in the consumer node, wherein the column-level data is used to respond to analysis queries in the target database.

[0219] Embodiments of this application may provide an electronic device, including: a memory storing an executable program; and a processor for running the program, wherein the program executes any of the foregoing data processing methods or task processing methods during runtime.

[0220] Figure 17 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 17 As shown, the electronic device 170 may include one or more (only one is shown in the figure) processors 172, memory 174, memory controller, and peripheral interfaces.

[0221] The aforementioned electronic device can be understood as an integrated smart terminal, including but not limited to servers, desktop computers, personal computers (PCs), all-in-one model machines, etc., and the electronic device may have the model described in the above embodiments of this application pre-installed.

[0222] Specifically, this electronic device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this electronic device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this electronic device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other types of models), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this electronic device can also create applications based on models, providing Application Programming Interface (API) invocation capabilities. Models can be invoked into the created applications through the API interface, and application management tools are provided to manage and monitor the applications.

[0223] Furthermore, the electronic device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing rich training resources to help users learn and master artificial intelligence (AI) technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0224] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the data processing method, task processing method, data processing device, and task processing device in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the data processing method or task processing method in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the computing terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0225] The processor can invoke an executable program stored in the memory via a transmission device to execute any of the above-described data processing methods or task processing methods in the above embodiments.

[0226] Those skilled in the art will understand that, Figure 17 The structure shown is for illustrative purposes only. Electronic devices can also be smart mobile devices (such as smartphones), tablets, PDAs, and mobile internet devices (MIDs) and other terminal devices. Figure 17 This does not limit the structure of the aforementioned electronic device. For example, electronic device 170 may also include components that are more... Figure 17 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 17 The different configurations shown.

[0227] Those skilled in the art will understand that all or part of the steps in the various data processing methods or task processing methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0228] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium includes a stored executable program, wherein, when the executable program runs, it controls the device where the computer-readable storage medium is located to execute any of the aforementioned data processing methods or task processing methods.

[0229] Optionally, in this embodiment, the storage medium may be located in an electronic device.

[0230] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, which, when the executable program is running, controls the device where the computer-readable storage medium is located to execute any of the data processing methods or task processing methods described in the above embodiments.

[0231] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the data processing method or task processing method provided in the above embodiments.

[0232] Embodiments of this application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which can be used to store a computer program. When the computer program is executed by a processor, it implements the data processing method or task processing method provided in the above embodiments.

[0233] Embodiments of this application also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the data processing method or task processing method provided in the above embodiments.

[0234] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0235] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0236] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0237] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0238] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the data processing method or task processing method described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, ROM, RAM, portable hard drives, magnetic disks, or optical disks.

[0239] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: In response to a synchronization trigger signal issued by a production node, the composite log record stored by the production node in the target storage space is read. The composite log record is generated by the production node based on transaction operation information when row-level data changes occur in the target transaction in the target database. The composite log record is used to characterize the physical redo information and change logic semantics of the row-level data change. The change logic semantics include the change type. The composite log record is generated by the production node by associating the change type with the log position corresponding to the physical redo information. Based on the transaction sequence number corresponding to the target transaction and the change primary key of at least one row record in the composite log record, a primary key index is written to obtain primary key index data, wherein the primary key index data is used to determine the column storage location corresponding to the row record; Based on the primary key index data, the row change content of the at least one row record is written into the columnar storage data in the consumer node, wherein the columnar storage data is used to respond to the analysis query of the target database.

2. The data processing method according to claim 1, characterized in that, The primary key index is written based on the transaction sequence number and the changed primary key of at least one row record, and the resulting primary key index data includes: According to the transaction sequence number and the changed primary key, the at least one row record is distributed to at least one primary key index processing process, wherein the primary key index processing process is used to process multiple row records corresponding to the same changed primary key in multiple transactions in sequence according to the transaction sequence number; In at least one of the primary key index processing processes, the column storage location is allocated for the row record, and the modified primary key and the column storage location corresponding to the row record are written into the primary key index data.

3. The data processing method according to claim 1, characterized in that, Writing the row change content into the columnar storage data based on the primary key index data includes: Based on the columnar structure corresponding to the columnar data, the row change content of the row record is converted into columnar data; Based on the primary key index data, the columnar data is written to the data unit corresponding to the columnar storage location in the columnar storage data to update the columnar storage data.

4. The data processing method according to any one of claims 1 to 3, characterized in that, The data units in the columnar data are associated with mask markers. The marker value of the mask marker is used to characterize whether the data unit is visible to the analysis query. The mask type of the mask marker is determined by the change type corresponding to the data unit.

5. The data processing method according to claim 1, characterized in that, The method further includes: After reading the composite log record, a transaction status identifier is created in the slot bucket structure for the target transaction, and the transaction status identifier is configured to the initial state. In response to the completion of the primary key index write, the transaction status flag is configured to the index commit status; In response to the completion of writing the row change content into the columnar storage data, the transaction status identifier is configured to the data commit status, the global visible sequence number queue of the target database is updated according to the transaction sequence number of the target transaction, and the mask mark of the columnar storage data is updated so that the data unit corresponding to the target transaction is visible to the analysis query. In response to the completion of updating the mask flag, the transaction status identifier is configured to a global commit state, and the slot resources corresponding to the transaction status identifier in the slot bucket structure are released; In response to the completion of releasing the slot resource, the transaction status flag is configured to a reclaimed state.

6. The data processing method according to claim 1, characterized in that, The method further includes: In response to the data reorganization task applied to the columnar data, the row records to be distributed to the primary key index processing process are cleared, and the data reorganization task is distributed to the primary key index processing process according to a preset batch granularity. In the primary key index processing process, according to the data reorganization task, the columnar storage data to be reorganized is modified to obtain the reorganized columnar storage data, and the reorganization mapping data corresponding to the data reorganization task is batch inserted into the primary key index data to obtain the reorganized primary key index data.

7. The data processing method according to claim 6, characterized in that, The method further includes: In response to the completion of a batch of remodeled mapping data insertion in the primary key index processing process, the entry delay of the transaction to be entered is obtained; In response to the entry delay being greater than a first threshold, the primary key index processing process is controlled to interrupt the execution of the data reorganization task.

8. The data processing method according to claim 1, characterized in that, The target transaction is the original transaction that enters the target database, or the target transaction includes multiple sub-transactions obtained by splitting the original transaction.

9. The data processing method according to claim 1, characterized in that, The method further includes: After writing the row change content into the columnar storage data, the undo log of the target transaction is recorded, wherein the undo log includes the change type, the change primary key, and the columnar storage location; In response to the rollback trigger event corresponding to the target transaction, a reverse change operation is performed on the primary key index data according to the reversal log to obtain the rolled-back primary key index data, and a reverse change operation is performed on the columnar storage data according to the reversal log to obtain the rolled-back columnar storage data.

10. The data processing method according to claim 1, characterized in that, The method further includes: Obtain the parsing latency of the composite log record; In response to the parsing delay being less than the second threshold, after reading the composite log record, the regular parsing thread of the consumer node is reused to parse and obtain at least one line record in the composite log record; In response to the parsing delay being greater than or equal to the second threshold, an independent parsing thread is started on the consumer node; Within a preset transaction process interval after the independent parsing thread is started, the parsing status of the regular parsing thread is monitored; In response to the parsing state remaining unchanged within the preset transaction process interval, skip reading the composite log record, use the independent parsing thread to parse at least one line record in the composite log record, and read the at least one line record.

11. A data processing method, characterized in that, include: In response to row storage data changes triggered by a target transaction in the target database, a composite log record is generated based on transaction operation information. The composite log record is used to characterize the physical redo information and change logic semantics of the row storage data change. The change logic semantics includes the change type. The composite log record is generated by the production node by associating the change type with the log position corresponding to the physical redo information. Store the composite log records to the target storage space; Send a synchronization trigger signal, wherein the synchronization trigger signal is used to notify the consumer node to read the composite log record in the target storage space, and to perform transaction replay based on the composite log record and the transaction timing information corresponding to the target transaction, so as to update the columnar storage data in the consumer node, wherein the columnar storage data is used to respond to the analysis query of the target database; The transaction sequence information includes a transaction sequence number, and the synchronization trigger signal is also used to notify the consumer node to perform the following steps: write a primary key index based on the transaction sequence number and the change primary key of at least one row record in the composite log record to obtain primary key index data, wherein the primary key index data is used to determine the column storage location corresponding to the row record; and write the row change content of the at least one row record into the column storage data according to the primary key index data.

12. The data processing method according to claim 11, characterized in that, Generating the composite log record based on the transaction operation information includes: Based on the transaction operation information, determine the physical redo information and the change type of at least one modified row, wherein the physical redo information includes the primary key and row change content corresponding to the modified row, and the change type includes at least addition change and deletion change; The change type is associated with the log position corresponding to the physical redo information to generate the composite log record, wherein the composite log record includes the row record corresponding to the change row.

13. The data processing method according to claim 11, characterized in that, The target storage space is a persistent storage space in the target database that is independent of the producer node and the consumer node. Storing the composite log records in the target storage space includes: The composite log records are written to the log partition of the target storage space in an append-only manner, wherein the writing order of the composite log records in the log partition corresponds to the commit order of the target transaction.

14. A task processing method, characterized in that, The method, applied to the consumer node of the target database, includes: Receive analysis and query tasks initiated against the target database; The analysis query task is processed using the target columnar storage data to obtain the analysis processing result. The target columnar storage data includes columnar storage data that is updated in real time in response to row storage data changes according to the data processing method of any one of claims 1 to 10. The row storage data changes are triggered by a transaction processing task initiated by the production node of the target database.

15. A task processing method, characterized in that, The method, applied to the production node of the target database, includes: Receive transaction processing tasks initiated against the target database; The transaction processing task is processed using the target row storage data to obtain the transaction processing result; In response to row-store data changes triggered during the processing of the transaction processing task, the column-store data in the consumer node of the target database is updated according to the data processing method of any one of claims 11 to 13, wherein the column-store data is used to respond to the analysis query of the target database.

16. A data processing system, characterized in that, include: The production node is configured to generate a composite log record based on the transaction operation information in response to row storage data changes triggered by the target transaction in the target database. The composite log record is used to characterize the physical redo information and change logic semantics of the row storage data change. The target storage space is configured to store the composite log records; The consumer node is configured to read the composite log record in the target storage space in response to the synchronization trigger signal issued by the producer node, and to perform transaction replay based on the composite log record and the transaction timing information corresponding to the target transaction, so as to update the columnar storage data in the consumer node. The columnar storage data is used to respond to the analysis query of the target database, and the transaction timing information includes the transaction sequence number. The consumer node is further configured to: write a primary key index based on the transaction sequence number and the change primary key of at least one row record in the composite log record to obtain primary key index data, wherein the primary key index data is used to determine the column storage location corresponding to the row record, and to write the row change content of the at least one row record into the column storage data according to the primary key index data.

17. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method according to any one of claims 1 to 15.

19. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images