Data processing method of row-column hybrid storage database and related product

Abstract

The application provides a data processing method of a row-column mixed storage database and related products. The data processing method comprises the following steps: obtaining an instruction for performing storage mode conversion on an initial partition, wherein the initial partition adopts a row storage mode; constructing a target partition, wherein the target partition adopts a column storage mode; performing columnar reorganization on data of the initial partition and migrating the data to the target partition, wherein the columnar reorganization is used for converting data based on the row storage mode into data based on the column storage mode; and atomically updating a metadata pointer of the initial partition to direct a corresponding query operation to the target partition. The data processing method overall realizes effective compression of storage space, acceleration of analysis query, and comprehensive optimization of system resource utilization, and is particularly suitable for mixed transaction and analysis processing scenarios.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a data processing method for a row-column mixed storage database, a computer-readable storage medium, a computer program product, and a computer device. Background Technology

[0002] Partitioned tables are a technique for dividing a large table into multiple smaller, more manageable partitions. By rationally partitioning data, partitioned tables can improve query efficiency, maintenance performance, and the speed of data loading and exporting. The core of partitioned table row-column mixed storage functionality is that the system supports dividing the main table into multiple sub-partitions and configuring different storage modes (row storage mode or column storage mode) for each sub-partition. In column-stored databases, data is stored column-wise, rather than row-wise as in traditional row storage. This storage method has significant advantages in querying specific columns and data compression because it can efficiently read and process data sets of specified columns. A batch in a column-stored database contains data from multiple rows.

[0003] In existing technologies, database systems face a trade-off between storage efficiency and computational performance when handling large-scale data queries. Traditional row-based storage offers high efficiency for data write and update operations but low space utilization, while column-based storage is more suitable for aggregation queries but has lower efficiency for data write and update operations. Summary of the Invention

[0004] One object of the present invention is to provide a data processing method, computer-readable storage medium, computer program product and computer device for a mixed row and column database, which realizes dynamic adjustment of the storage mode of the database partition to improve operational efficiency and reduce resource waste.

[0005] Specifically, according to one aspect of the present invention, the present invention provides a data processing method for a row-column mixed storage database, comprising: Obtain instructions to perform storage mode conversion on an initial partition, wherein the initial partition is in row storage mode; Construct the target partition, which adopts a columnar storage mode; The data in the initial partition is reorganized into columns and migrated to the target partition. The reorganization is used to transform the data based on the row storage mode into data based on the column storage mode. The metadata pointer of the initial partition is atomically updated to direct the corresponding query operation to the target partition.

[0006] Optionally, obtaining the instruction to perform storage mode conversion on the initial partition includes: Obtain the transaction operation records of the initial partition; Based on the transaction operation record, it is determined whether the data in the initial partition is in a stable state. The stable state indicates that the probability of the data being modified within a preset time is lower than a preset value. When the data in the initial partition is in the stable state, an instruction is generated to perform a storage mode conversion on the initial partition.

[0007] Optionally, determining whether the data in the initial partition is in a stable state based on the transaction operation record includes: Based on the transaction operation record, the operation parameters of the initial partition within a preset period are obtained. The operation parameters include at least one of data access frequency, number of writes, update timestamp, and query type. Based on the aforementioned operating parameters and the corresponding preset operating parameter thresholds, it is determined whether the data in the initial partition is in the stable state.

[0008] Optionally, the columnar reorganization of the data in the initial partition includes: Convert the data in the initial partition from the row storage mode to the column storage mode; The transformed data is compressed using a compression algorithm based on the column storage pattern.

[0009] Optionally, before the step of converting the data of the initial partition from the row storage mode to the column storage mode, the method further includes: A conversion failure detection program is initiated, which is configured to monitor the columnar reorganization process to trigger a system rollback mechanism in the event of an anomaly. The system rollback mechanism is used to restore the data during the conversion process to the data using the row storage mode. Following the step of atomically updating the metadata pointer of the initial partition, the method further includes: Stop the conversion failure detection program.

[0010] Optionally, before the step of columnar reorganizing the data in the initial partition and migrating it to the target partition, the method further includes: Holds a write lock on the initial partition; Following the step of atomically updating the metadata pointer of the initial partition to direct the corresponding query operation to the target partition, the method further includes: Delete the physical files in the initial partition; Release the write lock.

[0011] Optionally, obtaining the instruction to perform storage mode conversion on the initial partition includes: Obtain the user-input command for performing storage mode conversion on the initial partition.

[0012] According to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the data processing method for the row-column mixed storage database described above are implemented.

[0013] According to another aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the steps of the data processing method for any of the above-described row-column mixed-database methods.

[0014] According to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the data processing method for the row-column mixed storage database described above.

[0015] The data processing method for a hybrid row-column storage database of the present invention improves the efficiency of real-time data writing and updating operations by constructing the initial partition using a row-based storage mode. By setting a storage mode conversion trigger mechanism, the storage mode of the initial partition is converted from row-based to column-based storage mode, which reduces storage space usage and improves data analysis and query efficiency. This data processing method achieves effective compression of storage space, acceleration of analysis and querying, and comprehensive optimization of system resource utilization, and is particularly suitable for scenarios involving mixed transaction and analysis processing.

[0016] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0017] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings: Figure 1 This is a flowchart illustrating a data processing method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the process of obtaining storage mode conversion instructions in a data processing method according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating the process of determining whether the initial partition data is in a stable state according to a data processing method of an embodiment of the present invention. Figure 4 This is a schematic diagram of the data processing method for performing storage mode conversion according to an embodiment of the present invention; Figure 5This is a schematic diagram of the data processing method for performing storage mode conversion according to another embodiment of the present invention; Figure 6 This is a flowchart illustrating a data processing method according to another embodiment of the present invention; Figure 7 This is a schematic diagram of a computer program product according to an embodiment of the present invention; Figure 8 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention; Figure 9 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0018] Row-column hybrid databases are typically used in scenarios that combine transactional and analytical operations. Row-based storage supports high-concurrency read and write operations, making it suitable for Online Transaction Processing (OLTP) scenarios where real-time transactions are performed. Column-based storage supports optimized query performance and is suitable for complex aggregation calculations and large-scale data scanning, i.e., Online Analytical Processing (OLAP) scenarios.

[0019] Generally, recent data may require frequent write, update, or delete operations, making row-based storage suitable; while historical data needs to support efficient analysis and querying, thus column-based storage is more suitable. However, in related technologies, the storage mode is determined when data is written and does not dynamically adjust the storage mode throughout the data's lifecycle, resulting in low overall database space utilization and poor overall operational efficiency.

[0020] The purpose of the data processing method for the row-column mixed storage database in this embodiment is to dynamically adjust the storage mode of the database partition in order to improve operational efficiency and reduce resource waste.

[0021] Figure 1 This is a flowchart illustrating a data processing method for a row-column mixed-database according to an embodiment of the present invention. The method generally includes: S100, obtain the instruction to perform storage mode conversion on the initial partition, the initial partition adopts row storage mode; S200, construct the target partition, which uses columnar storage mode; S300 performs columnar reorganization on the data in the initial partition and migrates it to the target partition. Columnar reorganization is used to transform data based on row storage mode into data based on column storage mode. S400 atomically updates the metadata pointer of the initial partition to direct the corresponding query operation to the target partition.

[0022] In this embodiment, the partitions of the row-column hybrid storage database are constructed using row storage mode by default. Although row storage mode occupies a large amount of storage space, it can efficiently perform real-time write and update operations, thereby improving transaction processing efficiency.

[0023] In this embodiment, the database management system can set a storage mode conversion trigger mechanism, which may include manual triggering and automatic triggering.

[0024] The manual triggering mechanism provides dedicated management commands, allowing users to specify any initial partition for storage mode conversion. For example, users can determine whether the data in each initial partition is "cold data" in a stable state based on actual data usage scenarios and experience. A stable state means that the probability of writing, modifying, or performing other operations on the initial partition data in the future is low. Users can manually enter a command to perform storage mode conversion on a specified initial partition; upon receiving the command, the database management system generates storage mode conversion instructions.

[0025] An automatic triggering mechanism can use a program to monitor the data lifecycle of each initial partition, analyze the data access characteristics of each initial partition, and thus determine the lifecycle stage of the initial partition data. For example, if an initial partition's data is frequently being written to, modified, or deleted, it indicates that the probability of such operations occurring in the near future is high, classifying it as "hot data" in an unstable state. In this case, row-based storage mode can be maintained to improve the efficiency of write, modify, and delete operations. If an initial partition's data has not been written to, modified, or deleted for a long time, it indicates that the initial partition's data may be entering a stable state. In this case, a storage mode conversion instruction can be triggered, changing the storage mode of the initial partition from row-based to column-based storage mode. This reduces storage space usage and improves data analysis and query efficiency.

[0026] Specifically, in the automatic triggering mechanism mode, the database management system can use a background scheduled task to periodically scan each initial partition. If an initial partition has not been modified for more than a preset threshold, a storage mode conversion instruction will be automatically triggered. The database management system can also obtain transaction operation records of each initial partition in real time or periodically, and determine whether the data in each initial partition has entered a stable state by analyzing the transaction operation records.

[0027] In this embodiment, after receiving the instruction to convert the storage mode of the initial partition, the system automatically initiates the storage mode conversion process. Specifically, a target partition can be created first, which can be a temporary columnar storage mode partition. Next, the data in the initial partition is reorganized into a columnar storage format, and then migrated to the target partition.

[0028] After data migration is complete, the metadata pointers of the initial partition can be atomically updated, directing queries to the new columnar storage partition, i.e., the target partition. By atomically updating the metadata pointers of the initial partition, the data transformation process becomes completely transparent to upper-layer applications, requiring no additional adjustments to existing business logic and achieving seamless integration. Furthermore, it ensures data atomicity and consistency, guaranteeing system availability.

[0029] This embodiment of the data processing method for a hybrid row-column storage database improves the efficiency of real-time data write and update operations by constructing the initial partition using a row-based storage mode. By setting a storage mode conversion trigger mechanism, the initial partition's storage mode is converted from row-based to column-based, reducing storage space usage and improving data analysis and query efficiency. Overall, this data processing method achieves effective storage space compression, accelerated analysis and querying, and comprehensive optimization of system resource utilization, making it particularly suitable for hybrid transaction / analytical processing (HTAP) scenarios.

[0030] In some embodiments of the data processing method of the present invention, such as Figure 2 As shown, the instructions for obtaining storage mode conversion for the initial partition include: S111, retrieve the transaction operation records of the initial partition; S113, Based on the transaction operation record, determine whether the data in the initial partition is in a stable state. A stable state indicates that the probability of the data being modified within a preset time is lower than a preset value. S115, if the data in the initial partition is in a stable state, generates an instruction to perform a storage mode conversion on the initial partition.

[0031] In this embodiment, the database management system can periodically obtain the transaction operation records of the initial partition. These transaction operation records document various access records of the initial partition over multiple periods, including query records, write records, update records, and delete records. By analyzing the transaction operation records of the initial partition over multiple periods, the probability of the data in each initial partition being modified in one or more future periods can be estimated relatively accurately. If the probability of modification is high, it indicates that the data in that initial partition is in an unstable state. If the probability of modification is low, it indicates that the data in that initial partition is in a stable state.

[0032] The data lifecycle monitoring operation in this embodiment is relatively simple and reliable, which can significantly reduce the amount of manual maintenance, reduce the complexity of operation and the risk of error, and can promptly detect the initial partitions that have entered the stable state stage and promptly trigger the storage mode conversion instruction.

[0033] In some embodiments of the data processing method of the present invention, such as Figure 3 As shown, based on the transaction operation records, it is determined whether the data in the initial partition is in a stable state, including: S131, based on transaction operation records, obtain the operation parameters of the initial partition within a preset period. The operation parameters include at least one of the following: data access frequency, number of writes, update timestamp, and query type. S133, based on each operation parameter and the corresponding preset operation parameter threshold, determine whether the data in the initial partition is in a stable state.

[0034] In this embodiment, the database management system can periodically collect and statistically analyze operation parameters such as data access frequency, write count, update timestamp, and query type of each initial partition to form a data usage feature profile, which is used to determine the current status of each initial partition.

[0035] For example, the database management system can periodically collect write and update operation timestamps and set corresponding preset operation parameter thresholds. When the interval between the last write and update operation timestamp of an initial partition and the current time exceeds a preset time interval, the system can automatically determine that the data in the initial partition is in a stable state and trigger a storage mode conversion.

[0036] The following explanation uses an e-commerce order form as an example: First, the system creates an order table partitioned by month. The new partition uses row storage mode by default for real-time order writing.

[0037] Next, for the current month partition, since orders will be updated frequently every day, we will maintain row storage mode to support fast order insertion and modification.

[0038] For each partition from previous months, since no more data is written after the end of the current month, the age of the partition data can be monitored starting from the end of the current month.

[0039] For example, a preset age threshold can be set to 30 days. On the 31st day, the system automatically detects that the previous month's partition meets the conditions, and then triggers the storage mode conversion mechanism to start the row-to-column conversion and complete the column reorganization.

[0040] After the conversion, the monthly partition is converted to columnar storage mode, which significantly reduces storage space and makes it available for subsequent reports and analysis.

[0041] In some embodiments of the data processing method of the present invention, obtaining an instruction to perform storage mode conversion on an initial partition includes: Retrieves user input of commands for converting the storage mode of the initial partition.

[0042] In this embodiment, users can manually convert specific initial partitions at any time via commands to meet temporary analysis needs or for customized maintenance in special business scenarios, balancing flexibility and controllability.

[0043] In some embodiments of the data processing method of the present invention, such as Figure 4 As shown, the data in the initial partition is reorganized into columns, including: S311, converts the data in the initial partition from row storage mode to column storage mode; S313 uses a column-based compression algorithm to compress the transformed data.

[0044] In this embodiment, compression encoding is applied simultaneously during the process of reorganizing the original row-stored data into column-stored format, which can further reduce the storage space occupied by partitioned data.

[0045] In column-oriented storage, data is stored column-wise in a table, with similar data blocks stored together. Therefore, compared to row-oriented storage, column-oriented storage typically has a higher compression ratio, which can significantly reduce storage space usage.

[0046] Compression algorithms for column storage mode can include Huffman coding, arithmetic coding, LZ77 algorithm, LZW algorithm, run-length encoding (RLE algorithm), null value compression algorithm, etc., and there are no restrictions here.

[0047] In some embodiments of the data processing method of the present invention, such as Figure 5 As shown, before the step of converting the initial partition's data from row storage mode to column storage mode, the following steps are also included: S511, Start the conversion failure detection program. The conversion failure detection program is configured to monitor the columnar reorganization process to trigger the system rollback mechanism when an anomaly occurs. The system rollback mechanism is used to restore the data in the conversion process to data using row storage mode. Following the step of atomically updating the metadata pointer of the initial partition, the following steps are also included: S515, Stop the conversion failure detection program.

[0048] The conversion failure detection program monitors the columnar reorganization process. If an anomaly is detected during the conversion, the system will automatically trigger a rollback mechanism to restore the original row storage mode, ensuring data consistency and system reliability.

[0049] In some embodiments of the data processing method of the present invention, such as Figure 5 As shown, before the steps of columnar reorganizing the data in the initial partition and migrating it to the target partition, the following steps are also included: S513 holds the write lock for the initial partition; After the step of atomically updating the metadata pointer of the initial partition to direct the corresponding query operation to the target partition, the following steps are also included: S517, delete the physical files of the initial partition; S519, release write lock.

[0050] In this embodiment, when a storage mode conversion is triggered, the system locks the initial partition to prevent concurrent writes, ensuring data consistency during the storage mode conversion. After the storage mode conversion is completed, an atomic metadata switching operation replaces the original row storage partition with the column storage partition, thereby completing the storage mode update without affecting upper-layer business access.

[0051] In some embodiments of the data processing method of the present invention, such as Figure 6 As shown, it includes: S611 initializes each partition using row storage mode; S613 monitors the last update time of each partition; S615, all partitions are working normally; S617: Periodically check whether the last update interval of the partition exceeds the preset threshold; if yes, execute S619; otherwise, return to execute S615. S619 atomically modifies the corresponding partition to columnar storage mode.

[0052] The flowchart provided in this embodiment is not intended to indicate that the operations of the method will be performed in any particular order, or that all operations of the method are included in every case. Furthermore, the method may include additional operations. Within the scope of the technical concept provided by the method in this embodiment, additional variations can be made to the above method.

[0053] It should be understood that in some embodiments, the components may be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods may be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0054] This invention also provides a computer program product 10, a computer-readable storage medium 20, and a computer device 30. Figure 7 This is a schematic diagram of a computer program product 10 according to an embodiment of the present invention. Figure 8 This is a schematic diagram of a computer-readable storage medium 20 according to an embodiment of the present invention. Figure 9 This is a schematic diagram of a computer device 30 according to an embodiment of the present invention. The computer program product 10 includes a computer program 11, which, when executed by the processor 32, implements the steps of any of the data processing methods described above. A computer-readable storage medium 20 stores the computer program 11 thereon, which, when executed by the processor 32, implements the steps of the data processing method of any of the embodiments described above. The computer device 30 may include a memory 31, a processor 32, and the computer program 11 stored in the memory 31 and running on the processor 32.

[0055] The computer program 11 used to perform the operations of this invention may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​and procedural programming languages. The computer program 11 may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a Local Area Network (LAN) or Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, Field-Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information from computer-readable program instructions.

[0056] For the purposes of this embodiment, computer program product 10 is a related product that includes computer program 11.

[0057] For the purposes of this embodiment, the computer-readable storage medium 20 is a tangible device capable of holding and storing a computer program 11. It can be any device capable of containing, storing, communicating, propagating, or transmitting the computer program 11 for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable storage medium 20 include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanical encoding device, and any suitable combination thereof.

[0058] Computer device 30 can be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer device 30 can be a cloud computing node. Computer device 30 can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer device 30 can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0059] Computer device 30 may include a processor 32 adapted to execute stored instructions and a memory 31 that provides temporary storage space for the operation of said instructions during operation. The processor 32 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. The memory 31 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0060] Computer device 30 may also include a network adapter / interface and an input / output (I / O) interface. The I / O interface allows external devices that can be connected to the computer device to input and output data. The network adapter / interface provides communication between the computer device and a network, typically represented as a communication network.

[0061] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A data processing method for a mixed row-column database, characterized in that, include: Obtain instructions to perform storage mode conversion on an initial partition, wherein the initial partition is in row storage mode; Construct the target partition, which adopts a columnar storage mode; The data in the initial partition is reorganized into columns and migrated to the target partition. The reorganization is used to transform the data based on the row storage mode into data based on the column storage mode. The metadata pointer of the initial partition is atomically updated to direct the corresponding query operation to the target partition.

2. The data processing method according to claim 1, characterized in that, The instruction to obtain the storage mode conversion of the initial partition includes: Obtain the transaction operation records of the initial partition; Based on the transaction operation record, it is determined whether the data in the initial partition is in a stable state. The stable state indicates that the probability of the data being modified within a preset time is lower than a preset value. When the data in the initial partition is in the stable state, an instruction is generated to perform a storage mode conversion on the initial partition.

3. The data processing method according to claim 2, characterized in that, The step of determining whether the data in the initial partition is in a stable state based on the transaction operation record includes: Based on the transaction operation record, the operation parameters of the initial partition within a preset period are obtained. The operation parameters include at least one of data access frequency, number of writes, update timestamp, and query type. Based on the aforementioned operating parameters and the corresponding preset operating parameter thresholds, it is determined whether the data in the initial partition is in the stable state.

4. The data processing method according to claim 1, characterized in that, The columnar reorganization of the data in the initial partition includes: Convert the data in the initial partition from the row storage mode to the column storage mode; The transformed data is compressed using a compression algorithm based on the column storage pattern.

5. The data processing method according to claim 4, characterized in that, Before the step of converting the data of the initial partition from the row storage mode to the column storage mode, the method further includes: A conversion failure detection program is initiated, which is configured to monitor the columnar reorganization process to trigger a system rollback mechanism in the event of an anomaly. The system rollback mechanism is used to restore the data during the conversion process to the data using the row storage mode. Following the step of atomically updating the metadata pointer of the initial partition, the method further includes: Stop the conversion failure detection program.

6. The data processing method according to claim 1, characterized in that, Before the step of columnar reorganizing the data in the initial partition and migrating it to the target partition, the method further includes: Holds a write lock on the initial partition; Following the step of atomically updating the metadata pointer of the initial partition to direct the corresponding query operation to the target partition, the method further includes: Delete the physical files in the initial partition; Release the write lock.

7. The data processing method according to claim 1, characterized in that, The instruction to obtain the storage mode conversion of the initial partition includes: Obtain the user-input command for performing storage mode conversion on the initial partition.

8. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the steps of the data processing method for a row-column mixed storage database as described in any one of claims 1 to 7.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method for a row-column mixed database as described in any one of claims 1 to 7.

10. A computer device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the data processing method for a row-column mixed-database as described in any one of claims 1 to 7.

Images