A data writing method, device, apparatus and storage medium

Abstract

The application discloses a data writing method and device, equipment and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: obtaining CDC log data of a database table in a target database from a single partition of a message queue topic; dividing the CDC log data to obtain a plurality of divided log data and distributing the divided log data to a plurality of different partitions; performing table division processing on the divided log data in each different partition to obtain a plurality of log data tables; re-partitioning the log data in each log data table to obtain a plurality of divided data tables, locally reordering the log data in each divided data table, and concurrently writing the reordered log data to a distributed storage table. The application can improve the data writing efficiency, reduce the system load, and reduce the system operation cost, while ensuring the real-time performance, accuracy and reliability of data synchronization.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data writing method, apparatus, device and storage medium. Background Art

[0002] Currently, to ensure the real-time, accuracy, and reliability of data, incremental change data from databases such as MySQL (a relational database management system) / TdSQL (Tencent Distributed SQL, a distributed database) / Oracle (a relational database management system) is obtained in real time through CDC (Change Data Capture) logs. This incremental change data is then synchronized in a timely manner through big data storage to ensure data accuracy and avoid data inconsistencies caused by problems such as data duplication or data loss.

[0003] For example, in enterprise applications, multiple business systems and applications are often involved, and these business systems and applications often use different databases or even different data storage methods. Because different businesses need to process different data, they need to be stored and managed through different source tables. In addition, to meet the different performance requirements of enterprise applications, different data storage methods are also needed to store data. In addition, in enterprise applications, different data sources are often used to integrate different systems to achieve data sharing and interaction. In order to facilitate timely data analysis and decision-making, this data is also saved in real time to distributed storage systems (such as HDFS (Hadoop Distributed Filesystem) and Hudi (a general-purpose big data storage system)) to facilitate data analysis and data query, thereby improving the efficiency and accuracy of data analysis and decision-making.

[0004] However, enterprise databases typically contain numerous tables, and the CDC log data from these tables is written to a message queue topic. This consumes data from the consuming message queue and then concurrently writes it to numerous distributed storage tables. Currently, the mainstream approach for writing to distributed storage systems uses a single-threaded distributed computing engine. Due to the limitations of a single thread, this approach can lead to queued write requests, low write efficiency, and failure to meet the performance requirements for low-latency data synchronization to distributed storage systems. Furthermore, having multiple applications concurrently consume message queues to write to tables consumes significant computing resources, resulting in a surge in message queue consumption throughput and the duplication of large amounts of data, wasting computing resources and reducing efficiency. Furthermore, during data synchronization, if the data volume in one partition of a message queue topic is excessive while the data volume in other partitions is small, data skew within the message queue partition can be exacerbated, further increasing data synchronization latency and potentially failing to meet real-time synchronization requirements. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a data writing method, apparatus, device, and storage medium that can reduce the writing pressure on a single table, improve the efficiency of data writing, avoid potential performance bottlenecks and data loss, reduce system load, and lower system operating costs, while ensuring the real-time, accuracy, and reliability of data synchronization. The specific solution is as follows:

[0006] In a first aspect, the present application discloses a data writing method, comprising:

[0007] Get CDC log data for a database table in the target database from a single partition of a message queue topic;

[0008] Partitioning the CDC log data to obtain a plurality of partitioned log data, and allocating the partitioned log data to a plurality of different partitions;

[0009] Performing table processing on the divided log data in each of the different partitions to obtain multiple log data tables;

[0010] The log data in each of the log data tables are repartitioned to obtain a plurality of partitioned data tables, and the log data in each of the partitioned data tables are partially reordered to obtain sorted log data, which are then concurrently written into the distributed storage table.

[0011] Optionally, the divided log data in each of the different partitions is processed into separate tables to obtain multiple log data tables, including:

[0012] Cache the divided log data in each of the different partitions, and save the cache in a memory;

[0013] The divided log data in the memory is processed into tables to obtain multiple log data tables.

[0014] Optionally, the log data in each of the log data tables are repartitioned to obtain a plurality of partitioned data tables, including:

[0015] The log data in the log data table is repartitioned according to the primary key of the log data table to obtain a plurality of partitioned data tables.

[0016] Optionally, the locally reordering the log data in each of the divided data tables to obtain sorted log data includes:

[0017] The log data in the divided data table is partially reordered according to the primary key of the divided data table and the offset to be entered into the subject partition to obtain sorted log data.

[0018] Optionally, the concurrently writing the sorted log data into a distributed storage table includes:

[0019] Creating a distributed storage table writing task for each of the sorted log data, and performing a hash modulo calculation on each of the distributed storage table writing tasks according to a preset partition size to obtain a calculation result;

[0020] Using the calculation result to label each of the distributed storage table writing tasks, and grouping the distributed storage table writing tasks according to the label to obtain multiple groups of task partitions, and then sequentially pushing the multiple groups of task partitions into a pre-created distributed storage writing task concurrent queue; wherein the label is smaller than the preset partition size;

[0021] Submitting all the distributed storage table writing tasks in the distributed storage writing task concurrent queue to a pre-created thread pool in sequence to concurrently write the sorted log data into the distributed storage table; the thread pool runs in a distributed application main driver.

[0022] Optionally, the data writing method further includes:

[0023] The execution process of the distributed storage table writing task is monitored by a listener pre-configured for each distributed storage table writing task, and the execution result of the distributed storage table writing task is packaged and saved in a pre-created execution result storage queue.

[0024] Optionally, the data writing method further includes:

[0025] After all the distributed storage table writing tasks are submitted to the thread pool, the distributed application main driver is blocked by calling a counting lock, and the distributed application main driver is unblocked after the distributed storage table writing tasks are completed.

[0026] In a second aspect, the present application discloses a data writing device, comprising:

[0027] The log data acquisition module is used to obtain the CDC log data of the database table in the target database from a single partition of the message queue topic;

[0028] A data partitioning and allocation module is used to partition the CDC log data to obtain a plurality of partitioned log data, and allocate the partitioned log data to a plurality of different partitions;

[0029] A log data table partitioning module is used to perform table partitioning on the divided log data in each of the different partitions to obtain multiple log data tables;

[0030] A log data partitioning module is used to repartition the log data in each of the log data tables to obtain multiple partitioned data tables;

[0031] A local sorting module, configured to locally re-sort the log data in each of the divided data tables to obtain sorted log data;

[0032] The concurrent writing module is used to concurrently write the sorted log data into the distributed storage table.

[0033] In a third aspect, the present application discloses an electronic device comprising a processor and a memory; wherein the processor implements the aforementioned data writing method when executing a computer program stored in the memory.

[0034] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned data writing method is implemented.

[0035] It can be seen that the present application first obtains the CDC log data of the database table in the target database from a single partition of the message queue topic, and then divides the CDC log data to obtain multiple divided log data, and distributes the divided log data to multiple different partitions, and then performs table processing on the divided log data in each of the different partitions to obtain multiple log data tables, and then repartitions the log data in each of the log data tables to obtain multiple divided data tables, and locally reorders the log data in each of the divided data tables to obtain sorted log data, and finally writes the sorted log data concurrently to the distributed storage table. The present application performs secondary repartitioning and local sorting on the log data in a single partition, and flushes the CDC log data in a single partition into multiple distributed storage tables in a low-latency manner, which can reduce the writing pressure on a single table, improve the efficiency of data writing, avoid potential performance bottlenecks and data loss, reduce the load on the system, and reduce the operating cost of the system, while ensuring the real-time, accuracy and reliability of data synchronization, and effectively avoiding the phenomenon of single partition data skew. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0037] Figure 1 This is a flow chart of a data writing method disclosed in this application;

[0038] Figure 2 This is a specific schematic diagram of multiple data partitioning disclosed in this application;

[0039] Figure 3 A schematic diagram of a specific table writing process disclosed in this application;

[0040] Figure 4 This is a schematic diagram of the working principle of a specific counting lock disclosed in this application;

[0041] Figure 5 This is a flow chart of a specific data writing method disclosed in this application;

[0042] Figure 6 This is a structural diagram of a data writing device disclosed in this application;

[0043] Figure 7 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0045] The embodiments of the present application disclose a data writing method, as shown in Figure 1 The method comprises the following steps.

[0046] Step S11: Obtain CDC log data of a database table in a target database from a single partition of a message queue topic.

[0047] It can be understood that in some databases, such as enterprise databases, there are usually a large number of database tables, and these database tables are written into a topic of a message queue, which will cause the data in the message queue in the topic to be consumed and then written into a large number of distributed storage tables in a concurrent manner. However, in some specific scenarios, the data volume of a certain database table is very large, and the data volume of other database tables is relatively small, which causes the data volume in a certain partition of the message queue topic to be very large, and further causes a serious data skew phenomenon, which affects the subsequent writing speed of the database table with a relatively small data volume, causes a serious data synchronization delay, and even cannot meet the real-time synchronization requirement. In addition, when a parallel message queue consumption manner is used for writing table operation, a large amount of computing resources such as CPU (Central Processing Unit, central processor), memory, network bandwidth, etc. need to be consumed, and if these computing resources are not fully utilized, the computing efficiency will be reduced. In addition, due to the independent consumption mechanism of the message queue, the consumption throughput of the message queue may increase sharply and a large amount of data may be repeatedly consumed, thereby wasting computing resources and reducing the efficiency of the entire data writing. In order to solve the above problems, in the embodiments, when a large amount of data existing in a single partition needs to be written into a distributed storage table, CDC log (i.e., CDC log) data of a database table in a target database to be written is obtained from a single partition of a message queue topic.

[0048] Step S12: Divide the CDC log data to obtain a plurality of divided log data, and distribute the divided log data to a plurality of different partitions.

[0049] In this embodiment, after obtaining the CDC log data of the database table in the target database from a single partition of the message queue topic, in order to improve the degree of parallelism, solve the problem of low efficiency of single partition and single concurrency caused by the single thread scheduling of the distributed computing engine, and the inability to meet the performance requirements of low-latency synchronization of data to the distributed storage table, the above CDC log data can be further repartitioned. Figure 2 As shown, the CDC log data is distributed into n different partitions by shuffling. Specifically, the acquired CDC log data is first partitioned to obtain a plurality of corresponding partitioned log data, and then the partitioned log data is distributed into a plurality of different partitions.

[0050] Step S13: performing table partitioning processing on the divided log data in each of the different partitions to obtain multiple log data tables.

[0051] It should be noted that after the CDC log data is distributed to multiple different partitions, that is, after the CDC log data is scattered to multiple different partitions, the CDC log data in each different partition is out of order. If it is directly written to the distributed storage table, it will cause data confusion. In order to support multiple database tables to be written to the distributed storage table at the same time, see Figure 2 As shown, the divided log data in each partition is split into tables to obtain multiple corresponding log data tables. In other words, the CDC logs from different source database tables contained in the message queue partition are split into individual tables, and then the CDC logs from different source database tables are processed and correctly written to the designated distributed storage table.

[0052] Step S14: repartitioning the log data in each of the log data tables to obtain a plurality of partitioned data tables, and locally reordering the log data in each of the partitioned data tables to obtain sorted log data, and then concurrently writing the sorted log data into the distributed storage table.

[0053] In this embodiment, see Figure 2As shown, after the divided log data in each of the different partitions is divided into tables to obtain multiple log data tables, in order to further improve the parallelism of writing a single database table into the distributed storage table and avoid the data skew phenomenon caused by the large amount of data in a certain database table, the log data in each of the above-mentioned log data tables can be re-partitioned for a second time according to preset rules to obtain multiple divided data tables. It should be pointed out that the data in the divided data table after the second re-partitioning is still CDC log data, and the data is still out of order. In order to ensure the accuracy and consistency of the data written into the distributed storage table, as well as to ensure the orderliness of the data, the log data in each of the above-mentioned divided data tables can be partially re-sorted, and then the sorted log data can be written concurrently to multiple distributed storage tables.

[0054] In a specific embodiment, repartitioning the log data in each log data table to obtain a plurality of partitioned data tables may include repartitioning the log data in the log data table according to the primary key of the log data table to obtain the plurality of partitioned data tables. In other words, the log data in the log data table may be repartitioned according to the primary key of the log data table.

[0055] Specifically, the locally reordering the log data in each of the partitioned data tables to obtain the sorted log data may include: locally reordering the log data in the partitioned data table according to the primary key of the partitioned data table and the offset of the subject partition to be entered to obtain the sorted log data. In this embodiment, the log data in the partitioned data table may be locally reordered according to the primary key of the data table and the offset of the subject partition to be entered. It should be noted that the reason for locally reordering the partitioned data table instead of fully sorting is that full sorting degenerates into a single degree of parallelism when the amount of data increases sharply, thereby greatly affecting the efficiency of data synchronization.

[0056] In this embodiment, the concurrent writing of the sorted log data into the distributed storage table can specifically include: creating a distributed storage table write task for each of the sorted log data, and performing a hash modulus calculation on each of the distributed storage table write tasks according to a preset partition size to obtain a calculation result; using the calculation result to label each of the distributed storage table write tasks, and grouping the distributed storage table write tasks according to the label to obtain multiple groups of task partitions, and then pushing the multiple groups of task partitions into a pre-created distributed storage write task concurrent queue in sequence; wherein the label is smaller than the preset partition size; submitting all the distributed storage table write tasks in the distributed storage write task concurrent queue in sequence to a pre-created thread pool to concurrently write the sorted log data into the distributed storage table; the thread pool runs in a distributed application main driver. It should be noted that after performing the above-mentioned secondary repartitioning and local sorting on the CDC log data in a large number of database tables contained in a single partition of the message queue, the concurrency of the table write can be greatly improved. However, if the concurrent table write is not controlled, a large number of write tasks will be submitted to the main driver (Driver) at one time, resulting in all write tasks being very slow and unable to obtain sufficient resources to synchronize the data to the distributed storage table in a timely manner. In this embodiment, in order to solve the above-mentioned problem, a distributed storage table write task (including a deletion task) is created for each sorted log data, and then a hash remainder calculation is performed on the distributed storage table write task according to the preset partition size to obtain a calculation result, and then the calculation result is used to label each distributed storage table write task, and then the distributed storage table write task is grouped according to the above-mentioned label to obtain corresponding multiple groups of task partitions, and then the multiple groups of task partitions are sequentially pushed into the pre-created distributed storage write task concurrent queue. It should be noted that the distributed storage write task concurrent queue supports concurrent reading and writing, and serves as a container for the distributed storage table write tasks. The container stores a list of tasks one by one, and the label is smaller than the preset partition size. Furthermore, all distributed storage table write tasks in the above-mentioned distributed storage write task concurrent queue are submitted in sequence to the pre-created thread pool running in the distributed application main driver, and then each of the distributed storage table write tasks is executed in the above-mentioned thread pool, so that the above-mentioned sorted log data is concurrently written into the distributed storage table. It should be noted that since only the distributed storage table write task is submitted to the thread pool, the distributed storage table write task is executed in an independent thread to execute the scheduling of the main driver, which can support concurrent writing of multiple distributed storage tables instead of blocking the main driver from scheduling other tasks, thereby improving the submission speed of each distributed storage table write task. See Figure 3As shown in the figure, the specific table writing process can concurrently write each sorted log data into multiple distributed storage tables through the executor in the distributed computing engine.

[0057] Furthermore, the execution process of the distributed storage table writing task can be monitored by a listener pre-configured for each distributed storage table writing task, and the execution results of the distributed storage table writing task can be encapsulated and saved in a pre-created execution result storage queue. In this embodiment, a listener can be configured for each distributed storage table writing task. The listener can specifically monitor the following types of events: events before execution counting (i.e., counting table writing data), events after execution counting, events when there are no data rows to be written, events before table writing, events after table writing, exception actions thrown, task completion events, etc. By monitoring the above series of events, the execution process of each distributed storage table writing task can be managed and the execution progress of the current distributed storage table writing task can be understood. When an exception occurs, the exception event can be monitored through thread scheduling, and all distributed storage table writing tasks can be canceled. At the same time, regardless of whether the distributed storage table writing task is completed or an exception occurs during execution, the execution result will be encapsulated and placed in a pre-created execution result storage queue. In addition, in the main driver, if an exception is detected in the execution result, the execution result of the abnormal distributed storage table write task is directly encapsulated into the runtime exception execution result storage queue, and some partial messages and configuration information for troubleshooting are saved to trigger the abnormal stop. At the same time, it can also ensure that the checkpoint is valid and no data is lost.

[0058] In this embodiment, after all the distributed storage table writing tasks are submitted to the thread pool, it may also include: blocking the distributed application main driver by calling the counting lock, and unblocking the distributed application main driver after the distributed storage table writing task is completed; wherein the counting lock is used to control the execution of the distributed storage table writing task. It should be pointed out that after the thread pool receives all the distributed storage table writing tasks, the distributed application main driver can block the main thread by calling the waiting method of the counting lock, and after each distributed storage table writing task is completed, the counting lock will be reduced by one. When all tasks are completed, the counting lock is released and the main thread continues to execute. In this way, the synchronous control of the distributed storage table writing tasks is realized, ensuring that the data can be effectively synchronized to the distributed storage table to continue to perform subsequent data operations. For example, see Figure 4As shown, the execution process (thread0 to thread3) of the distributed storage table write task (writetask) in each partition (partition) is counted through a counting lock, and the corresponding execution results (JobResult) are counted. When an abnormal distributed storage table write task is detected, all current tasks are blocked, and the blockage is released when the abnormal task is completed.

[0059] In addition, for the distributed storage deletion table task, it should be pointed out that during the execution of all distributed storage table writing tasks, all deleted CDC log data will also be written into the distributed storage table, and a special deletion flag will be set for it; when all distributed storage table writing tasks are completed, the submission of the distributed storage deletion table task will be started. If the entire database table is directly scanned due to the deletion of a certain database table, a full table scan for some database tables with a large amount of data in the billions will cause a large delay in the deletion table task. Therefore, it is necessary to optimize the distributed storage deletion table task and narrow the scanning range of the deleted data. In order to solve the above problem, this application can first load the operation date of the CDC log data, and then obtain the distributed storage partition where the data needs to be deleted by taking the union method, and then perform a deletion scan on the distributed storage partition list within the small range, thereby improving the deletion efficiency of the distributed storage deletion table task.

[0060] Furthermore, to ensure the consistency and reliability of the data ultimately written to the distributed storage tables, effective data quality control and error handling can be performed. Furthermore, to enhance scalability and flexibility, dynamic adjustments can be made based on actual user needs and data size, further improving the efficiency and performance of data synchronization. Through resource scheduling, even in resource-constrained situations, it is possible to synchronize massive amounts of data from multiple tables to distributed storage tables with minimal system resources. This not only reduces system load and operating costs, but also ensures the accuracy and integrity of data synchronization.

[0061] It can be seen that the embodiment of the present application first obtains the CDC log data of the database table in the target database from a single partition of the message queue topic, and then divides the CDC log data to obtain multiple divided log data, and distributes the divided log data to multiple different partitions, and then performs table processing on the divided log data in each of the different partitions to obtain multiple log data tables, and then repartitions the log data in each of the log data tables to obtain multiple divided data tables, and locally reorders the log data in each of the divided data tables to obtain sorted log data, and finally writes the sorted log data concurrently to the distributed storage table. The embodiment of the present application reduces the writing pressure on a single table, improves the efficiency of data writing, avoids potential performance bottlenecks and data loss, reduces the load on the system, reduces the operating cost of the system, and at the same time ensures the real-time, accuracy and reliability of data synchronization, and effectively avoids the phenomenon of single partition data skew by performing secondary repartitioning and local sorting on the log data in a single partition, and flushes the CDC log data in a single partition into multiple distributed storage tables in a low-latency manner.

[0062] The present application discloses a specific method for writing data. Figure 5 As shown, the method includes:

[0063] Step S21: Obtain CDC log data of a database table in a target database from a single partition of a message queue topic.

[0064] Step S22: dividing the CDC log data to obtain a plurality of divided log data, and allocating the divided log data to a plurality of different partitions.

[0065] Step S23: Cache the divided log data in each of the different partitions, and save the cache in the memory.

[0066] In this embodiment, in order to improve the reading efficiency of the repartitioned data and avoid the data read from the message queue from being repeatedly repartitioned, the repartitioned data of the divided log data in each of the above-mentioned different partitions can be cached and stored in the memory, thereby facilitating iterative calculation.

[0067] Step S24: performing table division processing on the divided log data in the memory to obtain multiple log data tables.

[0068] In this embodiment, after the cache is saved in the memory, the divided log data in the memory is further processed into tables, thereby obtaining a corresponding plurality of log data tables.

[0069] Step S25: Repartition the log data in the log data table according to the primary key of the log data table to obtain multiple partitioned data tables, and locally reorder the log data in the partitioned data table according to the primary key of the partitioned data table and the offset to be entered into the subject partition to obtain sorted log data.

[0070] Step S26: creating a distributed storage table writing task for each of the sorted log data, and performing a hash modulo calculation on each of the distributed storage table writing tasks according to a preset partition size to obtain a calculation result.

[0071] Step S27: Use the calculation result to label each of the distributed storage table writing tasks, and group the distributed storage table writing tasks according to the label to obtain multiple groups of task partitions, and then push the multiple groups of task partitions into the pre-created distributed storage write task concurrent queue in sequence; wherein the label is smaller than the preset partition size.

[0072] Step S28: Submit all the distributed storage table writing tasks in the distributed storage writing task concurrent queue to a pre-created thread pool in sequence to concurrently write the sorted log data into the distributed storage table; the thread pool runs in the distributed application main driver.

[0073] Among them, for more specific processing procedures of the above steps S21, S22, S25 to S28, reference can be made to the corresponding contents disclosed in the above embodiments, which will not be repeated here.

[0074] It can be seen that the embodiment of the present application ensures both high concurrent read and write expansion of data and data consistency by performing secondary repartitioning and local sorting on the log data in a single partition. By flushing the CDC log data of a single partition into multiple distributed storage tables in a low-latency, batch manner, the writing pressure on a single table can be reduced, thereby improving the efficiency of data writing, facilitating data analysis and query, improving the efficiency and accuracy of data analysis, and avoiding potential performance bottlenecks and data losses. At the same time, the high reliability and security of data can be ensured by backing up multiple copies of data, and data recovery and backup operations can be efficiently implemented. In addition, this solution has effective system resource management capabilities and can perform concurrency control according to the system resource situation, such as controlling the usage of resources such as the first-level cache and the second-level cache, thereby ensuring efficient and stable data synchronization.

[0075] Correspondingly, the embodiment of the present application also discloses a data writing device, see Figure 6 As shown, the device includes:

[0076] The log data acquisition module 11 is used to obtain the CDC log data of the database table in the target database from a single partition of the message queue topic;

[0077] A data partitioning and allocation module 12 is configured to partition the CDC log data to obtain a plurality of partitioned log data, and allocate the partitioned log data to a plurality of different partitions;

[0078] The log data table partitioning module 13 is configured to perform table partitioning on the divided log data in each of the different partitions to obtain multiple log data tables;

[0079] The log data partitioning module 14 is used to repartition the log data in each of the log data tables to obtain multiple partitioned data tables;

[0080] A local sorting module 15 is used to locally re-sort the log data in each of the divided data tables to obtain sorted log data;

[0081] The concurrent writing module 16 is used to concurrently write the sorted log data into the distributed storage table.

[0082] Among them, the specific work processes of the above modules can refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0083] It can be seen that in the embodiment of the present application, the CDC log data of the database table in the target database is first obtained from a single partition of the message queue topic, and then the CDC log data is divided to obtain a plurality of divided log data, and the divided log data is distributed to a plurality of different partitions, and then the divided log data in each of the different partitions is subjected to table processing to obtain a plurality of log data tables, and then the log data in each of the log data tables is repartitioned to obtain a plurality of divided data tables, and the log data in each of the divided data tables is locally reordered to obtain sorted log data, and finally the sorted log data is concurrently written to the distributed storage table. The embodiment of the present application reduces the writing pressure on a single table, improves the efficiency of data writing, avoids potential performance bottlenecks and data loss, reduces the load on the system, reduces the operating cost of the system, and at the same time ensures the real-time, accuracy and reliability of data synchronization, and effectively avoids the single partition data skew phenomenon by performing secondary repartitioning and local sorting on the log data in a single partition and flushing the CDC log data in a single partition into multiple distributed storage tables in a low-latency manner.

[0084] In some specific embodiments, the log data table partitioning module 13 may specifically include:

[0085] a cache making unit, configured to make caches of the divided log data in each of the different partitions and save the caches into a memory;

[0086] The log data table division unit is used to perform table division processing on the divided log data in the memory to obtain multiple log data tables.

[0087] In some specific embodiments, the log data partitioning module 14 may specifically include:

[0088] The log data partitioning unit is used to repartition the log data in the log data table according to the primary key of the log data table to obtain multiple partitioned data tables.

[0089] In some specific embodiments, the local sorting module 15 may specifically include:

[0090] The local sorting unit is used to locally reorder the log data in the partitioned data table according to the primary key of the partitioned data table and the offset to be entered into the subject partition to obtain sorted log data.

[0091] In some specific embodiments, the concurrent writing module 16 may specifically include:

[0092] A table writing task creation unit, configured to create a distributed storage table writing task for each of the sorted log data;

[0093] A hash calculation unit, configured to perform a hash remainder calculation on each of the distributed storage table writing tasks according to a preset partition size to obtain a calculation result;

[0094] a labeling unit, configured to label each of the distributed storage table writing tasks using the calculation result;

[0095] a table writing task grouping unit, configured to group the distributed storage table writing tasks according to the labels to obtain a plurality of task partitions;

[0096] A task pushing unit, configured to sequentially push a plurality of task partitions into a pre-created distributed storage write task concurrent queue; wherein the number is smaller than the preset partition size;

[0097] A task submission unit is used to submit all the distributed storage table writing tasks in the distributed storage writing task concurrent queue to a pre-created thread pool in sequence, so as to concurrently write the sorted log data into the distributed storage table; the thread pool runs in the distributed application main driver.

[0098] In some specific embodiments, the data writing device may further include:

[0099] A monitoring unit, configured to monitor the execution process of the distributed storage table writing task through a monitor pre-configured for each distributed storage table writing task;

[0100] The encapsulation and storage unit is used to encapsulate the execution result of the distributed storage table writing task and save it in a pre-created execution result storage queue.

[0101] In some specific embodiments, the data writing device may further include:

[0102] A blocking unit, configured to block the distributed application main driver by calling a counting lock after all the distributed storage table writing tasks are submitted to the thread pool;

[0103] The unblocking unit is used to unblock the distributed application main driver after the distributed storage table writing task is completed.

[0104] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0105] Figure 7 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the data writing method disclosed in any of the aforementioned embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0106] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0107] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0108] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of implementing the data writing method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program capable of implementing other specific tasks.

[0109] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned data writing method is implemented. The specific steps of this method can be referred to the corresponding contents disclosed in the aforementioned embodiments and will not be repeated here.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0111] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0112] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0113] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0114] The above is a detailed introduction to a data writing method, device, equipment and storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A data writing method, characterized in that: include: Get CDC log data for a database table in the target database from a single partition of a message queue topic; Partitioning the CDC log data to obtain a plurality of partitioned log data, and allocating the partitioned log data to a plurality of different partitions; Performing table processing on the divided log data in each of the different partitions to obtain multiple log data tables; Repartitioning the log data in each of the log data tables to obtain a plurality of partitioned data tables, locally reordering the log data in each of the partitioned data tables to obtain sorted log data, and then concurrently writing the sorted log data into a distributed storage table; The concurrent writing of the sorted log data into the distributed storage table includes: creating a distributed storage table writing task for each of the sorted log data, and performing a hash modulus calculation on each of the distributed storage table writing tasks according to a preset partition size to obtain a calculation result; using the calculation result to label each of the distributed storage table writing tasks, and grouping the distributed storage table writing tasks according to the label to obtain multiple groups of task partitions, and then pushing the multiple groups of task partitions into a pre-created distributed storage writing task concurrent queue in sequence; wherein the label is smaller than the preset partition size; submitting all the distributed storage table writing tasks in the distributed storage writing task concurrent queue in sequence to a pre-created thread pool to concurrently write the sorted log data into the distributed storage table; the thread pool runs in a distributed application main driver.

2. The data writing method according to claim 1, wherein: The divided log data in each of the different partitions is processed into tables to obtain multiple log data tables, including: Cache the divided log data in each of the different partitions, and save the cache in a memory; The divided log data in the memory is processed into tables to obtain multiple log data tables.

3. The data writing method according to claim 1, wherein: The log data in each log data table is repartitioned to obtain a plurality of partitioned data tables, including: The log data in the log data table is repartitioned according to the primary key of the log data table to obtain a plurality of partitioned data tables.

4. The data writing method according to claim 1, wherein: The locally reordering the log data in each of the divided data tables to obtain sorted log data includes: The log data in the divided data table is partially reordered according to the primary key of the divided data table and the offset to be entered into the subject partition to obtain sorted log data.

5. The data writing method according to claim 1, wherein: Also includes: The execution process of the distributed storage table writing task is monitored by a listener pre-configured for each distributed storage table writing task, and the execution result of the distributed storage table writing task is packaged and saved in a pre-created execution result storage queue.

6. The data writing method according to claim 4, wherein: Also includes: After all the distributed storage table writing tasks are submitted to the thread pool, the distributed application main driver is blocked by calling a counting lock, and the distributed application main driver is unblocked after the distributed storage table writing tasks are completed.

7. A data writing device, characterized in that: include: The log data acquisition module is used to obtain the CDC log data of the database table in the target database from a single partition of the message queue topic; A data partitioning and allocation module is used to partition the CDC log data to obtain a plurality of partitioned log data, and allocate the partitioned log data to a plurality of different partitions; A log data table partitioning module is used to perform table partitioning on the divided log data in each of the different partitions to obtain multiple log data tables; A log data partitioning module is used to repartition the log data in each of the log data tables to obtain multiple partitioned data tables; A local sorting module, configured to locally re-sort the log data in each of the divided data tables to obtain sorted log data; A concurrent writing module is used to concurrently write the sorted log data into a distributed storage table; The concurrent writing module is specifically used to create a distributed storage table writing task for each of the sorted log data, and perform hash remainder calculation on each of the distributed storage table writing tasks according to a preset partition size to obtain a calculation result; use the calculation result to label each of the distributed storage table writing tasks, and group the distributed storage table writing tasks according to the label to obtain multiple groups of task partitions, and then push the multiple groups of task partitions into a pre-created distributed storage writing task concurrent queue in sequence; wherein the label is smaller than the preset partition size; all the distributed storage table writing tasks in the distributed storage writing task concurrent queue are submitted in sequence to a pre-created thread pool to concurrently write the sorted log data into the distributed storage table; The thread pool runs in the distributed application main driver.

8. An electronic device, characterized in that: The method comprises a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the data writing method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that Used to store a computer program; wherein, when the computer program is executed by a processor, the data writing method according to any one of claims 1 to 6 is implemented.

Images