A kudu database reading method and device based on DataX

Abstract

The application discloses a Kudu database reading method based on DataX, comprising the following steps: constructing a custom KuduReader plug-in in a DataX framework and connecting an Impala query engine; obtaining Kudu target table metadata; logically slicing to-be-read data according to a preset slicing strategy; generating a plurality of parallel reading tasks and querying corresponding sliced data through the Impala concurrent query; writing the query result into a DataX channel after processing the query result; and recording task states, slicing boundaries and offset information in the reading process to realize fault recovery and incremental resuming. The application fills the blank of the native support of DataX to the Kudu data source, utilizes the Impala as a unified query engine to consider the SQL compatibility and permission control, realizes load balancing through adaptive slicing, supports breakpoint resuming and failure retry, and significantly improves the efficiency, stability and universality of Kudu data synchronization.

Description

Technical Field

[0001] This application relates to the field of big data integration and database access technology, specifically to a Kudu database reading method, apparatus, computer-readable storage medium, and electronic device based on DataX. Background Technology

[0002] Kudu is a columnar distributed storage engine designed for analytical scenarios, suitable for high-throughput, low-latency random access, and real-time analytics. Kudu is often used in conjunction with computing engines such as Impala and Spark. Impala, in particular, supports direct SQL queries on Kudu tables, providing strong interactive analytical capabilities.

[0003] DataX is a widely used offline data synchronization tool that enables data exchange between different heterogeneous data sources through a Reader and Writer plugin mechanism. While DataX natively supports various data sources such as relational databases, HDFS, Hive, and ODPS, it does not natively provide a Reader plugin for the Kudu data source, making it difficult to directly integrate data from Kudu into the existing DataX synchronization system.

[0004] Currently, the main methods used for reading Kudu data are as follows:

[0005] Method 1: Indirect export via computing engines such as Spark and Flink. This involves connecting to Kudu via Spark / Flink, writing the data to HDFS files, Hive tables, or other intermediate storage, and then reading it from DataX. The disadvantages of this method include: long pipeline, system complexity; increased overhead for intermediate storage and transformation; poor real-time performance; and high task deployment and maintenance costs.

[0006] Method 2: Accessing Kudu tables via Impala using JDBC. This involves using the Impala JDBC driver to query the data tables mapped to Kudu, and then exporting the data. The disadvantages of this method include: traditional JDBC reads are typically single-channel and low-concurrency; full extraction of large tables is inefficient; it cannot well adapt to DataX's parallel partitioning model; and it lacks breakpoint recovery and task sharding scheduling capabilities.

[0007] Method 3: Develop a simple Kudu client program yourself. This involves directly scanning Tablet data using the Kudu Java Client. The disadvantages of this method include: complex development, requiring handling of Kudu's native API compatibility issues; insufficient support for table structure changes, type mapping, predicate pushdown, and sharded reads; difficulty in seamless integration with the existing DataX ecosystem; and the need to additionally address security authentication, concurrency scheduling, and fault recovery issues.

[0008] In summary, the existing technologies have the following problems: (1) DataX does not natively support Kudu reading and cannot be directly used for data synchronization from Kudu to the target system; (2) Direct reading via Impala JDBC lacks high parallelism and is insufficient in performance in large-scale data synchronization scenarios; (3) Implementation via the native Kudu client is costly and incompatible with the SQL ecosystem; (4) There is a lack of data partitioning and task distribution mechanisms suitable for Kudu scenarios, which cannot fully utilize the concurrency capabilities of DataX; (5) There is a lack of adaptive adaptation capabilities for Kudu table structures and data types, which can easily lead to problems such as field incompatibility and synchronization failure; (6) There is a lack of unified fault tolerance, resume and consistency control mechanisms, which cannot meet the stable synchronization requirements of the production environment.

[0009] Therefore, there is an urgent need to develop a technical solution that is compatible with the DataX framework, can efficiently query Kudu data using Impala, and has the capabilities of parallel reading, type mapping, and fault tolerance and recovery. Summary of the Invention

[0010] To overcome the aforementioned deficiencies in the existing technology, this application provides a novel Kudu database reading method and apparatus based on DataX. More specifically, this invention achieves efficient, scalable, and fault-tolerant reading of Kudu data by adding a custom Kudu reading plugin to DataX and using Impala as the query engine to access Kudu stored data. This solves the problems of DataX's inability to directly read Kudu, low reading efficiency, poor scalability, and insufficient stability in the existing technology.

[0011] To achieve the above objectives, the present invention employs the following technical strategies:

[0012] (1) Data reading plugin mechanism for Kudu in DataX

[0013] DataX currently does not natively support Kudu reading. This invention is the first to build a data reading plugin for Kudu, enabling Kudu to be used as a standard data source to access the DataX ecosystem, thus achieving framework-level extension.

[0014] (2) Unified access architecture based on Impala instead of Kudu native scanning

[0015] This invention constructs a novel "query engine proxy reading architecture" that does not directly rely on the underlying Kudu scanning interface, but instead uses Impala as a unified query engine to access Kudu data, thereby achieving the following: SQL compatibility, unified access control, simplified metadata acquisition, and reduced maintenance costs.

[0016] (3) Adaptive segmentation algorithm based on Kudu data distribution characteristics

[0017] This invention, based on primary key range, time interval, hash rules, and statistical information, enables dynamic adjustment of sharding granularity and secondary splitting of hotspot shards, solving the problems of ineffective parallelism and uneven load distribution in traditional JDBC read methods.

[0018] (4) Co-management mechanism for reading status and incremental sites

[0019] This invention integrates fragmentation status, boundary information, and incremental positions into status management. After a task fails, only the failed fragments are recovered, and it supports full / incremental integrated synchronization, significantly improving stability and practicality.

[0020] (5) Type adaptive mapping mechanism for heterogeneous synchronization scenarios

[0021] This invention improves the versatility of the solution in complex field types, null value scenarios, and pattern change scenarios by establishing an adaptive mapping from Impala / Kudu types to the DataX internal Record model and a degradation and fault tolerance strategy.

[0022] Specifically, this application provides the following technical solutions:

[0023] The first aspect of this application provides a Kudu database reading method based on DataX, such as... Figure 1 As shown, the method includes the following steps:

[0024] S1: Build a custom KuduReader plugin in the DataX framework and connect to the Impala query engine through the KuduReader plugin;

[0025] S2: Obtain the metadata information of the Kudu target table using the Impala query engine;

[0026] S3: Based on the metadata information and the preset segmentation strategy, logically segment the data to be read to generate multiple data segments;

[0027] S4: For each data shard, multiple parallel read tasks are generated, and the DataX framework schedules each read task to concurrently query the corresponding shard data through the Impala query engine;

[0028] S5: Parse the fields, convert the data types, and encapsulate the standard Records of the query results returned by the Impala query engine, and write the encapsulated data into the DataX channel for downstream Writer plugins to write out;

[0029] S6: Record task status, split boundaries and offset information during the reading process to achieve fault tolerance recovery and incremental resume.

[0030] Furthermore, in the method of this application, in step S1, the KuduReader plugin establishes a communication connection with the Impala query engine through a JDBC or ImpalaThrift interface, and completes authentication, session initialization and parameter verification.

[0031] Furthermore, in the method of this application, in step S2, the metadata information includes at least one of table structure information, field information, primary key information, partition information, statistical information, and field data distribution information.

[0032] Furthermore, in the method of this application, in step S3, the preset sharding strategy includes at least one of the following: sharding based on primary key range, sharding based on time field range, sharding based on hash modulo, and adaptive sharding based on Impala statistical information.

[0033] Furthermore, in the method of this application, the adaptive sharding based on Impala statistical information includes: pre-obtaining the number of rows, primary key distribution, partition information, or Tablet distribution information of the target table; dynamically adjusting the sharding granularity according to the data volume of each shard; performing secondary splitting on hot shards with data volume exceeding a threshold; and merging shards with data volume below the threshold to balance the load of each DataX task channel.

[0034] Furthermore, in the method of this application, in step S4, each reading task executes the corresponding shard's SQL query statement through the Impala query engine, and the SQL query statement is automatically generated according to the sharding conditions and filtering conditions configured by the user.

[0035] The parallel read task is configured with at least one of the following: query timeout control, result set retrieval size control, retry mechanism, and sequential cursor mode.

[0036] Furthermore, in the method of this application, step S5, the data type conversion includes: converting the Kudu / Impala data type into a DataX internal Column object, wherein the Kudu / Impala data type includes at least one of INT, BIGINT, STRING, BOOLEAN, FLOAT, DOUBLE, DECIMAL, TIMESTAMP, DATE, and BINARY; and the DataX internal Column object includes at least one of LongColumn, StringColumn, BoolColumn, DoubleColumn, DateColumn, and BytesColumn.

[0037] Furthermore, in the method of this application, step S5 also includes:

[0038] Perform NULL value safety handling, precision preservation, time format normalization, and special character escaping;

[0039] When the target field cannot be directly mapped, a configurable degradation strategy is used to convert the corresponding field to a StringColumn to avoid task interruption.

[0040] Furthermore, in the method of this application, in step S6, the fault-tolerant recovery includes: saving the completed shard identifier and the query condition boundary of the current shard; retrying only the failed shard when the task restarts abnormally; and for incremental synchronization scenarios, recording the last maximum timestamp or maximum primary key value as the starting point for the next synchronization.

[0041] A second aspect of this application provides a DataX-based Kudu database reading device, which, when running, implements the steps of the aforementioned DataX-based Kudu database reading method, such as... Figure 2 As shown, the device includes:

[0042] The connection management module is used to build custom KuduReader plugins in the DataX framework and establish communication connections between the KuduReader plugins and the Impala query engine.

[0043] The metadata acquisition module is used to acquire metadata information of the Kudu target table through the Impala query engine;

[0044] The task segmentation module is used to logically segment the data to be read based on the metadata information and the preset segmentation strategy, and generate multiple data segments.

[0045] The parallel reading module is used to generate multiple parallel reading tasks for each data shard and schedule each reading task to concurrently query the corresponding shard data through the Impala query engine.

[0046] The data conversion module is used to parse fields, convert data types, and encapsulate standard Records in the query results returned by the Impala query engine, and write the encapsulated data into the DataX channel for downstream Writer plugins to write out.

[0047] The status management module is used to record task status, split boundaries, and offset information during the reading process to achieve fault tolerance recovery and incremental resume.

[0048] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0049] Memory: Used to store computer programs;

[0050] Processor: Used to execute the computer program to implement the steps of the aforementioned DataX-based Kudu database reading method.

[0051] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned DataX-based Kudu database reading method.

[0052] In summary, compared with the prior art, the present invention has the following significant advantages:

[0053] (1) Filling the gaps in DataX's native capabilities

[0054] This invention enables DataX to directly read Kudu data by using a custom KuduReader plugin, thus solving the problem that DataX does not natively support Kudu.

[0055] (2) Using Impala as a unified query engine provides stronger compatibility.

[0056] This invention does not rely directly on the complex native scanning logic of Kudu, but instead uses Impala to uniformly execute SQL queries on Kudu tables, making it more compatible with existing data platforms, permission systems, and operation and maintenance systems.

[0057] (3) Supports parallel slice reading, significantly improving performance.

[0058] By designing a task splitting and parallel scheduling mechanism for the Kudu scenario, large table synchronization tasks can be split into multiple concurrent read tasks, significantly improving read throughput.

[0059] (4) Provides adaptive splitting capability to improve load balancing performance.

[0060] Compared to traditional fixed interval partitioning, this invention dynamically adjusts the granularity of partitioning based on table statistics and data distribution, which can reduce the performance bottleneck caused by hotspot partitioning.

[0061] (5) Supports resume download and retry on failure to improve stability.

[0062] This invention records fragment boundaries and synchronization points, enabling recovery from failed fragments when a task is interrupted, avoiding full repeated readings, and improving task stability and resource utilization.

[0063] (6) Supports automatic mapping of multiple field types, improving versatility

[0064] Through type mapping and null value compatibility mechanisms, this invention can adapt to common field types in Kudu / Impala, reducing business access costs.

[0065] (7) Supports integrated synchronization of full and incremental data.

[0066] This invention supports both full table reads and incremental synchronization based on timestamps or primary keys, making it more applicable to various business scenarios. Attached Figure Description

[0067] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this invention will be briefly introduced below. It should be noted that the drawings only show some embodiments of the invention. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0068] Figure 1 This is a flowchart illustrating the operational steps of the Kudu database reading method based on DataX in this application.

[0069] Figure 2 This is a structural diagram of the Kudu database reading device based on DataX in this application.

[0070] Figure 3 This is a diagram illustrating the overall implementation architecture of the proposed solution.

[0071] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0073] In this document, the term "including" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not an exhaustive list and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors.

[0074] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0075] This invention provides the following technical solution:

[0076] Figure 3 This is a diagram illustrating the overall implementation architecture of the present invention.

[0077] In a first aspect, the present invention provides a Kudu database reading method based on DataX, the method comprising:

[0078] (1) Build a custom KuduReader plugin in the DataX framework;

[0079] (2) Connect to the Impala query engine through the KuduReader plugin;

[0080] (3) Impala performs a query on the target table mapped to Kudu storage;

[0081] (4) Logically segment the data to be read according to the preset segmentation strategy;

[0082] (5) Generate multiple parallel reading tasks corresponding to multiple data shards;

[0083] (6) Each read task concurrently queries the corresponding data shard through Impala;

[0084] (7) Parse the query results, convert the data types, and encapsulate them into standard Records;

[0085] (8) Write the encapsulated data into the DataX channel for downstream Writer plugins to write out;

[0086] (9) Record the task status, split boundary and offset information during the reading process to achieve fault-tolerant recovery and incremental resume.

[0087] Optionally, the segmentation strategy in step (4) includes at least one of the following:

[0088] (1) Segmentation based on primary key range;

[0089] (2) Segmentation based on time field range;

[0090] (3) Segmentation based on hash modulo;

[0091] (4) Adaptive fragmentation based on Impala statistical information.

[0092] The adaptive fragmentation refers to:

[0093] (a) Obtain the number of rows, primary key distribution, partition information, or Tablet distribution information of the target table in advance;

[0094] (b) Dynamically adjust the granularity of the data segmentation based on the amount of data in each segment;

[0095] (c) Perform secondary splitting on the hotspot shards;

[0096] (d) Merge small fragments to balance the load on each DataX task channel.

[0097] Optionally, the data type conversion in step (7) includes:

[0098] The data types in Kudu / Impala include INT, BIGINT, STRING, BOOLEAN, FLOAT, DOUBLE, DECIMAL, TIMESTAMP, and DATE.

[0099] Convert to DataX internal Column objects, including LongColumn, StringColumn, BoolColumn, DoubleColumn, DateColumn, BytesColumn, etc.

[0100] Optionally, the fault-tolerant recovery in step (9) includes:

[0101] (a) Save the completed fragmentation identifier;

[0102] (b) Save the query condition boundaries for the current shard;

[0103] (c) Retry only the failed fragments when the task restarts abnormally;

[0104] (d) For incremental synchronization scenarios, record the last maximum timestamp or maximum primary key value as the starting point for the next synchronization.

[0105] Secondly, the present invention provides a Kudu database reading device based on DataX, the device comprising:

[0106] (1) Connection Management Module

[0107] It is used to establish a communication connection between the DataX custom plugin and Impala, and to complete authentication, session initialization and parameter validation.

[0108] (2) Metadata Acquisition Module

[0109] Used to obtain table structure information, field information, primary key information, partition information, and statistical information of the target table in Kudu.

[0110] (3) Query generation module

[0111] Used to generate SQL query statements for Impala based on user configuration and sharding strategy, including full query, conditional query, incremental query and sharded query.

[0112] (4) Task splitting module

[0113] This is used to divide a read task into multiple subtasks based on the primary key range, time range, hash rules, or statistical information.

[0114] (5) Parallel reading module

[0115] Used to control the concurrent execution of multiple DataX reader tasks, which read corresponding data shards in parallel using Impala.

[0116] (6) Data conversion module

[0117] It is used to convert the result set returned by Impala into a standard DataX Record object, and to perform field type adaptation, null value handling, and outlier compatibility handling.

[0118] (7) Status Management Module

[0119] It is used to record task splitting status, execution progress, exception information and resume points, and supports failure recovery and breakpoint retry.

[0120] (8) Output adapter module

[0121] Used to write the converted Record to the DataX Channel for the Writer plugin to write out later.

[0122] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0123] Example 1: DataX Reading Method for Kudu Table Query Based on Impala

[0124] Step S1: Receiving Task Configuration

[0125] DataX receives the synchronization task configuration submitted by the user. The configuration includes parameters such as: Impala connection address, port number, authentication information, database name, table name, list of fields to be read, where filter conditions, splitPk or splitField, concurrency, incremental field and starting position, fetchSize, and timeout.

[0126] For example, as specified in the configuration:

[0127] Source table: ods.user_profile

[0128] Query fields: id, name, age, update_time

[0129] Splitting field: id

[0130] Concurrent channels: 8

[0131] Incremental condition: update_time > '${last_time}'

[0132] Step S2: Connect to Impala and obtain metadata

[0133] After the KuduReader plugin is initialized, it first establishes a connection with the Impala service through JDBC or Impala Thrift interface.

[0134] Then perform a metadata query operation, for example:

[0135] Get the field name and field type;

[0136] Retrieve the primary key field;

[0137] Retrieve table partition information;

[0138] Get the minimum value, maximum value, and total number of rows;

[0139] Obtain field cardinality and data distribution information.

[0140] For example, execute:

[0141] SELECT MIN(id), MAX(id), COUNT(1) FROM ods.user_profile;

[0142] Or execute:

[0143] SHOW COLUMN STATS ods.user_profile;

[0144] This information provides a basis for subsequent sharding.

[0145] Step S3: Generate a segmentation strategy

[0146] The task splitting module automatically selects the splitting method based on user-specified parameters and metadata.

[0147] Scenario 1: User-specified primary key partitioning

[0148] If splitPk = id, then the interval is split into multiple consecutive ranges based on MIN(id) and MAX(id), for example:

[0149] task1: id >= 1 and id < 100000

[0150] task2: id >= 100000 and id < 200000 ...

[0152] task8: id >= 700000 and id <= 800000

[0153] Scenario 2: User-specified time field incremental splitting

[0154] If splitField=update_time, then the data will be split according to a time window, for example:

[0155] task1: update_time >= '2026-01-01 00:00:00' and update_time < '2026-01-01 03:00:00'

[0156] task2: update_time >= '2026-01-01 03:00:00' and update_time < '2026-01-01 06:00:00'

[0157] Scenario 3: Adaptive secondary partitioning with data skew

[0158] If the number of rows in a certain interval is too large, it will be further subdivided; if the number of rows in a certain interval is too small, it will be merged with the adjacent interval to achieve load balancing.

[0159] Note: This step is one of the key innovations of this invention. Instead of mechanically averaging the data, it involves dynamic segmentation and load balancing based on statistical information.

[0160] Step S4: Construct Impala sharded query SQL

[0161] For each shard, generate the corresponding Impala query statement.

[0162] For example:

[0163] SELECT id, name, age, update_time

[0164] FROM ods.user_profile

[0165] WHERE id >= 1 AND id < 100000;

[0166] If user filtering conditions exist, the data will be automatically concatenated:

[0167] SELECT id, name, age, update_time

[0168] FROM ods.user_profile

[0169] WHERE status = 1

[0170] AND update_time > '2026-01-01 00:00:00'

[0171] AND id >= 1 AND id < 100000;

[0172] To prevent full table locking or prolonged blocking, additional features can be added, such as query timeout control, result set retrieval size control, retry mechanism, and sequential cursor mode.

[0173] Step S5: DataX executes read tasks in parallel.

[0174] The DataX framework distributes multiple sharded tasks to different Reader Tasks for concurrent execution. Each Task independently completes the following: establishing an Impala connection, executing the corresponding SQL, streaming the result set, converting each row into a DataX Record, and writing to a DataXChannel.

[0175] High-throughput extraction is achieved through multi-task concurrency.

[0176] Step S6: Result Set Type Conversion

[0177] The data transformation module performs type mapping on the Impala return results:

[0178] TINYINT / SMALLINT / INT / BIGINT → LongColumn

[0179] FLOAT / DOUBLE / DECIMAL → DoubleColumn / StringColumn

[0180] STRING / VARCHAR / CHAR → StringColumn

[0181] BOOLEAN → BoolColumn

[0182] TIMESTAMP / DATE → DateColumn / StringColumn

[0183] BINARY → BytesColumn

[0184] It also supports: safe handling of NULL values, precision preservation, time format standardization, and escape of special characters.

[0185] If the target field cannot be directly mapped, a configurable degradation strategy is adopted, such as converting it to a StringColumn, to avoid task interruption.

[0186] Step S7: Status Recording and Breakpoint Recovery

[0187] Before each shard begins execution, the status management module records: task ID, shard ID, shard SQL, start and end boundaries, and current status (pending execution, in execution, completed, failed).

[0188] If the task succeeds, update the status to "completed"; if it fails, record the reason for the error and the last point of success.

[0189] During rescheduling, the system only re-initiates reading for the failed shards, without having to rerun the entire task.

[0190] For incremental synchronization, the maximum update_time of the current batch and / or the maximum primary key value of the current batch can also be recorded; the next synchronization will use this value as the lower boundary to achieve incremental resuming.

[0191] Example 2: Kudu Reading Method Based on Hash Segmentation

[0192] For Kudu tables with non-contiguous primary keys or highly uneven data distribution, a hash partitioning strategy can be adopted.

[0193] For example, generate the following SQL:

[0194] SELECT id, name, age

[0195] FROM ods.user_profile

[0196] WHERE MOD(HASH(id), 8) = 0;

[0197] The other tasks correspond to:

[0198] MOD(HASH(id), 8) = 1 ...

[0200] MOD(HASH(id), 8) = 7

[0201] This method is particularly suitable for situations where: the primary key has a large number of gaps, the range is unevenly divided, and there are obvious local hot spots in the data.

[0202] This approach further enhances the applicability of the invention to complex Kudu data distributions.

[0203] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0204] like Figure 4 As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the DataX-based Kudu database reading method described above.

[0205] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0206] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types:

[0207] Object-oriented programming languages, such as Java, Smalltalk, C++, etc.

[0208] Conventional procedural programming languages, such as "C" or similar programming languages.

[0209] The execution methods of program code include, but are not limited to:

[0210] It runs entirely on the user's computer;

[0211] Part of it executes on the user's computer, and part of it executes on a remote computer;

[0212] Execute as a standalone software package;

[0213] It is executed entirely on a remote computer or server.

[0214] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0215] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the DataX-based Kudu database reading method disclosed in this application.

[0216] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following:

[0217] (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0218] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0219] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0220] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0221] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0222] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0223] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0224] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0225] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0226] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0227] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0228] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program includes program code for executing the DataX-based Kudu database reading method disclosed in this application. When the computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0229] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0230] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A Kudu database reading method based on DataX, characterized in that, Includes the following steps: S1: Build a custom KuduReader plugin in the DataX framework and connect to the Impala query engine through the KuduReader plugin; S2: Obtain the metadata information of the Kudu target table using the Impala query engine; S3: Based on the metadata information and the preset segmentation strategy, logically segment the data to be read to generate multiple data segments; S4: For each data shard, multiple parallel read tasks are generated, and the DataX framework schedules each read task to concurrently query the corresponding shard data through the Impala query engine; S5: Parse the fields, convert the data types, and encapsulate the standard Records of the query results returned by the Impala query engine, and write the encapsulated data into the DataX channel for downstream Writer plugins to write out; S6: Record task status, split boundaries and offset information during the reading process to achieve fault tolerance recovery and incremental resume.

2. The method according to claim 1, characterized in that, In step S1, the KuduReader plugin establishes a communication connection with the Impala query engine through a JDBC or Impala Thrift interface, and completes authentication, session initialization, and parameter verification.

3. The method according to claim 1, characterized in that, In step S2, the metadata information includes at least one of table structure information, field information, primary key information, partition information, statistical information, and field data distribution information.

4. The method according to claim 1, characterized in that, In step S3, the preset sharding strategy includes at least one of the following: sharding based on primary key range, sharding based on time field range, sharding based on hash modulo, and adaptive sharding based on Impala statistics.

5. The method according to claim 4, characterized in that, The adaptive sharding based on Impala statistics includes: pre-obtaining the number of rows, primary key distribution, partition information, or Tablet distribution information of the target table; dynamically adjusting the sharding granularity according to the data volume of each shard; performing secondary splitting on hot shards with data volume exceeding the threshold; and merging shards with data volume below the threshold to balance the load of each DataX task channel.

6. The method according to claim 1, characterized in that, In step S4, each reading task executes the corresponding shard's SQL query statement through the Impala query engine. The SQL query statement is automatically generated based on the sharding conditions and filtering conditions configured by the user. The parallel read task is configured with at least one of the following: query timeout control, result set retrieval size control, retry mechanism, and sequential cursor mode.

7. The method according to claim 1, characterized in that, In step S5, the data type conversion includes: converting the Kudu / Impala data type into a DataX internal Column object, wherein the Kudu / Impala data type includes at least one of INT, BIGINT, STRING, BOOLEAN, FLOAT, DOUBLE, DECIMAL, TIMESTAMP, DATE, and BINARY; and the DataX internal Column object includes at least one of LongColumn, StringColumn, BoolColumn, DoubleColumn, DateColumn, and BytesColumn.

8. The method according to claim 1, characterized in that, Step S5 also includes: Perform NULL value safety handling, precision preservation, time format normalization, and special character escaping; When the target field cannot be directly mapped, a configurable degradation strategy is used to convert the corresponding field to a StringColumn to avoid task interruption.

9. The method according to claim 1, characterized in that, In step S6, the fault tolerance recovery includes: saving the completed shard identifier and the query condition boundary of the current shard; retrying only the failed shard when the task restarts abnormally; and for incremental synchronization scenarios, recording the last maximum timestamp or maximum primary key value as the starting point for the next synchronization.

10. A Kudu database reading device based on DataX, characterized in that, When the device is in operation, it implements the steps of the Kudu database reading method based on DataX as described in any one of claims 1-9, including: The connection management module is used to build custom KuduReader plugins in the DataX framework and establish communication connections between the KuduReader plugins and the Impala query engine. The metadata acquisition module is used to obtain metadata information of the Kudu target table through the Impala query engine; The task segmentation module is used to logically segment the data to be read based on the metadata information and the preset segmentation strategy, and generate multiple data segments. The parallel reading module is used to generate multiple parallel reading tasks for each data shard and schedule each reading task to concurrently query the corresponding shard data through the Impala query engine. The data conversion module is used to parse fields, convert data types, and encapsulate standard Records in the query results returned by the Impala query engine, and write the encapsulated data into the DataX channel for downstream Writer plugins to write out. The status management module is used to record task status, split boundaries, and offset information during the reading process to achieve fault tolerance recovery and incremental resume.

Images