A data processing method and device, electronic equipment and computer readable medium

Abstract

This application discloses a data processing method, apparatus, electronic device, and computer-readable medium, relating to the field of computer technology. A specific embodiment includes: obtaining a data stream and a cluster identifier based on a received data processing request; obtaining node data corresponding to the cluster identifier; generating a first hash value based on the node data; calculating a second hash value for each data item in the data stream; determining the target node corresponding to each data item in the data stream based on the first hash value, the second hash value, and a hash ring; determining the number of data items to be written for each target node; configuring an independent connection pool for each target node based on the number; and writing each data item to the corresponding target node based on the connection pool. This improves write performance and throughput when writing data to the cluster, avoids single-point overload, ensures uniform data distribution in the cluster, increases data write speed, and shortens data write latency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, and computer-readable medium. Background Technology

[0002] Apache Flink is an open-source stream processing framework suitable for real-time data processing and analysis. Flink provides various data source and sink interfaces, with JdbcSink responsible for writing data to database systems that support the JDBC protocol. ClickHouse is an open-source columnar database management system for online analytical processing (OLAP), characterized by its fast query and data analysis capabilities. ClickHouse clusters achieve horizontal scaling through data sharding to enhance query and storage capabilities. When writing data from Flink to a ClickHouse cluster, a single point of overload can occur, resulting in slow data write speeds and high latency. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data processing method, apparatus, electronic device, and computer-readable medium that can solve the problems of single-point overload, slow data writing speed, and long latency when writing data from Flink to a ClickHouse cluster.

[0004] To achieve the above objectives, according to one aspect of the embodiments of this application, a data processing method is provided, comprising:

[0005] Obtain the data stream and cluster identifier based on the received data processing request;

[0006] Obtain the node data corresponding to the cluster identifier, and generate the first hash value based on the node data;

[0007] Calculate the second hash value of the data items contained in the data stream, and determine the target node corresponding to each data item in the data stream based on the first hash value, the second hash value, and the hash ring;

[0008] Determine the number of data items to be written for each target node, and configure an independent connection pool for each target node based on the number.

[0009] Each data item is written to the corresponding target node based on the connection pool.

[0010] Optionally, determining the target node corresponding to each data item in the data stream includes:

[0011] The first position of the node data on the hash ring is determined based on the first hash value;

[0012] The second position of the data item in the data stream on the hash ring is determined based on the second hash value;

[0013] Based on the first position, the second position, and the preset allocation direction, determine the target node corresponding to each data item in the data stream.

[0014] Optionally, based on the first position, the second position, and the preset allocation direction, the target node corresponding to each data item in the data stream is determined, including:

[0015] At each second position, a search is performed according to the preset allocation direction, and the node corresponding to the closest first position is determined as the target node.

[0016] Optionally, each data item is written to the corresponding target node based on the connection pool, including:

[0017] When the space occupied by the data items to be written in the memory cache reaches a preset threshold, a batch write operation is performed to write the data items to be written in the memory cache to the corresponding target node based on the connection pool.

[0018] Optionally, each data item is written to the corresponding target node based on the connection pool, including:

[0019] In response to the triggering of a timed task or trigger associated with the memory cache, a batch write operation is performed to write the data items to be written from the memory cache to the corresponding target nodes based on the connection pool.

[0020] Optionally, each data item is written to the corresponding target node based on the connection pool, including:

[0021] Generate a unique identifier for each data item, and associate each data item with its corresponding unique identifier to obtain the associated data;

[0022] When writing each data item to the corresponding target node based on the connection pool, anti-duplicate verification is performed based on the associated data. If the anti-duplicate verification is successful, the corresponding data item is written to the corresponding target node. If the anti-duplicate verification fails, the corresponding data item is discarded.

[0023] Optionally, each data item is written to the corresponding target node based on the connection pool, including:

[0024] Each data item is written to the data table configured with a uniqueness strategy in the corresponding target node based on the connection pool.

[0025] In addition, this application also provides a data processing apparatus, including:

[0026] The acquisition unit is configured to acquire the data stream and cluster identifier based on the received data processing request;

[0027] The hash value generation unit is configured to obtain the node data corresponding to the cluster identifier and generate the first hash value based on the node data;

[0028] The target node determination unit is configured to calculate the second hash value of the data items contained in the data stream, and determine the target node corresponding to each data item in the data stream based on the first hash value, the second hash value and the hash ring;

[0029] The connection pool configuration unit is configured to determine the number of data items to be written for each target node, and configure an independent connection pool for each target node based on the number.

[0030] The data item writing unit is configured to write each data item to the corresponding target node based on the connection pool.

[0031] Optionally, the target node determination unit is further configured to:

[0032] The first position of the node data on the hash ring is determined based on the first hash value;

[0033] The second position of the data item in the data stream on the hash ring is determined based on the second hash value;

[0034] Based on the first position, the second position, and the preset allocation direction, determine the target node corresponding to each data item in the data stream.

[0035] Optionally, the target node determination unit is further configured to:

[0036] At each second position, a search is performed according to the preset allocation direction, and the node corresponding to the closest first position is determined as the target node.

[0037] Optionally, the data item writing unit is further configured to:

[0038] When the space occupied by the data items to be written in the memory cache reaches a preset threshold, a batch write operation is performed to write the data items to be written in the memory cache to the corresponding target node based on the connection pool.

[0039] Optionally, the data item writing unit is further configured to:

[0040] In response to the triggering of a timed task or trigger associated with the memory cache, a batch write operation is performed to write the data items to be written from the memory cache to the corresponding target nodes based on the connection pool.

[0041] Optionally, the data item writing unit is further configured to:

[0042] Generate a unique identifier for each data item, and associate each data item with its corresponding unique identifier to obtain the associated data;

[0043] When writing each data item to the corresponding target node based on the connection pool, anti-duplicate verification is performed based on the associated data. If the anti-duplicate verification is successful, the corresponding data item is written to the corresponding target node. If the anti-duplicate verification fails, the corresponding data item is discarded.

[0044] Optionally, the data item writing unit is further configured to:

[0045] Each data item is written to the data table configured with a uniqueness strategy in the corresponding target node based on the connection pool.

[0046] In addition, this application also provides a data processing electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the data processing method described above.

[0047] In addition, this application also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method described above.

[0048] To achieve the above objectives, according to another aspect of the embodiments of this application, a computer program product is provided.

[0049] A computer program product according to an embodiment of this application includes a computer program that, when executed by a processor, implements the data processing method provided in the embodiment of this application.

[0050] One embodiment of the above invention has the following advantages or beneficial effects: This application obtains a data stream and a cluster identifier according to a received data processing request; obtains the node data corresponding to the cluster identifier, and generates a first hash value based on the node data; calculates a second hash value for the data items contained in the data stream, and determines the target node corresponding to each data item in the data stream based on the first hash value, the second hash value, and the hash ring; determines the number of data items to be written for each target node, and configures an independent connection pool for each target node according to the number; and writes each data item to the corresponding target node based on the connection pool. This improves write performance and throughput when writing data to the cluster, avoids single-point overload, ensures uniform data distribution in the cluster, increases data write speed, and shortens data write latency.

[0051] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0052] The accompanying drawings are provided to better understand this application and do not constitute an undue limitation thereof. Wherein:

[0053] Figure 1 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application;

[0054] Figure 2 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application;

[0055] Figure 3 This is a schematic diagram illustrating the structure of a data processing method according to an embodiment of this application;

[0056] Figure 4 This is a schematic diagram of the system architecture of a data processing method according to an embodiment of this application;

[0057] Figure 5 This is a schematic diagram of the hash ring structure data distribution of a data processing method according to an embodiment of this application;

[0058] Figure 6 This is a schematic diagram of the main units of a data processing apparatus according to an embodiment of this application;

[0059] Figure 7 This is an exemplary system architecture diagram to which embodiments of this application can be applied;

[0060] Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers in the embodiments of this application. Detailed Implementation

[0061] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These embodiments should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solutions of this application comply with relevant national laws and regulations. It should also be noted that certain software, components, models, and other existing industry solutions may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solutions of this application, and do not imply that the applicant has already used or necessarily used such solutions. The collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solutions of this application all comply with relevant laws and regulations, are used for legal and reasonable purposes, do not violate public order and good morals, are not shared, disclosed, or sold outside of these legal uses, and are subject to supervision and management by regulatory authorities. Necessary measures should be taken to prevent unauthorized access to user personal information, safeguard user personal information security, cybersecurity, and national security, and ensure that those authorized to access personal information comply with relevant laws and regulations. Once this user personal information is no longer needed, risks should be minimized by restricting or even prohibiting data collection and / or deleting the data.

[0062] When used, including in certain relevant applications, data is deidentified to protect user privacy, for example by removing specific identifiers, controlling the amount or specificity of stored data, controlling how data is stored, and / or other methods.

[0063] Figure 1 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application, as shown below. Figure 1 As shown, the data processing method mainly includes the following steps S101-S105.

[0064] Step S101: Obtain the data stream and cluster identifier based on the received data processing request.

[0065] The following are explanations of some technical terms that may be involved in this application:

[0066] Flink: Apache Flink is an open-source stream processing framework for handling unbounded and bounded data streams. It provides high throughput, low latency stream processing capabilities and supports event-driven processing, state management, and fault tolerance mechanisms.

[0067] ClickHouse: ClickHouse is an open-source columnar database management system designed for Online Analytical Processing (OLAP), capable of efficiently handling large-scale data query and analysis tasks. Its features include high performance, real-time data writing and querying, and support for distributed architecture.

[0068] Consistent hashing: Consistent hashing is a hashing algorithm in distributed systems designed to distribute requests evenly across multiple nodes. It improves system scalability and load balancing by reducing the amount of data remapping when nodes change.

[0069] JdbcSink: JdbcSink is a data output interface provided by Flink, used to write processed data to a relational database. It supports batch writes and transaction management, and is suitable for application scenarios that require data persistence to a database.

[0070] Hash ring: In consistent hashing algorithms, hash rings are used to evenly distribute data items across different nodes. Each node and data item has a unique position on the hash ring, and data items are always assigned to the first node in a clockwise direction.

[0071] Idempotent write: Idempotent write is a write strategy that ensures the system state does not change when the same data is written repeatedly. It guarantees data consistency and accuracy by generating unique identifiers or using unique constraints in the database.

[0072] Connection pooling: Connection pooling is a technology used to manage database connections. By pre-creating and maintaining a certain number of database connections, it reduces the overhead of connection creation and destruction, thereby improving the performance and efficiency of database access.

[0073] Data caching: Data caching is a technique that temporarily stores data in memory. It is used to temporarily store data items to be written until the batch size or time limit is reached, and then perform batch write operations to improve write efficiency and reduce database pressure.

[0074] Batch write: Batch write is a technique that writes multiple data items to the database at once, reducing the overhead of database connections and transactions and improving write efficiency. It is usually used in conjunction with data caching, and batch write is triggered when certain conditions are met.

[0075] In this embodiment, the execution entity of the data processing method (e.g., a server) can receive data processing requests via wired or wireless connections. Based on the received data processing request, it obtains the data stream and cluster identifier. The data stream can be a Flink data stream from a Flink cluster. The cluster identifier can be the encoding or name of the ClickHouse cluster to which the data stream is to be written.

[0076] Step S102: Obtain the node data corresponding to the cluster identifier, and generate the first hash value based on the node data.

[0077] After obtaining the cluster identifier corresponding to the data processing request, the executing entity can obtain the node data corresponding to the cluster identifier, such as ClickHouse node A, ClickHouse node B, and ClickHouse node C. Based on the name of the node data, a first hash value is generated using a hash algorithm.

[0078] Step S103: Calculate the second hash value of the data items contained in the data stream, and determine the target node corresponding to each data item in the data stream based on the first hash value, the second hash value and the hash ring.

[0079] The data items in the data stream are hashed to obtain a second hash value. A hash mapping is then used to map each first hash value and each second hash value onto the same hash ring. Based on the mapping results on the same hash ring, the target node corresponding to each data item in the data stream is accurately determined.

[0080] Step S104: Determine the number of data items to be written for each target node, and configure an independent connection pool for each target node based on the number.

[0081] After determining the target node corresponding to each data item in the data stream, the execution entity can count the number of data items to be written for each target node in the current data stream, and promptly call the connection pool configuration process to configure an independent connection pool for each target node in real time, corresponding to the number of data items to be written. The number of connections in the connection pool can be the same as the number of data items to be written for the corresponding target node, so as to facilitate efficient data writing.

[0082] Step S105: Write each data item to the corresponding target node based on the connection pool.

[0083] Specifically, writing each data item to the corresponding target node based on the connection pool includes: in response to the space occupied by the data item to be written in the memory cache reaching a preset threshold, performing a batch write operation to write the data item to be written in the memory cache to the corresponding target node based on the connection pool.

[0084] For example, when the space occupied by the data items to be written in the memory cache reaches a preset threshold, a batch write operation is performed. That is, when the cache reaches the set batch size, a batch write operation is performed. This allows for efficient writing of the data items to be written in the memory cache to the corresponding target nodes on the hash ring based on a connection pool.

[0085] Specifically, writing each data item to the corresponding target node based on the connection pool includes: generating a unique identifier for each data item, associating each data item with the corresponding unique identifier to obtain associated data; when writing each data item to the corresponding target node based on the connection pool, performing anti-duplicate verification based on the associated data, and writing the corresponding data item to the corresponding target node in response to anti-duplicate verification, and discarding the corresponding data item in response to failure to perform anti-duplicate verification.

[0086] A unique identifier is generated for each data item to ensure its uniqueness. When writing each data item to the corresponding target node using the connection pool, a deduplication check is performed based on the association data between the data item and the unique identifier. The check verifies whether the data has been written repeatedly. If the deduplication check passes (i.e., it is not a duplicate write), the corresponding data item is written to the corresponding target node; otherwise, it is discarded. This ensures the correctness and consistency of data items written to the target node.

[0087] Specifically, writing each data item to the corresponding target node based on the connection pool includes: writing each data item to the data table configured with a uniqueness strategy in the corresponding target node based on the connection pool.

[0088] An idempotent write strategy is used to write each data item to the corresponding target node on the hash ring, configured with a uniqueness strategy, based on the connection pool. The idempotent write strategy involves configuring the table in ClickHouse with unique constraints or the replaceMergeTreeyiq engine to ensure data correctness and consistency even if data is written repeatedly.

[0089] This embodiment obtains the data stream and cluster identifier based on the received data processing request; obtains the node data corresponding to the cluster identifier; generates a first hash value based on the node data; calculates the second hash value of the data items contained in the data stream; and determines the target node corresponding to each data item in the data stream based on the first hash value, the second hash value, and the hash ring; determines the number of data items to be written for each target node; configures an independent connection pool for each target node based on the number; and writes each data item to the corresponding target node based on the connection pool. This improves write performance and throughput when writing data to the cluster, avoids single-point overload, ensures uniform data distribution in the cluster, increases data write speed, and shortens data write latency.

[0090] Figure 2 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application, as shown below. Figure 2 As shown, the data processing method mainly includes the following steps S201-S208.

[0091] Step S201: Obtain the data stream and cluster identifier based on the received data processing request.

[0092] The data stream can be a Flink data stream from a Flink cluster. The cluster identifier can be the code or name of the ClickHouse cluster to which the data stream is to be written.

[0093] Step S202: Obtain the node data corresponding to the cluster identifier, and generate the first hash value based on the node data.

[0094] After obtaining the cluster identifier corresponding to the data processing request, the node data corresponding to that cluster identifier can be obtained, such as ClickHouse node A, ClickHouse node B, and ClickHouse node C. The first hash value is generated based on the name of the node data using a hash algorithm.

[0095] First hash value (assuming the calculation result):

[0096] - Node A (first hash value 100)

[0097] - Node B (first hash value 500)

[0098] - Node C (first hash value 800)

[0099] Step S203: Calculate the second hash value of the data items contained in the data stream.

[0100] The second hash value is obtained by calculating the data items contained in the data stream using a hash algorithm.

[0101] Second hash value (assuming the calculation result):

[0102] - Data 1 (Second hash value 150)

[0103] - Data 2 (Second hash value 550)

[0104] - Data 3 (Second hash value 750)

[0105] Step S204: Determine the first position of the node data on the hash ring based on the first hash value.

[0106] - Node A's first position 100 on the hash ring

[0107] - Node B at position 500 on the hash ring

[0108] - Node C at position 800 on the hash ring

[0109] Step S205: Determine the second position of the data item contained in the data stream on the hash ring based on the second hash value.

[0110] - Data 1 at position 150 on the hash ring

[0111] - Data 2 at position 550 on the hash ring

[0112] - Data 3 at position 750 on the hash ring

[0113] Step S206: Determine the target node corresponding to each data item in the data stream based on the first position, the second position, and the preset allocation direction.

[0114] The default assignment direction is, for example, clockwise.

[0115] Specifically, based on the first position, the second position, and the preset allocation direction, the target node corresponding to each data item in the data stream is determined, including: at each second position, searching according to the preset allocation direction, and determining the node corresponding to the closest first position as the target node.

[0116] Data item allocation on a hash ring is as follows: Figure 5 As shown:

[0117] - Data 1 is located at position 150 on the hash ring. The nearest node in the clockwise direction is node B (500), which means the target node is node B (500).

[0118] - Data 2 is located at the second position 550 on the hash ring. The nearest node in the clockwise direction is node C (800), which means the target node is node C (800).

[0119] - Data 3 is located at the second position 750 on the hash ring. The nearest node in the clockwise direction is node A (100), which means the target node is node A (100).

[0120] Step S207: Determine the number of data items to be written for each target node, and configure an independent connection pool for each target node based on the number.

[0121] After determining the target node corresponding to each data item in the data stream, the execution entity can count the number of data items to be written for each target node in the current data stream, and promptly call the connection pool configuration process to configure an independent connection pool for each target node in real time, corresponding to the number of data items to be written. The number of connections in the connection pool can be the same as the number of data items to be written for the corresponding target node, so as to facilitate efficient data writing.

[0122] Step S208: Write each data item to the corresponding target node based on the connection pool.

[0123] Specifically, writing each data item to the corresponding target node based on the connection pool includes: in response to the triggering of a timed task or trigger associated with the memory cache, performing a batch write operation to write the data items to be written in the memory cache to the corresponding target node based on the connection pool.

[0124] The cache can be checked periodically by scheduled tasks or triggers, and batch write operations can be performed at appropriate times, after which the cache is cleared.

[0125] The embodiments of this application can improve write performance and throughput when writing data to the cluster, avoid single point overload, ensure uniform distribution of data in the cluster, improve data write speed, and shorten data write latency.

[0126] Apache Flink is an open-source stream processing framework suitable for real-time data processing and analysis. Flink provides various data source and data sink interfaces, among which JdbcSink is responsible for writing data to a database system that supports the JDBC protocol.

[0127] ClickHouse is an open-source columnar database management system for online analytical processing (OLAP), characterized by its fast query and data analysis capabilities. ClickHouse clusters achieve horizontal scaling through data sharding to enhance query and storage capabilities.

[0128] Figure 3This is a schematic diagram illustrating the structure of a data processing method according to an embodiment of this application. Figure 3 As shown, Flink data streams are efficiently written to the ClickHouse cluster, and Flink data streams are written to the ClickHouse cluster efficiently and idempotently using FlinkJdbcSink enhanced with consistent hashing to improve reliability. Node selection is achieved through consistent hashing for data distribution. Specifically, data is sharded using a consistent hashing algorithm and a sharding strategy, and the sharded data items are evenly distributed across ClickHouse nodes to ensure load balancing and efficient writing. System architecture: Implements data sharding and routing to efficiently write data items to various nodes in the ClickHouse cluster.

[0129] Figure 4 This is a schematic diagram of the system architecture of a data processing method according to an embodiment of this application. Figure 4 As shown, the Flink cluster's Flink data stream processes and generates data streams; data items need to be written to ClickHouse. A custom JdbcSink extends Flink's JdbcSink class to implement data sharding and write logic. A consistent hashing algorithm determines the target ClickHouse node for each data item, ensuring even data distribution. The ClickHouse cluster consists of multiple nodes (ClickHouse node A, ClickHouse node B, and ClickHouse node C) responsible for data storage and retrieval.

[0130] Extending JdbcSink: Since the native Flink JdbcSink only supports writing to a single JDBC URL and does not support writing data to multiple database shards simultaneously, we extend the functionality of the Flink JdbcSink module here. We create a CustomClickHouseSink class, which inherits from JdbcSink, and implements consistent hashing sharding and data writing.

[0131] Create a custom Sink class: Inherit from the JdbcSink class and create a CustomClickHouseSink. The constructor receives ClickHouse node information and a consistent hash parameter.

[0132] Implement the `invoke` method: Override the `invoke` method to process data received from the Flink data stream. Use a consistent hashing algorithm to calculate the hash value of the data item and determine the target ClickHouse node. Route the data item to the appropriate node for writing.

[0133] Connection pool management: Use a connection pool library (such as HikariCP) to create an independent connection pool for each node. Manage the connection pool in the CustomClickHouseSink class to ensure efficient connection management.

[0134] Connection pool creation and management:

[0135] Create a connection pool: Configure an independent connection pool for each ClickHouse node, such as using a library like HikariCP for management;

[0136] Configure connection pool parameters: Configure the maximum number of connections, minimum number of connections, connection timeout, etc., to optimize connection creation and management;

[0137] Integrated Connection Pool: Integrate a connection pool in CustomClickHouseSink to perform data write operations using efficient database connections.

[0138] Consistent hashing algorithm - hash mapping - sharding strategy - cache storage - cache update strategy - deduplication logic - data allocation - cache expiration strategy - idempotency check - idempotency handling - duplicate data processing - consistent hashing algorithm - node selection - data caching mechanism - data refresh.

[0139] Specifically, the consistent hashing algorithm is implemented as follows:

[0140] Hash function design: Use a stable and uniform hash function (such as MD5, SHA1, or MurmurHash) to hash data items and ClickHouse nodes in the Flink data stream. Ensure that the hash function has good uniformity in data distribution to avoid data hotspots.

[0141] Hash ring (also known as virtual ring) structure: Data items and node hashes are mapped to a fixed-size hash ring. Each node and data item has a unique position on the ring. Nodes are distributed clockwise on the ring, and data items are assigned to the first node in the clockwise direction. Example of a hash ring (virtual ring) structure: In consistent hashing algorithms, the virtual ring structure is used to evenly distribute data items across different nodes. Each node and data item has a unique position on the virtual ring, and data items are always assigned to the first node in the clockwise direction.

[0142] Example data:

[0143] - Nodes: Node A, Node B, Node C

[0144] - Data items: Data1, Data2, Data3

[0145] Hash value (assuming the result of calculation):

[0146] - Node A at position 100

[0147] - Node B at position 500

[0148] - Node C at position 800

[0149] - Data 1 at position 150

[0150] - Data 2 at position 550

[0151] - Data 3 at position 750

[0152] Data item allocation on a hash ring is as follows: Figure 5 As shown:

[0153] - Data 1 is at position 150, and the first node in the clockwise direction is node B (500).

[0154] - Data 2 is at position 550, and the first node in the clockwise direction is node C (800).

[0155] - Data 3 is at position 750, and the first node in the clockwise direction is node A (100).

[0156] illustrate:

[0157] - The label following each node and data item indicates its location information.

[0158] - Solid arrows indicate connections in a clockwise direction.

[0159] - Dashed arrow --> Indicates the node to which the data item is assigned.

[0160] Data caching and batch write optimization:

[0161] Data Caching - Data Provision: A memory cache is maintained in the custom Sink (CustomClickHouseSink) for storing data items to be written. The cache is updated based on a cache update policy and expired items are cleared based on a cache expiration policy. The cached data is then refreshed using the data caching mechanism to ensure its validity and accuracy. Deduplication logic is performed based on the unique identifier of each data item in the cache. Data items are cached until the batch size or timeout limit is reached.

[0162] Batch Write - Optimized Write: When the cache reaches the set batch size, a batch write operation is performed. The cache can be periodically checked via a scheduled task or trigger, and batch write operations can be performed at appropriate times, followed by clearing the cache after the write operation.

[0163] Idempotent write support: Unique identifier: Generates a unique identifier for each data record to ensure data uniqueness. Idempotent write strategy: When configuring tables in ClickHouse, use unique constraints or unique strategies such as the replaceMergeTreeyiq engine to ensure that the system maintains data correctness and consistency even if data is written repeatedly.

[0164] Idempotency check: After data allocation, that is, after the data items of the shards are evenly distributed to the ClickHouse nodes, an idempotency check is performed based on the cache expiration policy and deduplication logic, and idempotency processing is performed based on the idempotency check results, which may be duplicate data processing.

[0165] Exception handling and logging:

[0166] Exception Handling: Implement an exception handling mechanism, including exception capture, error processing, logging, and sending / storing to the alarm system. Specifically, capture and handle write exceptions, and design a retry mechanism for automatic retries for temporary failures. Logging: Record data sharding, routing, write operations, and exception information. Use a logging framework (such as Log4j or SLF4J) to record system status and performance metrics.

[0167] Configuration Options: Configuration File: Design the configuration file format to support ClickHouse node information, consistent hashing parameters, and connection pool settings. Configuration Parsing: Parse the configuration file in CustomClickHouseSink and initialize the consistent hashing algorithm and connection pool settings according to the parameters.

[0168] Besides the consistent hashing-enhanced Flink JdbcSink algorithm for efficient idempotent writes to ClickHouse clusters, here are some possible alternatives:

[0169] Using Kafka for data distribution and buffering: Kafka is introduced as an intermediate layer between Flink and ClickHouse. Flink writes data to Kafka, Kafka is responsible for data distribution and buffering, and ClickHouse reads data from Kafka in batches for writing.

[0170] Data buffering using Flink's State Backend: Utilize Flink's State Backend (such as RocksDB) for data buffering and batch processing. Data is cached in Flink's state, and when a certain batch size or time window is reached, it is written to ClickHouse in batches.

[0171] Data management using Apache Hudi or Apache Iceberg: Use Apache Hudi or Apache Iceberg for data management in Flink. These frameworks provide efficient incremental data processing and writing capabilities, supporting idempotency and consistency of data.

[0172] These alternatives can be selected based on specific needs and scenarios, enabling efficient and idempotent writing of data to ClickHouse cluster shards.

[0173] This application extends Flink's JdbcSink to achieve efficient sharded writing in a ClickHouse cluster based on a consistent hashing algorithm. A custom `CustomClickHouseSink` class optimizes the data writing process and introduces technologies such as data caching, batch writing, idempotent writing, and connection pool management to improve system performance and reliability. Idempotent writing ensures data consistency and accuracy; even if data is written repeatedly, the system maintains data correctness. The integration of exception handling and logging further enhances system robustness. This application ensures balanced data distribution and efficient storage, improving overall data processing capabilities.

[0174] This system leverages consistent hashing enhancements to achieve high-efficiency idempotent writing to ClickHouse clusters using Flink's JdbcSink. JdbcSink incorporates a consistent hashing sharding data routing strategy: in Flink data stream processing, the consistent hashing sharding algorithm is implemented by extending JdbcSink, writing data shards to multiple shards across the ClickHouse cluster. A pre-configured hash ring ensures even data distribution across multiple shards, guaranteeing load balancing. This improves data distribution uniformity, avoids single-point overload, and enhances system scalability and stability. Multi-level caching and batch write optimizations are implemented: a multi-level caching mechanism maintains a data cache in memory. When the cache reaches a set batch size or timeout limit, batch write operations are performed. Scheduled tasks or triggers periodically check the cache to ensure efficient and reliable data writing. This reduces frequent database write operations, improves write efficiency, and lowers system resource consumption. Efficient connection pool management and dynamic adjustment are achieved: an independent connection pool is configured for each ClickHouse node, managed using a high-performance connection pool library. Based on system load and database performance, dynamically adjust parameters such as the maximum number of connections, minimum number of connections, and connection timeout in the connection pool. This improves database connection utilization and management efficiency, enhancing system stability and performance. Idempotent writes and data consistency guarantees: Generate a unique identifier for each data record to ensure data uniqueness and idempotency. Use unique constraints or the replaceMergeTree engine when configuring tables in ClickHouse to ensure data correctness and consistency even if data is written repeatedly. This ensures data consistency and correctness, preventing data pollution caused by duplicate data. Exception handling and intelligent retry mechanisms: Implement a fine-grained exception handling mechanism to capture and handle write exceptions. Design an intelligent retry strategy to dynamically adjust the number of retries and intervals based on the exception type and system load, ensuring reliable data writes. This improves system fault tolerance and data write reliability, reducing data loss or write failures caused by exceptions.

[0175] Figure 6 This is a schematic diagram of the main units of a data processing apparatus according to an embodiment of this application. Figure 6 As shown, the data processing device 600 includes an acquisition unit 601, a hash value generation unit 602, a target node determination unit 603, a connection pool configuration unit 604, and a data item writing unit 605.

[0176] The acquisition unit 601 is configured to acquire the data stream and cluster identifier based on the received data processing request.

[0177] The hash value generation unit 602 is configured to obtain the node data corresponding to the cluster identifier and generate the first hash value based on the node data.

[0178] The target node determination unit 603 is configured to calculate the second hash value of the data items contained in the data stream, and determine the target node corresponding to each data item in the data stream based on the first hash value, the second hash value and the hash ring.

[0179] The connection pool configuration unit 604 is configured to determine the number of data items to be written for each target node and configure an independent connection pool for each target node based on the number.

[0180] The data item writing unit 605 is configured to write each data item to the corresponding target node based on the connection pool.

[0181] In some embodiments, the target node determination unit 603 is further configured to: determine a first position of node data on a hash ring based on a first hash value; determine a second position of data items contained in the data stream on a hash ring based on a second hash value; and determine the target node corresponding to each data item in the data stream according to the first position, the second position, and a preset allocation direction.

[0182] In some embodiments, the target node determination unit 603 is further configured to: at each second position, search according to a preset allocation direction, and determine the node corresponding to the closest first position as the target node.

[0183] In some embodiments, the data item writing unit 605 is further configured to: in response to the space occupied by the data item to be written in the memory cache reaching a preset threshold, perform a batch write operation to write the data item to be written in the memory cache to the corresponding target node based on the connection pool.

[0184] In some embodiments, the data item writing unit 605 is further configured to perform a batch write operation in response to a timed task or trigger associated with the memory cache being triggered, so as to write the data items to be written in the memory cache to the corresponding target node based on the connection pool.

[0185] In some embodiments, the data item writing unit 605 is further configured to: generate a unique identifier for each data item, associate each data item with the corresponding unique identifier to obtain associated data; when writing each data item to the corresponding target node based on the connection pool, perform anti-duplicate verification based on the associated data, and write the corresponding data item to the corresponding target node in response to anti-duplicate verification, and discard the corresponding data item in response to failure to perform anti-duplicate verification.

[0186] In some embodiments, the data item writing unit 605 is further configured to write each data item to a data table configured with a uniqueness strategy in the corresponding target node based on the connection pool.

[0187] It should be noted that the data processing method and data processing device in this application are related in specific implementation, so repeated content will not be described again.

[0188] Figure 7 An exemplary system architecture 700 is shown that can be applied to the data processing method or data processing apparatus of the embodiments of this application.

[0189] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, and 703, a network 704, and a server 705. Network 704 serves as the medium for providing communication links between terminal devices 701, 702, and 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0190] Users can use terminal devices 701, 702, and 703 to interact with server 705 via network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0191] Terminal devices 701, 702, and 703 can be various electronic devices with data processing screens and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0192] Server 705 can be a server providing various services, such as a backend management server supporting data processing requests submitted by users using terminal devices 701, 702, and 703 (for example only). The backend management server can obtain the data stream and cluster identifier based on the received data processing request; obtain the node data corresponding to the cluster identifier; generate a first hash value based on the node data; calculate the second hash value of the data items contained in the data stream; determine the target node corresponding to each data item in the data stream based on the first hash value, the second hash value, and the hash ring; determine the number of data items to be written for each target node; configure an independent connection pool for each target node based on the number; and write each data item to the corresponding target node based on the connection pool. This improves write performance and throughput when writing data to the cluster, avoids single-point overload, ensures uniform data distribution in the cluster, increases data write speed, and shortens data write latency.

[0193] It should be noted that the data processing method provided in this application embodiment is generally executed by server 705, and correspondingly, the data processing device is generally located in server 705.

[0194] It should be understood that Figure 7 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0195] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing a terminal device according to the embodiments of this application. Figure 8 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0196] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the computer system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0197] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0198] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this application.

[0199] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0200] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0201] The units described in the embodiments of this application can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a hash value generation unit, a target node determination unit, a connection pool configuration unit, and a data item writing unit. The names of these units do not necessarily limit the specific unit itself.

[0202] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to obtain a data stream and a cluster identifier according to a received data processing request; obtain node data corresponding to the cluster identifier; generate a first hash value based on the node data; calculate a second hash value for each data item in the data stream; determine the target node corresponding to each data item in the data stream based on the first hash value, the second hash value, and the hash ring; determine the number of data items to be written for each target node; configure an independent connection pool for each target node based on the number; and write each data item to the corresponding target node based on the connection pool.

[0203] The computer program product of this application includes a computer program that, when executed by a processor, implements the data processing method in the embodiments of this application.

[0204] According to the technical solution of the embodiments of this application, the write performance and throughput when writing data to the cluster can be improved, single point overload can be avoided, the data can be evenly distributed in the cluster, the data write speed can be improved, and the data write latency can be shortened.

[0205] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized by, include: Obtain the data stream and cluster identifier based on the received data processing request; Obtain the node data corresponding to the cluster identifier, and generate a first hash value based on the node data; Calculate the second hash value of the data items contained in the data stream, and determine the target node corresponding to each data item in the data stream based on the first hash value, the second hash value, and the hash ring; Determine the number of data items to be written for each target node, and configure an independent connection pool for each target node based on the number. Each data item is written to the corresponding target node based on the connection pool.

2. The method of claim 1, wherein, Determining the target node corresponding to each data item in the data stream includes: The first position of the node data on the hash ring is determined based on the first hash value; The second position of the data item contained in the data stream on the hash ring is determined based on the second hash value; Based on the first position, the second position, and the preset allocation direction, the target node corresponding to each data item in the data stream is determined.

3. The method of claim 2, wherein, The step of determining the target node corresponding to each data item in the data stream based on the first position, the second position, and the preset allocation direction includes: At each of the second positions, a search is performed according to a preset allocation direction, and the node corresponding to the closest first position is determined as the target node.

4. The method of claim 1, wherein, The step of writing each data item to the corresponding target node based on the connection pool includes: In response to the space occupied by the data items to be written in the memory cache reaching a preset threshold, a batch write operation is performed to write the data items to be written in the memory cache to the corresponding target node based on the connection pool.

5. The method of claim 1, wherein, The step of writing each data item to the corresponding target node based on the connection pool includes: In response to the triggering of a timed task or trigger associated with the memory cache, a batch write operation is performed to write the data items to be written in the memory cache to the corresponding target node based on the connection pool.

6. The method according to any one of claims 1 to 5, characterized in that, The step of writing each data item to the corresponding target node based on the connection pool includes: Generate a unique identifier for each data item, and associate each data item with its corresponding unique identifier to obtain the associated data; When writing each data item to the corresponding target node based on the connection pool, anti-duplicate verification is performed based on the associated data. If the anti-duplicate verification is successful, the corresponding data item is written to the corresponding target node. If the anti-duplicate verification fails, the corresponding data item is discarded.

7. The method according to any one of claims 1 to 5, characterized in that, The step of writing each data item to the corresponding target node based on the connection pool includes: Based on the connection pool, each data item is written to the data table configured using a uniqueness strategy in the corresponding target node.

8. A data processing apparatus, characterized by, include: The acquisition unit is configured to acquire the data stream and cluster identifier based on the received data processing request; The hash value generation unit is configured to obtain the node data corresponding to the cluster identifier and generate a first hash value based on the node data; The target node determination unit is configured to calculate the second hash value of the data items contained in the data stream, and determine the target node corresponding to each data item in the data stream based on the first hash value, the second hash value and the hash ring; The connection pool configuration unit is configured to determine the number of data items to be written for each target node, and configure an independent connection pool for each target node according to the number. The data item writing unit is configured to write each data item to the corresponding target node based on the connection pool.

9. A data processing electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer readable medium having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

11. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images