Method for distributing data to system, data processing method and related device

Abstract

The embodiment of the invention discloses a method for distributing data to a system, a data processing method and a related device, which are used for realizing efficient and rapid writing. The method is applied to a client and comprises the steps that a data stream containing multiple pieces of source data is obtained, and each piece of source data has a unique identifier; obtaining a unique identifier of each table node device in a plurality of table node devices included in the distributed database system; according to the unique identifier of each piece of source data and the unique identifier of each piece of table node equipment, determining target table node equipment corresponding to each piece of source data in the multiple pieces of table node equipment; wherein the unique identifier of each piece of source data is mapped and only mapped as the unique identifier of the target table node equipment corresponding to the source data; and grouping the source data groups corresponding to the same target table node equipment into the same data distribution, and distributing the data distribution to the target table node equipment corresponding to the data distribution.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of databases, and in particular to a method for distributing data to a system, a data processing method, and related devices. Background Art

[0002] Clickhouse is a high-performance, MPP-based, column-based OLAP database with complete DBMS functions. It can process large amounts of data, support precise queries and complex aggregation operations, and is used in data warehouses, log analysis, real-time reports, and business intelligence. Exactly once semantics means that during data processing, each piece of data is processed only once, and no more or more data can be processed. In order to promptly handle the duplication of clickhouse data after the processing process recovers from an abnormal failure, it is necessary to ensure the implementation of exactly once semantics, and specifically process the data through idempotent writing. In the existing solution, a single table node device of clickhouse distributes data, that is, the data of the distributed table is randomly distributed to a table node device of a clickhouse, and the table node device distributes the data, and each table node device is equipped with a deduplication engine, which can deduplicate the data in the same table node device, so as to ensure that the data is not repeated.

[0003] However, the existing solution uses a single table node device of ClickHouse to distribute data. In the high-throughput processing flow of big data, a single table node device cannot withstand such a large performance loss and has insufficient distribution capabilities. Summary of the invention

[0004] The embodiments of the present application provide a method for distributing data to a system, a data processing method and related devices, for achieving efficient and fast writing.

[0005] A first aspect of an embodiment of the present application provides a method for distributing data to a distributed database system, which is applied to a client and includes:

[0006] Obtaining a data stream containing multiple source data, wherein each source data has a unique identifier;

[0007] Obtaining a unique identifier of each of the plurality of table node devices included in the distributed database system;

[0008] According to the unique identifier of each source data and the unique identifier of each table node device, determine the target table node device corresponding to each source data in the multiple table node devices; wherein the unique identifier of each source data is mapped and only mapped to the unique identifier of the target table node device corresponding to the source data;

[0009] The source data groups corresponding to the same target table node device are divided into the same data stream, and the data stream is distributed to the target table node device corresponding to the data stream.

[0010] Optionally, if the unique identifier of the source data is a primary key value of a preset primary key, and the unique identifier of the table node device is an index value uniquely corresponding to the table node device index value table, then determining the target table node device corresponding to each source data in the plurality of table node devices according to the unique identifier of each source data and the unique identifier of each table node device includes:

[0011] For each piece of the source data, determine in the table node device index value table the target index value to which the primary key value of the source data is mapped and only mapped, and determine that the table node device corresponding to the target index value is the target table node device corresponding to the source data.

[0012] Optionally, determining in the table node device index value table that the primary key value of the source data is mapped and only mapped to a target index value includes:

[0013] The primary key value is operated based on a preset hash algorithm to obtain a target operation value;

[0014] The target index value is obtained by calculating the target operation value and the maximum index value in the table node device index value table.

[0015] Optionally, the calculating by using the target operation value and the maximum index value in the table node device index value table to obtain the target index value includes:

[0016] Performing a modulus operation on the target operation value and the maximum index value in the table of index values ​​of the table node device to obtain a modulus value, and determining the modulus value as the target index value corresponding to the primary key value; or,

[0017] A modulo operation is performed on the target operation value and the maximum index value in the table node device index value table to obtain a remainder value, and the remainder value is determined as the target index value corresponding to the primary key value.

[0018] Optionally, the source data is stored in a JSON object, and the method further includes:

[0019] The primary key value of the preset primary key is extracted from the JSON object based on a primary key value acquisition function.

[0020] Optionally, distributing the data stream to the target table node device corresponding to the data stream includes:

[0021] For each of the data splits, obtaining a target connection between the client and a target table node device corresponding to the data split from the connections pre-established between the client and each of the table node devices;

[0022] The multiple data streams are distributed in parallel to the target table node devices corresponding to the data streams through their respective corresponding target connections.

[0023] A second aspect of an embodiment of the present application provides a data processing method, which is applied to a table node device of a distributed database system, and the method includes:

[0024] Receive a data stream sent by a client, wherein the data stream includes at least one source data, the source data has a unique identifier, and the unique identifier of each source data is mapped and only mapped to the unique identifier of the table node device receiving the data stream;

[0025] A local table stored in the table node device is obtained, and source data in the local table and source data in the data split are deduplicated, and the deduplicated source data in the data split is stored in the local table.

[0026] A third aspect of an embodiment of the present application provides a data distribution device, including:

[0027] An acquisition unit, configured to obtain a data stream comprising a plurality of source data, wherein the source data has a unique identifier;

[0028] The acquisition unit is further used to obtain a unique identifier of each of the multiple table node devices included in the distributed database system;

[0029] A determination unit, configured to determine, according to the unique identifier of each source data and the unique identifier of each table node device, a target table node device corresponding to each source data in the plurality of table node devices; wherein the unique identifier of each source data is mapped and only mapped to the unique identifier of the target table node device corresponding to the source data;

[0030] The distribution unit is used to group the source data corresponding to the same target table node device into the same data stream, and distribute the data stream to the target table node device corresponding to the data stream.

[0031] A fourth aspect of the embodiments of the present application provides a data processing device, including:

[0032] A receiving unit, configured to receive a data stream sent by a client, wherein the data stream includes at least one source data, the source data has a unique identifier, and the unique identifier of each source data is mapped and only mapped to the unique identifier of the table node device receiving the data stream;

[0033] A deduplication unit is used to obtain a local table stored in the table node device, deduplicate the source data in the local table and the source data in the data split, and store the deduplicated source data in the data split in the local table.

[0034] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the aforementioned method.

[0035] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0036] First, a data stream containing multiple source data is obtained, and then the unique identifier of each table node device in the multiple table node devices included in the distributed database system is obtained. Then, according to the unique identifier of each source data and the unique identifier of each table node device, the target table node device corresponding to each source data is determined in the multiple table node devices. Finally, the source data corresponding to the same target table node device are grouped into the same data diversion, and the data diversion is distributed to the target table node device corresponding to the data diversion. The existing scheme is to use a clickhouse table node device of the distributed database system for distribution. The table node device needs to be responsible for the data distribution and storage tasks of all clients, and the performance consumption is large. The data distribution method of the embodiment of the present application is transferred to the client. The client determines the target table node device to be stored for its own source data and distributes and stores the diversion of the source data to the target table node device, thereby reducing the performance consumption of the distributed database system. The target table node device determined by the client has a unique mapping relationship with the source data, which can ensure the idempotent write capability of the distributed database system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of a data distribution framework disclosed in this application;

[0038] Figure 2 A schematic diagram of an embodiment of a method for distributing data to a distributed database system disclosed in this application;

[0039] Figure 3 This is a schematic diagram of another method embodiment of distributing data to a distributed database system disclosed in this application;

[0040] Figure 4 A schematic diagram of an embodiment of the data processing method disclosed in this application;

[0041] Figure 5 A schematic diagram of a process of data distribution and data processing disclosed in this application;

[0042] Figure 6 A schematic diagram of an embodiment of a data distribution device disclosed in the present application;

[0043] Figure 7 This is a schematic diagram of another embodiment of the data distribution device disclosed in this application. DETAILED DESCRIPTION

[0044] The present application is further described in detail below in conjunction with the accompanying drawings.

[0045] The embodiments of the present application provide a method for distributing data to a system, a data processing method and related devices, for achieving efficient and fast writing.

[0046] Clickhouse is an open source columnar database that can store large amounts of data in columns and supports online analytical processing, complex aggregation operations and other functions. In order to support the transmission of streaming data to the Clickhouse columnar database, a special connector (clickhousesink connector) that connects the device and the Clickhouse columnar database has been designed to process massive amounts of data, transfer data from the device to the Clickhouse columnar database, and ensure data availability and integrity. The Flink CDC (Flink Change Data Capture) computing engine development framework has also been designed. The Flink computing engine development framework is used to capture database change logs, which can obtain source data in real time and convert it into streaming data, and then transmit the streaming data to various specified storage components (such as Clickhouse, Elasticsearch, Hive, etc.), with source (source data acquisition) operators, transform (transformation processing) operators and sink (transmission landing) operators.

[0047] At present, the ClickHouse sink connector and the Flink computing engine development framework can be used to transfer and store source data into the ClickHouse column database, realizing at least once semantics, where at least once means that an operation is performed at least once when processing data. Repeated operations may occur, but the integrity and accuracy of the data can be ensured, and data will not be lost. However, when encountering abnormal failures, the ClickHouse column database will have data duplication problems and cannot achieve exactly once semantics, where exactly once means that each piece of data is only processed once during the data processing process. In order to achieve exactly once semantics, the existing solution uses idempotent writes to process, where idempotent writes mean that an operation can be repeated multiple times, but no matter how many times it is executed, the final result is the same. The existing solution will randomly assign a table node device to each piece of data with the same identifier, and a single table node device in the clickhouse column database will perform data distribution, and each table node device will have a deduplication engine that can deduplicate data on the same node, thus ensuring that the data is not repeated, wherein the deduplication engine can deduplicate data with the same identifier and can delete duplicate data when merging partitions, such as the ReplicatedReplacingMergeTree engine. However, the existing solution only relies on a single table node device in the clickhouse column database to distribute data, and the table node device cannot withstand large performance losses and has insufficient distribution capabilities. In order to solve the above problems, the present application provides a method for distributing data to a distributed database system, a data processing method, and related devices, which utilize a client to distribute data to each table node device in the clickhouse column database. Since the client can distribute at the same time, while a single table node device cannot distribute at the same time, the client has sufficient distribution capabilities in a short period of time and can withstand large performance losses. Please refer to Figure 1 , the client can distribute data to each table node device in the clickhouse column database at the same time. This application allocates the table node device corresponding to the data based on the unique identifier, and then cooperates with the deduplication engine owned by each table node device in each clickhouse column database, which can not only achieve efficient and fast multi-concurrency writing, but also realize excatly once semantics, ensuring the accuracy of online query business results.

[0048] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0049] The terms "first", "second", "third", "fourth", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0050] The following describes a method for distributing data to a distributed database system in the present application. Figure 2 , an embodiment of a method for distributing data to a distributed database system of the present application is applied to a client, and the method includes:

[0051] 201. Obtain a data stream containing multiple source data, wherein each source data has a unique identifier;

[0052] A data stream containing multiple source data is obtained, wherein each source data has a unique identifier. This embodiment can implement the following steps based on various suitable computing engine development frameworks, and can be implemented through the flink computing engine development framework or the spark computing engine development framework. It can be understood that the steps of this embodiment can also be performed without using the computing engine development framework. The clickhouse column database can be directly connected to the client, for example, using the pre-developed and designed function BalancedClickhouseDataSource, etc., which is not specifically limited here. In this embodiment, the flink computing engine development framework is used for illustration. The database of this embodiment can be clickhouse, elasticsearch, or hive, etc., which is not specifically limited here. This embodiment is illustrated by clickhouse. Among them, the source data is generally the target category data in the distributed publish-subscribe message system, such as kafkatopic data, etc., and can also be the data obtained by the target website, which is not specifically limited here. Specifically, in the flink computing engine development framework, relevant functions can be pre-designed, such as generating a kafka data stream through the StreamExecutionEnvironment.addSource(FlinkKafkaConsumer) function, and each source data has a unique identifier.

[0053] 202. Obtain a unique identifier of each table node device among a plurality of table node devices included in the distributed database system;

[0054] Get the unique identifier of each table node device in the multiple table node devices included in the distributed database system. Specifically, get the unique identifiers of multiple table node devices in the clickhouse column database. Among them, the table node device is the physical node device where the local table is located. Generally, a physical node device has one local table, and the local table is the physical storage table of the data. Compared with the distributed table, the local table is only stored on a single table node device, rather than distributed throughout the cluster. Specifically, the unique identifier of the table node device can be an index value or a unique device number, etc., which is not limited here and can be obtained according to actual conditions.

[0055] 203. Determine, according to the unique identifier of each source data and the unique identifier of each table node device, a target table node device corresponding to each source data from among the multiple table node devices;

[0056] According to the unique identifier of each source data and the unique identifier of each table node device, the target table node device corresponding to each source data is determined in multiple table node devices, wherein the unique identifier of each source data is mapped and only mapped to the unique identifier of the target table node device corresponding to the source data. Simply put, the one-to-one correspondence between the unique identifier of the source data and the unique identifier of the target table node device is pre-set, and the corresponding target table node device can be obtained by analyzing and querying the unique identifier of the source data. It should be noted that this one-to-one correspondence can ensure that the same source data (such as the same primary key value indicates that the source data is the same source data) will only be mapped to the target table node device, and will not be mapped to other table node devices, thereby avoiding the same source data being mapped to different table node devices for storage, resulting in repeated storage of source data (because this does not meet the requirements for the idempotent write capability of the distributed database system). This step can be specifically implemented by the following mapping algorithm, that is, the mapping algorithm can ensure this one-to-one correspondence between the source data and the target table node device.

[0057] 204. Group the source data corresponding to the same target table node device into the same data stream, and distribute the data stream to the target table node device corresponding to the data stream.

[0058] The source data corresponding to the same target table node device are grouped into the same data diversion, and the data diversion is distributed to the target table node device corresponding to the data diversion. Specifically, the source data of the same target table node device is marked as the diversion of the corresponding target table node device, and the source data of the same target table node device is assembled in batches to obtain the same data diversion, and the data diversion is distributed to the corresponding target table node device. Among them, each data diversion is executed by an execution unit task of the flink computing engine development framework, so that multiple tasks can simultaneously execute the task of transmitting the corresponding data diversion to the target table node device.

[0059] In an embodiment of the present application, a data stream containing multiple source data is first obtained, and then the unique identifier of each table node device in the multiple table node devices included in the distributed database system is obtained, and then the target table node device corresponding to each source data is determined in the multiple table node devices according to the unique identifier of each source data and the unique identifier of each table node device, and finally the source data corresponding to the same target table node device are grouped into the same data diversion, and the data diversion is distributed to the target table node device corresponding to the data diversion. The present application uses the client for distribution instead of using one of the table node devices of the clickhouse column database for distribution. The client executes the data distribution method without the need for a single table node device to perform the execution, which reduces the performance consumption of the system. The client can distribute to multiple table node devices at the same time, while a single table node device cannot distribute at the same time. The client can withstand a large performance consumption in a short time, and the distribution capacity is sufficient.

[0060] It is understandable that the unique identifier of the source data can be a primary key value or other values, which are not specifically limited here. The unique identifier of the table node device can be an index value or other values, which are not specifically limited here. For the convenience of description, the unique identifier of the source data in this embodiment is the primary key value of the preset primary key, and the unique identifier of the table node device is the unique corresponding index value in the table of table node device index values. This application can be executed using languages ​​such as Java, Scala, Python, etc., which are not specifically limited here. This embodiment uses Java as an example, and the source data is stored in a JSON object. Please refer to Figure 3 Another embodiment of a method for distributing data to a distributed database system of the present application is applied to a client, and the method includes:

[0061] 301. Obtain a data stream containing multiple source data, wherein each source data has a unique identifier;

[0062] A data stream containing multiple source data is obtained, wherein each source data has a unique identifier. This embodiment can implement the following steps based on various suitable computing engine development frameworks, and can be implemented through the flink computing engine development framework or the spark computing engine development framework. It can be understood that the steps of this embodiment can also be performed without using the computing engine development framework. The clickhouse column database can be directly connected to the client, for example, using the pre-developed and designed function BalancedClickhouseDataSource, etc., which is not specifically limited here. In this embodiment, the flink computing engine development framework is used for illustration. The database of this embodiment can be clickhouse, elasticsearch or hive, etc., which is not specifically limited here. This embodiment is illustrated by clickhouse. Among them, the source data is generally the target category data in the distributed publish-subscribe message system, such as kafkatopic data, etc., and can also be the data obtained by the target website, which is not specifically limited here. Specifically, in the flink computing engine development framework, relevant functions can be pre-designed, such as generating a kafka data stream through the StreamExecutionEnvironment.addSource(FlinkKafkaConsumer) function, and each source data has a unique identifier, that is, the unique primary key value of the preset primary key. The preset primary key can be set according to actual needs, and there is no specific limitation here. For example, there are three key-value pairs in the source data, the "ID" value is "1", the "Name" value is "Zhang San", and the "Age" value is "18". If the preset primary key is "ID", the primary key value of the source data is 1.

[0063] 302. Obtain a unique identifier of each table node device among a plurality of table node devices included in the distributed database system;

[0064] Obtain the unique identifier of each table node device in the multiple table node devices included in the distributed database system. Specifically, obtain the unique identifiers of multiple table node devices in the clickhouse column database. Among them, the table node device is the physical node device where the local table is located. Generally, a physical node device has one local table, and the local table is the physical storage table of the data. Compared with the distributed table, the local table is only stored on a single table node device, rather than distributed throughout the cluster. Specifically, the unique identifier of the table node device is the unique corresponding index value in the table node device index value table. The table node device index table is pre-set and stores the index values ​​of the table node devices of the surviving clickhouse column database. Each table node device corresponds to only one index value. The table node device index value table is stored in the system configuration, and the table node device index table can be retrieved from the system configuration. Alternatively, another device sends the table node device index value table to the client, and the client receives and uses it.

[0065] 303. Extracting the primary key value of the preset primary key from the JSON object based on the primary key value acquisition function;

[0066] Extract the primary key value of the preset primary key from the JSON object based on the primary key value acquisition function. The source data includes multiple JSON objects, each of which has at least one key-value pair. Specifically, you can use the primary key value acquisition function, such as the jsonobject.getstring() function. The preset primary key is the selected identification field in the data, which can be selected according to actual needs, and is not limited here. Parse the source data, that is, query the preset primary key from the JSON object, and obtain the value corresponding to the preset primary key to obtain the primary key value.

[0067] 304. For each source data, determine in the table node device index value table the target index value to which the primary key value of the source data is mapped and only mapped, and determine that the table node device corresponding to the target index value is the target table node device corresponding to the source data;

[0068] For each source data, determine in the table node device index value table that the primary key value of the source data is mapped and only mapped to the target index value, and determine that the table node device corresponding to the target index value is the target table node device corresponding to the source data, wherein the primary key value of the preset primary key of each source data is mapped and only mapped to the unique target index value in the table node device index table corresponding to the source data. In simple terms, the primary key value of each source data corresponds to the target index value one-to-one, and a unique target index value can be determined based on the primary key value, and the table node device corresponding to the target index value is the target table node device. Specifically, the primary key value is first calculated based on a preset hash algorithm to obtain a target calculation value, and then the target index value is calculated using the target calculation value and the maximum index value in the table node device index value table. There are many preset hash algorithms, such as MurMurHash algorithm or cityhash64 algorithm, etc., which are not specifically limited here. This embodiment takes cityhash64 algorithm as an example. Cityhash64 algorithm can generate fixed-length hash values ​​and variable-length hash values. Compared with many other hash algorithms, cityhash64 has higher hash performance and lower conflict rate, and has distributed hash consistency and strong random uniform distribution capabilities. Cityhash64 can be used to perform multiplication and bit operations on the primary key value to obtain the corresponding hash value, that is, the target operation value, and then calculate according to the target operation value and the maximum index value in the table node device index value table. Specifically, in one embodiment, a modulus operation is performed on the target operation value and the maximum index value in the table node device index value table to obtain a modulus value, and the modulus value is determined as the target index value corresponding to the primary key value. For example, the target operation value corresponding to the primary key value of a certain source data is 8, there are only three table node devices in the clickhouse column database, and the index values ​​in the table node device index value table correspond to 1, 2, and 3, respectively. 8 is modulo the maximum index value 3 in the table node device index value table to obtain 2, and the source data is distributed to the second table node device. In another embodiment, a remainder operation is performed on the target operation value and the maximum index value in the table node device index value table to obtain a remainder value, and the remainder value is determined as the target index value corresponding to the primary key value. For example, the target operation value corresponding to the primary key value of a certain source data is 8, there are only five table node devices in the clickhouse column database, and the index values ​​in the table node device index value table correspond to 1, 2, 3, 4, and 5, respectively. 8 is divided by the maximum index value 5 in the table node device index value table, and the remainder obtained is 3, and the source data is distributed to the third table node device.

[0069] 305. Group the source data corresponding to the same target table node device into the same data stream, and distribute the data stream to the target table node device corresponding to the data stream.

[0070] The source data corresponding to the same target table node device are grouped into the same data diversion, and the data diversion is distributed to the target table node device corresponding to the data diversion. Simply put, the source data of the same target table node device is marked as the diversion of the corresponding target table node device, and the source data of the same target table node device is assembled in batches to obtain the same data diversion, and the data diversion is distributed to the corresponding target table node device. Among them, each data diversion is executed by an execution unit task of the flink computing engine development framework, so that multiple tasks can simultaneously execute the task of transmitting the corresponding data diversion to the target table node device. Specifically, for each data diversion, the target connection between the client and the target table node device corresponding to the data diversion is obtained in the pre-established connection between the client and each table node device, and then multiple data diversions are distributed in parallel to the target table node device corresponding to the data diversion through their respective corresponding target connections. Simply put, the source data with the same primary key value are grouped together and assembled into the same data stream in batches. Based on the clickhousesink connector, a connection is established between the client and each table node device. Then, the target table node device corresponding to the data stream is determined. Then, based on multiple tasks, multiple data streams are distributed to multiple target table node devices in parallel and stored in the corresponding local table.

[0071] In this embodiment, the application uses the client to distribute, rather than using one of the clickhouse column database table node devices to distribute. The client executes the data distribution method without the need for a single table node device to perform the execution, thereby reducing the performance consumption of the system. The client can distribute to multiple table node devices at the same time, while a single table node device cannot distribute at the same time. The client can withstand greater performance consumption in a short period of time, and the distribution capacity is sufficient.

[0072] A method for distributing data to a distributed database system in an embodiment of the present application is described above. A method for processing data in an embodiment of the present application is described below. Figure 4 , a data processing method in an embodiment of the present application is applied to a table node device of a distributed database system, and the method includes:

[0073] 401. Receive data diversion sent by the client;

[0074] Receive a data diversion sent by a client, wherein the data diversion includes at least one source data, the source data has a unique identifier, and the unique identifier of each source data is mapped and only mapped to the unique identifier of the table node device receiving the data diversion. Specifically, each table node device will receive the data diversion corresponding to it, and the unique identifiers of the source data in the data diversion are the same. Preferably, the unique identifier of the source data can be the primary key value of a preset primary key, and the unique identifier of the table node device can be the unique corresponding index value in the table node device index value table.

[0075] 402. Obtain a local table stored in a table node device, deduplicate source data in the local table and source data in the data split, and store the deduplicated source data in the data split in the local table.

[0076] The local table stored in the table node device is obtained, and the source data in the local table and the source data in the data diversion are deduplicated, and the deduplicated source data in the data diversion is stored in the local table. Specifically, the table node device is provided with a deduplication engine, such as a ReplicatedReplacingMergeTree engine, which deduplicates data with the same primary key value, deletes duplicate data, and stores the deduplicated data for subsequent use.

[0077] In the embodiment of the present application, after receiving the data diversion sent by the client, the local table stored in the table node device is obtained, and the source data in the local table and the source data in the data diversion are deduplicated, and the deduplicated source data in the data diversion is stored in the local table. The deduplication engine is used to ensure that there is no duplicate data in the local table of the same table node device. Combined with the data distribution method of the above embodiment, the excatly once semantics can be realized to ensure the accuracy and reliability of the data.

[0078] Please refer to the data distribution and data processing process Figure 5 , use the Flink computing engine development framework and ClickHouse sink connector to implement client computing processing and distribution operations. The table node devices in the ClickHouse column database receive data diversion and use the deduplication engine to deduplicate, realizing exactly once semantics.

[0079] The following is a detailed description of a data distribution device in an embodiment of the present application. Figure 6 , a data distribution device according to an embodiment of the present application includes:

[0080] An acquisition unit 601 is used to obtain a data stream containing multiple source data, wherein the source data has a unique identifier;

[0081] The acquisition unit 601 is further configured to obtain a unique identifier of each of the table node devices in the plurality of table node devices included in the distributed database system;

[0082] A determination unit 602 is used to determine, according to the unique identifier of each source data and the unique identifier of each table node device, the target table node device corresponding to each source data in the plurality of table node devices; wherein the unique identifier of each source data is mapped and only mapped to the unique identifier of the target table node device corresponding to the source data;

[0083] The distribution unit 603 is used to group the source data corresponding to the same target table node device into the same data stream, and distribute the data stream to the target table node device corresponding to the data stream.

[0084] In the embodiment of the present application, the acquisition unit 601 first obtains a data stream containing multiple source data, and then obtains the unique identifier of each table node device in the multiple table node devices included in the distributed database system. Then the determination unit 602 determines the target table node device corresponding to each source data in the multiple table node devices according to the unique identifier of each source data and the unique identifier of each table node device. Finally, the distribution unit 603 groups the source data corresponding to the same target table node device into the same data stream, and distributes the data stream to the target table node device corresponding to the data stream. The present application uses the client for distribution instead of using one of the table node devices of the clickhouse column database for distribution. The client can distribute to multiple node devices at the same time, while a single table node device cannot. The client has stronger performance than the table node device, can withstand greater performance consumption, and has sufficient distribution capabilities.

[0085] The data distribution device of the present application is described in detail below. Another embodiment of the data distribution device of the present application includes:

[0086] An acquisition unit, configured to obtain a data stream comprising a plurality of source data, wherein the source data has a unique identifier;

[0087] The acquisition unit is further used to obtain a unique identifier of each of the multiple table node devices included in the distributed database system;

[0088] A determination unit, configured to determine, according to the unique identifier of each source data and the unique identifier of each table node device, a target table node device corresponding to each source data in the plurality of table node devices; wherein the unique identifier of each source data is mapped and only mapped to the unique identifier of the target table node device corresponding to the source data;

[0089] The distribution unit is used to group the source data corresponding to the same target table node device into the same data stream, and distribute the data stream to the target table node device corresponding to the data stream.

[0090] If the unique identifier of the source data is the primary key value of the preset primary key, and the unique identifier of the table node device is the unique corresponding index value in the table node device index value table, the determination unit is specifically used to:

[0091] For each piece of the source data, determine in the table node device index value table the target index value to which the primary key value of the source data is mapped and only mapped, and determine that the table node device corresponding to the target index value is the target table node device corresponding to the source data.

[0092] Determine the unit, specifically for:

[0093] The primary key value is operated based on a preset hash algorithm to obtain a target operation value;

[0094] The target index value is obtained by calculating the target operation value and the maximum index value in the table node device index value table.

[0095] Determine the unit, specifically for:

[0096] Performing a modulus operation on the target operation value and the maximum index value in the table of index values ​​of the table node device to obtain a modulus value, and determining the modulus value as the target index value corresponding to the primary key value; or,

[0097] A modulo operation is performed on the target operation value and the maximum index value in the table node device index value table to obtain a remainder value, and the remainder value is determined as the target index value corresponding to the primary key value.

[0098] The source data is stored in a JSON object. The data distribution device further includes an extraction unit 604, which is specifically used to:

[0099] The primary key value of the preset primary key is extracted from the JSON object based on a primary key value acquisition function.

[0100] Distribution unit, specifically used for:

[0101] For each of the data splits, obtaining a target connection between the client and a target table node device corresponding to the data split from the connections pre-established between the client and each of the table node devices;

[0102] The multiple data streams are distributed in parallel to the target table node devices corresponding to the data streams through their respective corresponding target connections.

[0103] The functions and processes performed by each unit in the data distribution device of this embodiment are similar to those described above. Figures 2 to 3 The functions and processes performed by the data distribution device are similar and will not be repeated here.

[0104] Figure 7 It is a structural schematic diagram of a data distribution device provided in an embodiment of the present application. The data distribution device 700 may include one or more central processing units (CPU) 701 and a memory 705. The memory 705 stores one or more application programs or data.

[0105] The memory 705 may be a volatile storage or a persistent storage. The program stored in the memory 705 may include one or more modules, each of which may include a series of instruction operations in the data distribution device 700. Furthermore, the central processor 701 may be configured to communicate with the memory 705 and execute a series of instruction operations in the memory 705 on the data distribution device 700.

[0106] The data distribution device 700 may also include one or more power supplies 702, one or more wired or wireless network interfaces 703, one or more input and output interfaces 704, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0107] The CPU 701 can execute the aforementioned Figures 2 to 3 The operations performed by the data distribution device in the illustrated embodiment will not be described in detail here.

[0108] The data distribution device of the present application is described above, and the data processing device of the present application is described below. An embodiment of the data processing device of the present application includes:

[0109] A receiving unit, configured to receive a data stream sent by a client, wherein the data stream includes at least one source data, the source data has a unique identifier, and the unique identifier of each source data is mapped and only mapped to the unique identifier of the table node device receiving the data stream;

[0110] A deduplication unit is used to obtain a local table stored in the table node device, deduplicate the source data in the local table and the source data in the data split, and store the deduplicated source data in the data split in the local table.

[0111] In the embodiment of the present application, after the receiving unit receives the data diversion sent by the client, the deduplication unit obtains the local table stored in the table node device, and deduplicates the source data in the local table and the source data in the data diversion, and stores the deduplicated source data in the data diversion in the local table. The deduplication engine is used to ensure that there will be no duplicate data in the local table of the same table node device. Combined with the data distribution method of the above embodiment, the excatly once semantics can be realized to ensure the accuracy and reliability of the data.

[0112] An embodiment of the present application provides a data processing device, which may include one or more central processing units (CPU) and a memory, wherein the memory stores one or more application programs or data.

[0113] The memory may be a volatile memory or a persistent memory. The program stored in the memory may include one or more modules, each of which may include a series of instruction operations in the data processing device. Furthermore, the central processing unit may be configured to communicate with the memory and execute a series of instruction operations in the memory on the data processing device.

[0114] The data processing device may also include one or more power supplies, one or more wired or wireless network interfaces, one or more input and output interfaces, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0115] The CPU can execute the aforementioned Figure 4 The operations performed by the data processing device in the illustrated embodiment will not be described in detail herein.

[0116] An embodiment of the present application also provides a computer-readable storage medium, including instructions, which, when executed on a computer, enable the computer to execute the method in the aforementioned embodiment.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0118] It should be noted that, although the steps in the flowcharts involved in the embodiments are drawn in sequence according to the indications of the arrows, unless otherwise clearly stated in this document, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0119] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0122] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A method for distributing data to a distributed database system, characterized in that: Applied to the client, including: Obtaining a data stream containing multiple source data, wherein each source data has a unique identifier; Obtaining a unique identifier of each of the plurality of table node devices included in the distributed database system; According to the unique identifier of each source data and the unique identifier of each table node device, determine the target table node device corresponding to each source data in the multiple table node devices; wherein the unique identifier of each source data is mapped and only mapped to the unique identifier of the target table node device corresponding to the source data; The source data groups corresponding to the same target table node device are divided into the same data stream, and the data stream is distributed to the target table node device corresponding to the data stream.

2. The method for distributing data to a distributed database system according to claim 1, characterized in that: If the unique identifier of the source data is a primary key value of a preset primary key, and the unique identifier of the table node device is a uniquely corresponding index value in the table node device index value table, then determining the target table node device corresponding to each source data in the plurality of table node devices according to the unique identifier of each source data and the unique identifier of each table node device includes: For each piece of the source data, determine in the table node device index value table the target index value to which the primary key value of the source data is mapped and only mapped, and determine that the table node device corresponding to the target index value is the target table node device corresponding to the source data.

3. The method for distributing data to a distributed database system according to claim 2, characterized in that: Determining in the table node device index value table that the primary key value of the source data is mapped and only mapped to the target index value includes: The primary key value is operated based on a preset hash algorithm to obtain a target operation value; The target index value is obtained by calculating the target operation value and the maximum index value in the table node device index value table.

4. The method for distributing data to a distributed database system according to claim 3, characterized in that: The target index value is obtained by calculating the target operation value and the maximum index value in the table node device index value table, including: Performing a modulus operation on the target operation value and the maximum index value in the table of index values ​​of the table node device to obtain a modulus value, and determining the modulus value as the target index value corresponding to the primary key value; or, A modulo operation is performed on the target operation value and the maximum index value in the table node device index value table to obtain a remainder value, and the remainder value is determined as the target index value corresponding to the primary key value.

5. The method for distributing data to a distributed database system according to claim 2, characterized in that: The source data is stored in a JSON object, and the method further includes: The primary key value of the preset primary key is extracted from the JSON object based on a primary key value acquisition function.

6. The method for distributing data to a distributed database system according to claim 1, characterized in that: The distributing the data stream to the target table node device corresponding to the data stream includes: For each of the data splits, obtaining a target connection between the client and a target table node device corresponding to the data split from the connections pre-established between the client and each of the table node devices; The multiple data streams are distributed in parallel to the target table node devices corresponding to the data streams through their respective corresponding target connections.

7. A data processing method, characterized in that: A table node device applied to a distributed database system, the method comprising: Receive a data stream sent by a client, wherein the data stream includes at least one source data, the source data has a unique identifier, and the unique identifier of each source data is mapped and only mapped to the unique identifier of the table node device receiving the data stream; A local table stored in the table node device is obtained, and source data in the local table and source data in the data split are deduplicated, and the deduplicated source data in the data split is stored in the local table.

8. A data distribution device, characterized in that: include: An acquisition unit, configured to obtain a data stream comprising a plurality of source data, wherein the source data has a unique identifier; The acquisition unit is further used to obtain a unique identifier of each of the multiple table node devices included in the distributed database system; A determination unit, configured to determine, according to a unique identifier of each piece of the source data and a unique identifier of each of the table node devices, a target table node device corresponding to each piece of the source data in the plurality of table node devices; The unique identifier of each source data is mapped and only mapped to the unique identifier of the target table node device corresponding to the source data; The distribution unit is used to group the source data corresponding to the same target table node device into the same data stream, and distribute the data stream to the target table node device corresponding to the data stream.

9. A data processing device, characterized in that: include: A receiving unit, configured to receive a data stream sent by a client, wherein the data stream includes at least one source data, the source data has a unique identifier, and the unique identifier of each source data is mapped and only mapped to the unique identifier of the table node device receiving the data stream; A deduplication unit is used to obtain a local table stored in the table node device, deduplicate the source data in the local table and the source data in the data split, and store the deduplicated source data in the data split in the local table.

10. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 7.

Images