Data writing method and device for column database, electronic equipment and readable medium

Abstract

The embodiment of the invention provides a data writing method and device of a column database, electronic equipment and a readable medium, the data writing method and device are applied to a data flow execution engine, and the method comprises the steps that connection with at least one node in the column database is established; acquiring data to be written into the column database; aiming at a value of a preset field in the data, calculating by adopting a Hash algorithm to obtain a Hash value; based on the hash value, selecting a target node corresponding to the data from the nodes; and writing the data into the target node. The data writing of the column database is taken over through the data stream execution engine, and the target node to which the data should be written is quickly determined by identifying the preset field hash value in the data. Therefore, the data with the same preset field is always written into the same node, and the problem of data hopping can be effectively avoided.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for writing data into a column-based database, a device for writing data into a column-based database, an electronic device, and a computer-readable medium. Background Art

[0002] In the prior art, column-based databases are usually used for big data analysis scenarios. Column-based databases are databases that store data in a column-related storage architecture, which are suitable for batch data processing and instant queries, so that they can have faster response efficiency during data analysis and improve data processing efficiency. Column-based databases can usually be ClickHouse, Hbase, etc.

[0003] For column-based databases, the same data may be written to different nodes during the data writing process. As a result, the data corresponding to different update times may exist in different nodes for the same data. If the data needs to be overwritten and updated again, the query results of the data to be overwritten are inconsistent, resulting in data jumps. This problem can easily lead to incorrect data being obtained during subsequent data analysis. Summary of the invention

[0004] The embodiment of the present invention provides a method, device, electronic device and computer-readable storage medium for writing data into a column-based database to solve the problem of data jump in a column-based database.

[0005] An embodiment of the present invention discloses a method for writing data into a column-based database, which is applied to a data flow execution engine. The method includes:

[0006] Establishing a connection with at least one node in the column-based database;

[0007] Acquire data to be written into the column-based database;

[0008] A hash value is calculated using a hash algorithm for the value of a preset field in the data;

[0009] Based on the hash value, selecting a target node corresponding to the data from among the nodes;

[0010] The data is written into the target node.

[0011] An embodiment of the present invention further provides a data writing device for a column-based database, which is applied to a data flow execution engine, and the device includes:

[0012] A node connection establishment module, used to establish a connection with at least one node in the column database;

[0013] A data acquisition module, used to acquire data to be written into the column-based database;

[0014] A hash calculation module, used to calculate a hash value using a hash algorithm for a value of a preset field in the data;

[0015] A target node determination module, configured to select a target node corresponding to the data from among the nodes based on the hash value;

[0016] The target node writing module is used to write the data into the target node.

[0017] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0018] The memory is used to store computer programs;

[0019] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0020] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0021] The embodiment of the present invention further discloses a computer program product, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the steps of the method described in the embodiment of the present invention are implemented.

[0022] The embodiments of the present invention include the following advantages:

[0023] Through the data writing method of the column-based database provided by the embodiment of the present invention, a connection with at least one node in the column-based database is established; the data to be written into the column-based database is obtained; for the value of the preset field in the data, a hash value is calculated using a hash algorithm; based on the hash value, a target node corresponding to the data is selected from the node; and the data is written into the target node. The data writing to the column-based database is taken over by the data flow execution engine, and the target node to which the data should be written is quickly determined by identifying the hash value of the preset field in the data. Therefore, data with the same preset field is always written to the same node, which can effectively avoid the problem of data jump. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flowchart of the steps of a method for writing data into a column-based database provided in an embodiment of the present invention;

[0025] Figure 2 is a flowchart of the steps of another method for writing data into a column-based database provided in an embodiment of the present invention;

[0026] Figure 3 It is a structural block diagram of a data writing device for a column-based database provided in an embodiment of the present invention;

[0027] Figure 4 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0028] Figure 5 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Reference Figure 1 , shows a flowchart of a method for writing data into a column-based database provided in an embodiment of the present invention, which may specifically include the following steps:

[0031] Step 101, establishing a connection with at least one node in a column-based database;

[0032] In an embodiment of the present invention, in order to avoid the data jump problem caused by random writing in a column-based database, especially ClickHouse, a data flow execution engine is introduced to enable it to take over the data writing of the column-based database, so that the writing mode of the column-based database can be changed through the data flow execution engine to avoid the data jump problem. Among them, the data flow execution engine can be Flink.

[0033] Specifically, the column-based database can also be a distributed database, so that it can be managed by a cluster, and the cluster can have multiple nodes for storing data. In order to allow the data flow execution engine to take over the data access of the column-based database, a connection between the data flow execution engine and at least one node in the column-based database can be established first, and the data flow execution engine can subsequently write information to the node through the connection.

[0034] Step 102, obtaining data to be written into the column-based database;

[0035] In an embodiment of the present invention, in the process of a network service provider providing users with various services such as video playback, product purchase, news browsing, etc., the data in the website can be continuously updated. For example, a user clicks to purchase a product to generate a new order, a user clicks to browse a video to generate a new browsing record, and the network service provider uploads new videos, news, products, etc. on the Internet, which can all generate new data. In order to facilitate subsequent data analysis, data with analytical value can be stored in a columnar database. Thereby, data to be written to the columnar database can be generated. In the case where the data flow execution engine takes over the data writing of the columnar database, the data flow execution engine can obtain the data to be written to the columnar database.

[0036] Step 103, using a hash algorithm to calculate a hash value for a value of a preset field in the data;

[0037] Thereafter, data with the same value of the preset field may be always stored in the same node, so as to avoid data jump problems caused by storing data with the same value of the preset field in different nodes.

[0038] The preset fields may be determined according to actual needs. For example, when recording in a key-value format, at least one key field may be used as a preset field, and when the values ​​of the key fields are the same, the data is stored in the same node.

[0039] For the data flow execution engine, in order to quickly and accurately identify the nodes where different data should be stored, it can use a hash algorithm to calculate a hash value for the value of a preset field in the data, and then determine the node where the data to be written should be stored based on the hash value.

[0040] Specifically, a hash value is a fixed-length digital value used to uniquely identify data. When the value of a preset field remains unchanged, the hash value can be fixed. At the same time, the hash value can represent a large amount of data with a much smaller digital value, so that the data flow execution engine can more efficiently determine the target node corresponding to the data.

[0041] Step 104, based on the hash value, selecting a target node corresponding to the data from the nodes;

[0042] The node to which the data to be written should be written can be determined based on the hash value. In the case where the hash values ​​are the same, the nodes to which the data should be written can be the same.

[0043] In a specific implementation, the corresponding relationship between the hash value and the node can be preset, and the target node corresponding to the data can be determined based on the corresponding relationship between the two. The target node can also be determined by taking the number at a specific position in the hash value and according to the corresponding relationship between the number at the specific position and the node. A numerical value can also be calculated by adding and taking a modulus operation, and the target node can be determined based on the corresponding relationship between the numerical word and the node, and the present invention does not limit this.

[0044] Step 105: write the data into the target node.

[0045] After determining the target node corresponding to the data, the data can be written into the target node, thereby completing the data writing. Data with the same value of the preset field is always written into the same node, which can effectively avoid the problem of data jump.

[0046] Through the data writing method of the column-based database provided by the embodiment of the present invention, a connection with at least one node in the column-based database is established; the data to be written into the column-based database is obtained; for the value of the preset field in the data, a hash value is calculated using a hash algorithm; based on the hash value, a target node corresponding to the data is selected from the node; and the data is written into the target node. The data writing to the column-based database is taken over by the data flow execution engine, and the target node to which the data should be written is quickly determined by identifying the hash value of the preset field in the data. Therefore, data with the same preset field is always written to the same node, which can effectively avoid the problem of data jump.

[0047] Reference Figure 2 , shows a flowchart of a method for writing data into a column-based database provided in an embodiment of the present invention, which may specifically include the following steps:

[0048] Step 201, establishing a connection with at least one node in a column-based database;

[0049] In an embodiment of the present invention, in order to avoid the data jump problem caused by random writing in a column-based database, especially ClickHouse, a data flow execution engine is introduced to enable it to take over the data writing of the column-based database, so that the writing mode of the column-based database can be changed through the data flow execution engine to avoid the data jump problem. Among them, the data flow execution engine can be Flink.

[0050] Specifically, the column-based database can also be a distributed database, so that it can be managed by a cluster, and the cluster can have multiple nodes for storing data. In order to allow the data flow execution engine to take over the data access of the column-based database, a connection between the data flow execution engine and at least one node in the column-based database can be established first, and the data flow execution engine can subsequently write information to the node through the connection.

[0051] In one embodiment of the present invention, the step of establishing a connection with at least one node in the column-based database includes:

[0052] S11, obtaining node configuration information of the column database;

[0053] In a specific implementation, in order to establish a connection with a column-based database, node configuration information of the column-based database may be first obtained. Exemplarily, the node configuration information may be Java Database Connectivity (JDBC), which may at least record the mapping port information (Hostport) of the node.

[0054] S12: Use the node configuration information to establish a connection with at least one node in the column database.

[0055] Thereafter, the data flow execution engine may use the node configuration information to access the nodes in the column database and establish a connection with the nodes. Specifically, when the node configuration information is mapping port information, the data flow execution engine may access and establish a connection with the nodes through the mapping port.

[0056] In one embodiment of the present invention, the method further comprises:

[0057] S21, extracting data writing configuration information through a dynamic table factory construction method provided by a preset custom column-based database connector, and constructing a dynamic table factory based on the data writing configuration information;

[0058] In order to enable the data flow execution engine to have the ability to connect to and write data to a columnar database, a dynamic table factory (DynamicTableSink) can be constructed through the method provided in the data flow execution engine, which can have the ability to write data in the data flow execution engine.

[0059] Specifically, you can create a custom column-based database connector (clickhouseconnector), and call the dynamic table factory construction method (DynamicTableSinkFactory) in the custom column-based database connector to create a dynamic table factory (createDynamicTableSink). In the process of creating a dynamic table factory, it can inherit the pre-configured data writing configuration information to subsequently write data to the column-based database in the manner specified by the data writing configuration information.

[0060] Specifically, the data writing configuration information may include configuration information such as the number of nodes existing in the column database, the mapping interface information of the nodes, the designated data writing node locations, the number of data written in a single batch, etc., and the present invention does not impose any limitation on this.

[0061] S22, obtaining data writing configuration information in the dynamic table factory, and determining output configuration information for the column-based database; the output configuration information includes node configuration information and / or data writing mode.

[0062] Afterwards, after the dynamic table factory is created, the dynamic table factory can be further inherited to obtain the data writing configuration information in the dynamic table factory, and the output configuration information (OutputFormat) for the column-based database can be defined to further specify how to write data to the column-based database. In this way, the data writing configuration information in the dynamic table factory can be obtained, and the output configuration information for the column-based database can be further generated.

[0063] Specifically, the output configuration information may include node configuration information and / or data writing mode, wherein the node configuration information may include the number of nodes, mapping interface information of the nodes, designated data writing node positions, and the like.

[0064] The data writing mode may include at least one of the sharding writing mode (sharding key), balanced mode (balanced), random mode (rand / shuffle), and value mode (value). Among them, the sharding writing mode can be used to determine which shard the data should be stored on according to the specified key. The balanced mode can be a polling writing method, in which the data is written to the idle node when the node is determined to be idle. The random mode can be to randomly write data to any node. The value mode can be to uniformly write data to a specified node.

[0065] In the embodiment of the present invention, in order to facilitate storing data with the same value of a preset field in the same node, a sharding writing mode may be used to write the data.

[0066] In addition, the output configuration information can be further used to configure the data to be written into the specified data table in the node, the amount of data written by the node at a time, the method of verifying whether the data table exists, etc., and the present invention does not impose any restrictions on this.

[0067] After determining the output configuration information for the column-based database, the data flow execution engine can use the node configuration information therein to establish a connection with the node in the column-based database, and subsequently write data to the column-based database in a manner specified by the data writing mode.

[0068] Step 202, obtaining data to be written into the column-based database;

[0069] In an embodiment of the present invention, in the process of a network service provider providing users with various services such as video playback, product purchase, news browsing, etc., the data in the website can be continuously updated. For example, a user clicks to purchase a product to generate a new order, a user clicks to browse a video to generate a new browsing record, and the network service provider uploads new videos, news, products, etc. on the Internet, which can all generate new data. In order to facilitate subsequent data analysis, data with analytical value can be stored in a columnar database. Thereby, data to be written to the columnar database can be generated. In the case where the data flow execution engine takes over the data writing of the columnar database, the data flow execution engine can obtain the data to be written to the columnar database.

[0070] Step 203, using a hash algorithm to calculate a hash value for a value of a preset field in the data;

[0071] Thereafter, data with the same value of the preset field may be always stored in the same node, so as to avoid data jump problems caused by storing data with the same value of the preset field in different nodes.

[0072] The preset fields may be determined according to actual needs. For example, when recording in a key-value format, at least one key field may be used as a preset field, and when the values ​​of the key fields are the same, the data is stored in the same node.

[0073] For the data flow execution engine, in order to quickly and accurately identify the nodes where different data should be stored, it can use a hash algorithm to calculate a hash value for the value of a preset field in the data, and then determine the node where the data to be written should be stored based on the hash value.

[0074] Thus, a hash value can be first calculated using a hash algorithm for the value of a preset field in the data. The hash value is a fixed-length digital value used to uniquely identify the data. When the content of the preset field remains unchanged, the hash value can be fixed. At the same time, the hash value represents a large amount of data with a much smaller digital value, which facilitates the data flow execution engine to more efficiently determine the target node corresponding to the data.

[0075] Step 204, performing a modulus operation on the hash value to obtain a node-related value;

[0076] Specifically, in order to improve the efficiency of node selection, a modulo operation can be performed on the hash value. When the preset fields are the same, the hash values ​​are the same, and the values ​​obtained by the modulo operation can also be the same. At the same time, the value can be associated with the node. For example, a correspondence between a node-associated value and a node can be preset. After obtaining the node-associated value, the target node corresponding to the data can be determined based on the correspondence between the node-associated value and the node.

[0077] In one embodiment of the present invention, the step of performing a modulus operation on the hash value to obtain a node-associated value includes:

[0078] S31, taking the number of nodes in the column database as a modulus;

[0079] In a specific implementation, the modulus used in the modulus operation can be the number of nodes in the column database. Therefore, according to the result of the modulus operation, the target node corresponding to the data can be directly determined, thereby further improving the efficiency of node selection.

[0080] For example, if the number of nodes in the column database is 20, when 20 is used as the modulus of the modulus operation, the final result will be in the range of 0 to 19. Therefore, each node can be matched one by one with a node-associated value. After the result is calculated, the target node corresponding to the data can be directly determined.

[0081] S32, using the modulus to perform a modulus operation on the hash value to obtain a node-associated value.

[0082] Thereafter, the modulus may be used to perform a modulus operation on the hash value to obtain a node-related value. The node-related data may directly correspond to the target node.

[0083] For example, for the preset field of order number, its value can be 66004810491, and 20 is used as the modulus for modulus operation. At this time, the node association value obtained is 5, that is, the corresponding target node can be directly determined to be node 6. For the order number 66003442847, 20 is used as the modulus for modulus operation. At this time, the node association value obtained is 19, that is, the corresponding target node can be directly determined to be node 20.

[0084] Step 205: Based on the correspondence between the node-associated value and the node, the node corresponding to the node-associated value is used as the target node corresponding to the data.

[0085] In the embodiment of the present invention, the correspondence between the node association value and the nodes may be preset. After the node association value is obtained, the target node corresponding to the data may be determined according to the correspondence between the node association value and the nodes.

[0086] Step 206: write the data into the target node.

[0087] After determining the target node corresponding to the data, the data can be written into the target node, thereby completing the data writing. Data with the same preset field is always written into the same node, which can effectively avoid the problem of data jump.

[0088] In one embodiment of the present invention, the step of writing the data into the target node includes:

[0089] S41, creating at least one task pool; the task pool includes channels corresponding to the nodes one by one;

[0090] In the specific implementation, if every time a piece of data is obtained, the data is written to the corresponding target, it is easy to cause the node to write too many times, which will cause excessive pressure on the node. In order to avoid affecting the normal operation of the node, the data can be written to the node after accumulating a certain amount of data.

[0091] Thus, at least one task pool can be created for temporarily storing data to be written to the node, and the task pool can include channels corresponding to the nodes one by one. The data can be stored in the channel corresponding to its target node.

[0092] S42, adding the data to the channel corresponding to the target node in the task pool;

[0093] After the data is acquired, the data can be added to the channel corresponding to the target node according to the target node information corresponding to the data. Each channel can be set with a preset threshold, which represents the upper limit of the data that can be stored.

[0094] S43, if the amount of data in at least one channel in the task pool reaches a preset threshold, write the data into a node corresponding to the channel.

[0095] If the amount of data in at least one channel in the task pool reaches a preset threshold, all the data in the channel can be written to the node corresponding to the channel, thereby completing the writing of the data to the target node.

[0096] Through the data writing method of the column-based database provided by the embodiment of the present invention, a connection with at least one node in the column-based database is established; the data to be written into the column-based database is obtained; for the value of the preset field in the data, a hash value is calculated using a hash algorithm; a modulus operation is performed on the hash value to obtain a node-associated value; based on the correspondence between the node-associated value and the node, the node corresponding to the node-associated value is used as the target node corresponding to the data; and the data is written into the target node. The data writing to the column-based database is taken over by the data flow execution engine, and the target node to which the data should be written is quickly determined by identifying the hash value of the preset field in the data. Therefore, data with the same preset field is always written to the same node, which can effectively avoid the problem of data jump.

[0097] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0098] Reference Figure 3 , shows a structural block diagram of a data writing device for a column-based database provided in an embodiment of the present invention, which may specifically include the following modules:

[0099] A node connection establishment module 301 is used to establish a connection with at least one node in a column-based database;

[0100] A data acquisition module 302 is used to acquire data to be written into the column-based database;

[0101] A hash calculation module 303 is used to calculate a hash value using a hash algorithm for a value of a preset field in the data;

[0102] A target node determination module 304, configured to select a target node corresponding to the data from among the nodes based on the hash value;

[0103] The target node writing module 305 is used to write the data into the target node.

[0104] Optionally, the target node determination module;

[0105] A modulo operation submodule, used to perform a modulo operation on the hash value to obtain a node-associated value;

[0106] The target node determination submodule is used to take the node corresponding to the node associated value as the target node corresponding to the data based on the corresponding relationship between the node associated value and the node.

[0107] Optionally, the modulo operation submodule includes:

[0108] A modulus selection unit, used for taking the number of nodes in the column database as the modulus;

[0109] The modulus operation unit is used to use the modulus to perform a modulus operation on the hash value to obtain a node-associated value.

[0110] Optionally, the target node writing module includes:

[0111] A task pool creation submodule, used to create at least one task pool; the task pool includes channels corresponding to the nodes one by one;

[0112] A channel adding submodule, used for adding the data to the channel corresponding to the target node in the task pool;

[0113] The target node writing submodule is used to write the data into the node corresponding to the channel if the amount of the data in at least one channel in the task pool reaches a preset threshold.

[0114] Optionally, the node connection establishment module includes:

[0115] A node configuration acquisition submodule, used to acquire node configuration information of the column database;

[0116] The node connection establishment submodule is used to establish a connection with at least one node in the column database using the node configuration information.

[0117] Optionally, the device further comprises:

[0118] A dynamic table factory construction module, used to extract data writing configuration information through a dynamic table factory construction method provided by a preset custom column-based database connector, and to construct a dynamic table factory based on the data writing configuration information;

[0119] The output configuration determination module is used to obtain the data writing configuration information in the dynamic table factory and determine the output configuration information for the column database; the output configuration information includes node configuration information and / or data writing mode.

[0120] Optionally, the data writing mode includes at least one of a shard writing mode, a balanced mode, a random mode, and a value mode.

[0121] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0122] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0123] Memory 403, used for storing computer programs;

[0124] The processor 401 is used to execute the program stored in the memory 403 to implement the following steps:

[0125] Establishing a connection with at least one node in the column-based database;

[0126] Acquire data to be written into the column-based database;

[0127] A hash value is calculated using a hash algorithm for the value of a preset field in the data;

[0128] Based on the hash value, selecting a target node corresponding to the data from among the nodes;

[0129] The data is written into the target node.

[0130] Optionally, the step of selecting a target node corresponding to the data from the nodes based on the hash value includes:

[0131] Performing a modulo operation on the hash value to obtain a node-associated value;

[0132] Based on the corresponding relationship between the node-associated value and the node, the node corresponding to the node-associated value is used as the target node corresponding to the data.

[0133] Optionally, the step of performing a modulus operation on the hash value to obtain a node-associated value includes:

[0134] Taking the number of nodes in the column database as the modulus;

[0135] The modulus is used to perform a modulus operation on the hash value to obtain a node-associated value.

[0136] Optionally, the step of writing the data into the target node includes:

[0137] Creating at least one task pool; the task pool includes channels corresponding to the nodes one by one;

[0138] Add the data to the channel corresponding to the target node in the task pool;

[0139] If the amount of data in at least one channel in the task pool reaches a preset threshold, the data is written into a node corresponding to the channel.

[0140] Optionally, the step of establishing a connection with at least one node in the column-based database includes:

[0141] Obtaining node configuration information of the column database;

[0142] The node configuration information is used to establish a connection with at least one node in the column database.

[0143] Optionally, the method further comprises:

[0144] Extract data writing configuration information through the dynamic table factory construction method provided by the preset custom column database connector, and build a dynamic table factory based on the data writing configuration information;

[0145] The data writing configuration information in the dynamic table factory is obtained, and the output configuration information for the column-based database is determined; the output configuration information includes node configuration information and / or data writing mode.

[0146] Optionally, the data writing mode includes at least one of a shard writing mode, a balanced mode, a random mode, and a value mode.

[0147] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0148] The communication interface is used for communication between the above terminal and other devices.

[0149] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0150] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0151] like Figure 5 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 501 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the data writing method of the column-based database described in the above embodiment.

[0152] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the method for writing data to a column-based database described in the above embodiment.

[0153] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0154] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0155] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0156] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for writing data into a column-based database, characterized in that: It is applied to a data flow execution engine, and the method comprises: Establishing a connection with at least one node in the column-based database; Acquire data to be written into the column-based database; A hash value is calculated using a hash algorithm for the value of a preset field in the data; Based on the hash value, selecting a target node corresponding to the data from among the nodes; The data is written into the target node.

2. The method according to claim 1, characterized in that The step of selecting a target node corresponding to the data from the node based on the hash value includes: Performing a modulo operation on the hash value to obtain a node-associated value; Based on the corresponding relationship between the node-associated value and the node, the node corresponding to the node-associated value is used as the target node corresponding to the data.

3. The method according to claim 2, characterized in that The step of performing a modulo operation on the hash value to obtain a node-associated value includes: Taking the number of nodes in the column database as the modulus; The modulus is used to perform a modulus operation on the hash value to obtain a node-associated value.

4. The method according to claim 1, characterized in that The step of writing the data into the target node comprises: Creating at least one task pool; the task pool includes channels corresponding to the nodes one by one; Add the data to the channel corresponding to the target node in the task pool; If the amount of data in at least one channel in the task pool reaches a preset threshold, the data is written into a node corresponding to the channel.

5. The method according to claim 1, characterized in that The step of establishing a connection with at least one node in the column-based database comprises: Obtaining node configuration information of the column database; The node configuration information is used to establish a connection with at least one node in the column database.

6. The method according to claim 5, characterized in that The method further comprises: Extract data writing configuration information through the dynamic table factory construction method provided by the preset custom column database connector, and build a dynamic table factory based on the data writing configuration information; The data writing configuration information in the dynamic table factory is obtained, and the output configuration information for the column-based database is determined; the output configuration information includes node configuration information and / or data writing mode.

7. The method according to claim 6, characterized in that The data writing mode includes at least one of a shard writing mode, a balanced mode, a random mode, and a value mode.

8. A data writing device for a column-based database, characterized in that: It is applied to a data stream execution engine, and the device comprises: A node connection establishment module, used to establish a connection with at least one node in the column database; A data acquisition module, used to acquire data to be written into the column-based database; A hash calculation module, used to calculate a hash value using a hash algorithm for a value of a preset field in the data; A target node determination module, configured to select a target node corresponding to the data from among the nodes based on the hash value; The target node writing module is used to write the data into the target node.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. One or more computer-readable media having instructions stored thereon, which when executed by one or more processors cause the processors to perform the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images