Data processing method and device, server, medium and product

Abstract

The application belongs to the technical field of distributed technology, and specifically provides a data processing method and device, a server, a medium and a product. The method comprises the following steps: obtaining a preset message category corresponding to message data to be stored by a database server; consuming corresponding target message data from a message queue according to the preset message category and a preset consumption limit quantity; and sending the target message data to the database server, so that the database server stores the target message data. According to the method, the data transmission middleware in the ClickHouse system consumes the message queue data, and at the same time, the consumed message data is stored in the database server of ClickHouse. Compared with the data consumption from the message queue engine, the stability is higher, so that the quality and efficiency of data transmission are higher. In addition, the data volume of the target message data can be limited according to the preset consumption limit quantity, so as to avoid the problem of reduced data consumption performance caused by excessive data volume.

Description

Technical Field

[0001] This application relates to the field of distributed technology, and in particular to a data processing method, apparatus, server, medium and product. Background Technology

[0002] ClickHouse is a complete columnar database management system that allows you to create tables and databases, load data, and run queries at runtime. Message queues, such as the commonly used Kafka distributed publish-subscribe messaging system, are containers that store messages during message transmission.

[0003] ClickHouse typically consumes data from message queues and stores the consumed data. Consuming data generally refers to the process of the consuming server reading data. When ClickHouse consumes data from message queues, it primarily uses ClickHouse's native message queue engine, which is matched to the message queue. Currently, ClickHouse's native message queue engine has poor stability, resulting in poor data transmission quality and efficiency. Summary of the Invention

[0004] This application provides a data processing method, apparatus, server, medium, and product to solve the problem that the poor stability of ClickHouse's native message queue engine leads to poor data transmission quality and efficiency.

[0005] This application provides a data processing method applied to a data transmission middleware in a ClickHouse system, wherein the ClickHouse system further includes a database server, and the method includes:

[0006] Retrieve the preset message category corresponding to the message data to be stored on the database server;

[0007] Consume the corresponding target message data from the message queue according to the preset message category and preset consumption limit;

[0008] The target message data is sent to the database server so that the database server stores the target message data.

[0009] Furthermore, in the method described above, the data transmission middleware includes: a first configuration file corresponding to the database server;

[0010] The preset message category corresponding to the message data to be stored on the database server includes:

[0011] The preset message category corresponding to the message data to be stored is determined from the first configuration file.

[0012] Furthermore, in the method described above, the data transmission middleware further includes: a second configuration file corresponding to the message queue;

[0013] The step of consuming the corresponding target message data from the message queue according to the preset message category and preset consumption limit includes:

[0014] The network address information of the server cluster corresponding to the message queue is determined from the second configuration file according to the preset message category; the server cluster stores target message data to be consumed;

[0015] Consume the corresponding target message data from the corresponding server cluster based on the network address information;

[0016] The actual amount of target message data consumed is controlled based on the preset consumption limit.

[0017] Further, in the method described above, the step of consuming the corresponding target message data from the corresponding server cluster based on the network address information includes:

[0018] A data consumption request is sent to the corresponding server cluster based on the network address information;

[0019] Receive the consumption permission information from the corresponding server cluster;

[0020] Consume the corresponding target message data from the corresponding server cluster according to the permitted consumption information.

[0021] Furthermore, in the method described above, controlling the actual amount of data consumed by the target message data according to the preset consumption limit includes:

[0022] Monitor whether the actual amount of data consumed by the target message exceeds the preset consumption limit.

[0023] If it is determined that the actual amount of data consumed is greater than the preset consumption limit, then the actual amount of data consumed is made equal to the preset consumption limit.

[0024] Furthermore, in the method described above, the first configuration file further includes: the Internet Protocol address of the database server;

[0025] Sending the target message data to the database server so that the database server stores the target message data includes:

[0026] Determine the Internet Protocol address corresponding to the database server from the first configuration file;

[0027] The target message data is sent to the database server according to the Internet Protocol address, so that the database server stores the target message data.

[0028] Furthermore, in the method described above, the database server employs a dual-replica mechanism; the database server includes a primary replica node server and a secondary replica node server;

[0029] Sending the target message data to the database server according to the Internet Protocol address, so that the database server stores the target message data, includes:

[0030] The target message data is sent to the primary replica node server according to the Internet Protocol address, so that the primary replica node server stores the target message data and the secondary replica node server synchronizes the target message data.

[0031] Furthermore, in the method described above, before obtaining the preset message category corresponding to the message data to be stored in the database server, the method further includes:

[0032] Retrieve the preset message category and Internet protocol address corresponding to the database server;

[0033] The first configuration file is generated based on the preset message category and Internet protocol address corresponding to the database server.

[0034] Obtain the network address information and server cluster identifier of the server cluster;

[0035] A second configuration file is generated based on the network address information and server cluster identifier of the server cluster.

[0036] A second aspect of this application provides a data processing apparatus, a data transmission middleware located in a ClickHouse system, the ClickHouse system further including a database server, the apparatus comprising:

[0037] The acquisition module is used to acquire the preset message category corresponding to the message data to be stored in the database server;

[0038] The processing module is used to consume the corresponding target message data from the message queue according to the preset message category and preset consumption limit.

[0039] The sending module is used to send the target message data to the database server so that the database server stores the target message data.

[0040] Furthermore, in the apparatus described above, the data transmission middleware includes: a first configuration file corresponding to the database server;

[0041] The acquisition module is specifically used for:

[0042] The preset message category corresponding to the message data to be stored is determined from the first configuration file.

[0043] Furthermore, in the apparatus described above, the data transmission middleware further includes: a second configuration file corresponding to the message queue;

[0044] The processing module is specifically used for:

[0045] The network address information of the server cluster corresponding to the message queue is determined from the second configuration file according to the preset message category; the server cluster stores target message data to be consumed; the target message data is consumed from the corresponding server cluster according to the network address information; and the actual amount of target message data consumed is controlled according to the preset consumption limit.

[0046] Furthermore, in the apparatus described above, when the processing module consumes the corresponding target message data from the corresponding server cluster based on the network address information, it is specifically used for:

[0047] Send a data consumption request to the corresponding server cluster based on the network address information; receive the consumption permission information from the corresponding server cluster; and consume the corresponding target message data from the corresponding server cluster based on the consumption permission information.

[0048] Furthermore, in the apparatus described above, when the processing module controls the actual amount of target message data consumed according to the preset consumption limit, it is specifically used for:

[0049] Monitor whether the actual amount of data consumed by the target message is greater than the preset consumption limit; if it is determined that the actual amount of data consumed is greater than the preset consumption limit, then make the actual amount of data consumed equal to the preset consumption limit.

[0050] Furthermore, in the apparatus described above, the first configuration file further includes: the Internet Protocol address of the database server;

[0051] The sending module is specifically used for:

[0052] The Internet Protocol address corresponding to the database server is determined from the first configuration file; the target message data is sent to the database server according to the Internet Protocol address, so that the database server stores the target message data.

[0053] Furthermore, in the apparatus described above, the database server employs a dual-replica mechanism; the database server includes a primary replica node server and a secondary replica node server.

[0054] When the sending module sends the target message data to the database server according to the Internet Protocol address, so that the database server stores the target message data, it is specifically used for:

[0055] The target message data is sent to the primary replica node server according to the Internet Protocol address, so that the primary replica node server stores the target message data and the secondary replica node server synchronizes the target message data.

[0056] Furthermore, in the apparatus described above, the data processing apparatus further includes:

[0057] The configuration module is used to obtain the preset message category and Internet protocol address corresponding to the database server; generate a first configuration file based on the preset message category and Internet protocol address corresponding to the database server; obtain the network address information and server cluster identifier of the server cluster; and generate a second configuration file based on the network address information and server cluster identifier of the server cluster.

[0058] A third aspect of this application provides a middleware server, including: a processor, a memory, and a transceiver;

[0059] The processor, the memory, and the transceiver circuit are interconnected;

[0060] The memory stores computer-executed instructions; the transceiver is used for sending and receiving data.

[0061] The processor executes computer execution instructions stored in the memory to implement the data processing method as described in any of the first aspects.

[0062] A fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the data processing method described in any of the first aspects.

[0063] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method described in any of the first aspects.

[0064] This application provides a data processing method, apparatus, server, medium, and product applied to a data transmission middleware in the ClickHouse system. The ClickHouse system also includes a database server. The method includes: obtaining a preset message category corresponding to message data to be stored on the database server; consuming corresponding target message data from a message queue according to the preset message category and a preset consumption limit; and sending the target message data to the database server so that the database server stores the target message data. This data processing method consumes message queue data through the data transmission middleware in the ClickHouse system and simultaneously stores the consumed message data in the ClickHouse database server. Compared to consuming data from a message queue engine, this method offers higher stability, resulting in higher data transmission quality and efficiency. Furthermore, the preset consumption limit can restrict the amount of target message data, preventing performance degradation due to excessive data volume. Attached Figure Description

[0065] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0066] Figure 1 This is a scenario diagram illustrating how the data processing method described in the embodiments of this application can be implemented;

[0067] Figure 2 Flowchart of the data processing method provided in this application Figure 1 ;

[0068] Figure 3 Flowchart of the data processing method provided in this application Figure 2 ;

[0069] Figure 4 Schematic diagram of the data processing apparatus provided in this application Figure 1 ;

[0070] Figure 5 Schematic diagram of the data processing apparatus provided in this application Figure 2 ;

[0071] Figure 6 A schematic diagram of the middleware server provided in this application.

[0072] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0073] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0074] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions of this application comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0075] It should be noted that the data processing methods, apparatus, servers, media, and products disclosed herein can be used in the field of distributed technology. They can also be used in any field other than distributed systems, such as the financial sector. The application areas of the data processing methods, apparatus, equipment, media, and products disclosed herein are not limited.

[0076] The technical solutions of this application will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0077] To clearly understand the technical solution of this application, we will first provide a detailed introduction to the solutions of existing technologies. In recent years, the amount of monitoring data in distributed systems has grown rapidly, with the daily storage of data soon to exceed 100 billion records. Currently, when ClickHouse consumes data from message queues such as Kafka, it primarily uses ClickHouse's native Kafka engine to directly consume Kafka data, avoiding the use of middleware. However, using ClickHouse's native Kafka engine to directly consume Kafka data still has some drawbacks: the Kafka engine is complex, prone to failure, and difficult to debug; the Kafka engine runs within the database process, resulting in poor stability and consequently, poor quality and efficiency of Kafka data transmission. The same applies to other message queue engines; ClickHouse's native message queue engine also suffers from poor stability, leading to poor data transmission quality and efficiency.

[0078] Therefore, in response to the problem that the native message queue engine of ClickHouse has poor stability, resulting in poor data transmission quality and efficiency, the inventors discovered in their research that, in order to solve this problem, a data transmission middleware such as Sinker middleware in ClickHouse can be used. By leveraging the compatibility between the data transmission middleware and ClickHouse, as well as its effect as middleware, the quality and efficiency of data transmission can be improved.

[0079] Specifically, the system retrieves the preset message category corresponding to the message data to be stored on the database server. Based on the preset message category and preset consumption limit, it consumes the corresponding target message data from the message queue. The target message data is then sent to the database server for storage. Compared to consuming data from a message queue engine, this method offers higher stability, resulting in higher data transmission quality and efficiency. Furthermore, the preset consumption limit can restrict the amount of target message data, preventing performance degradation due to excessive data volume.

[0080] Based on the above-mentioned inventive discovery, the inventor has proposed the technical solution of this application.

[0081] The following describes the application scenarios of the data processing method provided in the embodiments of this application. For example... Figure 1 As shown, 1 is the middleware server, 2 is the message queue, and 3 is the ClickHouse database server. The network architecture of the application scenario corresponding to the data processing method provided in this embodiment includes: middleware server 1, message queue 2, and ClickHouse database server 3 (hereinafter referred to as database server 3). Message queue 2 generally stores the collected data in a server cluster; therefore, message queue 2 can be referred to as server cluster 2. Server cluster 2 stores message data published by various producers, such as applications, and stores them according to message categories. Each server in server cluster 2 can store message data corresponding to different message categories. There can be one or more database servers 3.

[0082] For example, a data consumption task can be pre-configured on middleware server 1. If a data consumption task needs to be executed, middleware server 1 obtains the preset message category corresponding to the message data to be stored on database server 3. The preset message category can be set according to the actual application, such as by message entity or message range. Simultaneously, middleware server 1 consumes the corresponding target message data from server cluster 2 according to the preset message category and preset consumption limit, and sends the target message data to database server 3 so that database server 3 can store the target message data. Furthermore, if the amount of target message data consumed is too large, the amount of data consumed can be limited according to the preset consumption limit, thereby reducing the data processing pressure on the internal processor.

[0083] The embodiments of this application are described below with reference to the accompanying drawings.

[0084] Figure 2 Flowchart of the data processing method provided in this application Figure 1 ,like Figure 2 As shown, in this embodiment, the execution entity of this application embodiment is a data processing device, which can be integrated into a middleware server. The method of this embodiment is applied to the data transmission middleware in the ClickHouse system. The data transmission middleware is a native middleware in the ClickHouse system and can be the Sinker middleware. The Sinker middleware does not have a common Chinese name; it can be translated as "sinking thing" and has high compatibility with ClickHouse. Meanwhile, the ClickHouse system also includes a database server for storing data. Therefore, the data processing method provided in this embodiment includes the following steps:

[0085] Step S101: Obtain the preset message category corresponding to the message data to be stored on the database server.

[0086] In this embodiment, the message data to be stored refers to the message data that the database server needs or plans to store. For example, if the database server wants to store a certain push notification, then that push notification is the message data to be stored. For each message data, there is a corresponding preset message category. The preset message category can be set according to the needs of the actual application scenario, such as setting the preset message category according to message entity, message scope, or the field involved in the message.

[0087] There can be multiple database servers, which can form a database server cluster.

[0088] Step S102: Consume the corresponding target message data from the message queue according to the preset message category and preset consumption limit.

[0089] In this embodiment, the server clusters commonly used for message queues, such as Kafka, can be distributed according to preset message categories. For example, each server cluster corresponds to a major category within a preset message category, and each server in the server cluster corresponds to a subcategory within that major category. Kafka is used as an example below. Consuming data generally refers to the process of a consuming server reading data from Kafka; in this embodiment, the consuming server is a middleware server.

[0090] The target message data refers to the message data consumed from the Kafka system that the database server requires or is expected to store.

[0091] As a message queue, Kafka creates a corresponding message category (topic) when a producer publishes message data to the Kafka server. To consume this message data, the corresponding message category must be configured, and the message data in Kafka must be consumed according to the specified message category.

[0092] Step S103: Send the target message data to the database server so that the database server stores the target message data.

[0093] If there are multiple database servers, when sending target message data to the database servers, it can be sent to the corresponding database server according to the message category for storage. This allows each database server to store data according to the message category, which facilitates subsequent data retrieval and improves the efficiency of subsequent data query.

[0094] This application provides a data processing method, which includes: obtaining a preset message category corresponding to message data to be stored on a database server; consuming corresponding target message data from a message queue according to the preset message category and a preset consumption limit; and sending the target message data to the database server so that the database server stores the target message data.

[0095] The data processing method in this application consumes message queue data through the data transmission middleware in the ClickHouse system. Simultaneously, the consumed message data is stored in the ClickHouse database server. Compared to consuming data directly from the message queue engine, this method offers higher stability, resulting in higher data transmission quality and efficiency. Furthermore, the amount of target message data can be limited according to a preset consumption limit to avoid performance degradation caused by excessive data volume.

[0096] Figure 3 Flowchart of the data processing method provided in this application Figure 2 ,like Figure 3As shown, the data processing method provided in this embodiment is a further refinement based on the data processing method provided in the previous embodiment of this application. Since the data transmission middleware used in this embodiment is the Sinker middleware, the data processing method provided in this embodiment includes the following steps.

[0097] It should be noted that the Sinker middleware includes: a first configuration file corresponding to the database server and a second configuration file corresponding to the message queue.

[0098] Step S201: Determine the preset message category corresponding to the message data to be stored from the first configuration file.

[0099] In this embodiment, the first configuration file includes the database server port, the preset message category corresponding to the database server, etc. The first configuration file is pre-configured, and information such as the database server port and the preset message category corresponding to the message data to be stored can be determined from the first configuration file. The preset message category generally corresponds to the database server.

[0100] Optionally, in this embodiment, before step S201, a configuration file generation process is also included, as follows:

[0101] Retrieve the preset message category and Internet protocol address corresponding to the database server.

[0102] The first configuration file is generated based on the preset message category and Internet protocol address corresponding to the database server.

[0103] Obtain the network address information and server cluster identifier of the server cluster.

[0104] A second configuration file is generated based on the network address information and server cluster identifier of the server cluster.

[0105] Taking Kafka as an example, ClickHouse's Sinker, acting as middleware for transmitting Kafka messages to ClickHouse, requires initial configuration in its configuration file. This includes configuring the Kafka server cluster from which Sinker will consume data (including the cluster's network address and identifier) ​​and its corresponding version. It also requires configuring the ClickHouse server port and its corresponding cluster, representing the database server storing the data in ClickHouse and its internet protocol address. Secondly, the consumption task schedule can be configured, setting corresponding consumption information for each table, including the table name and its corresponding topic configuration. All fields and types for this table must be configured in Sinker, and the field types must match the table structure of the corresponding topic in Kafka and the table structure created in the ClickHouse database; otherwise, data consumption will fail.

[0106] Step S202: Determine the network address information of the server cluster corresponding to the message queue from the second configuration file according to the preset message category. The server cluster stores the target message data to be consumed.

[0107] In this embodiment, since a first configuration file and a second configuration file are pre-configured, the relevant information of the Kafka end and the ClickHouse end can be directly determined based on the data stored in the configuration files.

[0108] The network address information of the server cluster is used for data access and data consumption.

[0109] Step S203: Consume the corresponding target message data from the corresponding server cluster based on the network address information.

[0110] In this embodiment, since the network address information is already determined, the corresponding target message data can be consumed directly from the corresponding server cluster based on the network address information. For example, a consumption request can be sent to the corresponding server cluster based on the network address information, thereby realizing data consumption.

[0111] Optionally, in this embodiment, step S203 can specifically be:

[0112] Data consumption requests are sent to the corresponding server cluster based on the network address information.

[0113] Receive consumption permission information from the corresponding server cluster.

[0114] Consume the corresponding target message data from the corresponding server cluster based on the allowed consumption information.

[0115] In this embodiment, sending a data consumption request and providing feedback allowing consumption can improve data transmission security. After sending the data consumption request, to further enhance information security, the data consumption request can be authenticated, such as through password verification of the public and private keys, or identity verification.

[0116] Step S204: Control the actual amount of target message data consumed according to the preset consumption limit.

[0117] Considering the scenario where the amount of data received during a certain period of the day may continuously increase in practical applications, this embodiment modifies the original Sinker program by adding a rate-limiting mechanism during the consumption process. This limits the rate-limiting parameters of Sinker, i.e., it adds a preset consumption limit. This avoids the problem of reduced data consumption performance caused by excessive data volume.

[0118] Optionally, in this embodiment, step S204 can be further refined as follows:

[0119] Monitor whether the actual amount of data consumed by the target message exceeds the preset consumption limit.

[0120] If the actual consumption data is determined to be greater than the preset consumption limit, then the actual consumption data will be made equal to the preset consumption limit.

[0121] When the actual amount of data consumed exceeds the preset consumption limit, the rate limiting parameter remains at the set value, and this rate limiting parameter configuration is added to the Sinker configuration file. During peak data inflow periods, the maximum actual data consumption is limited, maintaining consumption at the preset limit. After the peak data inflow period ends, the data inflow gradually decreases, and the actual data consumption also gradually decreases until the remaining data on the Kafka end is consumed. This effectively alleviates the pressure on the middleware server's memory and processor, preventing memory spikes and performance degradation caused by a sudden increase in data consumption at a particular moment, thus improving the stability of the middleware server.

[0122] It should be noted that the first configuration file also includes the Internet Protocol address of the database server.

[0123] Step S205: Determine the Internet Protocol address corresponding to the database server from the first configuration file.

[0124] In this embodiment, determining the Internet Protocol address corresponding to the database server from the first configuration file is to determine the transmission address for subsequent message data. This provides a basis for sending the target message data to the database server.

[0125] Step S206: Send the target message data to the database server according to the Internet Protocol address so that the database server stores the target message data.

[0126] In this embodiment, the target message data to be consumed can be cached. At the same time, appropriate data transmission can be performed according to the data processing efficiency of the database server to avoid causing excessive processor pressure on the database server.

[0127] Optionally, in this embodiment, to reduce the probability of data loss, a dual-replica mechanism can be further adopted for the database server. In this embodiment, the database server includes a primary replica node server and a secondary replica node server, then step S206 can specifically be as follows:

[0128] The target message data is sent to the primary replica node server according to the Internet Protocol address, so that the primary replica node server stores the target message data and the secondary replica node server synchronizes the target message data.

[0129] In this embodiment, two ClickHouse servers are used as examples, acting as replicas of each other. After initialization, the two replica nodes first select a primary replica node. During data writing, data is first written to the primary replica node. Once the data writing is complete, a log is pushed. The secondary replica node continuously listens for log changes and then detects from the log that the remote primary replica node has successfully written the data. At this point, the secondary replica node needs to synchronize this data and initiates a download request to the primary replica node. Upon receiving the request, the primary replica node responds and sends the written data to the other replica. The other replica completes the data writing and saves it locally. In this way, the two servers store the same data as replicas of each other. Even if one server fails, the data can be recovered from the other, which not only increases data redundancy but also reduces the probability of data loss to some extent.

[0130] Meanwhile, since ClickHouse adopts a multi-master architecture, meaning each server node is peer-to-peer, insert, modify, and query operations can be performed on any replica node with identical results. Each replica node can serve as a data read / write entry point, effectively distributing the node load. To effectively utilize data storage space, this embodiment also employs a multi-path storage JBOD (Just a Bunch Of Disks) strategy for replica nodes. This is a round-robin strategy, determined by the order in which JBOD is configured. Each data shard generated by writing data is written to one disk, and the next data shard generated by writing data is written to the next disk, and so on, writing to multiple disk paths sequentially. This avoids excessive data pressure on a single disk due to frequent writes, and under certain conditions, increases the performance of parallel data read / write.

[0131] Figure 4 Schematic diagram of the data processing apparatus provided in this application Figure 1 ,like Figure 4 As shown, in this embodiment, the data processing device 300 is located in the data transmission middleware of the ClickHouse system, and the data processing device 300 includes:

[0132] The acquisition module 301 is used to acquire the preset message category corresponding to the message data to be stored in the database server.

[0133] The processing module 302 is used to consume the corresponding target message data from the message queue according to the preset message category and preset consumption limit.

[0134] The sending module 303 is used to send the target message data to the database server so that the database server can store the target message data.

[0135] The data processing device provided in this embodiment can perform... Figure 2 The technical solution of the method embodiment shown has the same implementation principle and technical effect as... Figure 2 The methods and embodiments shown are similar and will not be described in detail here.

[0136] Figure 5 Schematic diagram of the data processing apparatus provided in this application Figure 2 The data processing apparatus provided in this application is a further refinement of the data processing apparatus provided in the previous embodiment. For better explanation, the data processing apparatus 400 is used as an example in this embodiment. The data processing apparatus 400 includes:

[0137] Optionally, in this embodiment, the data transmission middleware includes a first configuration file corresponding to the database server.

[0138] Module 301 is used specifically for:

[0139] The preset message category corresponding to the message data to be stored is determined from the first configuration file.

[0140] Optionally, in this embodiment, the data transmission middleware further includes a second configuration file corresponding to the message queue.

[0141] Processing module 302 is specifically used for:

[0142] The network address information of the server cluster corresponding to the message queue is determined from the second configuration file based on the preset message category. The server cluster stores the target message data to be consumed. The corresponding target message data is consumed from the corresponding server cluster based on the network address information. The actual amount of target message data consumed is controlled according to the preset consumption limit.

[0143] Optionally, in this embodiment, when the processing module 302 consumes the corresponding target message data from the corresponding server cluster based on the network address information, it is specifically used for:

[0144] A data consumption request is sent to the corresponding server cluster based on the network address information. The server cluster then replies with permission to consume the data. Based on this permission, the target message data is consumed from the corresponding server cluster.

[0145] Optionally, in this embodiment, when the processing module 302 controls the actual amount of data consumed by the target message data according to the preset consumption limit, it is specifically used for:

[0146] Monitor whether the actual amount of data consumed by the target message exceeds the preset consumption limit; if it is determined that the actual amount of data consumed exceeds the preset consumption limit, then make the actual amount of data consumed equal to the preset consumption limit.

[0147] Optionally, in this embodiment, the first configuration file further includes: the Internet Protocol address of the database server.

[0148] The sending module 303 is specifically used for:

[0149] The Internet Protocol address (IPA) corresponding to the database server is determined from the first configuration file. The target message data is then sent to the database server according to the IPA address, so that the database server stores the target message data.

[0150] Optionally, in this embodiment, the database server employs a dual-replica mechanism. The database server includes a primary replica node server and a secondary replica node server.

[0151] When sending target message data to the database server according to the Internet Protocol address, so that the database server stores the target message data, the sending module 303 is specifically used for:

[0152] The target message data is sent to the primary replica node server according to the Internet Protocol address, so that the primary replica node server stores the target message data and the secondary replica node server synchronizes the target message data.

[0153] Optionally, in this embodiment, the data processing device 400 further includes:

[0154] Configuration module 401 is used to obtain the preset message category and Internet protocol address corresponding to the database server. It generates a first configuration file based on the preset message category and Internet protocol address corresponding to the database server. It also obtains the network address information and server cluster identifier of the server cluster. Finally, it generates a second configuration file based on the network address information and server cluster identifier of the server cluster.

[0155] The data processing device provided in this embodiment can perform... Figures 2-3 The technical solution of the method embodiment shown has the same implementation principle and technical effect as... Figures 2-3 The methods and embodiments shown are similar and will not be described in detail here.

[0156] According to embodiments of this application, this application also provides a middleware server, a computer-readable storage medium, and a computer program product.

[0157] like Figure 6 As shown, Figure 6 This is a schematic diagram of the middleware server provided in this application. The middleware server is intended for various forms of electronic devices, such as laptops, desktop computers, workstations, personal digital assistants, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0158] like Figure 6 As shown, the middleware server includes a processor 501, a memory 502, and a transceiver 503. The various components are interconnected via different buses and can be installed on a common motherboard or otherwise as needed. The processor processes instructions executed within the middleware server. The transceiver 503 is used for sending and receiving data.

[0159] The memory 502 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause at least one processor to perform the data processing method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the data processing method provided in this application.

[0160] Memory 502, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions / modules corresponding to the data processing method in the embodiments of this application (e.g., appendix). Figure 4 The acquisition module 301, processing module 302, and sending module 303 are shown. The processor 501 executes various functional applications and data processing of the electronic device by running non-transient software programs, instructions, and modules stored in the memory 502, thereby implementing the data processing method in the above method embodiment.

[0161] In addition, this embodiment also provides a computer product, which, when the instructions in the computer product are executed by the processor of an electronic device, such as a middleware server, enables the middleware server to execute the data processing method of the above embodiment.

[0162] Other embodiments of the present application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the embodiments of this application that follow the general principles of the embodiments of this application and include common knowledge or customary techniques in the art not disclosed in the embodiments of this application.

[0163] It should be understood that the embodiments of this application are not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from their scope. The scope of the embodiments of this application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, A data transfer middleware applied in the ClickHouse system, which also includes a database server, the method comprising: Retrieve the preset message category corresponding to the message data to be stored on the database server; Consume the corresponding target message data from the message queue according to the preset message category and preset consumption limit; The target message data is sent to the database server so that the database server stores the target message data. The data transmission middleware includes: a first configuration file corresponding to the database server; The preset message category corresponding to the message data to be stored on the database server includes: Determine the preset message category corresponding to the message data to be stored from the first configuration file; The data transmission middleware further includes: a second configuration file corresponding to the message queue; The step of consuming the corresponding target message data from the message queue according to the preset message category and preset consumption limit includes: The network address information of the server cluster corresponding to the message queue is determined from the second configuration file according to the preset message category; the server cluster stores target message data to be consumed; Consume the corresponding target message data from the corresponding server cluster based on the network address information; The actual amount of target message data consumed is controlled based on the preset consumption limit.

2. The method according to claim 1, characterized in that, The step of consuming the corresponding target message data from the corresponding server cluster based on the network address information includes: A data consumption request is sent to the corresponding server cluster based on the network address information; Receive consumption permission information from the corresponding server cluster; Consume the corresponding target message data from the corresponding server cluster according to the permitted consumption information.

3. The method according to claim 1, characterized in that, The method of controlling the actual amount of target message data consumed according to a preset consumption limit includes: Monitor whether the actual amount of data consumed by the target message exceeds the preset consumption limit. If it is determined that the actual amount of data consumed is greater than the preset consumption limit, then the actual amount of data consumed is made equal to the preset consumption limit.

4. The method according to claim 3, characterized in that, The first configuration file also includes: the Internet Protocol address of the database server; Sending the target message data to the database server so that the database server stores the target message data includes: Determine the Internet Protocol address corresponding to the database server from the first configuration file; The target message data is sent to the database server according to the Internet Protocol address, so that the database server stores the target message data.

5. The method according to claim 4, characterized in that, The database server employs a dual-replica mechanism; the database server includes a primary replica node server and a secondary replica node server. Sending the target message data to the database server according to the Internet Protocol address, so that the database server stores the target message data, includes: The target message data is sent to the primary replica node server according to the Internet Protocol address, so that the primary replica node server stores the target message data and the secondary replica node server synchronizes the target message data.

6. The method according to claim 4, characterized in that, Before obtaining the preset message category corresponding to the message data to be stored in the database server, the process also includes: Retrieve the preset message category and Internet protocol address corresponding to the database server; The first configuration file is generated based on the preset message category and Internet protocol address corresponding to the database server. Obtain the network address information and server cluster identifier of the server cluster; A second configuration file is generated based on the network address information and server cluster identifier of the server cluster.

7. A data processing apparatus, characterized in that, The device comprises a data transmission middleware located in the ClickHouse system, which also includes a database server, and the apparatus includes: The acquisition module is used to acquire the preset message category corresponding to the message data to be stored in the database server; The processing module is used to consume the corresponding target message data from the message queue according to the preset message category and preset consumption limit. A sending module is used to send the target message data to the database server so that the database server stores the target message data; The data transmission middleware includes: a first configuration file corresponding to the database server; The acquisition module is specifically used for: Determine the preset message category corresponding to the message data to be stored from the first configuration file; The data transmission middleware further includes: a second configuration file corresponding to the message queue; The processing module is specifically used for: The network address information of the server cluster corresponding to the message queue is determined from the second configuration file according to the preset message category; the server cluster stores target message data to be consumed; Consume the corresponding target message data from the corresponding server cluster based on the network address information; The actual amount of target message data consumed is controlled based on the preset consumption limit.

8. A middleware server, characterized in that, include: Processor, memory, and transceiver; The processor, the memory, and the transceiver circuit are interconnected; The memory stores computer-executed instructions; the transceiver is used for sending and receiving data. The processor executes computer execution instructions stored in the memory to implement the data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data processing method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 6.

Images