Data transmission method and device

Abstract

The present application discloses a data transmission method and device. The data transmission method includes: configuring a speed limit rate in a data stream processing engine using a declarative query language; the data stream processing engine limits the rate at which the data stream processing engine reads / writes data from a data source based on the speed limit rate. The present application can decouple data users from data sources, so that users can set the speed limit for the consumed data topic in a specific task, and different data users can consume the same data topic at different rates.

Description

Technical Field

[0001] The present application relates to the technical field of data stream processing, and in particular to a data transmission method and device. Background Art

[0002] Although the data source provides a quota management function, allowing users to set the data transmission rate through user identity authentication, the setting process is relatively cumbersome, and after the rate is set for the same data topic in the data source, all data users will be subject to speed limits when reading. Summary of the invention

[0003] The present application provides a data transmission method and device, which can decouple data users and data sources, so that users can set the speed limit for the consumed data topics in specific tasks, and different data users can consume the same data topic at different rates.

[0004] To achieve the above object, the present application provides a data transmission method, the method comprising:

[0005] Use declarative query language to configure rate limits in the data stream processing engine;

[0006] The data stream processing engine limits the rate at which the data stream processing engine reads / writes data from the data source based on the rate limit.

[0007] In order to achieve the above object, the present application also provides a data transmission device, which includes:

[0008] Configuration module, used to configure the rate limit in the data stream processing engine using a declarative query language;

[0009] The current limiting module is used to enable the data stream processing engine to limit the rate at which the data stream processing engine reads / writes data from the data source based on the speed limit rate.

[0010] To achieve the above objectives, the present application also provides an electronic device, which includes a processor; the processor is used to execute instructions to implement the above method.

[0011] To achieve the above objectives, the present application also provides a computer-readable storage medium, which is used to store instructions / program data, and the instructions / program data can be executed to implement the above method.

[0012] The present application proposes a data transmission method, which uses a declarative query language to configure a speed limit rate in a data stream processing engine; then the data stream processing engine limits the rate at which the data stream processing engine reads / writes data from the data source based on the speed limit rate. Compared with the related technical solutions, the speed limit rate used to limit the rate at which the data stream processing engine reads / writes data from the data source is configured through a declarative query language, without the need to write more complex codes, and can conveniently set the consumption speed required for each stream through new configuration items to implement a flow control solution. The execution efficiency is relatively high, the parameters can be flexibly adjusted, and the usability is improved; and the flow rate can be controlled by the data stream processing engine end, without the need to configure through the data source to implement flow control, and can be decoupled from the data source, so that the user can set the speed limit of the consumed data topic in a specific task, so that different data users can consume the same data topic at different rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0014] Figure 1 It is a flowchart of an implementation method of the data transmission method of the present application;

[0015] Figure 2 This is a flow chart of an embodiment of a data transmission method of the present application;

[0016] Figure 3 It is a schematic diagram of the working principle of the current limiter in the data transmission method of the present application;

[0017] Figure 4 is a schematic diagram of a current limiter in the data transmission method of the present application;

[0018] Figure 5 This is a flow chart of an embodiment of a data transmission method of the present application;

[0019] Figure 6 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0020] Figure 7 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. In addition, unless otherwise specified (for example, "or in addition" or "or in an alternative"), the term "or" as used herein refers to a non-exclusive "or" (that is, "and / or"). Moreover, the various embodiments described herein are not necessarily mutually exclusive, because some embodiments can be combined with one or more other embodiments to form new embodiments.

[0022] Although there are related technical solutions for implementing the dynamic adjustment of the flow control function in the data flow processing engine, they control the flow rate between the data flow processing engine and the data source through JAVA statements. This related technical solution requires writing relatively complex codes and has low execution efficiency.

[0023] Based on this, the present application proposes a data transmission method, which uses a declarative query language to configure a speed limit rate in a data stream processing engine; then the data stream processing engine limits the rate at which the data stream processing engine reads / writes data from the data source based on the speed limit rate. Compared with the related technical solutions, the speed limit rate used to limit the rate at which the data stream processing engine reads / writes data from the data source is configured through a declarative query language, without the need to write more complex code, and can conveniently set the required consumption speed of each stream through new configuration items to implement a flow control solution. The execution efficiency is relatively high, the parameters can be flexibly adjusted, and the usability is improved; and the flow rate can be controlled by the data stream processing engine end, without the need to configure through the data source to implement flow control, and can be decoupled from the data source, so that the user can set the speed limit of the consumed data topic in a specific task, so that different data users can consume the same data topic at different rates.

[0024] like Figure 1 As shown, the data transmission method of this embodiment includes the following steps. It should be noted that the following step numbers are only used to simplify the description and are not intended to limit the execution order of the steps. The execution order of each step of this embodiment can be changed arbitrarily without violating the technical idea of ​​this application.

[0025] S110: Use a declarative query language to configure the rate limit in the data stream processing engine.

[0026] The rate limit can be configured in the data stream processing engine using a declarative query language first, so that the subsequent data stream processing engine can limit the rate at which the data stream processing engine reads / writes data from the data source based on the rate limit.

[0027] Declarative query languages ​​are a type of programming language or language feature that allows users to describe the data they want to query or operate on, rather than specifying how to perform these operations. Such languages ​​are often used in database queries and other data processing scenarios. Compared to the JAVA language, which requires specific instructions on how to perform operations, it is more convenient and efficient to configure the speed limit rate through a declarative query language. Optionally, the declarative query language can be SQL or Table language.

[0028] Optionally, a data definition language (DDL) in a declarative query language may be used to configure a rate parameter, where the rate parameter is used to set a rate limit.

[0029] In one embodiment, when creating a connector for connecting a data source of a data stream processing engine through a data definition language, a rate parameter (e.g., a "Rate" parameter) can be set in the configuration options of the connector so that the user can set a speed limit in the rate parameter, thereby allowing the user to fine-tune the interaction between the data stream processing engine and the data source based on the configuration options of the connector (including the speed limit set in the rate parameter), so that the data stream processing engine can control the reading or writing of data based on the configuration options of the connector.

[0030] The configuration options of the connector may also include the connector type (e.g., the 'connector' parameter), the data source topic name (e.g., the 'topic' parameter), the bootstrap server information of the data source (e.g., the 'properties.bootstrap.servers' parameter), the data format (e.g., the 'format' parameter), and / or the starting position parameter for reading messages (e.g., the 'scan.startup.mode' parameter).

[0031] S120: The data stream processing engine limits the rate at which the data stream processing engine reads / writes data from the data source based on the rate limit.

[0032] When a declarative query language is used to configure a rate limit in a data stream processing engine, the data stream processing engine can limit the rate at which the data stream processing engine reads / writes data from a data source based on the rate limit.

[0033] The data stream processing engine may be a stream processing engine such as Flink, and the data source may be Kafka, etc.

[0034] Among them, limiting the rate at which the data stream processing engine reads / writes data from the data source based on the speed limit rate can be reflected as: limiting the rate at which the data stream processing engine reads data from the data source based on the speed limit rate; and / or, limiting the rate at which the data stream processing engine writes data to the data source based on the speed limit rate.

[0035] Optionally, the number of data transmissions can be limited based on the speed limit rate, so as to limit the data transmission rate by limiting the number of data transmissions, such as limiting the number of read data based on the speed limit rate, or limiting the number of write data based on the speed limit rate. Alternatively, the average data transmission rate can be limited based on the speed limit rate, such as limiting the average read data rate based on the speed limit rate, or limiting the average write data rate based on the speed limit rate; compared with limiting the number of data transmissions, limiting the average data transmission rate can allow a certain degree of burst, as long as it does not exceed the rate limit, it can be processed, and 3 data transmission requests or 4 data transmission requests can be processed at a time, and at the beginning or after a long period of no requests, the limiter can accumulate some additional request permissions to allow requests higher than the average rate to pass. This mode is suitable for scenarios that occasionally need to handle burst traffic.

[0036] Optionally, the set speed limit rate can be passed to a current limiting module set on the data flow processing engine, so that the current limiting module limits the rate at which the data flow processing engine reads / writes data from the data source based on the speed limit rate. Exemplarily, the connector in the data flow processing engine is directly connected to the subject of the data source, and the connector in the data flow processing engine adjusts the rate at which data is read based on the byte size of the data arriving from the subject and the speed limit rate, so that subsequent stream processing operations such as aggregation and / or conversion can be performed based on the read data, thereby limiting the rate at which the data flow processing engine reads data from the data source based on the speed limit rate. Figure 2 As shown in the figure, different topics represent the streams that flink (i.e., data stream processing engine) needs to consume. Each topic has several partitions. Kafka (i.e., data source) topic1 and topic2 are directly connected to the flink end. The flink connector adjusts the poll rate according to the byte size of the data coming from the topic. The data pulled by the connector can be seen in the figure. Figure 2 The thickness of stream 1 and stream 2 in the figure represents the effect of different rate limits on the two streams; the connector in the data stream processing engine also adjusts the rate of writing data according to the data byte size and the rate limit rate of data arriving from other units of the data stream processing engine, so as to limit the rate at which the data stream processing engine writes data from the data source based on the rate limit rate.

[0037] In one implementation, the current limiting module can control the data transmission rate through a token bucket algorithm. Figure 3As shown, the control of data transmission rate by token bucket algorithm may include: in response to data transmission request, taking out a preset number of tokens from the current token balance, wherein the current token balance includes the most recently generated new tokens, and the new tokens are generated at a fixed rate; in response to taking out the preset number of tokens, transmitting the data corresponding to the data transmission request; in response to not taking out the preset number of tokens, waiting until the preset number of tokens are obtained. The preset number may be a fixed value, such as 1, 2 or 3. Alternatively, the preset number corresponds to the data transmission request, wherein the maximum transmission rate allowed by a token is fixed. Assuming that the data transmission request corresponds to the data transmission rate, the preset number can be calculated based on the data transmission rate corresponding to the data transmission request and the maximum transmission rate allowed by a token. For example, the preset number may be equal to the ratio of the data transmission rate specified by the data transmission request to the maximum transmission rate allowed by a token. Alternatively, if the number of tokens is specified in the data transmission request, the preset number may be equal to the number of tokens specified in the data transmission request.

[0038] As mentioned above, the token bucket algorithm is a bucket for storing tokens. The bucket can be of fixed capacity or floating capacity, and tokens can be added to the bucket at a fixed rate. The number of tokens stored in the bucket can have a maximum upper limit, and it will be rejected if it exceeds the upper limit. When traffic or network requests arrive, each request must obtain a token. If it can be obtained, it will be processed directly, and the token bucket will delete a token. If it cannot be obtained, the request will be limited and wait in the buffer. The token bucket adds tokens to the bucket at a fixed rate. Whether the request is processed depends on whether there are enough tokens in the bucket. When the number of tokens is reduced to zero, new requests are rejected. Among them, the token bucket limits the average inflow rate, allowing a certain degree of burst. As long as there are tokens, it can be processed, and it supports taking 3 tokens and 4 tokens at a time.

[0039] As shown above, in this implementation, the current limiter achieves the principle of flow rate control by setting the token generation speed and acquisition speed, so that tokens can be accumulated in the token bucket to alleviate the problem of excessive traffic. Optionally, the fixed rate of new token generation can be related to the speed limit rate, that is, the speed limit rate is passed to the current limiter (for example, the internal method setrate of the current limiter) to affect the token generation rate of the bucket. Optionally, the data stream processing engine can convert the poll data of the connector into the number of bytes and obtain tokens through the acquire method, so that the number of tokens taken out (that is, the preset number) can be determined based on the number of bytes, and the data obtained from the token can be transmitted downward, so that the current limiter can be transplanted into the flink connector to limit the connector from pulling the data source according to the parameters set by humans. Optionally, the number of bytes is a factor that has a positive impact on the preset number, that is, if the number of bytes increases, the preset number may also increase or remain unchanged. Further, the preset number can be positively correlated with the number of bytes.

[0040] In another implementation, the current limiting module can control the data transmission rate by using fixed window current limiting. Specifically, the time can be divided into fixed-size windows, and only a certain number of requests are allowed to pass through each window. Once the limit is reached, subsequent requests will be rejected until the next window starts. This method is simple and easy to understand, but burst traffic problems may occur when switching windows.

[0041] In another implementation, the current limiting module can control the data transmission rate by limiting the current through an elastic buffer, wherein an expandable buffer can be used to smooth requests, and when the request volume is large, the buffer can temporarily store additional requests, and when the request volume is low, the buffer releases the stored requests, thereby alleviating the instantaneous peak pressure to a certain extent.

[0042] Among them, Figure 4 As shown, the internal implementation of the current limiter is relatively simple, mainly including an abstract class RateLimiter and an actual smooth rate current limiter SmoothRateLimiter, where smooth preheating SmoothWarmingUP and smooth burst SmoothBursty are two internal classes of SmoothRateLimiter, representing ordinary current limiting and current limiting with preheating, respectively. Preferably, SmoothBursty can be used to implement the current limiter.

[0043] Optionally, the above-mentioned passing the set speed limit rate to the current limiting module set on the data flow processing engine may include: passing the speed limit rate to the method of creating a data source to trigger the start of the current limiter in the connector. In this way, the triggering of the current limiter and the configuration of the speed limit rate are set in the process of creating the data source and its connector, and the creation process of the data source and its connector can be utilized without too many additional steps, so that flow control can be efficiently implemented.

[0044] Among them, Figure 5 As shown, passing the speed limit rate to the method for creating a data source to trigger the start of the current limiter in the connector may include: creating a dynamic data source through a data source dynamic table factory; verifying the rate parameter in the data definition language; and in response to setting the speed limit rate in the data definition language, executing the step of passing the speed limit rate to the method for creating a data source to trigger the start of the current limiter in the connector. In a specific example, as Figure 5 As shown, Figure 5It is the process of designing the limiter into the data source connector. Assuming that the data stream processing engine is Flink and the data source is Kafka, the KafkaDynamicTableFactory (data source dynamic table factory) in Flink will create a KafkaDynamicSource (dynamic data source). At this time, the LIMIT_Rate parameter (rate parameter) in KafkaConnectorOptions (Kafka connector configuration in the data definition language) is checked. If the Rate (rate limit) is set in the task, it will be passed to the createKafkaSource (create data source) method through this parameter, and then the Ratelimiter (limiter) in the KafkaSource (connector) will be triggered through the RuntimeContext (runtime context). The start of the connector will then continue to control the poll (read) rate of the connector to the Kafka topic (topic) data source in the KafkaPartitionSplitReader (Kafka partition slice reader). In this way, the data of the Kafka poll can be directly connected to the Ratelimiter to realize the function of the limiter.

[0045] Optionally, the data transmission method of the present application can be applied to scenarios where multiple streams are transmitted simultaneously, such as dual-stream join, so as to effectively solve the data accumulation and blocking problems caused by inconsistent matching amounts of data from different streams when processing Flink dual-stream join. Preferably, a token bucket-based current limiting module can be used to limit the speed of dual-stream join to deal with the problem of excessive traffic.

[0046] In one embodiment, if the application scenario is a scenario of simultaneous transmission of multiple streams such as dual stream join, the speed limit rate of each stream can be set through step S110, and then the rate at which the data stream processing engine reads / writes data of each stream of the data source is limited based on the speed limit rate of each stream.

[0047] In another embodiment, when the application scenario is a scenario where multiple streams are transmitted simultaneously, such as dual-stream join, the total limit rate of multiple streams can be set through step S110, and then the read / write rate of each stream can be dynamically limited through the current limiting module based on the total limit rate of multiple streams, thereby improving flexibility.

[0048] The present application also provides a data transmission device, the device comprising:

[0049] Configuration module, used to configure the rate limit in the data stream processing engine using a declarative query language;

[0050] The current limiting module is used to enable the data stream processing engine to limit the rate at which the data stream processing engine reads / writes data from the data source based on the speed limit rate.

[0051] Optionally, the configuration module is used to configure rate parameters using a data definition language in a declarative query language, wherein the rate parameters are used to set a rate limit.

[0052] The current limiting module is used to pass the speed limit rate to the method of creating a data source to trigger the start of the current limiter in the connector; the rate at which the connector reads / writes data from the data source is limited by the current limiter.

[0053] Optionally, the configuration module is used to set a rate parameter in a configuration option of the connector when creating a connector of the data stream processing engine through the data definition language, the connector being used to connect to the data source.

[0054] The current limiting module is used to create a dynamic data source through a data source dynamic table factory; verify the rate parameters in the data definition language; and in response to the speed limit rate set in the data definition language, execute the step of passing the speed limit rate to the data source creation method to trigger the start of the current limiter in the connector.

[0055] The current limiting module is used to respond to a data transmission request, take out a preset number of tokens from the current token balance, wherein the current token balance includes the most recently generated new tokens, and the new tokens are generated at a fixed rate; in response to taking out the preset number of tokens, transmit the data corresponding to the data transmission request; in response to not taking the preset number of tokens, wait until the preset number of tokens are obtained.

[0056] The current limiting module is used to limit the average rate at which the connector reads / writes data from the data source based on the speed limit rate.

[0057] The current limiting module is used to convert the read data of the connector into the number of bytes; and determine the preset number based on the number of bytes.

[0058] See also Figure 6 , Figure 6 The electronic device 20 of the present application includes a processor 22, which is used to execute instructions to implement the method of any of the above embodiments of the present application and any non-conflicting combination thereof.

[0059] The processor 22 may also be referred to as a CPU (Central Processing Unit). The processor 22 may be an integrated circuit chip having signal processing capabilities. The processor 22 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor, or the processor 22 may also be any conventional processor, etc.

[0060] The electronic device 20 may further include a memory 21 for storing instructions and data required for the processor 22 to run.

[0061] See also Figure 7 , Figure 7 It is a schematic diagram of the structure of the computer-readable storage medium in the implementation mode of the present application. The computer-readable storage medium 30 of the embodiment of the present application stores instructions / program data 31, and when the instructions / program data 31 are executed, the method provided by any embodiment of the above-mentioned method of the present application and any non-conflicting combination is implemented. Among them, the instructions / program data 31 can form a program file and be stored in the above-mentioned storage medium 30 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) executes all or part of the steps of the various implementation methods of the present application. The aforementioned storage medium 30 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a computer, a server, a mobile phone, a tablet and other devices.

[0062] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0063] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0064] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0065] The above are only implementation methods of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data transmission method, characterized in that: The method comprises: When creating a connector of a data stream processing engine by using a data definition language, the connector is used to connect to a data source, and a rate parameter is set in a configuration option of the connector, and the rate parameter is used to set a speed limit rate; Create a dynamic data source through the data source dynamic table factory; Verifying the rate parameter in the data definition language; In response to setting the rate limit in the data definition language, passing the rate limit to a method for creating a data source to trigger the start of a rate limiter in a connector; The rate at which the connector reads / writes data from the data source is limited by the rate limiter.

2. The data transmission method according to claim 1, characterized in that: The limiting the rate at which the connector reads / writes data from the data source by the current limiter includes: In response to a data transmission request, taking a preset number of tokens from a current token balance, wherein the current token balance includes a newly generated token, and the new token is generated at a fixed rate; In response to taking out a preset number of tokens, transmitting data corresponding to the data transmission request; In response to not obtaining a preset number of tokens, waiting until a preset number of tokens is obtained.

3. The data transmission method according to claim 2, characterized in that: The limiting the rate at which the connector reads / writes data from the data source by the current limiter includes: The average rate at which the connector reads / writes data from the data source is limited based on the rate limit.

4. The data transmission method according to claim 3, characterized in that: The step of taking out a preset number of tokens from a current token balance in response to a data transmission request comprises: Convert the data read by the connector into bytes; The preset number is determined based on the number of bytes.

5. A data transmission device, characterized in that: The device comprises: A configuration module, used for setting a rate parameter in a configuration option of a data stream processing engine when creating a connector of the data stream processing engine by a data definition language, the connector being used to connect a data source, the rate parameter being used to set a speed limit rate; A current limiting module is used to create a dynamic data source through a data source dynamic table factory; verify the rate parameters in the data definition language; in response to the speed limit rate set in the data definition language, pass the speed limit rate to the data source creation method to trigger the start of the current limiter in the connector; and limit the rate at which the connector reads / writes data from the data source through the current limiter.

6. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to execute instructions to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions / program data, and the instructions / program data are used to be executed to implement the method of any one of claims 1-4.

Images