Kafka data automated stream processing method and apparatus

Abstract

The disclosure provides a Kafka data automatic stream processing method, relates to the technical field of distribution, and can be applied to the technical field of finance. The method comprises the following steps: receiving stream processing rule information input by a user; verifying the stream processing rule information; and performing stream data processing according to the stream processing rule information after it is determined that the stream processing rule information passes the verification. The disclosure further provides a Kafka data automatic stream processing device, equipment, storage medium and program product.

Description

Technical Field

[0001] This disclosure relates to the field of distributed technology, specifically to the field of stream processing technology, and more specifically to a method, apparatus, device, storage medium, and program product for automated Kafka data stream processing. Background Technology

[0002] Kafka, as a high-throughput distributed messaging system, has been widely adopted in the actual business operations of many large companies. Although Kafka provides the stream processing component Kafka Streams in the community, it is not a stream processing framework, but rather a library for processing streaming data. Users who want to process Kafka streaming data need to call the relevant APIs themselves. This approach is not user-friendly, requiring users to learn how to use the APIs and handle exceptions. Furthermore, the APIs provided by Kafka Streams can only process messages within the cluster, lacking cross-cluster stream processing capabilities.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a service-oriented method, apparatus, device, storage medium and program product for automating cross-cluster Kafka data stream processing.

[0005] According to a first aspect of this disclosure, a method for automated stream processing of Kafka data is provided, the method comprising:

[0006] Receive user-inputted stream processing rule information;

[0007] The stream processing rule information is validated; and

[0008] After confirming that the stream processing rule information has passed the verification, streaming data processing is performed according to the stream processing rule information.

[0009] According to embodiments of this disclosure, performing streaming data processing based on the streaming processing rule information includes:

[0010] Start a new thread to parse the stream processing rule information;

[0011] Based on the parsed stream processing rule information, a stream processing service is provided to external parties, which is used to process Kafka message data.

[0012] According to embodiments of this disclosure, starting a new thread to parse the stream processing rule information includes:

[0013] Based on the stream processing rule information, determine the source cluster identifier, source topic name, destination cluster identifier, destination topic name, rule type, processing object, and specific rule content.

[0014] According to embodiments of this disclosure, providing stream processing services based on the parsed stream processing rule information includes:

[0015] Determine the message inflow target based on the source cluster identifier and source topic;

[0016] Determine the message outflow target based on the destination cluster identifier and destination topic;

[0017] The message processing interface to be invoked is determined based on the rule type;

[0018] The parsed details of the processing object and rules are then used as interface parameters for the message processing interface; and

[0019] Process Kafka message data according to the message inflow target, the message outflow target, and the interface parameters.

[0020] According to embodiments of this disclosure, the stream processing rule information further includes rule operation types, and the verification of the stream processing rule information includes:

[0021] The source cluster identifier, source topic name, destination cluster identifier, and destination topic name are validated according to the rule operation type, wherein the rule operation type includes adding a rule, modifying a rule, and deleting a rule.

[0022] According to embodiments of this disclosure, the method further includes:

[0023] Generate streaming reports periodically during streaming data processing.

[0024] According to embodiments of this disclosure, the operational status and performance data of the stream processing service are monitored periodically; and

[0025] Generate a stream processing report based on the operating status and performance data.

[0026] A second aspect of this disclosure provides an apparatus for automated streaming processing of Kafka data, the apparatus comprising:

[0027] The acquisition module is used to receive stream processing rule information input by the user;

[0028] A verification module is used to verify the stream processing rule information; and

[0029] The execution module is used to perform streaming data processing according to the streaming processing rule information after determining that the streaming processing rule information has passed the verification.

[0030] According to embodiments of this disclosure, the execution module includes a parsing submodule and a stream processing service submodule.

[0031] The parsing submodule is used to start a new thread to parse the stream processing rule information;

[0032] The stream processing service submodule is used to provide stream processing services to external parties based on the parsed stream processing rule information. The stream processing service is used to process Kafka message data.

[0033] According to an embodiment of this disclosure, the parsing submodule includes a first determining unit.

[0034] The first determining unit is used to determine the source cluster identifier, source topic name, destination cluster identifier, destination topic name, rule type, processing object, and specific rule content based on the stream processing rule information.

[0035] According to embodiments of this disclosure, the stream processing service submodule includes: a second determining unit, a third determining unit, a fourth determining unit, a fifth determining unit, and a processing unit.

[0036] The second determining unit is used to determine the message inflow target based on the source cluster identifier and the source topic;

[0037] The third determining unit is used to determine the message outflow target based on the destination cluster identifier and the destination topic;

[0038] The fourth determining unit is used to determine the message processing interface to be invoked based on the rule type;

[0039] The fifth determining unit is used to parse the processing object and the specific content of the rule, and then use them as interface parameters for the message processing interface; and

[0040] The processing unit is used to process Kafka message data according to the message inflow target, the message outflow target, and the interface parameters.

[0041] According to an embodiment of this disclosure, the verification module includes a verification submodule.

[0042] The verification submodule is used to verify the source cluster identifier, the source topic name, the destination cluster identifier, and the destination topic name according to the rule operation type, wherein the rule operation type includes adding rules, modifying rules, and deleting rules.

[0043] According to embodiments of this disclosure, the device further includes a monitoring module.

[0044] The monitoring module is used to generate stream processing reports periodically during the execution of streaming data processing.

[0045] According to embodiments of this disclosure, the monitoring module includes a monitoring submodule and a report generation submodule.

[0046] The monitoring submodule is used to periodically monitor the running status and performance data of the stream processing service; and

[0047] The report generation submodule is used to generate stream processing reports based on the running status and the performance data.

[0048] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the Kafka data automation stream processing method described above.

[0049] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the Kafka data automation stream processing method described above.

[0050] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described Kafka data automation stream processing method.

[0051] This disclosure provides an automated Kafka data stream processing method that receives user-input stream processing rule information, generated based on a template. To ensure successful execution of the stream processing rules, the rules are validated. Upon successful validation, streaming data processing is performed according to the rules. By implementing automated stream processing of Kafka data through a service-oriented approach to stream processing rules, this method offers cross-cluster stream processing capabilities compared to related technologies that use the Kafka Streams component for processing streaming data, while reducing user costs. Attached Figure Description

[0052] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0053] Figure 1 The illustration schematically depicts an application scenario of the Kafka data automation stream processing method, apparatus, device, storage medium, and program product according to embodiments of this disclosure;

[0054] Figure 2This schematically illustrates a system architecture diagram of a Kafka data automation stream processing apparatus provided according to embodiments of the present disclosure;

[0055] Figure 3 schematically illustrates a flowchart of an automated Kafka data stream processing method provided according to an embodiment of the present disclosure;

[0056] Figure 4 This illustration schematically shows one of the flowcharts of a method for performing streaming data processing based on the streaming rule information provided according to an embodiment of the present disclosure;

[0057] Figure 5 This schematically illustrates a second flowchart of a method for performing streaming data processing based on the streaming rule information provided in an embodiment of the present disclosure;

[0058] Figure 6 A flowchart illustrating a stream processing report generation method provided according to an embodiment of the present disclosure is shown schematically.

[0059] Figure 7 A schematic diagram illustrating a structural block diagram of a Kafka data automation stream processing apparatus according to an embodiment of the present disclosure; and

[0060] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a Kafka data automated stream processing method according to an embodiment of the present disclosure. Detailed Implementation

[0061] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0062] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0063] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0064] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0065] First, the terms appearing in the embodiments of this disclosure will be explained:

[0066] Kafka: Positioned as a distributed streaming processing platform, it is widely used for its high throughput, persistence, horizontal scalability, and support for streaming data processing.

[0067] Message Topic: A logical unit on the message engine that stores different types of messages, used to distinguish specific business logic. When a producer produces a message, it specifies the topic to which the message belongs. When a consumer consumes a message, it specifies the topic and retrieves the messages produced by the producer within that topic.

[0068] Kafka, as a high-throughput distributed messaging system, has been widely adopted in the actual business operations of many large companies. Although Kafka provides the stream processing component Kafka Streams, it is not a stream processing framework, but rather a library for processing streaming data. When users need to process Kafka streaming data, they need to manually call the relevant APIs. This approach requires users to have certain API usage experience and the ability to handle exceptions. Furthermore, due to the isolation between messages in the cluster, the APIs provided by Kafka Streams can only process messages within the cluster and lack cross-cluster stream processing capabilities.

[0069] Based on the above-mentioned technical problems, embodiments of this disclosure provide a method for automated streaming processing of Kafka data. The method includes: receiving streaming processing rule information input by a user; verifying the streaming processing rule information; and, after determining that the streaming processing rule information has passed verification, performing streaming data processing according to the streaming processing rule information.

[0070] Figure 1 The illustration schematically depicts an application scenario of the Kafka data automation stream processing method, apparatus, device, storage medium, and program product according to embodiments of the present disclosure. Figure 2 A schematic diagram of the system architecture of a Kafka data automation stream processing apparatus provided according to an embodiment of the present disclosure is shown.

[0071] like Figure 1As shown, application scenario 100 according to this embodiment may include a Kafka data stream processing scenario. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0072] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0073] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0074] Server 105 can be a Kafka data automation stream processing server, which executes the Kafka data automation stream processing method provided in this embodiment of the disclosure, receives stream processing rule information input by the user through terminal devices at 101, 102, and 103, verifies the legality of the stream processing rule information, and then performs streaming data processing according to the stream processing rule information.

[0075] It should be noted that the Kafka data automated stream processing method provided in this disclosure embodiment can generally be executed by server 105. Correspondingly, the Kafka data automated stream processing device provided in this disclosure embodiment can generally be located in server 105. The Kafka data automated stream processing method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the Kafka data automated stream processing device provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0076] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0077] It should be noted that the Kafka data automated stream processing method and apparatus determined in the embodiments of this disclosure can be used in the field of distributed technology, the field of financial technology, and any field other than the financial field. The application field of the Kafka data automated stream processing method and apparatus determined in the embodiments of this disclosure is not limited.

[0078] like Figure 2 As shown in the embodiments of this disclosure, the Kafka automated stream processing device includes a stream processing rule information receiving module, a rule verification module, a rule execution module, and a monitoring module. The Kafka cluster 1101 contains multiple topics 1102, each isolated from the others and without data transfer. When a user 1103 needs to process topic data and transfer it to another topic, this requirement can be met through the Kafka automated stream processing device. First, the user 1103 generates stream processing rule information 1104 based on the template provided by the device. The rule verification module 1105 is mainly responsible for verifying whether the stream processing rule information 1104 is reasonable and valid, such as whether the source and destination topics have been created; whether the corresponding topic already has the rule when adding a rule; and whether the corresponding topic exists when deleting a rule, etc. After successful verification, the stream processing rule information 1104 is input to the rule execution module 1106 for the next step. The rule execution module 1106 starts a new process to independently execute a rule. During process execution, it periodically sends heartbeats and some data to the monitoring module 1107. The monitoring module 1107 checks whether the heartbeat verification function sent by the process at regular intervals is normal according to the stream processing rules. If no heartbeat is received within the specified time, the process is considered to have crashed and relevant prompts are issued.

[0079] The following will be based on Figure 1 The scene described and Figure 2 The system architecture described above, through Figures 3-5 The present disclosure provides a detailed description of the Kafka data automated stream processing method according to embodiments.

[0080] Figure 3 A flowchart illustrating an automated Kafka data stream processing method according to an embodiment of this disclosure is shown schematically. Figure 3 As shown, the Kafka data persistence method in this embodiment includes operations S210 to S230, which can be executed by a server or other computing devices.

[0081] In operation S210, the user inputs stream processing rule information.

[0082] In one example, in some business scenarios, when a user needs to process topic data in the message middleware and transfer it to another topic, such as splitting message data containing a certain keyword in topic A to topic B; copying message data containing a certain keyword in topic A to topic B; or merging message data containing a certain keyword in topic A with message data in topic B, etc., in related technologies, users need to call relevant APIs to implement the above message data processing. Since message data can only flow within the cluster, it lacks cross-cluster stream processing capabilities. This disclosure provides a service-oriented approach to stream processing, allowing users to input stream processing rule information based on preset templates and business requirements. The rule operation types include three main categories: adding rules, modifying rules, and deleting rules. Each record represents a stream processing rule, including source cluster identifier, source topic name, destination cluster identifier, destination topic name, rule type (including splitting rules, merging rules, copying rules, and filtering rules), and processing object (including incremental message key, incremental message value, full message key, full message value, and the specific content of the rule). For example, the routing rule [A:topic1, B:topic2] means that messages containing the keyword A are placed in topic1, and messages containing the keyword B are placed in topic2.

[0083] In operation S220, the stream processing rule information is verified.

[0084] According to embodiments of this disclosure, the stream processing rule information includes rule operation types. The source cluster identifier, source topic name, destination cluster identifier, and destination topic name are validated based on the rule operation type, wherein the rule operation type includes adding rules, modifying rules, and deleting rules.

[0085] In one example, to ensure the smooth execution of stream processing rules, the stream processing rules in operation S210 are validated to determine their validity. The validation checks the source cluster identifier, source topic name, destination cluster identifier, and destination topic name based on the rule operation type. Specifically, this includes checking if the source and destination topics have been created; if the rule operation type is "add rule," checking if the corresponding topic already has the rule; and if the rule operation type is "delete rule," checking if the corresponding topic has the rule. If the rule validation fails, the user is prompted with the correct field in the specified row. If all rule validations pass, the stream processing rule information is input into the rule execution module for the next step.

[0086] In operation S230, after confirming that the stream processing rule information has passed the verification, streaming data processing is performed according to the stream processing rule information.

[0087] In one example, after verifying that the stream processing rule information has passed the check, the rule information is parsed to obtain the rule content and the processing object. An independent thread is then started to execute the rule, providing streaming data services and processing Kafka message data. For details, please refer to [link to relevant documentation]. Figure 4 Operations S231 and S232 shown will not be described in detail here.

[0088] This disclosure provides an automated Kafka data stream processing method that receives user-input stream processing rule information, generated based on a template. To ensure successful execution of the stream processing rules, the rules are validated. Upon successful validation, streaming data processing is performed according to the rules. By implementing automated stream processing of Kafka data through a service-oriented approach to stream processing rules, this method offers cross-cluster stream processing capabilities compared to related technologies that use the Kafka Streams component for processing streaming data, while reducing user costs.

[0089] The following will be through Figure 4 This invention describes the process of performing streaming data processing based on the streaming processing rule information in the embodiments of this disclosure. Figure 4 One of the flowcharts illustrating a method for performing streaming data processing based on the streaming rule information provided according to an embodiment of the present disclosure is shown schematically. Figure 5 A flowchart illustrating a method for performing streaming data processing based on the streaming rule information according to an embodiment of this disclosure is shown in the diagram as follows: Figure 4 As shown, operation S230 includes operations S231 to S232.

[0090] In operation S231, a new thread is started to parse the stream processing rule information.

[0091] According to embodiments of this disclosure, the source cluster identifier, source topic name, destination cluster identifier, destination topic name, rule type, processing object, and specific rule content are determined based on the stream processing rule information.

[0092] In operation S232, a stream processing service is provided to the outside world based on the parsed stream processing rule information. The stream processing service is used to process Kafka message data.

[0093] like Figure 5 As shown, operation S232 includes operations S2321 to S2325.

[0094] In operation S2321, the message inflow target is determined based on the source cluster identifier and source topic; in operation S2322, the message outflow target is determined based on the destination cluster identifier and destination topic; in operation S2323, the message processing interface to be called is determined based on the rule type; in operation S2324, the processing object and the specific content of the rule are parsed and used as the interface parameters of the message processing interface; in operation S2325, the Kafka message data is processed according to the message inflow target, the message outflow target, and the interface parameters.

[0095] In one example, a new thread is started to parse the stream processing rule information, determining the source cluster identifier, source topic name, destination cluster identifier, destination topic name, rule type, processing object, and rule content. Then, based on the source cluster identifier and source topic, the message inflow target is determined; based on the destination cluster identifier and destination topic, the message outflow target is determined; and based on the rule type, the message processing interface to be called is determined. The processing object and rule content are parsed and used as interface parameters. For example, it determines whether the operation is on a key or value, and which topics or partitions contain which keywords are output to.

[0096] In one example, suppose topicA contains business messages from all regions of China, where the message key stores the corresponding region information. Due to business requirements, downstream applications want to launch multiple consumers to consume messages from a single region, and messages from other regions cannot be included. The stream processing rule information generated according to this requirement is as follows:

[0097]

[0098] The rule execution module parses the above rules and then starts a new process to consume messages starting from the earliest message in topic 0 of Kafka1. Since the processing object is a full message key, consumption starts from the earliest message. If the processing object is an incremental message, consumption starts from the latest message. Messages with the key "Beijing" are sent to topic 1 of Kafka2, messages with the key "Guangzhou" are sent to topic 2 of Kafka2, and so on. All subsequent messages written to topic 0 will also be distributed according to this logic. In this way, topics 1, 2, ... in Kafka2 will store independent messages for each region, which are then consumed and processed independently by downstream applications.

[0099] To ensure the availability of the stream processing service, this embodiment of the disclosure also monitors the execution process of the stream processing rules and generates stream processing reports periodically during the execution of stream data processing. Figure 6 A flowchart illustrating a stream processing report generation method provided according to an embodiment of the present disclosure is shown schematically.

[0100] like Figure 6 As shown, this includes operations S310 to S320.

[0101] When operating S310, periodically monitor the running status and performance data of the stream processing service.

[0102] In operation S320, a stream processing report is generated based on the operating status and the performance data.

[0103] In one example, during process execution, heartbeats and other data are periodically sent to the monitoring module. The monitoring module verifies the heartbeat function sent periodically by the process according to the stream processing rules. If no heartbeat is received within a specified time, the process is deemed to have crashed, and relevant prompts are issued. Simultaneously, the stream processing rule execution process also compiles inflow and outflow performance data before and after message processing and sends it to the monitoring module. Performance data includes the average TPS, peak TPS, average traffic, peak traffic, process CPU utilization, process memory utilization, etc., within one minute. The monitoring module processes the running status and performance data of the stream processing service and then generates stream processing reports to provide feedback to the user.

[0104] The Kafka data automated stream processing method provided in this disclosure encapsulates a stream processing API based on Kafka, offering stream processing service capabilities. Users input stream processing rules according to preset templates and actual business needs. These user-input rules are then encapsulated into a stream processing service to process streaming data. This effectively reduces user costs and enables stream processing of messages between clusters.

[0105] Based on the above-described automated Kafka data stream processing method, this disclosure also provides an automated Kafka data stream processing apparatus. The following will combine... Figure 7 The device is described in detail.

[0106] Figure 7 The diagram illustrates a structural block diagram of a Kafka data automation stream processing apparatus according to an embodiment of the present disclosure.

[0107] like Figure 7 As shown, the Kafka data persistence device 700 in this embodiment includes an acquisition module 710, a verification module 720, and an execution module 730.

[0108] The acquisition module 710 is used to receive stream processing rule information input by the user. In one embodiment, the acquisition module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0109] The verification module 720 is used to verify the stream processing rule information. In one embodiment, the verification module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0110] The execution module 730 is used to perform streaming data processing according to the streaming rule information after determining that the streaming rule information has passed the verification. In one embodiment, the execution module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0111] According to embodiments of this disclosure, the execution module includes a parsing submodule and a stream processing service submodule.

[0112] The parsing submodule is used to start a new thread to parse the stream processing rule information. In one embodiment, the parsing submodule can be used to perform the operation S231 described above, which will not be repeated here.

[0113] The stream processing service submodule is used to provide stream processing services to external parties based on the parsed stream processing rule information. These stream processing services are used to process Kafka message data. In one embodiment, the stream processing service submodule can be used to execute the operation S232 described above, which will not be repeated here.

[0114] According to an embodiment of this disclosure, the parsing submodule includes a first determining unit.

[0115] The first determining unit is configured to determine the source cluster identifier, source topic name, destination cluster identifier, destination topic name, rule type, processing object, and specific rule content based on the stream processing rule information. In one embodiment, the first determining unit may be configured to perform the operation S231 described above, which will not be repeated here.

[0116] According to embodiments of this disclosure, the stream processing service submodule includes: a second determining unit, a third determining unit, a fourth determining unit, a fifth determining unit, and a processing unit.

[0117] The second determining unit is used to determine the message inflow target based on the source cluster identifier and the source topic. In one embodiment, the second determining unit can be used to perform the operation S2321 described above, which will not be repeated here.

[0118] The third determining unit is used to determine the message outflow target based on the destination cluster identifier and the destination topic. In one embodiment, the third determining unit can be used to perform the operation S2322 described above, which will not be repeated here.

[0119] The fourth determining unit is used to determine the message processing interface to be called based on the rule type. In one embodiment, the fourth determining unit can be used to perform the operation S2323 described above, which will not be repeated here.

[0120] The fifth determining unit is used to parse the processing object and the specific content of the rule, and then use them as interface parameters for the message processing interface. In one embodiment, the fifth determining unit can be used to execute the operation S2324 described above, which will not be repeated here.

[0121] The processing unit is used to process Kafka message data according to the message inflow target, the message outflow target, and the interface parameters. In one embodiment, the processing unit can be used to perform the operation S2325 described above, which will not be repeated here.

[0122] According to an embodiment of this disclosure, the verification module includes a verification submodule.

[0123] The verification submodule is used to verify the source cluster identifier, the source topic name, the destination cluster identifier, and the destination topic name according to the rule operation type, wherein the rule operation type includes adding rules, modifying rules, and deleting rules. In one embodiment, the verification submodule can be used to perform the operation S220 described above, which will not be repeated here.

[0124] According to embodiments of this disclosure, the device further includes a monitoring module.

[0125] The monitoring module is used to generate stream processing reports periodically during the execution of streaming data processing.

[0126] According to embodiments of this disclosure, the monitoring module includes a monitoring submodule and a report generation submodule.

[0127] The monitoring submodule is used to periodically monitor the running status and performance data of the stream processing service. In one embodiment, the monitoring submodule can be used to perform the operation S310 described above, which will not be repeated here.

[0128] The report generation submodule is used to generate stream processing reports based on the running status and the performance data. In one embodiment, the report generation submodule can be used to perform the operation S320 described above, which will not be repeated here.

[0129] According to embodiments of this disclosure, any plurality of modules among the acquisition module 710, verification module 720, and execution module 730 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the acquisition module 710, verification module 720, and execution module 730 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the acquisition module 710, verification module 720, and execution module 730 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0130] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a Kafka data automated stream processing method according to an embodiment of the present disclosure.

[0131] like Figure 8 As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0132] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0133] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0134] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the Kafka data automated stream processing method according to embodiments of this disclosure.

[0135] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0136] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the Kafka data automation stream processing method provided in embodiments of this disclosure.

[0137] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0138] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0139] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0140] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0142] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0143] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for automated stream processing of Kafka data, characterized in that, The method includes: Receive stream processing rule information input by the user; The stream processing rule information is validated; and After confirming that the stream processing rule information has passed the verification, stream data processing is performed according to the stream processing rule information; The step of performing streaming data processing according to the streaming processing rule information includes: Start a new thread to parse the stream processing rule information; Based on the parsed stream processing rule information, stream processing services are provided to external parties, and the stream processing services are used to process Kafka message data; The process of starting a new thread to parse the stream processing rule information includes: Based on the stream processing rule information, determine the source cluster identifier, source topic name, destination cluster identifier, destination topic name, rule type, processing object, and specific rule content; The provision of stream processing services based on the parsed stream processing rule information includes: Determine the message inflow target based on the source cluster identifier and source topic; Determine the message outflow target based on the destination cluster identifier and destination topic; The message processing interface to be invoked is determined based on the rule type; The parsed details of the processing object and rules are then used as interface parameters for the message processing interface; and Process Kafka message data according to the message inflow target, the message outflow target, and the interface parameters.

2. The method according to claim 1, characterized in that, The stream processing rule information also includes rule operation types, and the verification of the stream processing rule information includes: The source cluster identifier, source topic name, destination cluster identifier, and destination topic name are validated according to the rule operation type, wherein the rule operation type includes adding a rule, modifying a rule, and deleting a rule.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Generate streaming reports periodically during streaming data processing.

4. The method according to claim 3, characterized in that, During streaming data processing, periodically generate streaming processing reports, including: Regularly monitor the running status and performance data of the stream processing service; and Generate a stream processing report based on the operating status and performance data.

5. A Kafka data automated stream processing device, characterized in that, The device includes: The acquisition module is used to receive stream processing rule information input by the user; A verification module is used to verify the stream processing rule information; and The execution module is used to perform streaming data processing according to the streaming rule information after determining that the streaming rule information has passed the verification. The execution module includes a parsing submodule and a stream processing service submodule. The parsing submodule is used to start a new thread to parse the stream processing rule information. The stream processing service submodule is used to provide stream processing services to the outside world based on the parsed stream processing rule information. The stream processing service is used to process Kafka message data. The parsing submodule includes a first determining unit, which is used to determine the source cluster identifier, source topic name, destination cluster identifier, destination topic name, rule type, processing object, and specific rule content based on the stream processing rule information. The stream processing service submodule includes: a second determining unit, a third determining unit, a fourth determining unit, a fifth determining unit, and a processing unit; the second determining unit is used to determine the message inflow target based on the source cluster identifier and the source topic; the third determining unit is used to determine the message outflow target based on the destination cluster identifier and the destination topic; the fourth determining unit is used to determine the message processing interface to be called based on the rule type; the fifth determining unit is used to parse the processing object and the specific content of the rule and use them as interface parameters for the message processing interface; and the processing unit is used to process Kafka message data based on the message inflow target, the message outflow target, and the interface parameters.

6. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors execute the Kafka data automation stream processing method according to any one of claims 1 to 4.

7. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the Kafka data automation stream processing method according to any one of claims 1 to 4.

8. A computer program product comprising a computer program that, when executed by a processor, implements the Kafka data automation stream processing method according to any one of claims 1 to 4.

Images