Configuration method and device of real-time data processing service and related equipment

Abstract

The invention relates to the field of streaming data processing, in particular to a real-time data processing service configuration method and device and related equipment. The method comprises the steps of determining task information of a first task in response to process editing of a real-time data processing service corresponding to the first task on a first interface by a user; in response to a received starting signal used for indicating to start the first task, sending task information of the first task to the Zookeeper for storage according to the starting signal, and enabling a sentinel service monitoring the task information of the first task to call the task information of the first task stored in the Zookeeper to an Flink cluster, the task configuration module is used for the Flink cluster to configure real-time data processing service corresponding to the first task according to the task information of the first task. The technical problem that time and labor are consumed in the process of adjusting codes of data processing logic in the prior art can be solved.

Description

Technical Field

[0001] The present application relates to the field of streaming data processing, and in particular to a configuration method, apparatus and related equipment for real-time data processing services. Background Art

[0002] In the existing technology, the Flink cluster deployed based on the open source stream processing framework Apache Flink can meet the needs of many business scenarios such as data migration, data warehouse construction, data integration, data conversion and cleaning for real-time processing of real-time collected data streams. Therefore, the application of Apache Flink is becoming more and more extensive.

[0003] Before using the Flink cluster to process data streams in real time, you need to configure the real-time data processing service to the Flink cluster so that the Flink cluster has the ability to process data streams in real time. The process of configuring the real-time data processing service includes: developers write codes using the interfaces provided by the Flink cluster, and the Flink cluster runs the codes to implement the configuration of the real-time data processing service.

[0004] In actual implementation, since developers are required to write code during the configuration of real-time data processing services, if the data processing logic indicated by the code needs to be adjusted, the developers also need to adjust the code. The whole process is not only time-consuming and labor-intensive, but may also add bugs to the code during the adjustment process, ultimately having an adverse impact on the real-time processing of data streams by the Flink cluster. Summary of the invention

[0005] In view of this, the purpose of the present application is to provide a configuration method, device and related equipment for real-time data processing services to solve the technical problem in the prior art that adjustments to data processing logic also require developers to adjust the code, resulting in a time-consuming and labor-intensive adjustment process that is likely to have an adverse effect on the real-time processing of data streams.

[0006] In a first aspect, the present application provides a configuration method for a real-time data processing service, which is applied to a service configuration system, wherein the service configuration system establishes a connection with a distributed application coordination service Zookeeper, and the Zookeeper establishes a connection with a sentinel service configured in a Flink cluster; the method comprises:

[0007] In response to a user performing a process-based editing on a first interface for a real-time data processing service corresponding to a first task, determining task information of the first task;

[0008] Wherein, the first task indicates a task for processing data;

[0009] In response to receiving a start signal for instructing to start the first task, according to the start signal, the task information of the first task is sent to the Zookeeper for storage, and the Sentinel service that monitors the task information of the first task retrieves the task information of the first task stored in the Zookeeper to the Flink cluster, so that the Flink cluster configures a real-time data processing service corresponding to the first task according to the task information of the first task.

[0010] In a second aspect, the present application provides a service configuration system, the system comprising: a network device, the network device being used to execute the configuration method of the real-time data processing service as described above.

[0011] In a third aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the memory is used to store software programs, and the processor runs or executes the software programs stored in the memory so that the electronic device performs the configuration method of the real-time data processing service as described above.

[0012] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program codes executed by a processor, wherein the program codes include a configuration method for executing the above-mentioned real-time data processing service.

[0013] In a fifth aspect, the present application provides a computer program product, including program code, which enables a computer to execute the configuration method of a real-time data processing service as described above when the computer program product is run on the computer.

[0014] Beneficial effects:

[0015] The present application provides a configuration method for a real-time data processing service, which is applied to a service configuration system, wherein the service configuration system establishes a connection with the distributed application coordination service Zookeeper, and the Zookeeper establishes a connection with the sentinel service configured in the Flink cluster; the method comprises: in response to a user performing a process-based editing on a first interface for a real-time data processing service corresponding to a first task, determining task information of the first task; wherein the first task indicates a task for processing data; in response to receiving a start signal for indicating the start of the first task, according to the start signal, sending the task information of the first task to Zookeeper for storage, and causing the sentinel service that monitors the task information of the first task to retrieve the task information of the first task stored in Zookeeper to the Flink cluster, so that the Flink cluster can configure the real-time data processing service corresponding to the first task according to the task information of the first task;

[0016] In summary, in terms of task information writing, the configuration scheme of the real-time data processing service provided by the present application replaces the editing operation of the job code in the prior art by providing a first interface that can process-edit the real-time data processing service corresponding to the target task, which can effectively reduce the operational threshold for the configuration of the real-time data processing service. Even if the staff does not have the ability to write code, they can also implement the configuration of the real-time data processing service for the Flink cluster without considering the problem of code bugs. The configuration scheme of the real-time data processing service can also greatly reduce the configuration cost of the real-time data processing service and save the time and energy of the staff. In terms of the submission of task information, the configuration scheme of the real-time data processing service automatically retrieves the task information of the target task stored in Zookeeper through the sentinel service. The entire process does not require operations on the job code, and can effectively reduce the operational threshold for the configuration of the real-time data processing service. It can effectively avoid the adverse effects of real-time processing of data streams caused by adjustments to the code by developers, and can also save the time and energy of the staff. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. The following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A structural diagram of the service configuration system provided in the embodiment of the present application;

[0019] Figure 2 An example diagram of the first interface provided in an embodiment of the present application;

[0020] Figure 3 An example diagram of the second interface provided in an embodiment of the present application;

[0021] Figure 4 A flowchart of a method for configuring a real-time data processing service provided in an embodiment of the present application;

[0022] Figure 5 An example diagram of the second information area in the first interface provided in an embodiment of the present application;

[0023] Figure 6 An example diagram of the third interface corresponding to the newly created operator provided in the embodiment of the present application;

[0024] Figure 7 A logical diagram of operator information and task information provided in an embodiment of the present application;

[0025] Figure 8 An example diagram of selecting an operator in the third interface provided in an embodiment of the present application;

[0026] Fig. 9 A logic diagram of the first part and the second part of each operator provided in the embodiment of the present application;

[0027] Fig.10 An example diagram of the fourth interface corresponding to the general operator provided in the embodiment of the present application;

[0028] Fig.11 This is an example diagram of the fourth interface corresponding to the built-in operator provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In the prior art, a Flink cluster configured with a real-time data processing service has the advantages of being able to process high-throughput data streams while maintaining extremely low latency and ensuring the real-time nature of data processing, and therefore has very wide applications.

[0030] The process of building a Flink cluster with real-time data processing services based on Apache Flink usually includes: installing the open source stream processing framework Apache Flink on the server, deploying the Flink cluster by setting cluster parameters and cluster mode on Apache Flink, configuring data sources on the Flink cluster, and writing and submitting Flink jobs. Among them, the Flink job indicates the data processing logic of the real-time data processing service. The content of the Flink job mainly includes: defining data sources, data conversion logic, and data sinks. When the Flink cluster executes the submitted Flink job, the configuration of the real-time data processing service is completed.

[0031] Among them, Apache Flink provides an API (Application Programming Interface) for writing Flink jobs. Developers use the API to write the job code of the Flink job. After the job code is written, the job code is packaged into a JAR file. The JAR file is submitted to the Flink cluster through the command line tool provided by Apache Flink, and the Flink cluster runs the JAR file. After the Flink cluster runs the JAR file, the Flink cluster decomposes the Flink job into multiple work tasks, and then schedules and executes the above work tasks on multiple nodes in the Flink cluster. Among them, each work task corresponds to at least one operation step in the Flink job, and the operation step is executed by an operator. The operator defines the specific processing logic of the data, such as data conversion, filtering, and aggregation.

[0032] In actual applications, once the Flink cluster completes the configuration of the real-time data processing service, it can be used to perform real-time processing on the data stream. However, since the staff needs to write the job code during the configuration of the real-time data processing service, if the data processing logic indicated by the job code needs to be adjusted, the developers also need to adjust the job code. The whole process is not only time-consuming and labor-intensive, but may also add bugs to the code during the adjustment process, ultimately affecting the real-time processing of the data stream by the Flink cluster.

[0033] In order to solve the above technical problems, the present application proposes a configuration scheme for real-time data processing services. The configuration scheme for real-time data processing services is implemented based on a service configuration system 100, a distributed application coordination service Zookeeper 200, and a sentinel service 310 configured in a Flink cluster 300. The service configuration system 100 establishes a connection with Zookeeper 200, and the Zookeeper 200 establishes a connection with the sentinel service 310; Figure 1 As shown, Figure 1 The service configuration system 100 is a structural diagram of the service configuration system provided in the embodiment of the present application, and includes: a process configuration module 110 and a task status module 120. The service configuration system 100 is implemented through a network device; the network device may be a server or other terminal device with interactive functions.

[0034] The process configuration module 110 provides a first interface for editing the task information of the target task, such as Figure 2 As shown, Figure 2 This is an example diagram of the first interface provided in an embodiment of the present application. The first interface includes: a first information area 410 and a second information area 420; wherein the first information area 410 is used to edit the name, identification information and other information of the target task;

[0035] The second information area 420 is used to edit the task flow of the target task. Nodes and node relationships are displayed in the second information area 420. Nodes are used to visually represent operators. The square-shaped controls "New Operator" and "General Operator" in the second information area 420 are nodes, which are used to represent corresponding operators. The square-shaped controls "Input Source" and "Output Source" in the second information area 420 are also nodes, which are equivalent to the source operator and sink operator in the prior art, but are only used to indicate the "start execution" and "end execution" of the target task; the node relationship is used to visually represent the execution sequence of multiple nodes. The connecting line with an arrow between any two adjacent nodes in the second information area 420 is a node relationship, which can be used to represent the execution sequence of multiple operators.

[0036] In actual operation, the process-based editing process for the target task is a pre-construction process for the "real-time data processing service" corresponding to the target task; when the process-based editing of the target task is completed, after receiving the start signal for instructing to start the target task, the real-time data processing service corresponding to the target task can be called to enter the task execution process, and the task execution process is the process of real-time data processing through the real-time data service.

[0037] Preferably, to facilitate task management, not all real-time data processing services corresponding to target tasks that have completed the pre-build process will immediately enter the task execution process. In actual operation, the real-time data processing service should be in an adjustable state before entering the task execution process, that is, the user is allowed to make at least one adjustment to the real-time data processing service.

[0038] The embodiment of the present application inserts a task status adjustment process between the pre-construction process and the task execution process of the real-time data processing service, so that the user can adjust the real-time data processing service according to actual needs after completing the real-time data processing service corresponding to the target task of the pre-construction process; wherein the task status module 120 provides a second interface, which can be used to determine whether to send the task information of the target task to Zookeeper 200 for storage. Figure 3 As shown, Figure 3 This is an example diagram of the second interface provided in an embodiment of the present application. The second interface is used to edit the task status of each target task; the second interface displays multiple attribute columns for displaying and / or editing the task information of the target task.

[0039] In actual operation, when the real-time data processing service corresponding to the target task needs to be modified, the task can be stopped on the second interface, and the editing and adjustment page of the corresponding task can be selected; when the real-time data processing service corresponding to the target task is confirmed to be correct, the start of the target task can be indicated on the second interface, that is, the start signal is determined accordingly, and then the task information of the target task is sent to Zookeeper200 for storage according to the start signal.

[0040] In actual operation, if the task information of the target task is sent to Zookeeper200 for storage, the Sentinel service 310 will monitor the task information of the target task in Zookeeper200 and retrieve the task information of the target task to the Flink cluster 300, so that the Flink cluster 300 configures the real-time data processing service corresponding to the first task according to the task information of the first task; wherein the first task indicates a task for processing data.

[0041] In summary, in terms of writing task information, the configuration scheme of the real-time data processing service provided in the embodiment of the present application replaces the editing operation of the job code in the prior art by providing a first interface that can perform process-based editing of the real-time data processing service corresponding to the target task, which can effectively reduce the operational threshold for the configuration of the real-time data processing service. Even if the staff does not have the ability to write code, they can also implement the configuration of the real-time data processing service for the Flink cluster without considering the problem of code bugs. The configuration scheme of the real-time data processing service can also greatly reduce the configuration cost of the real-time data processing service and save the time and energy of the staff. In terms of submitting task information, the configuration scheme of the real-time data processing service automatically retrieves the task information of the target task stored in Zookeeper 200 through the sentinel service 310. The entire process does not require operations on the job code, which can effectively reduce the operational threshold for the configuration of the real-time data processing service, and can also save the time and energy of the staff.

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0043] First, the present application embodiment provides a configuration method for a real-time data processing service, such as Figure 4 As shown, Figure 4 This is a flow chart of a method for configuring a real-time data processing service provided in an embodiment of the present application. The method includes: S110 to S120, and the details are as follows:

[0044] S110: In response to the user performing a process-based editing on the first interface for the real-time data processing service corresponding to the first task, determining task information of the first task;

[0045] The first task indicates a task for processing data.

[0046] Specifically, after the data processing logic of the first task is determined according to actual needs, the first interface is displayed through the process configuration module 110, and the task information of the first task can be obtained by editing the task process of the first task according to the data processing logic of the first task on the first interface. Among them, the process configuration module 110 introduces technologies such as the Bpmn.js standard process engine as technical support, which is used to determine the task information by arranging the task process.

[0047] The first task in the embodiment of the present application corresponds to one of the multiple work tasks determined by the Flink cluster according to the job code in the prior art; in the embodiment of the present application, the task information corresponding to the first task and the second task mentioned later can be edited separately on the first interface provided by the process configuration module 110, without the need to write them uniformly in the form of code as in the prior art; since each task in the embodiment of the present application can be edited separately, the flexibility of making specific adjustments to each task is greatly increased, thereby improving the efficiency and accuracy of configuring real-time data processing services for the Flink cluster.

[0048] In one implementation, S110 includes: step (1), details of which are as follows:

[0049] Step (1) determining that the user edits the nodes and node relationships on the first interface to obtain task information of the first task;

[0050] The node indicates the operator used by the Flink cluster when processing data through the real-time data processing service; the task information includes: task identifier, operator identifier, and operator relationship.

[0051] Specifically, in the prior art, a Flink cluster processes jobs through multiple operators. Apache Flink supports multiple types of operators, including: transformation operators, aggregation operators, join operators, CoFlatMap operators, source operators, and sink operators.

[0052] In the prior art, the conversion operator is used to convert the format of the data according to preset rules, such as converting the data representing the date from "MM / DD / YYYY" to "YYYY-MM-DD"; the aggregation operator is used to summarize or calculate multiple elements in the data set according to certain rules, such as summing, averaging, and finding the maximum value; the connection operator is used to process table associations, integrating data from different data sources or different data tables according to one or more key fields; the union operator can be used to perform joint operations on data in two data streams; the source operator is used to define the input of the data, which is the starting point of data processing; the sink operator is used to define the output of the data, which is the end point of data processing; in the Flink cluster, the above operators can be combined to form a complex data processing pipeline to support various real-time data processing scenarios. For the sake of accurate description, the operators in the prior art mentioned above will be simply referred to as "existing operators" in the following, which are used to distinguish the operators newly constructed compared with the prior art in the embodiments of the present application.

[0053] In the embodiment of the present application, the process-based editing of the real-time data processing service corresponding to the first task on the first interface refers to editing the nodes and node relationships on the first interface.

[0054] like Figure 2 and Figure 5 As shown, Figure 5 This is an example diagram of the second information area in the first interface provided in an embodiment of the present application. Figure 2 In the example, the first information area 410 includes four edit boxes, which are used to fill in the task name, task identifier, task parallelism and task description respectively; the second information area 420 includes: an operator candidate area 421 and a process arrangement area 422.

[0055] In the first information area 410, the task name, task identifier and task description can all be filled in according to actual needs; the task parallelism indicates the number of nodes used to execute the first task, that is, the number of nodes that need to be configured with the real-time data processing service corresponding to the first task; in the second information area 420, the operator selection area 421 visually displays controls for multiple operators, specifically including: general operators, built-in operators and newly created operators; the operator selection area 421 also includes: controls indicating the start and end of the process, the input source is used to indicate the start of the process, and the output source is used to indicate the end of the process; among them, the general operator, the built-in operator and the newly created operator are different types of operators that are newly constructed in the embodiment of the present application compared to the prior art and are different from the existing operators; for the sake of accurate description, the operator newly constructed in the embodiment of the present application compared to the prior art will be simply referred to as a "constructed operator" to distinguish it from the "existing operator".

[0056] Among them, the difference between the "construction operator" and the "existing operator" is that: in the prior art, in order to ensure that it can be widely used in various data processing in this technical field, the "existing operator" only has the most basic data processing function. In actual operation, the user can only choose the above-mentioned "existing operator", but as the data processing requirements become higher and higher, the "existing operator" with only basic data processing functions obviously has very large application limitations and has become increasingly unable to adapt to actual data processing requirements; however, in the embodiment of the present application, the data processing function of the "construction operator" constructed based on the service configuration system 100 can be adaptively adjusted according to actual data processing requirements. In actual operation, any of the above-mentioned "existing operators" can be adjusted in different directions and to different degrees, so that the "construction operator" obtained after adaptive adjustment of the data processing function can meet various actual data processing requirements. In the process of editing nodes and node relationships through the first interface, first drag the "input source" in the operator selection area 421 to the process orchestration area 422 to indicate the start of the current process. Subsequently, according to the various "construction operators" required by the real-time data processing service corresponding to the first task in the data processing process, the "construction operator" in the operator selection area 421 can be dragged to the process orchestration area 422. Finally, drag the "output source" in the operator selection area 421 to the process orchestration area 422 to indicate the end of the current process. In the above process, it is necessary to connect the node relationships for each node; when the task process orchestration of the first task is completed, the task information of the first task can be determined according to the task process through the process configuration module 110.

[0057] In one implementation, the service configuration system 100 further includes: an operator configuration module 130; different types of "construction operators", namely, general operators, built-in operators and newly created operators are pre-constructed by the operator configuration module 130, and after construction, are used to be called in the process of arranging the task process of the first task by the process configuration module 110; the process of pre-constructing each "construction operator" by the operator configuration module 130 refers to the first part of the content A of each "construction operator" on the third interface displayed by the operator configuration module 130. 1 The editing process; the first part of the content A 1 The meaning of is described later.

[0058] like Figure 6 As shown, Figure 6 This is an example diagram of the third interface corresponding to the newly created operator provided in the embodiment of the present application. Figure 6The third information area 510, the fourth information area 520 and the fifth information area 530 in the third interface are shown; wherein the third information area 510 includes 4 edit boxes, and the 4 edit boxes are used to fill in the operator name, method name, class name and operator description respectively; in actual operation, the operator name is used to distinguish different "construction operators", which are not necessarily different types of "construction operators". The same type of "construction operators" can also be distinguished by the operator name; the method name is used when the target task corresponding to the method name is processed in the first interface. When different types of "construction operators" are selected in the operator selection area 421, the "construction operator" that meets the actual data processing needs can be dragged to the process arrangement area 422; the class name is the class name of the code filled in the fifth information area 530; the operator description is used to simply describe the purpose of the "construction operator" and / or the code. In actual operation, the class name in the third information area 510 can be filled in manually or automatically; in the process of automatically filling in the class name, when the class definition format "class xxxx{" is recognized in the code filled in the fifth information area 530, the class name "xxxx" can be extracted to the corresponding edit box in the third information area 510. Figure 5 The class name that should be identified and extracted is "myClass".

[0059] The fourth information area 520 also includes 4 edit boxes, which are respectively used to fill in the parameter information of the parameters involved in the code of the "construction operator". In the embodiment of the present application, the parameter information includes: parameter name, parameter type, whether it is a required item and description; wherein the parameter name and parameter type are used to define the parameter, and the parameter definition helps to clarify the input and expected behavior of the function or method in the code; "Determine whether it is a required item" indicates whether the parameter must be defined in advance, function parameters and method parameters must be defined, and the remaining parameters can be determined according to actual needs; description is used to briefly describe the parameter, and the specific content can be determined according to actual conditions;

[0060] For example, the four edit boxes in the fourth information area 520 may be filled with "ip", "String", "No", and "ewew" in sequence. In actual operation, filling in parameter information in the fourth information area 520 helps to quickly identify the parameter information in the code, and effectively avoids reading a large section of code to determine the parameter information.

[0061] The fifth information area 530 includes: a code editing box, which is used to fill in the code corresponding to the "construction operator", and the code indicates the data processing logic of the "construction operator".

[0062] It should be noted that in the embodiments of the present application, the formats of the third interfaces corresponding to different types of "construction operators", namely general operators, built-in operators and newly created operators, are usually the same, for example, they all include a third information area 510, a fourth information area 520 and a fifth information area 530; the difference is that the editable contents of different types of general operators, built-in operators and newly created operators in different information areas are different.

[0063] In actual applications, the contents of the four edit boxes in the third information area 510, the contents of the four edit boxes in the fourth information area 520, and the code filled in the fifth information area 530 can all be determined according to actual needs, and this application does not make any specific limitations on this.

[0064] In the prior art, the functions that can be achieved by "existing operators" such as conversion operators cannot be adaptively adjusted. In actual operation, if only multiple operators in the prior art can be used, when editing the target task, one can only stick to how to select from the multiple operators in the above-mentioned prior art, resulting in the inability to edit and obtain the accurate target task; however, from the above introduction to general operators, built-in operators and newly created operators, it can be seen that compared with the "construction operators" newly constructed in the prior art, that is, general operators, built-in operators and newly created operators, the embodiments of the present application can be roughly regarded as "code blocks" that can be constructed and / or adaptively adjusted according to actual needs and can achieve different functions, and have the advantage of "moving wherever it is useful"; it can be seen that the general operators, built-in operators and newly created operators provided in the embodiments of the present application are very practical.

[0065] In the embodiments of the present application, by executing "constructing and / or adaptively adjusting general operators, built-in operators and newly created operators" and "scheduling for different general operators, built-in operators and newly created operators" to replace the process of "writing code", the process-based editing of task information is made more flexible and more efficient. Therefore, it can not only greatly improve the efficiency and accuracy of constructing target tasks, but also improve the richness of the types of target tasks, greatly stimulating the "creativity" of data processing.

[0066] In the embodiment of the present application, the general operator, the built-in operator and the new operator are different types of "construction operators" determined according to the degree of editability. The degree of editability indicates the universality and confidentiality of the uneditable content in the "construction operator". In detail, Figure 7 As shown, Figure 7 The logical diagram of operator information and task information provided in the embodiment of the present application, general operators, built-in operators and newly created operators all include: the first part of the content A edited during the pre-construction process of "Construct Operator" 1 , the operator re-editing process to edit the second part of the content A 2; Among them, the first part of "Constructing Operators" is A 1 Edit through the third interface, the second part of "Construct Operator" A 2 Editing is performed through the fourth interface; the re-editing process of the operator is embedded in the process of arranging the task flow for the first task.

[0067] For example, if Figure 5 and Figure 8 As shown, Figure 8 An example diagram of a control for selecting an operator in the third interface provided in an embodiment of the present application. Figure 5 After a new operator and / or a general operator is inserted into the process arrangement area 422 shown, the control is selected and becomes editable; in actual operation, the fourth interface is displayed through the operator configuration module 130, and the fourth interface is used to re-edit the operator.

[0068] It should be emphasized that Figure 5 The process of inserting new operators and general operators in the "construction operator" is shown only for exemplary purposes. In actual operation, any type of new operators, general operators and / or built-in operators in the "construction operator" can be inserted into the process orchestration area 422 according to actual needs; in addition, the insertion quantity and insertion position of different types of "construction operators" can be determined according to actual data processing requirements, and this application does not make any specific limitations on this.

[0069] In actual operation, different types of "construction operators", namely general operators, built-in operators and new operators, are described in the first part. 1 and Part II Content A 2 are usually different; e.g. Fig. 9 As shown, Fig. 9 The logic diagram of the first part and the second part of the content in each "construction operator" provided in the embodiment of the present application, in the pre-construction process of the "construction operator", when each "construction operator" is edited through the first part of the content A in the third interface corresponding to each other 1 After saving, the first part of the content A 1 It cannot be edited during the process of arranging the task flow for the first task;

[0070] During the re-editing process of the "Construction Operator", the second part of the content A can be edited through the fourth interface corresponding to each "Construction Operator". 2 In the present embodiment, the above-mentioned editability specifically refers to the second part content A 2The degree to which the operator can be edited during the arrangement of the task flow for the first task; in actual execution, if the "construction operator" may be called during the arrangement of the task flow of multiple tasks, the degree of editability of the "construction operator" can be set higher, for example Fig. 9 The first part of the new operator A is shown in the new operator 1 The first part of the general and built-in operators A 1 If the code in the "Construction Operator" needs to be kept confidential, the editability of the operator can be set to a lower level, such as Fig. 9 The built-in operators shown in the first part of the built-in operators A 1 The first part of the general and built-in operators A 1 More.

[0071] In the embodiment of the present application, although it is possible to construct a "constructed operator" with various data processing functions, if it is still necessary to use an "existing operator", it can be constructed as a "built-in operator" for use. Fig.10 and Fig.11 As shown, Fig.10 This is an example diagram of the fourth interface corresponding to the general operator provided in the embodiment of the present application. Fig.11 This is an example diagram of the fourth interface corresponding to the built-in operator provided in the embodiment of the present application, the second part of the general operator A 2 Less than the second part of the built-in operator A 2 ; Among them, the code of built-in operators is not even editable.

[0072] In the embodiment of the present application, the fourth interface is set for re-editing the "construction operator" in order to improve the editing efficiency of the "construction operator"; in actual operation, all the "construction operators" selected for use can be completely edited in the third interface, that is, they are all constructed as "built-in operators" with a lower degree of editability, and when they are in the first interface, the edited "built-in operators" are selected; however, since the embodiment of the present application can be used to construct "construction operators" with various data processing functions, if all "construction operators" are completely edited in the third interface, even if the data processing functions of multiple "construction operators" are very similar, resulting in only a small difference in the second part A2, the user is still required to completely edit multiple "construction operators" respectively; in order to improve the editing efficiency of the "construction operator", the embodiment of the present application divides the editing process of the "construction operator" into the first part A 1 and Part II Content A 2 , which allows users to edit the second part of the "Construction Operator" of various types on the fourth interface according to actual data processing needs during the process of process editing for the target task. 2In practical applications, the type of "construction operator" can be determined according to actual needs, and it is not necessary to distinguish them according to the degree of editability; the names and editability of general operators, built-in operators, and new operators, as well as the specific contents of the first part A 1 and Part II Content A 2 All of them can be determined according to actual needs, and this application does not make any specific limitations on this.

[0073] When in the first part of each "construction operator" content A 1 The second part of the content A2 is edited and saved through their respective third interfaces and fourth interfaces to obtain the operator information of each "construction operator". The operator configuration module 130 sends the operator information of each "construction operator" to Zookeeper 200. Zookeeper 200 stores the operator information of each "construction operator" based on the operator identifier in the operator information.

[0074] In the embodiment of the present application, since general operators, built-in operators and new operators are different types of "construction operators" determined according to the degree of editability, the type of "construction operator" is irrelevant to the specific data processing function that the "construction operator" can implement. However, in the process of arranging the task flow for the first task in the first interface, it is necessary to clarify the functions of each "construction operator" and place the "construction operator" that performs the corresponding function in the corresponding place in the task flow. That is, in the process of arranging the task flow for the first task, the operator directly dragged into the process arrangement area 422 is not a "construction operator" with actual data processing functions, but a control on which a "construction operator" with actual functions can be further selected.

[0075] In actual operation, when the arrangement process of the task flow of the first task is completed, the task information of the first task can be obtained; wherein the task information includes: the task identifier, the operator identifiers of each operator used in the first task, the order relationship between the operators, and the second part of the content A edited in the fourth interface corresponding to each operator 2 .

[0076] S120: In response to receiving a start signal for instructing to start the first task, according to the start signal, the task information of the first task is sent to Zookeeper for storage, and the Sentinel service that monitors the task information of the first task retrieves the task information of the first task stored in Zookeeper to the Flink cluster, so that the Flink cluster can configure a real-time data processing service corresponding to the first task according to the task information of the first task.

[0077] Specifically, after obtaining the task information of the first task, the second interface is displayed through the task status module 120, and the task status of the first task can be set to start in the second interface. When the task status of the first task is set to start, the process configuration module 110 sends the task information of the first task to Zookeeper200, and Zookeeper200 stores the task information of the first task based on the task identifier in the task information.

[0078] The Sentinel service 300 can monitor the changes in task information and operator information in Zookeeper 200. If the Sentinel service 300 monitors that the task information of the first task has been sent to Zookeeper 200, it will retrieve the task information of the first task and send it to the Flink cluster for the Flink cluster to configure the real-time data processing service corresponding to the first task according to the task information of the first task.

[0079] In one implementation, S120 includes: step (2), details of which are as follows:

[0080] Step (2): Send the task information of the first task to Zookeeper 200 for storage, instruct the Sentinel service that monitors the task information of the first task to retrieve the task information of the first task stored in Zookeeper 200, and retrieve the operator information stored in Zookeeper 200 according to the task identifier in the retrieved task information, and send the retrieved task information and operator information to the Flink cluster 300, so that the Flink cluster 300 can configure the real-time data processing service corresponding to the first task according to the task information and the operator information.

[0081] Specifically, when each operator is pre-built, it includes the first part of content A 1 The operator information is sent to Zookeeper200 for storage, and then the task information of the first task is sent. What is sent is not the specific information of each operator, but the operator identifier, which is used to instantly retrieve the required operator information from Zookeeper200. In practical applications, the above steps can effectively reduce the amount of data included in the task information and improve data communication efficiency.

[0082] In one implementation, the start signal is determined when the user sets the task status of the task for processing data to start on the second interface; the method further includes: steps (3) to (4), the details of which are as follows:

[0083] Step (3): in response to the user making a process adjustment on the first interface for the real-time data processing service corresponding to the first task, determining task information of the second task;

[0084] Step (4): in response to the user setting the task state of the second task to start on the second interface, determining a start signal;

[0085] Step (5): According to the start signal, the task information of the second task is sent to Zookeeper 200 for storage, and the Sentinel service that monitors the task information of the second task retrieves the task information of the second task stored in Zookeeper 200 to the Flink cluster 300, so that the Flink cluster 300 can configure the real-time data processing service corresponding to the second task according to the task information of the second task.

[0086] Specifically, steps (3) to (5) provide steps for adjusting the task flow of the first task. Based on the process configuration module 110, the task flow of the first task can be adjusted. When the adjustment is completed, the first task is transformed into the second task. If the user sets the task state of the second task to start on the second interface, the process configuration module 110 generates a start signal accordingly in response to the user's operation. The process configuration module 110 sends the task information of the second task to Zookeeper 200 for storage according to the start signal, and the sentinel service 310 that monitors the task information of the second task retrieves the task information of the second task stored in Zookeeper 200 to the Flink cluster 300, so that the Flink cluster 300 can configure the real-time data processing service corresponding to the second task according to the task information of the second task.

[0087] In the process of real-time processing of data streams by the Flink cluster 300 configured with the real-time data processing service corresponding to the first task or the second task, Figure 3 The second interface shown dynamically adjusts the task status of each first task, so that the Flink cluster 300 based on the Sentinel service 310 can dynamically load and execute the code in the operator or update the operator information in real time, and the data processing results are also updated in real time, ensuring the timeliness and accuracy of the task information and operator information. The entire process does not require the Flink cluster 300 to be restarted, so it can provide strong support for the company's rapid response and decision support; among them, the Flink cluster 300 uses the JANINO lightweight code compiler to load the custom code in the operator.

[0088] In actual operation, whenever the user adjusts the task status of the first task or the second task on the second interface, an adjustment signal is generated accordingly, and the adjustment signal is used to indicate the next process; wherein, when the task status of the first task or the second task is set to start, the generated adjustment signal is a start signal, and the start signal is used to indicate the execution of step (5).

[0089] In actual operation, after the Flink cluster is configured through the above steps, the Flink cluster has the function of real-time processing of real-time collected data streams, which can ensure the instant transmission and consistency of data between different systems or components, and further improve the integrity and efficiency of the data processing flow. Second, the application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, steps S110 to S120 provided in the above embodiment are implemented.

[0090] Third, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, steps S110 to S120 of the above embodiment are executed.

[0091] Fourth, the computer program product provided in the present application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method in the previous method embodiment. The specific implementation can refer to steps S110 to S120 of the method embodiment, which will not be repeated here.

[0092] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0093] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0094] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0095] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM) random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0096] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0097] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for configuring a real-time data processing service, characterized in that: Applied to a service configuration system, the service configuration system establishes a connection with a distributed application coordination service Zookeeper, and the Zookeeper establishes a connection with a sentinel service configured in a Flink cluster; the method comprises: In response to a user performing a process-based editing on a first interface for a real-time data processing service corresponding to a first task, determining task information of the first task; Wherein, the first task indicates a task for processing data; In response to receiving a start signal for instructing to start the first task, according to the start signal, the task information of the first task is sent to the Zookeeper for storage, and the Sentinel service that monitors the task information of the first task retrieves the task information of the first task stored in the Zookeeper to the Flink cluster, so that the Flink cluster configures a real-time data processing service corresponding to the first task according to the task information of the first task.

2. The method according to claim 1, characterized in that The determining the task information of the first task includes: Determining that the user obtains task information of the first task after editing the nodes and node relationships on the first interface; The node indicates an operator used by the Flink cluster when performing data processing through the real-time data processing service; and the task information includes: a task identifier, an operator identifier, and an operator relationship.

3. The method according to claim 2, characterized in that The sending the task information of the first task to the Zookeeper for storage includes: The task information of the first task is sent to the Zookeeper, and the Zookeeper is instructed to store the corresponding task information according to the task identifier in the task information of the first task.

4. The method according to claim 2, characterized in that: Before determining that the user obtains the task information of the first task after editing the nodes and node relationships on the first interface, the method further includes: Determining operator information obtained after the user edits the edit box on the third interface; The editing box is used to edit the class information, parameter information or code of the operator; the parameter information is the information of the parameters contained in the code of the operator; the operator information includes: the operator identifier, the class information, the parameter information and the code; The operator information is sent to the Zookeeper, and the Zookeeper is instructed to store the corresponding operator information according to the operator identifier.

5. The method according to claim 4, characterized in that The sending the task information of the first task to the Zookeeper for storage includes: The task information of the first task is sent to the Zookeeper for storage, and the Sentinel service that monitors the task information of the first task retrieves the task information of the first task stored in the Zookeeper, and retrieves the operator information stored in the Zookeeper according to the operator identifier in the retrieved task information, and sends the retrieved task information and the operator information to the Flink cluster, so that the Flink cluster configures the real-time data processing service corresponding to the first task according to the task information and the operator information.

6. The method according to claim 2, characterized in that The start signal is determined when the user sets the task status of the task for processing data to start on the second interface; the method further includes: In response to the user making a process adjustment on the first interface for the real-time data processing service corresponding to the first task, determining task information of the second task; In response to the user setting the task state of the second task to start on the second interface, determining the start signal; According to the start signal, the task information of the second task is sent to the Zookeeper for storage, and the Sentinel service that monitors the task information of the second task retrieves the task information of the second task stored in the Zookeeper to the Flink cluster, so that the Flink cluster can configure a real-time data processing service corresponding to the second task according to the task information of the second task.

7. A service configuration system, characterized in that: The system comprises: a network device, wherein the network device is used to execute the configuration method of the real-time data processing service according to any one of claims 1 to 6.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory is used to store software programs, and the processor runs or executes the software programs stored in the memory so that the electronic device executes the configuration method for real-time data processing services as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes executed by a processor, wherein the program codes include a method for executing a configuration method of a real-time data processing service according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises program codes, and when a computer runs the computer program product, the computer executes the method for configuring a real-time data processing service according to any one of claims 1 to 6.

Images