K8S-based flink task log processing method and computer device

Abstract

The application discloses a flink task log processing method based on K8S. The flink task log processing method comprises the following steps: associating a task identifier with a flink task role; listening to state information of a K8S container corresponding to the flink task, and storing the task identifier of the flink task role; collecting and storing log information of the flink task in the K8S container; and querying log information of the flink task role associated with the task identifier. The application also discloses a flink task log processing method based on K8S, a flink task log processing device based on K8S, a computer device and a computer readable storage medium. The flink task log information collection, storage and display of K8S are realized, so that a user can complete flink task submission, state updating and log information checking through a unified flink big data computing platform, the running problem of the flink task of K8S is conveniently located, and the problem locating efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data platforms, and in particular to a flink task log processing method and device based on K8S, a computer device, and a computer readable storage medium. BACKGROUND

[0002] A current big data offline computing platform design uses flink to implement extraction, conversion, and storage of multiple data sources, and the underlying is to run flink tasks based on a Kubernetes (K8S) cluster, but K8S does not provide collection and display of container logs, and after K8S task execution succeeds or fails, the pod will be quickly recycled, and the task log will be immediately cleaned up, which makes it impossible to view the log, and this brings difficulties to task running status and task problem positioning. SUMMARY

[0003] To solve at least one of the technical problems in the background art, the embodiments of the present application provide a flink task log processing method and device based on K8S, a computer device, and a computer readable storage medium.

[0004] The flink task log processing method based on K8S of the embodiments of the present application comprises:

[0005] associating a task identifier with a flink task role;

[0006] listening to state information of a K8S container corresponding to the flink task, storing the task identifier of the flink task role, collecting and storing log information of the flink task in the K8S container, and

[0007] querying log information of the flink task role associated with the task identifier.

[0008] In some embodiments, the collecting and storing of the log information of the flink task in the K8S container comprises:

[0009] creating a fluented application based on K8S;

[0010] starting the fluented application on any node of K8S and collecting running logs of the K8S container; and

[0011] storing the collected running logs of the K8S container to elasticsearch.

[0012] In some embodiments, the creating of the fluented application based on K8S comprises:

[0013] A source configuration is collected.

[0014] A filter configuration of K8S metadata records is added; and

[0015] Data output configuration of fluented application collection is added.

[0016] In some embodiments, the source configuration includes associating log files of K8S containers corresponding to the flink task with collection marks.

[0017] The filter configuration of the K8S metadata records includes log files of K8S containers associated with the collection marks.

[0018] In some embodiments, the log information of the flink task role associated with the task identification includes:

[0019] A query condition is constructed, and the query condition includes the task identification.

[0020] An elasticsearch engine is called to query the log information of the flink task role associated with the task identification; and

[0021] The log information is assembled and displayed.

[0022] In some embodiments, the assembling and displaying of the log information includes:

[0023] It is determined whether the query condition carries scroll information.

[0024] If yes, the next page data of the log information is queried according to the scroll information and displayed; and

[0025] If no, a data display size is configured, and the log information is displayed according to the data display size.

[0026] In some embodiments, the processing method of the flink task log further includes:

[0027] A predetermined namespace is defined for the flink task; and

[0028] A naming rule of a K8S pod is set.

[0029] The processing device of the K8S-based flink task log according to the embodiments of the present application includes:

[0030] A task submission module is configured to associate a task identification with a flink task role.

[0031] The task monitoring module is configured to monitor state information of a K8S container corresponding to the Flink task, and store a task identifier of the Flink task role.

[0032] The log collection module is configured to collect and store log information of the Flink task in the K8S container.

[0033] The log display module is configured to query log information of the Flink task role associated with the task identifier.

[0034] The computer device of the embodiments of the present application comprises one or more processors, a memory, and one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and the one or more computer programs are configured to perform the processing method of the Flink task log based on K8S described in any embodiment of the present application.

[0035] The non-volatile computer readable storage medium of the embodiments of the present application stores a computer program, which, when executed by one or more processors, causes the processors to perform the processing method of the Flink task log based on K8S described in any embodiment of the present application.

[0036] In the processing method of the Flink task log based on K8S, the processing device of the Flink task log based on K8S, the computer device, and the computer readable storage medium of the embodiments of the present application, the log information collection, storage and display of the Flink task based on K8S are realized, so that the user can complete the submission, state update and log information viewing of the Flink task through a unified Flink big data computing platform, which facilitates the positioning of the running problem of the Flink task based on K8S and improves the efficiency of problem positioning.

[0037] Additional aspects and advantages of the embodiments of the present application will be in part apparent and in part pointed out hereinafter in the description of the embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0038] The above and / or additional aspects and advantages of the present application can become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings in which:

[0039] Figure 1 Flowchart of the processing method of the Flink task log based on K8S of certain embodiments of the present application;

[0040] Figure 2 Flowchart of the processing method of the Flink task log based on K8S of certain embodiments of the present application;

[0041] Figure 3 Flowchart of a processing method of a K8S-based flink task log according to some embodiments of the present application;

[0042] Figure 4 Flowchart of a processing method of a K8S-based flink task log according to some embodiments of the present application;

[0043] Figure 5 Flowchart of a processing method of a K8S-based flink task log according to some embodiments of the present application;

[0044] Figure 6 Flowchart of a processing method of a K8S-based flink task log according to some embodiments of the present application;

[0045] Figure 7 Module diagram of a processing device based on a K8S-based flink task log according to some embodiments of the present application;

[0046] Figure 8 Schematic diagram of a computer readable storage medium in communication with a processor according to some embodiments of the present application;

[0047] Figure 9 Schematic diagram of a computer device according to some embodiments of the present application. DETAILED DESCRIPTION

[0048] Embodiments of the present application are described in detail below with reference to the attached drawing figures, wherein the same or like designations denote the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the attached drawing figures are merely exemplary for explaining the embodiments of the present application and cannot be understood as limiting the embodiments of the present application.

[0049] Through investigation, three main ways are provided for K8S log collection in the industry: 1. Deploy a K8S DaemonSet object, run an agent on each node to collect logs; 2. Include a sidecar container in the pod to collect application logs; 3. Push log information in the application to the collection backend.

[0050] The second mode has a log container and a host computer, and there are double logs. The third mode requires intrusive code modification of the application program, and the logs need to be sent to the backend, which will cause resource waste, and the application logs cannot be accessed within the K8S cluster. Most of the current K8S application log collection solutions are based on the first mode, and the typical method is to use ELK (ElasticSearch Filebeat, Kibana), EFL (ElasticSearch Filebeat, LogStash) and other methods. These methods have the following disadvantages in flink on k8s feature platform log collection display:

[0051] 1. More than three services are needed for log collection and display.

[0052] 2. There is no way to recycle the pods of different roles of flink.

[0053] 3. There is no method to associate tasks on the platform and extract and display corresponding task logs.

[0054] The K8S-based flink task log processing method, K8S-based flink task log processing device, computer device and computer readable storage medium of the present application embodiment can be used to solve at least one of the above technical problems.

[0055] Please refer to Figure 1 , Figure 1 The flowchart of the K8S-based flink task log processing method of some embodiments of the present application is shown. The flink task log processing method of the present application embodiment includes the following steps:

[0056] 01: Associate a task identifier with a flink task role;

[0057] 02: Listen to the state information of the K8S container corresponding to the flink task, and store the task identifier of the flink task role;

[0058] 03: Collect and store the log information of the flink task in the K8S container; and

[0059] 04: Query the log information of the flink task role associated with the task identifier.

[0060] The processing method of the flink task log based on K8S of the embodiments of the present application realizes the collection, storage and display of the log information of the flink task of K8S, so that the user can complete the submission, state updating and log information viewing of the flink task through a unified flink big data computing platform, facilitate the positioning of the running problem of the flink task of K8S, and improve the efficiency of problem positioning.

[0061] Specifically, in the process of submitting the flink task, the corresponding task identifier is added when starting the flink driver task, and then the started Jobmanager and Taskmanager are associated with the corresponding flink driver task. The task identifier includes role information, which indicates whether the flink task belongs to the jobmanager or the taskmanager. In addition, the task identifier also includes the ID information of the flink task, which indicates which task the flink task belongs to. The task identifier is the unique identifier of the flink task, which is used for subsequent log query.

[0062] The subsequent program queries K8S to obtain the submitted task, judges the role of the flink task according to the role information carried in the task identifier, records the state of the task according to the ID information, and matches the task identifier when querying the log later.

[0063] In step 02, the backend writes logic based on K8S informer to capture the change information of the flink task K8S container, such as adding, deleting, state change and the like, and stores the container ID information and metadata information of each flink task corresponding role (referring to the Driver, Jobmanager and multiple TaskManager roles that constitute a flink task) in order to make subsequent log association query.

[0064] Please refer to Figure 2 , Figure 2 The flowchart of the processing method of the flink task log based on K8S of some embodiments of the present application is shown in FIG. 3. Step 03: collect and store the log information of the flink task in the K8S container, including the following steps:

[0065] 031: create a fluented application based on K8S;

[0066] 032: start the fluented application on any node of K8S, and collect the running log of the K8S container; and

[0067] 033: Store the collected running logs of K8S containers to elasticsearch.

[0068] In this embodiment, the collection of K8S cluster log information is completed by the fluentd Daemonset object deployed by K8S. The log information of the application is only saved in one copy in the K8S cluster, saving the K8S cluster disk space, and only fluented and elasticsearch need to be deployed, without the need to deploy ELK, EFL and other components. In addition, the collection of log information is separated from the business logic, without the need to modify the open source components K8S and flink and also without the need to modify the flink upper layer application, reducing the development and maintenance cost.

[0069] Please refer to Figure 3 , Figure 3 is a flowchart of the processing method of the K8S-based flink task log of some embodiments of the present application. Step 031: Create a K8S-based fluented application, including the following steps:

[0070] 0311: Collect source configuration;

[0071] 0312: Add filter configuration of K8S metadata record; and

[0072] 0313: Add data output configuration collected by the fluented application.

[0073] Specifically, the collection source source configuration, or the K8S RBAC rule configuration for the fluented application, can include associating the log file of the K8S container corresponding to the flink task with a collection mark, for example, according to regular matching to the log file of the corresponding K8S container belonging to the characteristic platform flink task (the container log of the flink task running), and marking with a collection mark, for example: flink_log.

[0074] For example, the configuration realizes DeamonSet metadata collection, flink application running image, resource request and limitation, log directory mounting, configuration file mounting, etc.

[0075] The filter configuration of the K8S metadata record can include recording the log file of the K8S container associated with the collection mark, for example, the filter matches the log file associated with the collection mark (such as the above-mentioned flink_log), and records the K8S container metadata, to record the basic information of the container running.

[0076] The data output configuration of the fluent application collection is increased, mainly including configuring the storage of log data to elasticsearch (ES), for example, ES server information configuration, buffer refresh frequency configuration, etc.

[0077] Please refer to Figure 4 , Figure 4 For the flowchart of the K8S-based flink task log processing method of some embodiments of the present application, step 04: query the log information of the flink task role associated with the task identification, including steps:

[0078] 041: build a query condition, the query condition including the task identification;

[0079] 042: call the elasticsearch engine to query the log information of the flink task role associated with the task identification; and

[0080] 043: assemble and display the log information.

[0081] Build a query condition, wherein the query condition includes the task identification. According to the task identification, the log information of the corresponding role can be queried by calling the elasticsearch engine. By assembling and displaying the log information, the developer can view the running situation and find problems. Of course, when the elasticsearch does not store the log information corresponding to the task identification, the result of ending is returned.

[0082] Please refer to Figure 5 , Figure 5 For the flowchart of the K8S-based flink task log processing method of some embodiments of the present application, step 043: assemble and display the log information, including steps:

[0083] 0431: determine whether the query condition carries scrolling information;

[0084] 0432: if yes, query the next page data of the log information according to the scrolling information and display; and

[0085] 0433: if no, configure the data display size, and display the log information according to the data display size.

[0086] Specifically, if the log information returned by one query result is too much, the log amount is too large, the time of one query is long, and all the log information cannot be displayed on one screen, the log information needs to be displayed in pages. If the query condition carries the scroll information, the next page data of the log information is queried and displayed according to the scroll information, and the query requirement is met. If the query condition does not carry the scroll information, the amount of one query data (one page data) is configured, and how many pages of the query result are returned.

[0087] Referring to Figure 6 , Figure 6 FIG. 1 is a flowchart of a processing method of a flink task log based on K8S according to some embodiments of the present application. The processing method of the flink task log based on K8S further includes the following steps:

[0088] 05: defining a predetermined namespace for the flink task; and

[0089] 06: setting a naming rule of the K8S pod.

[0090] The specific namespace is specified for the flink task, and the design related to the log mainly standardizes the naming of the pod name, generates a container log file with a specific file name at the beginning, and facilitates the subsequent collection of the relevant information of the log file by the fluented application.

[0091] Referring to Figure 7 , Figure 7 FIG. 1 is a flowchart of a processing method of a flink task log based on K8S according to some embodiments of the present application. The processing method of the flink task log based on K8S further includes the following steps:

[0092] Please continue to refer to Figure 7In some embodiments, the log collection module 13, when implementing step 03: collecting and storing log information of the flink task in the K8S container, can be specifically used for implementing steps 031, 032 and 033, that is, the log collection module 13 can be used to create a K8S-based fluented application; start the fluented application on any node of the K8S, and collect the running log of the K8S container; and store the collected running log of the K8S container to the elasticsearch.

[0093] Please continue to refer to Figure 7 In some embodiments, the log collection module 13, when implementing step 031: creating a K8S-based fluented application, can be specifically used for implementing steps 0311, 0312 and 0313, that is, the log collection module 13 can be used to collect source configuration; add filter configuration of K8S metadata record; and add data output configuration collected by the fluented application.

[0094] Please continue to refer to Figure 7 In some embodiments, the log display module 14, when implementing step 04: querying log information of the flink task role associated with the task identification, can be specifically used for implementing steps 041, 042 and 043, that is, the log display module 14 can be used to build a query condition, the query condition including the task identification; call the elasticsearch engine to query the log information of the flink task role associated with the task identification; and assemble and display the log information.

[0095] Please continue to refer to Figure 7 In some embodiments, the log display module 14, when implementing step 043: assembling and displaying the log information, can be specifically used for implementing steps 0431, 0432 and 0433, that is, the log display module 14 can be used to judge whether the query condition carries scrolling information; if yes, query and display the next page data of the log information according to the scrolling information; and if no, configure the data display size, and display the log information according to the data display size.

[0096] Please continue to refer to Figure 7 In some embodiments, the task submission module 11 can also be used to implement steps 05 and 06, that is, the task submission module 11 can be used to define a predetermined namespace for the flink task; and set the naming rule of the K8S pod.

[0097] It should be noted that the implementation details and effects of the flink task log processing device 10 when implementing the flink task log processing method of any embodiment of the present application can be referred to the description of the flink task log processing method above, which will not be repeated here.

[0098] Further, referring to Figure 8 The embodiments of the present application provide a computer readable storage medium, and a computer program is stored on the computer readable storage medium. The computer program is executed by a processor to implement the flink task log processing method in any of the above embodiments. The computer readable storage medium includes, but is not limited to, any type of disk (including a floppy disk, a hard disk, an optical disk, a CD-ROM, and a magneto-optical disk), a ROM (Read-Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a flash memory, a magnetic card, or an optical card. That is, the storage device includes any medium that stores or transmits information in a form capable of being read by an apparatus (for example, a computer, a mobile phone), and can be a read-only memory, a magnetic disk or an optical disk, etc.

[0099] The content of the method embodiments of the present application is applicable to the storage medium embodiments of the present application. The storage medium embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments. For details, refer to the description in the method embodiments, which will not be repeated here.

[0100] Further, referring to Figure 9 The embodiments of the present application further provide a computer device. The computer device in the embodiments can be a server, a personal computer, a network device, or the like. The computer device includes one or more processors, a memory, and one or more computer programs. The one or more computer programs are stored in the memory and configured to be executed by the one or more processors. The one or more computer programs are configured to execute the flink task log processing method in any of the above embodiments.

[0101] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the exemplary description of the above terms does not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in an appropriate manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0102] Any process or method descriptions or descriptions of the flow diagrams in the specification or otherwise described herein can be understood as representing the steps of the code modules, segments, or portions of the respective code that include one or more executable instructions for performing specific logical functions or steps in the process. The scope of preferred embodiments of the present application includes the additional implementation in which the functions can be performed in an order different from the order shown or discussed, including functions performed in substantially simultaneous, or in reverse order, as appropriate, depending on the functions involved, as would be understood by one skilled in the art.

[0103] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and those ordinarily skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. A method for processing Flink task logs based on Kubernetes, characterized in that, The method for processing Flink task logs includes: Associate task identifiers with Flink task roles; Monitor the status information of the K8S container corresponding to the Flink task and store the task identifier of the Flink task role; Collect and store the log information of the Flink task in the Kubernetes container; and Query the log information of the Flink task role associated with the task identifier; The collection and storage of log information of the Flink task in the Kubernetes container includes: Create fluented applications based on Kubernetes; Start the fluented application on any Kubernetes node and collect the runtime logs of the Kubernetes containers; and The collected runtime logs of the Kubernetes containers are stored in Elasticsearch.

2. The method for processing Flink task logs based on Kubernetes according to claim 1, characterized in that, The creation of fluented applications based on Kubernetes includes: Source configuration; Add filter configuration for Kubernetes metadata records; and Add configuration options for the data output collected by the Fluented application.

3. The method for processing Flink task logs based on Kubernetes according to claim 2, characterized in that, The source configuration includes associating collection tags with the log files of the K8S container corresponding to the Flink task; The filter configuration for the K8S metadata records includes logging files of K8S containers associated with collection tags.

4. The method for processing Flink task logs based on Kubernetes according to claim 1, characterized in that, The query for log information of the Flink task role associated with the task identifier includes: Construct query conditions, which include task identifiers; Call the Elasticsearch engine to query the log information of the Flink task role associated with the task identifier; and The log information is assembled and displayed.

5. The method for processing Flink task logs based on Kubernetes according to claim 4, characterized in that, The assembly and display of log information includes: Determine whether the query conditions include scrolling information; If so, retrieve and display the next page of log information based on the scrolling information; and If not, configure the data display size and display log information according to the data display size.

6. The method for processing Flink task logs based on Kubernetes according to claim 1, characterized in that, The method for processing Flink task logs also includes: Define a predefined namespace for Flink tasks; and Configure the naming rules for Kubernetes pods.

7. A processing device for Flink task logs based on Kubernetes, characterized in that, The Flink task log processing device includes: The task submission module is used to associate task identifiers with Flink task roles; The task monitoring module is used to monitor the status information of the K8S container corresponding to the Flink task and store the task identifier of the Flink task role. The log collection module is used to collect and store the log information of the Flink task in the Kubernetes container, including: creating a Kubernetes-based Fluented application; starting the Fluented application on any Kubernetes node and collecting the runtime logs of the Kubernetes container; and storing the collected runtime logs of the Kubernetes container in Elasticsearch; and The log display module is used to query log information for Flink task roles associated with task identifiers.

8. A computer device, characterized in that, include: One or more processors; Memory; and One or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, the one or more computer programs being configured to: perform the Kubernetes-based Flink task log processing method according to any one of claims 1 to 6.

9. A non-volatile computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by one or more processors, the processors perform the Kubernetes-based Flink task log processing method as described in any one of claims 1 to 6.

Images