A log service method and device based on kubernetes cluster

Abstract

The present invention discloses a log service method and device based on a kubernetes cluster, and relates to the field of computer technology. A specific implementation of the method includes: obtaining the running status of an application running on a kubernetes cluster; wherein the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain a local application log set; determining the application that has been completed according to the running status, and storing the locally stored application log corresponding to the completed application in the cloud to obtain a cloud application log set; responding to an application log query request initiated by a user, querying the local application log set and the cloud application log set in turn according to the application number included in the application log query request to determine the target application log, and sending the target application log to the user. This implementation saves storage resources, simplifies the log query process, improves the log query efficiency, and enhances the user experience.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a log service method and device based on a Kubernetes cluster. Background Art

[0002] Kubernetes is an open source framework for managing containerized applications on multiple hosts in a cloud platform. As a resource scheduling framework, Kubernetes cluster is often used to schedule online services. Its goal is to make the deployment of containerized applications simple and efficient (powerful). Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance. Currently, traditional distributed log solutions such as ElK (Elasticsearch, Logstash, and Kibana are log collection and management systems built on three open source frameworks) are commonly used in Kubernetes clusters.

[0003] Spark is a popular big data computing framework that is better suited for algorithms that require iteration, such as data mining and machine learning. Existing Spark usually uses the traditional Yarn scheduler for resource scheduling.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] 1. The existing log service method for running applications in a Kubernetes cluster mainly uses traditional distributed log solutions, which consumes storage resources and has low log query efficiency;

[0006] 2. When the Spark application runs in a Kubernetes cluster, logs need to be queried and displayed through specific pages, search, query, and display components. The log query process is complicated, resulting in a poor user experience. Summary of the invention

[0007] In view of this, the embodiments of the present invention provide a log service method and device based on a Kubernetes cluster, which can save storage resources, simplify the log query process, improve the log query efficiency, and enhance the user experience.

[0008] To achieve the above object, according to a first aspect of an embodiment of the present invention, a log service method based on a kubernetes cluster is provided, comprising:

[0009] Get the running status of the application running on the kubernetes cluster; the running node corresponding to the application locally stores the application logs generated during the running process of the application to obtain a local application log set;

[0010] Determine the application that has been completed according to the running status, and store the locally stored application logs corresponding to the completed application in the cloud to obtain a cloud application log set;

[0011] In response to an application log query request initiated by a user, the local application log set and the cloud application log set are queried in sequence according to the application number included in the application log query request to determine the target application log, and the target application log is sent to the user.

[0012] Furthermore, the steps of obtaining the running status of the application running on the kubernetes cluster include:

[0013] The application number list is obtained from the kubernetes cluster periodically, where the running status corresponding to each application number in the application number list is running on the kubernetes cluster.

[0014] Further, the step of determining the application that has completed running according to the running status includes:

[0015] The application number list and the local list are compared, and the applications in the complement of the application number list and the local list are determined to be the applications that have been run. The local list includes the application numbers corresponding to the applications applied to run on the kubernetes cluster.

[0016] Furthermore, after the step of storing the locally stored application logs corresponding to the completed applications in the cloud to obtain a local application log set, the method further includes:

[0017] Delete the application logs corresponding to the completed applications from the local application log collection.

[0018] Furthermore, the application is a Spark application, and there are multiple running nodes. Each running node locally stores application logs generated during the running of the Spark application, and obtains a local application log set corresponding to each running node.

[0019] Furthermore, the application log query request also includes the running node number;

[0020] The step of sequentially querying the local application log set and the cloud application log set according to the application number included in the application log query request to determine the target application log also includes:

[0021] According to the application number and running node number included in the log query request, the local application log set and the cloud application log set corresponding to the running node number are queried in turn to determine the target application log.

[0022] According to a second aspect of an embodiment of the present invention, a log service device based on a kubernetes cluster is provided, comprising a running status acquisition module, a log storage module and a log query module; wherein,

[0023] The running status acquisition module is used to obtain the running status of the application running on the kubernetes cluster and send the obtained running status to the log storage module, wherein the running node corresponding to the application locally stores the application logs generated during the running process of the application to obtain a local application log set;

[0024] A log storage module is used to determine the application that has been completed according to the running status, and store the locally stored application logs corresponding to the completed application in the cloud to obtain a cloud application log set;

[0025] The log query module is used to respond to the application log query request initiated by the user, query the local application log collection and the cloud application log collection in turn according to the application number included in the application log query request to determine the target application log, and send the target application log to the user.

[0026] Furthermore, the application is a Spark application, and there are multiple running nodes. Each running node locally stores application logs generated during the running of the Spark application, and obtains a local application log set corresponding to each running node.

[0027] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0028] one or more processors;

[0029] a storage device for storing one or more programs,

[0030] When one or more programs are executed by one or more processors, the one or more processors implement any of the above-mentioned log service methods based on the Kubernetes cluster.

[0031] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, any of the above-mentioned log service methods based on a Kubernetes cluster is implemented.

[0032] An embodiment of the above invention has the following advantages or beneficial effects: because the running status of the application running on the kubernetes cluster is obtained; wherein the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain a local application log set; the application that has been completed is determined according to the running status, and the locally stored application log corresponding to the completed application is stored in the cloud to obtain a cloud application log set; in response to the application log query request initiated by the user, the local application log set and the cloud application log set are sequentially queried according to the application number included in the application log query request to determine the target application log, and the target application log is sent to the user. The technical means overcomes the technical problem that the existing log service method for running applications in the kubernetes cluster mainly adopts the traditional distributed log solution, consumes storage resources, and has low log query efficiency, thereby achieving the technical effect of saving storage resources, simplifying the log query process, improving the log query efficiency, and improving the user experience; further, the above application is a Spark application, which solves the technical problem in the prior art that when the Spark application is running in the kubernetes cluster, the log needs to be queried and displayed through specific pages, search, query, display and other components, the log query process is complicated, and the user experience is poor, thereby achieving the technical effect of improving the user experience.

[0033] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0035] Figure 1 It is a schematic diagram of the main process of the log service method based on the kubernetes cluster provided according to the first embodiment of the present invention;

[0036] Figure 2 It is a schematic diagram of the main process of the log service method based on the kubernetes cluster provided according to the second embodiment of the present invention;

[0037] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation method corresponding to the method;

[0038] Figure 4 It is a schematic diagram of main modules of a log service device based on a kubernetes cluster provided according to an embodiment of the present invention;

[0039] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0040] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0042] Figure 1 Schematic diagram of the main process of the log service method based on kubernetes cluster provided according to the first embodiment of the present invention; the log service method of the embodiment of the present invention mainly involves the collection (storage), query and display of application logs. Figure 1 As shown, the log service method based on the kubernetes cluster provided by the embodiment of the present invention mainly includes:

[0043] Step S101, obtaining the running status of the application running on the kubernetes cluster; wherein the running node corresponding to the application locally stores the application log generated during the running process of the application to obtain a local application log set.

[0044] In the prior art, the collection of application logs of applications running on a kubernetes cluster is mainly carried out through traditional distributed log collection solutions such as ELK, that is, a log service system with multiple components (including log collection components, log storage management components, log analysis and display components, etc., with complex architectures and high coupling between components) is rebuilt. For example, for log storage, the application logs generated by applications running on the kubernetes cluster are mounted on network storage for storage, which often causes application startup delays when the application is running, and also makes the application startup failure rate higher.

[0045] In the implementation of the present invention, a local storage device is mounted on the running node corresponding to the application, so that the running node can directly store the application logs generated during the application running process locally. Since the local storage speed is fast, the log storage efficiency is improved, and the storage failure problem caused by the simultaneous running of multiple applications in the kubernetes cluster is solved.

[0046] Further, according to an embodiment of the present invention, the step of obtaining the running status of the application running on the kubernetes cluster includes:

[0047] The application number list is obtained from the kubernetes cluster periodically, where the running status corresponding to each application number in the application number list is running on the kubernetes cluster.

[0048] In the actual operation process, the operation status includes "running", "retrying", "failed", "submitting", "successful" (the last four statuses all indicate that the operation is completed at a certain stage or in full), and other intermediate statuses, not just "running" and "completed". Through the above settings, the application number list with the operation status of "running" can be obtained, and the running applications can be accurately obtained, and then the applications that are not running can be determined. It is conducive to collecting application logs corresponding to applications that are "not in the running state" and to avoiding the loss of application logs.

[0049] Step S102, determining the application that has been completed according to the running status, and storing the locally stored application logs corresponding to the application that has been completed in the cloud to obtain a cloud application log set.

[0050] Specifically, according to an embodiment of the present invention, the step of determining the application that has completed running according to the running status includes:

[0051] The application number list and the local list are compared, and the applications in the complement of the application number list and the local list are determined to be applications that have been run. The local list includes application numbers corresponding to the applications applied to run on the kubernetes cluster.

[0052] The local list records all the application numbers submitted for running in the kubernetes cluster. By comparing the application number list corresponding to the running applications with the local list, the complement of the two is the "not in running state" applications, that is, the running applications. For the application logs corresponding to this type of application, they are transferred from the local storage device to the cloud storage device for subsequent log query.

[0053] Furthermore, according to a specific implementation of an embodiment of the present invention, a first application number can also be determined from the local list to check whether the first application number exists in the received "list of application numbers corresponding to running applications". If so, it means that the application corresponding to the first application number is in the "running" state; if not, the application log corresponding to the first application number in the local storage device can be stored in the cloud.

[0054] Further, according to an embodiment of the present invention, after the step of cloud storing the locally stored application logs corresponding to the completed application to obtain a local application log set, it also includes: deleting the application logs corresponding to the completed application from the local application log set.

[0055] Through the above settings, after the corresponding application logs in the local storage device are stored in the cloud, the corresponding application logs in the local storage device are deleted, which can release the storage space of the local storage device and reduce the burden on the running node, so as to avoid affecting the running efficiency of the running node due to insufficient local storage space.

[0056] Step S103, in response to the application log query request initiated by the user, query the local application log set and the cloud application log set in sequence according to the application number included in the application log query request to determine the target application log, and send the target application log to the user.

[0057] Specifically, according to an embodiment of the present invention, the application log query request may further include the application running time (for application logs named after the running time as storage files, the efficiency and accuracy of querying the target application logs may be improved).

[0058] Further, according to an embodiment of the present invention, the above application is a Spark application, there are multiple running nodes, and each running node locally stores the application logs generated during the running of the application to obtain a local application log set corresponding to each running node.

[0059] According to an embodiment of the present invention, the above application is a distributed computing program, preferably a Spark application. Distributed computing programs involve large amounts of computation and require the coordinated action of multiple running nodes, and are more suitable for running on a kubernetes cluster.

[0060] Spark application is a popular big data computing framework with a wide range of applications. In the existing technology, Spark applications run in Yarn (another scheduling framework) cluster, but due to the insufficient scalability, low flexibility and low resource utilization of Yarn, offline and online hybrid deployment cannot be well achieved. By combining Spark applications with kubernetes clusters (containerized applications on multiple hosts in the management cloud platform), the scheduling framework can be unified and the computing process can be optimized, and the hybrid deployment of online and offline services can be achieved.

[0061] Preferably, according to an embodiment of the present invention, the application log query request also includes a running node number; the step of sequentially querying a local application log set and a cloud application log set according to the application number included in the application log query request to determine the target application log also includes:

[0062] According to the application number and running node number included in the log query request, the local application log set and the cloud application log set corresponding to the running node number are queried in turn to determine the target application log.

[0063] Specifically, after the application is started, tasks will be assigned to multiple running nodes, and the multiple running nodes will respectively perform part of the running tasks. The running node number is further specified in the application log query request, and the target application log corresponding to the running node can be obtained. Then, the running status and resource usage status of the application on the running node can be determined based on the target application log, so as to monitor and troubleshoot the running health of the application on each running node.

[0064] According to the technical solution of the embodiment of the present invention, the running status of the application running on the kubernetes cluster is obtained; wherein the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain a local application log set; the application that has been completed is determined according to the running status, and the locally stored application log corresponding to the completed application is stored in the cloud to obtain a cloud application log set; in response to the application log query request initiated by the user, the local application log set and the cloud application log set are sequentially queried according to the application number included in the application log query request to determine the target application log, and the target application log is sent to the user. The technical means overcomes the technical problem that the existing log service method for running applications in the kubernetes cluster mainly adopts the traditional distributed log solution, which consumes storage resources and has low log query efficiency, thereby achieving the technical effect of saving storage resources, simplifying the log query process, improving the log query efficiency, and improving the user experience; further, the above application is a Spark application, which solves the technical problem in the prior art that when the Spark application is running in the kubernetes cluster, the log needs to be queried and displayed through specific pages, search, query, display and other components, the log query process is complicated, and the user experience is poor, thereby achieving the technical effect of improving the user experience.

[0065] Figure 2 is a schematic diagram of the main process of a log service method based on a kubernetes cluster provided according to a second embodiment of the present invention; the application scenario of the embodiment of the present invention is a log service method of Spark running on a kubernetes cluster, such as Figure 2 As shown, it mainly includes:

[0066] Step S201, periodically obtain an application number list from the Kubernetes cluster, wherein the running status corresponding to each application number in the application number list is running on the Kubernetes cluster.

[0067] Specifically, the Spark application runs on a kubernetes cluster, and the Spark application runs on multiple running nodes. Each running node locally stores the application logs generated during the running process to obtain a local application log set. Specifically, according to a specific implementation of an embodiment of the present invention, Figure 3 As shown in the figure, after Spark submits the running application to the kubernetes cluster through submit, it allocates multiple tasks to multiple running nodes (for Spark, it is computing nodes, executors, computing programs) through driver, and each running node executes the task processing (i.e. runs the application) and stores the application logs generated during the running process to the local disk (local volume), i.e. performs local storage. The kubernetes cluster uses SparkTaskSvc (running status acquisition module) to query the running status of the application (which can be specific to each running node).

[0068] Since Spark has multiple intermediate states during its operation, not just "running" and "completed", through the above settings, a list of application numbers with a running state of "running" is obtained, so that the running applications can be accurately obtained, and then the applications that are not running can be determined.

[0069] Furthermore, according to an embodiment of the present invention, the above-mentioned application is not only applicable to Spark applications, but also to distributed computing programs, which are applicable to running on a Kubernetes cluster because of the large amount of computing involved in their running process and the need for the coordinated action of multiple running nodes.

[0070] Spark application is a popular big data computing framework with a wide range of applications. In the prior art, Spark applications run in Yarn (another scheduling framework) clusters, but due to the lack of scalability, low flexibility, and low resource utilization of Yarn, offline and online hybrid deployment cannot be well achieved. By combining Spark applications with kubernetes clusters (containerized applications on multiple hosts in the management cloud platform), the scheduling framework can be unified, the computing process can be optimized, and the hybrid deployment of online and offline services can be achieved.

[0071] Step S202: According to the application number list and the first application number, determine whether the application corresponding to the first application number has completed running. If yes, that is, the application corresponding to the first application number has completed running, then execute step S203; if no, that is, the running status corresponding to the first application number is "running", go to step S201.

[0072] Furthermore, according to an embodiment of the present invention, the local list can be directly compared with the above application number list. For applications in the intersection of the application number list and the local list, they are determined to be applications that have not been completed; for applications in the complement of the application number list and the local list, they are determined to be applications that have been completed. Since the local list records all application numbers submitted for application to run in the kubernetes cluster, by comparing the application number list corresponding to the running application with the local list, the complement of the two is the application "not in the running state", and the application logs corresponding to this type of application are transferred from the local storage device to the cloud storage device to facilitate subsequent log query.

[0073] Step S203: determine the application log corresponding to the first application number in the local storage device, store it in the cloud, and delete the corresponding application log in the local storage device.

[0074] Through the above settings, after the corresponding application logs in the local storage device are stored in the cloud, the corresponding application logs in the local storage device are deleted, which can release the storage space of the local storage device and reduce the burden on the running nodes, so as to avoid affecting the running efficiency of the running nodes due to insufficient local storage space. Specifically, according to a specific implementation of an embodiment of the present invention, by setting a cleaner (cleaning module, equivalent to the log storage module in the following text, which can be set in each node of the kubernetes cluster), the cleaner (log storage module) obtains the running status from SparkTaskSvc (running status acquisition module), and stores the application logs corresponding to the "running completed" application in the cloud (stored in CS, cloud storage), and at the same time, deletes the corresponding application logs in the local disk (local volume).

[0075] Step S204, in response to the application log query request initiated by the user, query the local application log set and the cloud application log set corresponding to the running node number in sequence according to the application number and the running node number included in the log query request to determine the target application log.

[0076] Specifically, according to an embodiment of the present invention, the application log query request may further include the application running time (for application logs named after the running time as storage files, the efficiency and accuracy of querying the target application logs may be improved).

[0077] Through the above settings, the target application log corresponding to the running node can be obtained, and then the running status and resource usage status of the application on the running node can be determined according to the target application log, so as to monitor and troubleshoot the running health status of the application on each running node. Figure 3 As shown, according to a specific implementation of an embodiment of the present invention, by setting a log service (log query module, providing log query service), after receiving a log query request initiated by a user, it queries the local disk and the cloud storage device in turn according to the information indicated in the log query request (including the application number and the running node number) to determine the target application log.

[0078] Step S205: Send the target application log to the user.

[0079] Specifically, Figure 3 As shown, according to a specific implementation of an embodiment of the present invention, a user can initiate a log query request through the SparkUI or HistoryServer (history service program) of the Spark application (HistoryServer and SparkUI pages provide a set of web ui user interfaces for users to use (the interface provides log query links for each running node), which are used to monitor the running status of the Spark application and the resource consumption of the kubernetes cluster, etc. Generally, SparkUI is used for running application tasks, and HistoryServer is used for completed application tasks). The user can initiate a log query request including the application running time, application number, and running node number to the log service (log query module) through the page, receive the target application log returned by the log service, and finally the SparkUI or HistoryServer page displays the target application log to the user. Through the above settings, the query process of the application log is hidden from the user, retaining the user's previous habit of viewing the application log through the SparkUI or HistoryServer page under the scheduling framework such as Yarn, thereby improving the user experience.

[0080] According to the technical solution of the embodiment of the present invention, the running status of the application running on the kubernetes cluster is obtained; wherein the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain a local application log set; the application that has been completed is determined according to the running status, and the locally stored application log corresponding to the completed application is stored in the cloud to obtain a cloud application log set; in response to the application log query request initiated by the user, the local application log set and the cloud application log set are sequentially queried according to the application number included in the application log query request to determine the target application log, and the target application log is sent to the user. The technical means overcomes the technical problem that the existing log service method for running applications in the kubernetes cluster mainly adopts the traditional distributed log solution, which consumes storage resources and has low log query efficiency, thereby achieving the technical effect of saving storage resources, simplifying the log query process, improving the log query efficiency, and improving the user experience; further, the above application is a Spark application, which solves the technical problem in the prior art that when the Spark application is running in the kubernetes cluster, the log needs to be queried and displayed through specific pages, search, query, display and other components, the log query process is complicated, and the user experience is poor, thereby achieving the technical effect of improving the user experience.

[0081] Figure 4 is a schematic diagram of main modules of a log service device based on a kubernetes cluster provided according to an embodiment of the present invention; Figure 4 As shown, the log service device 400 based on the kubernetes cluster provided by the embodiment of the present invention mainly includes:

[0082] The running status acquisition module 401 is used to obtain the running status of the application running on the kubernetes cluster; wherein the running node corresponding to the application locally stores the application logs generated during the running process of the application to obtain a local application log set.

[0083] In the implementation of the present invention, a local storage device is mounted on the running node corresponding to the application, so that the running node can directly store the application logs generated during the application running process locally. Since the local storage speed is fast, the log storage efficiency is improved, and the storage failure problem caused by the simultaneous running of multiple applications in the kubernetes cluster is solved.

[0084] Further, according to an embodiment of the present invention, the above-mentioned running status acquisition module 401 is also used to: regularly obtain an application number list from the kubernetes cluster, wherein the running status corresponding to each application number in the application number list is running on the kubernetes cluster.

[0085] In the actual operation process, the operation status is not limited to "running" and "run successfully". Through the above settings, the application ID list with the operation status of "running" can be obtained to accurately obtain the running applications, and then determine the applications that are not running. This is conducive to collecting application logs corresponding to applications that are "not in the running state" and avoiding the loss of application logs.

[0086] The log storage module 402 is used to determine the application that has been completed according to the running status, and store the locally stored application logs corresponding to the application that has been completed in the cloud to obtain a cloud application log set.

[0087] Specifically, according to an embodiment of the present invention, the log storage module 402 is also used for:

[0088] The application number list and the local list are compared, and the applications in the complement of the application number list and the local list are determined to be applications that have been run. The local list includes application numbers corresponding to the applications applied to run on the kubernetes cluster.

[0089] The local list records all the application numbers submitted for running in the kubernetes cluster. By comparing the application number list corresponding to the running applications with the local list, the complement of the two is the "not running" applications. For the application logs corresponding to this type of application, they are transferred from the local storage device to the cloud storage device to facilitate subsequent log queries.

[0090] Further, according to an embodiment of the present invention, after the step of cloud storing the locally stored application logs corresponding to the completed application to obtain a local application log set, the log storage module 402 is further used to:

[0091] Delete the application logs corresponding to the completed applications from the local application log collection.

[0092] Through the above settings, after the corresponding application logs in the local storage device are stored in the cloud, the corresponding application logs in the local storage device are deleted, which can release the storage space of the local storage device and reduce the burden on the running node, so as to avoid affecting the running efficiency of the running node due to insufficient local storage space.

[0093] The log query module 403 is used to respond to the application log query request initiated by the user, query the local application log set and the cloud application log set in sequence according to the application number included in the application log query request to determine the target application log, and send the target application log to the user.

[0094] Specifically, according to an embodiment of the present invention, the application log query request may further include the application running time (for application logs named after the running time as storage files, the efficiency and accuracy of querying the target application logs may be improved).

[0095] Further, according to an embodiment of the present invention, the above application is a Spark application, there are multiple running nodes, and each running node locally stores the application logs generated during the running of the application to obtain a local application log set corresponding to each running node.

[0096] According to an embodiment of the present invention, the above application is a distributed computing program, preferably a Spark application. The distributed computing program design has a large amount of calculation, requires the coordination of multiple running nodes, and is more suitable for running on a kubernetes cluster.

[0097] Spark application is a popular big data computing framework with a wide range of applications. In the existing technology, Spark application runs in Yarn (another scheduling framework) cluster. Due to the insufficient scalability, low flexibility and low resource utilization of Yarn, offline and online hybrid deployment cannot be well achieved. By combining Spark application with kubernetes cluster (containerized application on multiple hosts in the management cloud platform), the scheduling framework can be unified and the computing process can be optimized, and the hybrid deployment of online and offline services can be achieved.

[0098] Preferably, according to an embodiment of the present invention, the application log query request also includes the running node number; the log query module 403 is also used for:

[0099] According to the application number and running node number included in the log query request, the local application log set and the cloud application log set corresponding to the running node number are queried in turn to determine the target application log.

[0100] Specifically, after the application is started, tasks will be assigned to multiple running nodes, and the multiple running nodes will respectively perform part of the running tasks. The running node number is further specified in the application log query request, and the target application log corresponding to the running node can be obtained. Then, the running status and resource usage status of the application on the running node can be determined based on the target application log, so as to monitor and troubleshoot the running health of the application on each running node.

[0101] According to the technical solution of the embodiment of the present invention, the running status of the application running on the kubernetes cluster is obtained; wherein the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain a local application log set; the application that has been completed is determined according to the running status, and the locally stored application log corresponding to the completed application is stored in the cloud to obtain a cloud application log set; in response to the application log query request initiated by the user, the local application log set and the cloud application log set are sequentially queried according to the application number included in the application log query request to determine the target application log, and the target application log is sent to the user. The technical means overcomes the technical problem that the existing log service method for running applications in the kubernetes cluster mainly adopts the traditional distributed log solution, which consumes storage resources and has low log query efficiency, thereby achieving the technical effect of saving storage resources, simplifying the log query process, improving the log query efficiency, and improving the user experience; further, the above application is a Spark application, which solves the technical problem in the prior art that when the Spark application is running in the kubernetes cluster, the log needs to be queried and displayed through specific pages, search, query, display and other components, the log query process is complicated, and the user experience is poor, thereby achieving the technical effect of improving the user experience.

[0102] Figure 5 An exemplary system architecture 500 is shown to which a log service method based on a Kubernetes cluster or a log service device based on a Kubernetes cluster according to an embodiment of the present invention can be applied.

[0103] like Figure 5 As shown, the system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505 (this architecture is only an example, and the components included in the specific architecture may be adjusted according to the specific application). The network 504 is used to provide a medium for a communication link between the terminal devices 501, 502, 503 and the server 505. The network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0104] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as big data computing applications, data processing applications, log service applications, etc. (only for example).

[0105] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0106] The server 505 may be a server that provides various services, such as a background management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503. The background management server may analyze and process the received application log query request and other data, and feed back the processing results (such as the target application log - only an example) to the terminal device.

[0107] It should be noted that the log service method based on the kubernetes cluster provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the log service device based on the kubernetes cluster is generally arranged in the server 505 .

[0108] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0109] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 6 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0110] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0111] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0112] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0113] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0114] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor. For example, they may be described as: a processor includes a running status acquisition module, a log storage module, and a log query module. The names of these modules do not constitute a limitation on the module itself in some cases. For example, the running status acquisition module may also be described as "a module for obtaining the running status of an application running on a kubernetes cluster, and sending the obtained running status to a log storage module, wherein the running node corresponding to the application locally stores the application logs generated by the application during operation, and obtains a local application log collection module".

[0116] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining the running status of the application running on the kubernetes cluster; wherein the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain a local application log set; determining the application that has been completed according to the running status, and storing the locally stored application log corresponding to the application that has been completed in the cloud to obtain a cloud application log set; in response to the application log query request initiated by the user, querying the local application log set and the cloud application log set in turn according to the application number included in the application log query request to determine the target application log, and sending the target application log to the user.

[0117] According to the technical solution of the embodiment of the present invention, the running status of the application running on the kubernetes cluster is obtained; wherein the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain a local application log set; the application that has been completed is determined according to the running status, and the locally stored application log corresponding to the completed application is stored in the cloud to obtain a cloud application log set; in response to the application log query request initiated by the user, the local application log set and the cloud application log set are sequentially queried according to the application number included in the application log query request to determine the target application log, and the target application log is sent to the user. The technical means overcomes the technical problem that the existing log service method for running applications in the kubernetes cluster mainly adopts the traditional distributed log solution, which consumes storage resources and has low log query efficiency, thereby achieving the technical effect of saving storage resources, simplifying the log query process, improving the log query efficiency, and improving the user experience; further, the above application is a Spark application, which solves the technical problem in the prior art that when the Spark application is running in the kubernetes cluster, the log needs to be queried and displayed through specific pages, search, query, display and other components, the log query process is complicated, and the user experience is poor, thereby achieving the technical effect of improving the user experience.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A log service method based on kubernetes cluster, characterized in that: include: Obtaining the running status of an application running on the Kubernetes cluster, specifically comprising: periodically obtaining a list of application numbers from the Kubernetes cluster, wherein the running status corresponding to each application number in the list is running on the Kubernetes cluster; wherein a running node corresponding to the application locally stores an application log generated by the application during operation to obtain a local application log set; Determining the applications that have completed running according to the running status specifically includes: comparing the application number list with a local list, and determining the applications in the complement of the application number list and the local list as completed applications, wherein the local list includes application numbers corresponding to the applications that have applied to run on the Kubernetes cluster; Storing the locally stored application logs corresponding to the completed applications in the cloud to obtain a cloud application log set; In response to an application log query request initiated by a user, the local application log set and the cloud application log set are queried in sequence according to the application number included in the application log query request to determine the target application log, and the target application log is sent to the user.

2. The log service method based on the kubernetes cluster according to claim 1, characterized in that: After the step of storing the locally stored application logs corresponding to the completed applications in the cloud to obtain a local application log set, the method further includes: The application log corresponding to the application that has completed running is deleted from the local application log set.

3. The log service method based on the kubernetes cluster according to claim 1, characterized in that: The application is a Spark application, and there are multiple running nodes. Each running node locally stores application logs generated during the running of the Spark application to obtain a local application log set corresponding to each running node.

4. The log service method based on the kubernetes cluster according to claim 3, characterized in that: The application log query request also includes the running node number; The step of sequentially querying the local application log set and the cloud application log set according to the application number included in the application log query request to determine the target application log further includes: According to the application number and the running node number included in the log query request, the local application log set and the cloud application log set corresponding to the running node number are queried in sequence to determine the target application log.

5. A log service device based on a kubernetes cluster, characterized in that: It includes a running status acquisition module, a log storage module and a log query module; among which, The running status acquisition module is used to obtain the running status of the application running on the Kubernetes cluster, specifically to periodically obtain a list of application numbers from the Kubernetes cluster, wherein the running status corresponding to each application number in the application number list is running on the Kubernetes cluster; and send the obtained running status to the log storage module, wherein the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain a local application log set; The log storage module is configured to determine completed applications based on the running status, specifically to compare the application number list with a local list, and determine applications in the complement of the application number list and the local list as completed applications, wherein the local list includes application numbers corresponding to applications applying to run on the Kubernetes cluster; and to cloud store the locally stored application logs corresponding to the completed applications to obtain a cloud application log set; The log query module is used to respond to an application log query request initiated by a user, query the local application log set and the cloud application log set in sequence according to the application number included in the application log query request, to determine the target application log, and send the target application log to the user.

6. The log service device based on the kubernetes cluster according to claim 5, characterized in that: The application is a Spark application, and there are multiple running nodes. Each running node locally stores application logs generated during the running of the Spark application to obtain a local application log set corresponding to each running node.

7. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

8. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images