Log information processing method and device, electronic equipment, medium and program product
By sending log information to persistent storage and a log subscription consumer, the method addresses the loss of logs in Kubernetes environments, ensuring effective log maintenance and processing.
Patent Information
- Application Number
- CN202411149315.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-06
AI Technical Summary
In distributed environments like Kubernetes, log files generated by short-lived containers are often lost when the containers are destroyed, posing a challenge for effective log collection and maintenance.
Log information is sent to both a first and a second message middleware, with the log information from the first being stored in a persistent storage device and the second being sent to a log subscription consumer, ensuring persistence and enabling data processing activities.
This approach prevents log loss due to container destruction, enhances log maintenance, and allows for data processing and presentation of the log information to the log subscription consumer.
Smart Images

Figure CN119088617A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular to a method, device, electronic device, medium and program product for processing log information. Background Art
[0002] With the rise of cloud-native technology, containerization has become an important way of software development and delivery. As one of the most popular container orchestration platforms, K8s (Kubernetes) provides enterprises with powerful resource scheduling and management capabilities, facilitating the deployment and expansion of applications in distributed environments. However, in such a distributed environment, log collection becomes a very challenging task, because containers are short-lived and dynamic, and log files are often lost when containers are destroyed. It can be seen that in the related technologies, there is a problem that the logs generated by containers are easily lost when containers are destroyed. Summary of the invention
[0003] The present application provides a method, device, electronic device, medium and program product for processing log information, which can avoid the problem of corresponding log loss due to the destruction of the container, thereby facilitating the improvement of the maintenance effect of the log information.
[0004] In a first aspect, an embodiment of the present application provides a method for processing log information, the method comprising:
[0005] Sending log information generated by all containers in a server cluster to a first message middleware and a second message middleware respectively, wherein a plurality of containers are deployed in the server cluster;
[0006] The log information stored in the first message middleware is stored in a persistent storage device, and the log information stored in the second message middleware is pushed to a log subscription consumer end.
[0007] In a second aspect, an embodiment of the present application provides a log information processing device, including:
[0008] A sending module, used to send log information generated by all containers in the server cluster to the first message middleware and the second message middleware respectively, wherein multiple containers are deployed in the server cluster;
[0009] The processing module is used to store the log information stored in the first message middleware in a persistent storage device, and to push the log information stored in the second message middleware to a log subscription consumer.
[0010] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method steps of the first aspect described above are implemented.
[0011] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method steps of the first aspect described above are implemented.
[0012] In a fifth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method of the first aspect.
[0013] In an embodiment of the present application, the logs generated by all containers in the server cluster are sent to the first message middleware and the second message middleware respectively, and the log information stored in the first message middleware is stored in a persistent storage device. Since the persistent storage device can store the log information of all containers in the server cluster, the problem of corresponding log loss caused by the destruction of the container can be avoided, which is conducive to improving the maintenance effect of the log information. In addition, by pushing the log information stored in the second message middleware to the log subscription consumer end, the relevant consumer end can obtain the log information of the server cluster by subscription, and implement data processing activities related to the log information based on the obtained log information, which is conducive to enriching the presentation form of the log information to meet the diverse needs of users for log information. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0015] Figure 1 This is one of the flow charts of the method for processing log information provided in the embodiment of the present application;
[0016] Figure 2 This is the second flowchart of the method for processing log information provided in the embodiment of the present application;
[0017] Figure 3 This is one of the structural schematic diagrams of the log information processing device provided in the embodiment of the present application;
[0018] Figure 4This is the second structural diagram of the log information processing device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0020] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0021] See also Figure 1 and Figure 2 , Figure 1 and Figure 2 A flowchart of a log information processing method provided in this application, the method comprising:
[0022] Step 101: Send log information generated by all containers in a server cluster to a first message middleware and a second message middleware respectively, wherein a plurality of containers are deployed in the server cluster;
[0023] Step 102: Store the log information stored in the first message middleware in a persistent storage device, and push the log information stored in the second message middleware to a log subscription consumer.
[0024] The above-mentioned server cluster may be various server clusters in related technologies. For example, in some embodiments of the present application, the server cluster may be a Kubernetes cluster.
[0025] The above container can be a container corresponding to various service applications, that is, a specific service application can be deployed in the container, so that the container with the service application deployed can provide services corresponding to the service application, wherein the service application can be a common service application such as a query service application and a computing service application. It can be understood that the query service application can provide query services for users. For example, the user can send a data query request to the container with the query service application deployed through the client. After the container receives the data query request, it can respond to the data query request to query the data that the user needs to query in a specific database, and send the queried data to the client. Accordingly, the computing service application can provide specific computing services for users.
[0026] It is understandable that the server cluster may include at least two servers, each server may be deployed with at least one container, and the container types deployed in the same server may be the same or different. Accordingly, the types of containers deployed in different servers may be the same or different.
[0027] The above-mentioned sending the log information generated by all containers in the server cluster to the first message middleware and the second message middleware respectively means: sending the log information generated by all containers in the server cluster to the first message middleware, and sending the log information generated by all containers in the server cluster to the second message middleware. Among them, in the server cluster, each server can send the log information of the included containers to the first message middleware and the second message middleware separately, or a specific server in the server cluster can receive the log information of all containers and send the received log information to the first message middleware and the second message middleware.
[0028] The first message middleware and the second message middleware mentioned above can be various common message middleware in the relevant technology. Among them, the message middleware can also be called a message queue, which refers to the use of an efficient and reliable message transmission mechanism for platform-independent data exchange, and the integration of distributed systems based on data communication. By providing a message transmission and message queue model, the communication of processes can be extended in a distributed environment. The current mainstream message middlewares include RabbitMQ, Kafka, ActiveMQ, RocketMQ, etc. It can communicate between different platforms, and is often used to shield the characteristics between various platform protocols and realize the collaboration between applications. Its advantage is that it can connect synchronously and asynchronously between the client and the server, and can transmit and forward messages at any time. It is a very important component in a distributed system, mainly used to solve problems such as application coupling, asynchronous communication, and traffic peak shaving.
[0029] The first message middleware and the second message middleware may be two different message middlewares in the server cluster. In addition, the first message middleware and the second message middleware may also be two different message middlewares provided by two different third-party services. Alternatively, one of the first message middleware and the second message middleware is a message middleware in the server cluster, and the other is a message middleware provided by a third-party service.
[0030] The above-mentioned log subscription consumer end can be various consumer ends that provide log analysis services. For example, the log subscription consumer end can be a client for detecting the health status of a server cluster. In this way, the log subscription consumer end can obtain the log information generated by all containers in the server cluster from the second message middleware through log subscription, and can analyze the health status of the server cluster by analyzing the obtained log information. For example, when the log subscription consumer end determines through analysis of the log information that the failure rate of the target service in the server cluster in processing user requests is higher than a preset threshold, a first warning information can be output, wherein the first warning information indicates that the target service is abnormal.
[0031] The above-mentioned persistent storage device can be various storage devices in the related technology, for example, it can be a storage device corresponding to the storage service provided by a third-party service. It is understandable that the log information of the container is persistently stored through the persistent storage device, so that even if the container is destroyed, it will not affect the log information stored in the persistent storage device.
[0032] In this implementation, the logs generated by all containers in the server cluster are sent to the first message middleware and the second message middleware respectively, and the log information stored in the first message middleware is stored in a persistent storage device. Since the persistent storage device can store the log information of all containers in the server cluster, the problem of corresponding log loss caused by the destruction of the container can be avoided, which is conducive to improving the maintenance effect of the log information. In addition, by pushing the log information stored in the second message middleware to the log subscription consumer end, the relevant consumer end can obtain the log information of the server cluster by subscription, and implement data processing activities related to the log information based on the obtained log information, which is conducive to enriching the presentation form of the log information to meet the diverse needs of users for log information.
[0033] Optionally, see Figure 2 , storing the log information stored in the first message middleware in a persistent storage device includes:
[0034] The log statistical indicator calculation and analysis service pulls log information from the first message middleware, and analyzes the pulled log information to obtain log statistical indicators;
[0035] The task service reads the log statistical indicator generated by the log statistical indicator service, and creates a storage repository table corresponding to the log statistical indicator in the persistent storage device;
[0036] The storage service pulls log information from the first message middleware, performs data cleaning on the pulled log information, and stores the cleaned log information in a corresponding position indicated by the storage repository table according to the attributes of the cleaned log information.
[0037] The above-mentioned calculation and analysis log statistical indicator service, task service and storage service can all be pre-deployed third-party services.
[0038] The above-mentioned log statistics service can pull and consume logs from the "message middleware" and obtain various log statistics indicators through calculation and analysis, such as the log level (DEBUG, INFO, WARN, ERROR), which application Pod the log belongs to, and the time period to which the log belongs.
[0039] The above-mentioned task service reads the log statistical indicators generated by the log statistical indicator service, and creates a storage table structure according to the read log statistical indicators. For example, if the "persistent storage" is Elasticsearch, the "task service" creates Elasticsearch Indices according to the read statistical indicators.
[0040] The storage service pulls and consumes the information log in the first message middleware, and after data cleaning, stores the log persistently in the repository table created by the "task service".
[0041] Among them, the above-mentioned storage repository table can be used to describe the storage location of the logs of various log statistical indicators. For example, when the log statistical indicator includes a date, the storage repository table may include storage locations corresponding to different dates. Correspondingly, when the log statistical indicator includes a service type, the storage repository table may include storage locations corresponding to different service types. When the log statistical indicator includes a container ID, the storage repository table may include storage locations corresponding to different container IDs. It can be understood that since the same log information usually corresponds to a date, a service type and a container ID at the same time, according to different attributes, based on the storage repository table, it may be possible to determine more than two different storage locations corresponding to the same log information. In this case, the same log information can be stored in all corresponding locations. Among them, the date, container ID and service type are all attributes of the log information.
[0042] The data cleaning of the pulled log information may specifically be to clean the log information based on various data cleaning methods, for example, converting the log information into a unified format, or deleting data in a specific format in the log information.
[0043] Specifically, since the above-mentioned repository table can be used to describe the storage location of logs of various log statistical indicators, and according to the different attributes of the log information, the same log information may correspond to different storage locations in the persistent storage device, the cleaned log information can be stored in the repository table according to the attributes of the cleaned log information.
[0044] In the related art, since log information comes from a wide range of sources and has different formats, it is difficult to incorporate it into the analysis platform in a unified manner. Therefore, there is a problem of difficulty in integrating stored data. Based on this, in an embodiment of the present application, log information is pulled from the first message middleware by calculating and analyzing the log statistical indicator service, and the pulled log information is analyzed to obtain log statistical indicators, and based on the log statistical indicators, a repository table corresponding to the log statistical indicators is created in the persistent storage device, and then, according to the attributes of the cleaned log information, the cleaned log information is stored in the location indicated by the repository table. At the same time, before storing the log information in the location indicated by the repository table, the log information is cleaned to integrate the format of the log information, thereby solving the problem of difficult data integration and improving the real-time performance of log information processing.
[0045] In addition, the log processing methods in the related art still have a low degree of intelligence. The existing log analysis tools can only complete basic log classification and analysis tasks, and have limited support for higher-level threat detection and automation. In the embodiments of the present application, based on the low degree of intelligence, the existing log analysis tools can only complete basic log classification and analysis tasks, and have limited support for higher-level threat detection and automation, which is conducive to supporting higher-level threat detection and automation.
[0046] Optionally, the log statistical indicator includes a first indicator and a second indicator, wherein a first indicator corresponds to an attribute of the log information, and the second indicator is a composite indicator generated based on the log information corresponding to at least two first indicators.
[0047] Among them, the number of the first indicators and the second indicators included in the above-mentioned log statistical indicators can be multiple, wherein a first indicator corresponds to an attribute of the log information, for example, the first indicator can be: date, container ID, service type, whether the response is successful, etc.
[0048] The above-mentioned task service reads the log statistical indicator generated by the log statistical indicator service, and creates a storage repository table corresponding to the log statistical indicator in the persistent storage device, which specifically refers to: the task service reads the log statistical indicator generated by the log statistical indicator service, and creates a storage repository table corresponding to the first indicator in the log statistical indicator in the persistent storage device.
[0049] The above-mentioned calculation and analysis log statistical indicator service can pull log information from the first message middleware every preset time, and analyze the pulled log information to obtain log statistical indicators, wherein the duration of the preset time can be set according to specific needs, for example, half an hour, 1 hour, 1 day, etc. In this way, the indicator type of the first indicator can be analyzed in advance to construct the second indicator. For example, the second indicator can be the number of times a service response fails within the preset time, and the generated second indicator can be pushed to the relevant client for display, so that the relevant personnel can handle the abnormal situation in the server cluster in time, wherein the first indicator "whether the response is successful" can be used to determine all log information of failed responses, and then, the second indicator can be obtained by counting the number of log information identified by the response. In addition, the above-mentioned second indicator can also continuously record the changing trend of the number of user requests corresponding to a certain service, so as to adjust the business in the server cluster based on the second indicator, wherein the number of user requests for a certain service on each date can be determined by the first indicator "service type" and the first indicator "date", and based on this, the second indicator is generated.
[0050] Optionally, see Figure 2 , each container in the server cluster is pre-configured with a log collection plug-in, the server cluster includes a first server, the first server is pre-configured with a receiving service, and the log information generated by all containers in the server cluster is sent to the first message middleware and the second message middleware respectively, including:
[0051] The log collection plug-in in the container in the server cluster sends the collected log information to the receiving service when collecting log information of the container to which it belongs, and sends the collected log information to the second message middleware;
[0052] The receiving service sends the received log information to the first message middleware.
[0053] The log collection plug-in can be an application plug-in in various forms, for example, a software development kit (SDK) plug-in. Specifically, the log collection plug-in can be deployed in the container using SDK in the continuous integration (CI) and early stages.
[0054] In this implementation, the collection of log information of the container is achieved by adopting a log collection plug-in, so that there is no need for manual intervention to install and deploy the collection terminal, which is conducive to simplifying the log collection process.
[0055] Optionally, see Figure 2 , each server in the server cluster is pre-configured with an agent service, and the log information generated by all containers in the server cluster is sent to the first message middleware and the second message middleware respectively, including:
[0056] The proxy service obtains log information of all containers in the server to which it belongs;
[0057] The proxy service sends the acquired log information to the first message middleware and the second message middleware respectively.
[0058] The proxy service can be deployed to the Worker node of Kubernetes using Daemonset. Specifically, the proxy service can be deployed in the continuous delivery (CD) and later stages.
[0059] It can be understood that in some embodiments of the present application, based on any one of the above-mentioned pre-configured log collection plug-in and proxy service, the log information of the containers in the server cluster is collected.
[0060] In this implementation, the collection of log information of the container is achieved by adopting the form of proxy service, so that there is no need for manual intervention to install and deploy the collection terminal, which is conducive to simplifying the log collection process.
[0061] Optionally, see Figure 2 After storing the log information stored in the first message middleware in a persistent storage device, the method further includes:
[0062] Based on the preset analysis task, the log information stored in the first message middleware is analyzed to obtain analysis data;
[0063] The analysis data is sent to a target terminal for display.
[0064] Specifically, the above-mentioned preset analysis tasks can be tasks pre-configured by relevant personnel. For example, the preset analysis tasks may include various anomaly detection tasks, trend analysis tasks, etc. The anomaly detection task can be to analyze the log information in various ways to determine whether there is an anomaly in the server cluster. The trend analysis task can be to obtain the changing trend of the corresponding indicator by statistics on various data.
[0065] In some embodiments of the present application, the above-mentioned preset analysis tasks may include an analysis task for counting the number of times each service in the server cluster provides services to users. Accordingly, the analysis data may include a changing trend of the number of times each service provides services to users.
[0066] In other embodiments of the present application, the above-mentioned preset analysis tasks may include statistical tasks for service response time of each service in the server cluster, and accordingly, the analysis data may include the average response time of each service for one response.
[0067] In this implementation, the target terminal may be a front-end client of the server cluster. Thus, by sending the analysis data to the target terminal for display, relevant personnel can intuitively understand the current operating status of the server cluster based on the analysis data.
[0068] See also Figure 3 , Figure 3 The present invention provides a schematic diagram of a log information processing device 300 according to an embodiment of the present invention. The log information processing device 300 includes:
[0069] A sending module 301 is used to send log information generated by all containers in the server cluster to the first message middleware and the second message middleware respectively, wherein multiple containers are deployed in the server cluster;
[0070] The processing module 302 is used to store the log information stored in the first message middleware in a persistent storage device, and to push the log information stored in the second message middleware to a log subscription consumer.
[0071] Optionally, the processing module 302 includes:
[0072] An analysis submodule, used for calculating and analyzing the log statistical indicator service to pull log information from the first message middleware, and analyze the pulled log information to obtain log statistical indicators;
[0073] Create a submodule, which is used for the task service to read the log statistical indicator generated by the log statistical indicator service, and create a storage repository table corresponding to the log statistical indicator in the persistent storage device;
[0074] The storage submodule is used to store the storage service to pull log information from the first message middleware, clean the pulled log information, and store the cleaned log information in the location indicated by the storage repository table according to the attributes of the cleaned log information.
[0075] Optionally, the log statistical indicator includes a first indicator and a second indicator, wherein a first indicator corresponds to an attribute of the log information, and the second indicator is a composite indicator generated based on the log information corresponding to at least two first indicators.
[0076] Optionally, a log collection plug-in is pre-configured in each container in the server cluster, the server cluster includes a first server, a receiving service is pre-configured in the first server, and the sending module 301 is specifically used for the log collection plug-in in the container in the server cluster to send the collected log information to the receiving service when the log information of the container to which it belongs is collected, and to send the collected log information to the second message middleware;
[0077] The sending module 301 is further specifically configured for the receiving service to send the received log information to the first message middleware.
[0078] Optionally, each server in the server cluster is pre-configured with a proxy service, and the sending module 301 is specifically used for the proxy service to obtain log information of all containers in the server to which it belongs;
[0079] The sending module 301 is further specifically used for the proxy service to send the acquired log information to the first message middleware and the second message middleware respectively.
[0080] Optionally, the device further comprises:
[0081] An analysis module, used to analyze the log information stored in the first message middleware based on a preset analysis task to obtain analysis data;
[0082] The sending module 301 is further used to send the analysis data to a target terminal for display.
[0083] The log information processing device 300 provided in the embodiment of the present application can implement each process in the above-mentioned log information processing method embodiment, and will not be described again here to avoid repetition.
[0084] See also Figure 4 , Figure 4 is a structural diagram of a log information processing device 400 provided by another embodiment of the present application, such as Figure 4 As shown, the log information processing device 400 includes: a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor. The various components in the log information processing device 400 are coupled together via a bus interface 403. When the computer program is executed by the processor 401, the following steps are implemented:
[0085] Sending log information generated by all containers in the server cluster to the first message middleware and the second message middleware respectively, wherein multiple containers are deployed in the server cluster;
[0086] The log information stored in the first message middleware is stored in a persistent storage device, and the log information stored in the second message middleware is pushed to a log subscription consumer end.
[0087] Optionally, storing the log information stored in the first message middleware in a persistent storage device includes:
[0088] The log statistical indicator calculation and analysis service pulls log information from the first message middleware, and analyzes the pulled log information to obtain log statistical indicators;
[0089] The task service reads the log statistical indicator generated by the log statistical indicator service, and creates a storage repository table corresponding to the log statistical indicator in the persistent storage device;
[0090] The storage service pulls log information from the first message middleware, performs data cleaning on the pulled log information, and stores the cleaned log information in a corresponding position indicated by the storage repository table according to the attributes of the cleaned log information.
[0091] Optionally, the log statistical indicator includes a first indicator and a second indicator, wherein a first indicator corresponds to an attribute of the log information, and the second indicator is a composite indicator generated based on the log information corresponding to at least two first indicators.
[0092] Optionally, each container in the server cluster is pre-configured with a log collection plug-in, the server cluster includes a first server, the first server is pre-configured with a receiving service, and the log information generated by all containers in the server cluster is sent to the first message middleware and the second message middleware respectively, including:
[0093] The log collection plug-in in the container in the server cluster sends the collected log information to the receiving service when collecting log information of the container to which it belongs, and sends the collected log information to the second message middleware;
[0094] The receiving service sends the received log information to the first message middleware.
[0095] Optionally, each server in the server cluster is pre-configured with a proxy service, and the sending of log information generated by all containers in the server cluster to the first message middleware and the second message middleware respectively includes:
[0096] The proxy service obtains log information of all containers in the server to which it belongs;
[0097] The proxy service sends the acquired log information to the first message middleware and the second message middleware respectively.
[0098] Optionally, after storing the log information stored in the first message middleware in a persistent storage device, the method further includes:
[0099] Based on the preset analysis task, the log information stored in the first message middleware is analyzed to obtain analysis data;
[0100] The analysis data is sent to a target terminal for display.
[0101] An embodiment of the present application also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0102] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0103] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0104] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for an electronic device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.
[0105] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A method for processing log information, It is characterized in that The method comprises: Sending log information generated by all containers in a server cluster to a first message middleware and a second message middleware respectively, wherein a plurality of containers are deployed in the server cluster; The log information stored in the first message middleware is stored in a persistent storage device, and the log information stored in the second message middleware is pushed to a log subscription consumer end.
2. The method according to claim 1, It is characterized in that The storing the log information stored in the first message middleware in a persistent storage device includes: The log statistical indicator calculation and analysis service pulls log information from the first message middleware, and analyzes the pulled log information to obtain log statistical indicators; The task service reads the log statistical indicator generated by the log statistical indicator service, and creates a storage repository table corresponding to the log statistical indicator in the persistent storage device; The storage service pulls log information from the first message middleware, performs data cleaning on the pulled log information, and stores the cleaned log information in a corresponding position indicated by the storage repository table according to the attributes of the cleaned log information.
3. The method according to claim 2, It is characterized in that The log statistical index includes a first index and a second index, wherein a first index corresponds to an attribute of the log information, and the second index is a composite index generated based on the log information corresponding to at least two first indicators.
4. The method according to claim 1, It is characterized in that A log collection plug-in is pre-configured in each container in the server cluster, the server cluster includes a first server, a receiving service is pre-configured in the first server, and the log information generated by all containers in the server cluster is sent to the first message middleware and the second message middleware respectively, including: The log collection plug-in in the container in the server cluster sends the collected log information to the receiving service when collecting log information of the container to which it belongs, and sends the collected log information to the second message middleware; The receiving service sends the received log information to the first message middleware.
5. The method according to claim 1, It is characterized in that Each server in the server cluster is pre-configured with a proxy service, and the log information generated by all containers in the server cluster is sent to the first message middleware and the second message middleware respectively, including: The proxy service obtains log information of all containers in the server to which it belongs; The proxy service sends the acquired log information to the first message middleware and the second message middleware respectively.
6. The method according to any one of claims 1 to 5, It is characterized in that After storing the log information stored in the first message middleware in a persistent storage device, the method further includes: Based on the preset analysis task, the log information stored in the first message middleware is analyzed to obtain analysis data; The analysis data is sent to a target terminal for display.
7. A log information processing device, It is characterized in that include: A sending module, used to send log information generated by all containers in a server cluster to the first message middleware and the second message middleware respectively, wherein a plurality of containers are deployed in the server cluster; The processing module is used to store the log information stored in the first message middleware in a persistent storage device, and to push the log information stored in the second message middleware to a log subscription consumer.
8. An electronic device, It is characterized in that The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method steps according to any one of claims 1 to 6 when executed by the processor.
9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 6 are implemented.
10. A computer program product, It is characterized in that The computer program product is stored in a storage medium, and the computer program product is executed by at least one processor to implement the method steps according to any one of claims 1 to 6.
