Log collection method, device, storage medium and electronic device
By adding namespace information to the target container logs in the target cluster in the container cloud, only the logs are collected to collect account logs with the status of the open state, the problems of waste of resources and insufficient user rights management in the existing technology are solved, and dynamic logs are realized Collection and resource conservation.
Patent Information
- Application Number
- CN202210130090.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-11
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-02-11
AI Technical Summary
In the prior art, user permissions cannot be dynamically managed when collecting logs in the container cloud, resulting in waste of resources and the inability to meet the log collection needs of different users.
By obtaining the target container log in the target cluster, adding namespace information according to the container identifier in the log storage address, only the log corresponding to the account with the log collection status is on is pushed to the log storage component to realize dynamic logs for different users collection.
It solves the problem of resource waste, realizes the log collection needs for different users, and saves resources.
Smart Images

Figure CN114500582B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of communications, and in particular, to a log collection method, device, storage medium and electronic device. Background Art
[0002] In the related art, the log management solution used in the container cloud is generally EFK (Elasticsearch, Filebeat or Fluentd, Kibana) or Loki+Grafana, etc. First, the standard output log of the container stored in the docker directory on each node is pushed to the log storage component through the log collector component, and then queried. Although Kibana and Grafana have their own user permission system, they cannot be connected to the user permission system of the container cloud. And because the logs of all containers on the node are collected, the logs of useless containers may be collected, resulting in a waste of resources. It is impossible to meet the log collection needs of dynamically turning on or off containers deployed by different users.
[0003] It can be seen from this that there is a problem of waste of resources caused by collecting logs in the related technology.
[0004] Currently, no effective solution has been proposed for the above-mentioned problems existing in the related technologies. Summary of the invention
[0005] The embodiments of the present invention provide a log collection method, device, storage medium and electronic device to at least solve the problem of waste of resources caused by log collection in the related art.
[0006] According to an embodiment of the present invention, a log collection method is provided, comprising: obtaining logs stored in a target container included in a target cluster; adding namespace information to each log based on a container identifier included in a storage address of each log in the log to obtain a first log; pushing the first log to a target gateway to instruct the target gateway to send a target log included in the first log to a log storage component, wherein the namespace information of the target log is the namespace information included in a first namespace information list of a first account, and the first account is an account that uses the target container and whose log collection status is turned on.
[0007] According to another embodiment of the present invention, a log collection device is provided, including: an acquisition module, used to obtain logs stored in a target container included in a target cluster; an adding module, used to add namespace information to each log based on a container identifier included in a storage address of each log in the log, to obtain a first log; and a collection module, used to push the first log to a target gateway to instruct the target gateway to send a target log included in the first log to a log storage component, wherein the namespace information of the target log is the namespace information included in a first namespace information list of a first account, and the first account is an account that uses the target container and whose log collection status is turned on.
[0008] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program implements the steps of any of the above methods when executed by a processor.
[0009] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0010] Through the present invention, the log stored in the target container included in the target cluster is obtained, and the namespace information is added to the target log according to the container identifier included in the storage address of each log in the log, and the first log is obtained. The first log is pushed to the target gateway. After receiving the first log, the target gateway can send the target log included in the first log to the log storage component for storage, wherein the namespace information of the target log is the namespace information included in the first namespace information list of the first account, and the first account is an account that uses the target container, and the log collection state is an open state. Since when collecting logs, only the target log corresponding to the account whose log collection state is an open state is stored in the log storage component, there is no need to store the logs of all containers in the target cluster. Therefore, the problem of waste of resources caused by collecting logs in the related art can be solved, and the effect of saving resources can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 It is a hardware structure block diagram of a mobile terminal of a log collection method according to an embodiment of the present invention;
[0012] Figure 2 is a flow chart of a log collection method according to an embodiment of the present invention;
[0013] Figure 3 is a flow chart of collecting logs according to an exemplary embodiment of the present invention;
[0014] Figure 4 is a flow chart of log query according to an exemplary embodiment of the present invention;
[0015] Figure 5 It is a structural block diagram of a log collection device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0016] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.
[0017] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0018] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a log collection method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0019] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the log collection method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0020] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0021] In this embodiment, a log collection method is provided. Figure 2 is a flow chart of a log collection method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0022] Step S202, obtaining logs stored in a target container included in the target cluster;
[0023] Step S204, based on the container identifier included in the storage address of each log in the log, add namespace information to each log to obtain a first log;
[0024] Step S206: Push the first log to the target gateway to instruct the target gateway to send the target log included in the first log to the log storage component, wherein the namespace information of the target log is the namespace information included in the first namespace information list of the first account, and the first account is an account that uses the target container and whose log collection status is turned on.
[0025] In the above embodiment, the target cluster can be kubernetes, and the target cluster can include multiple nodes, each node can include one or more target containers, and the target container can store logs generated by different applications, that is, the target container can store logs of applications deployed by different users. When obtaining the logs stored in the target container, it can be obtained through the log collection component. After obtaining the log, namespace information can be added to each log according to the container identifier included in the log storage address to obtain the first log. And push the first log to the target gateway. After receiving the first log, the target gateway can determine the log collection status of the account connected to the target gateway, determine the account whose log collection status is turned on, and determine the namespace information list of the account whose log collection status is turned on, and determine that the namespace information included in the first log is the target log in the first namespace information list and store it in the log storage component. That is, after pushing the first log to the target gateway, the target gateway can determine whether the namespace information in the first log that reaches the gateway is in the namespace list of the account that turns on log collection. If the namespace information in the log is in the list of the namespace list of the account that turns on log collection, the log request is forwarded to the log storage component, and the response result of the log storage component is returned. Otherwise, the log push request is discarded. The first account is the account corresponding to the user who deployed the application to the target container. The flowchart of collecting logs can be found in the attached Figure 3 .
[0026] In the above embodiment, the order logs in the docker directory can be collected, and the storage address of the logs can be in / var / lib / docker / containers / <containerid> / *-json.log, you can get the corresponding pod information according to the containerId, that is, the container identifier, add the pod namespace information to the log and push it to the gateway.
[0027] In the above embodiment, the target gateway can monitor the changes of account configuration and namespace through apiserver, and maintain a namespace list of accounts with log collection enabled in memory. If the account configuration is changed to enable monitoring, all namespaces of the account are inserted into the namespace list, otherwise they are deleted from the namespace list.
[0028] In the above embodiment, the log collection status of the first account can be determined according to the account configuration stored in the container cloud platform. The user can adjust the log collection status of the first account by modifying the account configuration.
[0029] In the above embodiment, when the target log is a log of the first account, but there is no namespace information matching the namespace information list of the target log in the first namespace information list of the first account, a prompt message may be issued, wherein the prompt message is used to prompt the user that the target log does not exist in the first namespace information list, whether to create a new namespace information in the first namespace information list, or whether to delete the target log. And based on the received instruction, the creation or deletion operation is performed.
[0030] Optionally, the executor of the above steps may be a container cloud platform, a background processor, or other devices with similar processing capabilities, or a machine that at least integrates a data processing device, wherein the data processing device may include terminals such as computers and mobile phones, but is not limited thereto.
[0031] Through the present invention, the log stored in the target container included in the target cluster is obtained, and the namespace information is added to the target log according to the container identifier included in the storage address of each log in the log, and the first log is obtained. The first log is pushed to the target gateway. After receiving the first log, the target gateway can send the target log included in the first log to the log storage component for storage, wherein the namespace information of the target log is the namespace information included in the first namespace information list of the first account, and the first account is an account that uses the target container, and the log collection state is an open state. Since when collecting logs, only the target log corresponding to the account whose log collection state is an open state is stored in the log storage component, there is no need to store the logs of all containers in the target cluster. Therefore, the problem of waste of resources caused by collecting logs in the related art can be solved, and the effect of saving resources can be achieved.
[0032] In an exemplary embodiment, obtaining logs stored in a target container included in a target cluster includes: monitoring container group information in a container group included in the target cluster through an apiserver, wherein the container group includes a target container; and obtaining the logs stored in the target container when it is determined that the information of the target container included in the container group information is updated. In this embodiment, when obtaining logs, the log collection component can monitor pod resources, i.e., container group information, through an apiserver. When the information corresponding to the target container included in the container group information is updated, the logs stored in the target container are obtained.
[0033] In the above embodiment, logs may also be collected when a collection instruction is received. The collection instruction may include a container where the logs to be collected are located, and the logs stored in the container where the logs to be collected are located may be obtained according to the collection instruction.
[0034] In an exemplary embodiment, the method further includes: receiving a control instruction for instructing to adjust the first namespace information list; and adjusting the first namespace information list based on the control instruction. In this embodiment, the first namespace information list is also allowed to be adjusted. When the control instruction for adjusting the first namespace information list is received, the first namespace information list can be adjusted according to the control instruction. The control instruction can be an instruction triggered when the first namespace information list is selected.
[0035] In an exemplary embodiment, adjusting the first namespace information list based on the control instruction includes at least one of the following: when the control instruction is a delete instruction, deleting the namespace information to be deleted indicated in the delete instruction from the first namespace information list; when the control instruction is a create instruction, creating the namespace information to be newly created indicated in the create instruction in the first namespace information list. In this embodiment, when the control instruction is a delete instruction, the namespace information to be deleted can be deleted from the first namespace information list. The namespace information to be deleted can be the selected namespace information, or it can be the namespace information included in the control instruction. When the control instruction is a create instruction, the namespace information to be newly created indicated in the create instruction can be created in the first namespace information list. When creating the namespace information, the namespace information to be newly created can be entered in the first namespace information list.
[0036] In an exemplary embodiment, the method further includes: receiving a query instruction input by a second account; verifying the legitimacy of the second account; and if the second account is legal, querying the first log for a second log corresponding to the second account based on the query instruction. In this embodiment, when a user queries a log, a query instruction may be input, and when a query instruction input by a second account is received, the legitimacy of the second account may be verified, and if the second account is legal, the first log may query the second log corresponding to the second account according to the query instruction. Among them, the user may call the log query interface request header to add user information, and the query instruction may include a log query syntax. If there is no authentication information, that is, if the second account is illegal, the gateway may directly return an unauthenticated error.
[0037] In an exemplary embodiment, querying the second log corresponding to the second account in the first log based on the query instruction includes: obtaining a second namespace information list of the second account; querying the second log in the first log based on the namespace information included in the second namespace information list. In this embodiment, when the tenant (i.e., the user of the second account) adds a new namespace, the container cloud platform can add the tenant id, i.e., the second account, to the label of the namespace. The gateway can obtain the tenant's second namespace list based on the tenant id. After receiving the query instruction input by the second account, the second namespace information list of the second account can be obtained, and the second log can be queried in the first log based on the namespace information included in the second namespace information list.
[0038] In an exemplary embodiment, querying the second log in the first log based on the namespace information included in the second namespace information list includes: adding the namespace information included in the second namespace information list to the query statement included in the query instruction to obtain a target query instruction; sending the target query instruction to the log query component to instruct the log query component to query the second log with the same namespace information as that included in the target log query instruction. In this embodiment, the gateway can modify the log query syntax included in the query instruction, add a namespace filter condition to the query statement, and obtain the target query instruction, and the gateway forwards the modified target query instruction to the log query component. The log query component can query the first log for a log with the same namespace information as that included in the target log query instruction, determine this log as the second log, and return the query result. Among them, the flowchart of the log query can be found in the attached. Figure 4 .
[0039] In the aforementioned embodiment, for the logs of applications deployed by different tenants, the namespace is used as the minimum division unit. The namespace permissions owned by the tenant are associated with the log management permissions. The log collector collects the logs of all containers on the node and appends the namespace information by listening to the pod information through the apiserver. The address of the log storage component pushed in the log collection component is changed to the gateway address. The plug-in in the gateway obtains the log switch configuration of all tenants of the container cloud platform, and then obtains the namespace list of the tenants who enable log collection. The tenant configuration is stored in kubernetes as a custom resource. The plug-in listens to the changes in the tenant configuration and namespace through the apiserver, and stores the namespace list for enabling log collection in the memory. If the namespace in the collected log is not in the list, it is discarded. If it is in the list, it is forwarded to the log storage component by the gateway. When the tenant calls the interface to query the log, it needs to add user information in the request, and the request is first submitted to the gateway. The gateway first obtains the namespace list owned by the tenant based on the tenant information, and then adds the namespace information to the filter condition of the tenant's log query request.
[0040] By introducing a gateway plug-in, based on the namespace permissions owned by the tenant, logs are filtered during log collection and query according to the tenant permission configuration. The log management needs of multiple tenants are met without changing the technical architecture of the original log management solution. Not only can logs be viewed according to tenant permissions to prevent sensitive information leakage, but logs can also be collected according to tenant permissions to save resources. By storing tenant configurations as custom resources in kubernetes, and using namespaces as the configuration effectiveness unit. The association between tenants and namespaces is used as the basis for log reading and writing to avoid changes in tenant information that cannot be associated with logs. Cooperating with apiserver to monitor changes in tenant configurations and namespace resources can make configurations effective in real time.
[0041] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0042] In this embodiment, a log collection device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0043] Figure 5 is a structural block diagram of a log collection device according to an embodiment of the present invention, such as Figure 5 As shown, the device comprises:
[0044] An acquisition module 52 is used to acquire logs stored in a target container included in a target cluster;
[0045] An adding module 54, configured to add namespace information to each log based on a container identifier included in a storage address of each log in the log, to obtain a first log;
[0046] The collection module 56 is used to push the first log to the target gateway to instruct the target gateway to send the target log included in the first log to the log storage component, wherein the namespace information of the target log is the namespace information included in the first namespace information list of the first account, and the first account is an account that uses the target container and whose log collection status is turned on.
[0047] In an exemplary embodiment, the acquisition module 52 can acquire the logs stored in the target container included in the target cluster in the following manner: by monitoring the container group information in the container group included in the target cluster through the apiserver, wherein the container group includes the target container; and when it is determined that the information of the target container included in the container group information is updated, acquiring the logs stored in the target container.
[0048] In an exemplary embodiment, the apparatus is configured to receive a control instruction for instructing adjustment of the first namespace information list; and adjust the first namespace information list based on the control instruction.
[0049] In an exemplary embodiment, the device can adjust the first namespace information list based on the control instruction by at least one of the following: when the control instruction is a delete instruction, deleting the namespace information to be deleted indicated in the delete instruction from the first namespace information list; when the control instruction is a create instruction, creating the namespace information to be created indicated in the create instruction in the first namespace information list.
[0050] In an exemplary embodiment, the device is also used to receive a query instruction input by a second account; verify the legitimacy of the second account; and if the second account is legitimate, query the first log for a second log corresponding to the second account based on the query instruction.
[0051] In an exemplary embodiment, the device can query the second log corresponding to the second account in the first log based on the query instruction in the following manner: obtain a second namespace information list of the second account; query the second log in the first log based on the namespace information included in the second namespace information list.
[0052] In an exemplary embodiment, the device can query the second log in the first log based on the namespace information included in the second namespace information list in the following manner: add the namespace information included in the second namespace information list to the query statement included in the query instruction to obtain a target query instruction; send the target query instruction to the log query component to instruct the log query component to query the second log with the same namespace information as that included in the target log query instruction.
[0053] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0054] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.
[0055] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0056] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0057] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0058] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.
[0059] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.< / containerid>
Claims
1. A log collection method, It is characterized in that include: Get the logs stored in the target container included in the target cluster; Based on the container identifier included in the storage address of each log in the log, add namespace information to each log to obtain a first log; Pushing the first log to a target gateway to instruct the target gateway to send a target log included in the first log to a log storage component, wherein the namespace information of the target log is the namespace information included in the first namespace information list of the first account, and the first account is an account that uses the target container and whose log collection status is turned on; Acquiring the log stored in the target container included in the target cluster includes: monitoring, through an apiserver, container group information in a container group included in the target cluster, wherein the container group includes a target container; and acquiring the log stored in the target container when it is determined that information of the target container included in the container group information is updated.
2. The method according to claim 1, It is characterized in that The method further comprises: receiving a control instruction for instructing adjustment of the first namespace information list; The first namespace information list is adjusted based on the control instruction.
3. The method according to claim 2, It is characterized in that Adjusting the first namespace information list based on the control instruction includes at least one of the following: In a case where the control instruction is a deletion instruction, deleting the namespace information to be deleted indicated in the deletion instruction from the first namespace information list; In the case where the control instruction is a new creation instruction, the namespace information to be created indicated in the new creation instruction is newly created in the first namespace information list.
4. The method according to claim 1, It is characterized in that The method further comprises: Receiving a query instruction input by a second account; Verifying the legitimacy of the second account; If the second account is valid, the first log is queried for a second log corresponding to the second account based on the query instruction.
5. The method according to claim 4, It is characterized in that Querying the first log for a second log corresponding to the second account based on the query instruction includes: Obtain a second namespace information list of the second account; The second log is queried in the first log based on the namespace information included in the second namespace information list.
6. The method according to claim 5, It is characterized in that Querying the second log in the first log based on the namespace information included in the second namespace information list includes: Adding the namespace information included in the second namespace information list to the query statement included in the query instruction to obtain a target query instruction; The target query instruction is sent to the log query component to instruct the log query component to query the second log having the same namespace information as that included in the target log query instruction.
7. A log collection device, It is characterized in that include: An acquisition module, used to acquire logs stored in a target container included in a target cluster; An adding module, configured to add namespace information to each log based on a container identifier included in a storage address of each log in the log, to obtain a first log; a collection module, configured to push the first log to a target gateway to instruct the target gateway to send a target log included in the first log to a log storage component, wherein the namespace information of the target log is the namespace information included in the first namespace information list of the first account, and the first account is an account that uses the target container and whose log collection status is turned on; The acquisition module acquires the log stored in the target container included in the target cluster in the following manner: monitoring, through an apiserver, container group information in a container group included in the target cluster, wherein the container group includes a target container; and acquiring the log stored in the target container when it is determined that information of the target container included in the container group information is updated.
8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 6 when executed by a processor.
9. An electronic device comprising a memory and a processor, It is characterized in that A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.
