Log collection method, system and device

Abstract

The embodiment of the present invention provides a log collection method, system and device, which relates to the field of cloud computing technology. The first host sends a first collection task request to the control server through a first log collector, and the request includes an application identifier of an application deployed in the first host; if there is a collection task corresponding to the application identifier in the request, the collection task corresponding to the application identifier is sent to the first log collector; the first log collector collects the log of the application to which the application identifier belongs according to the received collection task corresponding to the application identifier. In this way, the first log collector of the first host can collect logs for the application according to the collection task. If there is no collection task corresponding to the application deployed by the first host, the first host will not be scanned, thereby avoiding the problem of computer resource waste caused by the invalid scanning operation of the first log collector, and ensuring the normal operation of the application.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a log collection method, system and device. Background Art

[0002] With the rapid development of cloud computing technology, more and more applications are deployed in the cloud using container technology. After the container is created, the container management tool can automatically change the host machine to which the container belongs according to business needs. The corresponding application in the container also changes its host machine accordingly, making it impossible for users to determine the current host machine of the application, and it is also impossible to collect the running log of the application in the host machine to which the application currently belongs.

[0003] In the prior art, log collectors are installed on all host machines, and the log collectors scan the host machines to ensure that the running logs of the application are scanned in time. However, in this way, if the container corresponding to the application is not created in the host machine, the log collector in the host machine will still scan, which will cause the log collector to perform invalid scanning operations, resulting in a waste of computer resources and affecting the normal operation of the application in the container of the host machine. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a log collection method, system and device, so that a first log collector of a first host machine can perform log collection for an application corresponding to a first collection task. That is to say, if there is no collection task corresponding to an application deployed on the first host machine, the first host machine will not be scanned, thereby avoiding the problem of wasting computer resources caused by invalid scanning operations performed by the first log collector, and ensuring the normal operation of the application.

[0005] The specific technical solutions are as follows:

[0006] In a first aspect of the implementation of the present invention, a log collection method is first provided, the method is applied to a control server, and the method comprises:

[0007] receiving a first collection task request sent by a first log collector in a first host machine; the first collection task request includes an application identifier of an application program deployed in each container in the first host machine;

[0008] For each application identifier in the first collection task request, determining whether there is a collection task corresponding to the application identifier among the specified multiple collection tasks;

[0009] If there is a collection task corresponding to the application identifier, the collection task corresponding to the application identifier is sent to the first log collector, so that the first log collector collects the log of the application program to which the application identifier belongs according to the collection task corresponding to the application identifier.

[0010] Optionally, before receiving the first collection task request sent by the first log collector in the first host machine, the method further includes:

[0011] Receive container information of a created target container sent by the first host; wherein the container information includes a target application identifier of a target application deployed in the target container;

[0012] Determine whether there is a collection task corresponding to the target application identifier among the specified multiple collection tasks;

[0013] If there is a collection task corresponding to the target application identifier, determining whether a log collector is not currently installed in the first host machine;

[0014] If the log collector is not currently installed in the first host machine, an installation reminder message is sent to the first host machine, so that the first host machine installs the log collector according to the installation reminder message.

[0015] Optionally, the determining whether a log collector is not currently installed in the first host machine includes:

[0016] Determine whether a collection task request sent by the log collector in the first host machine is received within a preset historical time period;

[0017] If the collection task request sent by the log collector in the first host machine is not received, it is determined that the log collector is not currently installed in the first host machine.

[0018] Optionally, a collection task corresponding to an application identifier includes: the application identifier, a storage path of a log generated by an application corresponding to the application identifier in a corresponding host machine, and a processing strategy for the log of the application corresponding to the application identifier.

[0019] Optionally, the method further includes:

[0020] Receiving a second collection task request sent by a second log collector in a second host machine;

[0021] The second collection task request includes the application identifier of the application deployed in each container in the second host; the second host and the first host have the same target application deployed; the target application deployed in the second host is responsible for different business processing from the target application deployed in the first host, and the target application deployed in the second host is obtained by dynamically expanding the capacity based on the target application deployed in the first host;

[0022] For each application identifier in the second collection task request, determining whether there is a collection task corresponding to the application identifier among the specified multiple collection tasks;

[0023] If there is a collection task corresponding to the application identifier, the collection task corresponding to the application identifier is sent to the first log collector, so that the second log collector collects the log of the application program to which the application identifier belongs according to the collection task corresponding to the application identifier.

[0024] In a second aspect of the present invention, a log collection method is provided. The method is applied to a first host machine and includes:

[0025] Sending a first collection task request to a control server through a first log collector, wherein the first collection task request includes an application identifier of an application program deployed in each container in the first host machine; so that the control server receives the first collection task request, and for each application identifier in the first collection task request, determines whether there is a collection task corresponding to the application identifier among the specified multiple collection tasks, and if there is a collection task corresponding to the application identifier, sends the collection task corresponding to the application identifier to the first log collector;

[0026] Through the first log collector, for each application identifier, according to the received collection task corresponding to the application identifier, the log of the application program to which the application identifier belongs is collected.

[0027] Optionally, the collection task corresponding to each application identifier includes: the application identifier and a storage path of a log generated by an application corresponding to the application identifier in a corresponding host machine;

[0028] The collecting of the logs of the application program to which the application identifier belongs by the first log collector for each application identifier according to the received collection task corresponding to the application identifier includes:

[0029] Through the first log collector, for each application identifier, the log generated by the application program corresponding to the application identifier is obtained from the storage path corresponding to the application identifier.

[0030] Optionally, the collection task corresponding to each application identifier further includes: a processing strategy for the log of the application program corresponding to the application identifier;

[0031] After obtaining, by the first log collector, for each application identifier, the log generated by the application program corresponding to the application identifier from the storage path corresponding to the application identifier, the method further includes:

[0032] Through the first log collector, according to the processing strategy, information of preset fields in the acquired log is extracted, a target log is generated, and the target log is saved to a preset storage location.

[0033] In a third aspect of the present invention, a log collection system is provided. The log collection system includes a control server and a first host machine, wherein:

[0034] The first host machine is used to send a first collection task request to the control server through a first log collector, where the first collection task request includes an application identifier of an application program deployed in each container in the first host machine;

[0035] The control server is configured to receive the first collection task request; for each application identifier in the first collection task request, determine whether there is a collection task corresponding to the application identifier among the specified multiple collection tasks; if there is a collection task corresponding to the application identifier, send the collection task corresponding to the application identifier to the first log collector;

[0036] The first host machine is further configured to collect, through the first log collector, for each application identifier and according to a received collection task corresponding to the application identifier, logs of the application program to which the application identifier belongs.

[0037] In a fourth aspect of the implementation of the present invention, a log collection device is further provided, the device is applied to a control server, and the device includes:

[0038] A receiving module, configured to receive a first collection task request sent by a first log collector in a first host machine; the first collection task request includes an application identifier of an application program deployed in each container in the first host machine;

[0039] A determination module, configured to determine, for each application identifier in the first acquisition task request, whether there is an acquisition task corresponding to the application identifier in the specified multiple acquisition tasks;

[0040] The first sending module is used to send the collection task corresponding to the application identifier to the first log collector if there is a collection task corresponding to the application identifier, so that the first log collector collects the log of the application program to which the application identifier belongs according to the collection task corresponding to the application identifier.

[0041] In a fifth aspect of the implementation of the present invention, a log collection device is further provided. The device is applied to a first host machine, and the device includes:

[0042] A second sending module is used to send a first collection task request to the control server through the first log collector, wherein the first collection task request includes an application identifier of an application program deployed in each container in the first host machine; so that the control server receives the first collection task request, and for each application identifier in the first collection task request, determines whether there is a collection task corresponding to the application identifier in the specified multiple collection tasks, and if there is a collection task corresponding to the application identifier, sends the collection task corresponding to the application identifier to the first log collector;

[0043] The collection module is used to collect the log of the application program to which the application identifier belongs through the first log collector for each application identifier according to the received collection task corresponding to the application identifier.

[0044] Another aspect of the embodiments of the present invention further provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0045] Memory, used to store computer programs;

[0046] The processor is used to implement any of the above-mentioned log collection methods when executing the program stored in the memory.

[0047] In another aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned log collection methods is implemented.

[0048] In another aspect of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute any of the log collection methods described above.

[0049] A log collection method provided by an embodiment of the present invention receives a first collection task request sent by a first log collector in a first host machine; the first collection task request includes an application identifier of an application deployed in each container in the first host machine; for each application identifier in the first collection task request, it is determined whether there is a collection task corresponding to the application identifier in multiple specified collection tasks; if there is a collection task corresponding to the application identifier, the collection task corresponding to the application identifier is sent to the first log collector, so that the first log collector collects the log of the application to which the application identifier belongs according to the collection task corresponding to the application identifier.

[0050] The first log collector in the first host can send a first collection task request to the controller to obtain the collection tasks corresponding to the applications deployed by the first host, and collect logs for the applications corresponding to each collection task according to the collection tasks. This enables the first log collector of the first host to collect logs for the applications corresponding to the collection tasks, that is, if there is no collection task corresponding to the applications deployed by the first host, the first host will not be scanned, thus avoiding the problem of wasting computer resources caused by invalid scanning operations by the log collector, and ensuring the normal operation of the application. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0052] Figure 1 It is a flow chart of a log collection method in the prior art;

[0053] Figure 2 A flow chart of a log collection method provided in an embodiment of the present invention;

[0054] Figure 3 A flowchart of another log collection method provided in an embodiment of the present invention;

[0055] Figure 4 A flow chart of a log collection method provided in an embodiment of the present invention;

[0056] Figure 5 A structural diagram of a log collection system provided in an embodiment of the present invention;

[0057] Figure 6 A flow chart of a log collection method provided in an embodiment of the present invention;

[0058] Figure 7 A schematic diagram of a log collection process of a video website provided in an embodiment of the present invention;

[0059] Figure 8 A system block diagram of a log collection system provided in an embodiment of the present invention;

[0060] Fig. 9 A structural diagram of a log collection device provided in an embodiment of the present invention;

[0061] Fig.10 A structural diagram of a log collection device provided in an embodiment of the present invention;

[0062] Fig.11 The present invention is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0064] See also Figure 1 , Figure 1 The figure is a flow chart of a log collection method in the prior art.

[0065] In the prior art, log collectors are manually installed in each host machine, and collection tasks are edited and applications are deployed to each host machine. Regardless of whether a container corresponding to the application is created in the host machine, the log collector installed in each host machine will scan the host machine. If a container corresponding to the application is not created in the host machine, the log collector will perform an invalid scanning operation, resulting in a waste of computer resources and affecting the normal operation of the application in the container of the host machine.

[0066] In order to solve the above problem, an embodiment of the present invention provides a log collection method applied to a control server.

[0067] Example 1, see Figure 2 , Figure 2 A flow chart of a log collection method provided by an embodiment of the present invention, the method may include the following steps:

[0068] S201: Receive a first collection task request sent by a first log collector in a first host machine.

[0069] The first collection task request includes an application identifier of an application program deployed in each container in the first host machine.

[0070] S202: For each application identifier in the first collection task request, determine whether there is a collection task corresponding to the application identifier in the specified multiple collection tasks.

[0071] S203: If there is a collection task corresponding to the application identifier, send the collection task corresponding to the application identifier to the first log collector, so that the first log collector collects the logs of the application program to which the application identifier belongs according to the collection task corresponding to the application identifier.

[0072] Based on the log collection method provided by the embodiment of the present invention, the first log collector in the first host can send a first collection task request to the controller to obtain the collection task corresponding to the application deployed by the first host, and collect logs for the application corresponding to each collection task according to the collection task. It is realized that the first log collector of the first host can collect logs for the application corresponding to the collection task, that is, if there is no collection task corresponding to the application deployed by the first host, the first host will not be scanned, avoiding the problem of waste of computer resources caused by invalid scanning operations of the log collector, and ensuring the normal operation of the application.

[0073] In the field of cloud computing technology, a host is a host that provides hardware resources, in which users can create containers and deploy applications in the containers. Applications can be website programs, game programs, etc., which are not limited here. Accordingly, the application will generate logs during operation, and the logs record key information, warning information, error information, etc. when the application is running. The logs of the container are the logs generated by the application deployed therein. For example, a container management tool (for example, Kubernetes) can be used to create a container in the first host and deploy an application in the container.

[0074] The log collector is a software application that can search for files in a specified path in the first host by running the software application in the first host. The application identifier of the application can be an application name specified by the application, or can also be an application number specified by the application.

[0075] In one implementation, for each application, the user can determine whether to collect the logs of the application according to business needs. If the logs of the application need to be collected, the user can record the collection task corresponding to the application in the control server, and the control server can obtain multiple specified collection tasks. The collection task records the application ID of the application, the storage path of the logs generated by the application in the host machine to which it belongs, and the processing strategy for the logs of the application.

[0076] In the subsequent steps, when the control server receives the first collection task request sent by the first log collector in the first host machine, the first collection task request includes the application identifier (first application identifier) ​​of the application (first application), and it can be determined whether there is a collection task (first collection task) recorded with the first application identifier among the collection tasks recorded locally. If there is a first collection task, it means that the logs of the first application need to be collected.

[0077] In the cloud computing architecture, the control server can communicate with multiple host machines, and the first host machine is any one of the multiple host machines. That is to say, based on the method provided in the embodiment of the present application, the control server can communicate with any host machine to determine whether the application deployed in the host machine needs to collect logs.

[0078] Example 2, see Figure 3 ,exist Figure 2 On the basis of, before step S201, the method may further include:

[0079] S204: Receive container information of the created target container sent by the first host.

[0080] S205: Determine whether there is a collection task corresponding to the target application identifier among the specified multiple collection tasks.

[0081] S206: If there is a collection task corresponding to the target application identifier, it is determined whether a log collector is not currently installed in the first host machine.

[0082] S207: If the log collector is not currently installed in the first host machine, an installation reminder message is sent to the first host machine, so that the first host machine installs the log collector according to the installation reminder message.

[0083] The container information includes a target application identifier of a target application deployed in the target container.

[0084] In one implementation, when the first host creates a new container (which may be referred to as a target container) and deploys a target application in the target container, the first host may send container information of the target container to the control server. The control server may determine whether there is a collection task (target collection task) that records the target application identifier among the various collection tasks recorded locally. If there is a target collection task, then it may be determined that the target collection task is the collection task corresponding to the target application identifier. In other words, the user needs to collect logs of the target application.

[0085] In one implementation, if there is a collection task corresponding to the target application identifier, and the log collector is not currently installed in the first host, the control server may send an installation reminder message to the first host. The installation reminder message may include: a software application package and installation instructions of the log collector, and the first host installs the software application of the log collector according to the software application package and installation instructions of the log collector in the installation reminder message. After the installation, the log collector can collect logs of the target application in the first host.

[0086] In another implementation, if there is no collection task corresponding to the target application identifier, it indicates that the user does not need to collect logs of the target application, and thus logs of the target application do not need to be collected, and there is no need to send an installation reminder message to the first host.

[0087] In another implementation, if there is a collection task corresponding to the target application identifier and the log collector is currently installed in the first host machine, there is no need to send an installation reminder message to the first host machine.

[0088] In the third embodiment, determining whether the log collector is not currently installed in the first host machine includes:

[0089] Step 1: Determine whether a collection task request sent by the log collector in the first host machine is received within a preset historical time period.

[0090] Step 2: If the collection task request sent by the log collector in the first host machine is not received, it is determined that the log collector is not currently installed in the first host machine.

[0091] In one implementation, the log collector can send a collection task request to the control server according to a preset period. The preset period can be set by a technician according to business needs. For example, the preset period can be 10 seconds, or the preset period can be 30 seconds, which is not limited here. The preset historical time period can be the previous preset period of the current moment.

[0092] In one implementation, the collection task request sent by the log collector may include the network address of the first host machine to which the log collector belongs.

[0093] In one implementation, if within a preset historical time period, a collection task request containing the network address of the first host is received, the control server can determine that the collection task request is sent by the log collector in the first host, and then determine that the log collector is currently installed in the first host. The duration of the above preset historical time period is greater than the duration of one cycle of the above preset period.

[0094] In another implementation, if no collection task request including the network address of the first host machine is received within a preset historical time period, the control server can determine that no collection task request sent by the log collector in the first host machine is received, and then determine that the log collector is not currently installed in the first host machine.

[0095] In the fourth embodiment, a collection task corresponding to an application identifier includes: the application identifier, a storage path of a log generated by an application corresponding to the application identifier in a corresponding host machine, and a processing strategy for the log of the application corresponding to the application identifier.

[0096] The control server can determine which application the collection task is for through the application identifier in the collection task. The log collector can find the logs generated by the application according to the save path corresponding to the application identifier. The log collector can process the logs according to the processing strategy for the logs of the application corresponding to the application identifier.

[0097] In embodiment 5, the method further comprises:

[0098] Step 1: receiving a second collection task request sent by a second log collector in a second host machine.

[0099] Among them, the second collection task request includes the application identifier of the application deployed in each container in the second host machine; the same target application is deployed in the second host machine and the first host machine; the target application deployed in the second host machine is responsible for different business processing from the target application deployed in the first host machine, and the target application deployed in the second host machine is obtained by dynamic expansion based on the target application deployed in the first host machine.

[0100] Step 2: for each application identifier in the second collection task request, determine whether there is a collection task corresponding to the application identifier in the specified multiple collection tasks.

[0101] Step three: if there is a collection task corresponding to the application identifier, the collection task corresponding to the application identifier is sent to the first log collector, so that the second log collector collects the log of the application program to which the application identifier belongs according to the collection task corresponding to the application identifier.

[0102] When the business volume of the target application increases and the target application deployed in the first host cannot meet the current business needs, the container management tool can dynamically expand the target application and deploy the target application in the second host. The target applications deployed in the first host and the second host are responsible for business processing together and are responsible for different businesses respectively.

[0103] In one implementation, when the number of users using the target application increases, the business volume of the target application increases, causing the target application in the first host to be unable to respond to business requests of all users. Then, the container management tool can dynamically expand the target application and deploy the target application in the second host. The target application in the first host and the target application in the second host are responsible for the business of different users respectively.

[0104] In another implementation, when the target application adds a new service, the service volume of the target application increases, causing the first host to be unable to run all services of the target application. The container management tool can dynamically expand the capacity of the target application and deploy the target application in the second host. The target application in the first host and the target application in the second host are both responsible for the services of all users, and the types of services they are responsible for are different.

[0105] Based on the same inventive concept, the present invention provides a log collection method, which is applied to a first host machine.

[0106] Example 6, see Figure 4 , Figure 4 A flow chart of a log collection method provided by an embodiment of the present invention, the method may include the following steps:

[0107] S401: Sending a first collection task request to a control server through a first log collector, where the first collection task request includes an application identifier of an application program deployed in each container in a first host machine.

[0108] The control server receives the first collection task request, and for each application identifier in the first collection task request, determines whether there is a collection task corresponding to the application identifier in the specified multiple collection tasks. If there is a collection task corresponding to the application identifier, the collection task corresponding to the application identifier is sent to the first log collector.

[0109] S402: For each application identifier, the first log collector collects the logs of the application program to which the application identifier belongs according to the received collection task corresponding to the application identifier.

[0110] Based on the log collection method provided by the embodiment of the present invention, the first log collector in the first host can send a first collection task request to the controller to obtain the collection task corresponding to the application deployed by the first host, and collect logs for the application corresponding to each collection task according to the collection task. It is realized that the first log collector of the first host can collect logs for the application corresponding to the collection task, that is, if there is no collection task corresponding to the application deployed by the first host, the first host will not be scanned, avoiding the problem of waste of computer resources caused by invalid scanning operations of the log collector, and ensuring the normal operation of the application.

[0111] In one implementation, the first log collector may send a first collection task request to the control server according to a preset period. The preset period may be set by a technician according to business requirements. For example, the preset period may be 10 seconds, or the preset period may be 30 seconds, which is not limited here.

[0112] For an introduction to the control server side, reference may be made to the introduction to the above steps S201 - S203 .

[0113] In the seventh embodiment, the collection task corresponding to each application identifier includes: the application identifier, and a storage path of a log generated by an application corresponding to the application identifier in a corresponding host machine;

[0114] Step S402 specifically includes: obtaining, by the first log collector, for each application identifier, the log generated by the application program corresponding to the application identifier from the storage path corresponding to the application identifier.

[0115] In the eighth embodiment, the collection task corresponding to each application identifier further includes: a processing strategy for the log of the application program corresponding to the application identifier;

[0116] After obtaining, by the first log collector, for each application identifier, the log generated by the application program corresponding to the application identifier from the storage path corresponding to the application identifier, the method further includes:

[0117] Through the first log collector, according to the processing strategy, the information of the preset fields in the acquired log is extracted, the target log is generated, and the target log is saved to a preset storage location.

[0118] The first log collector can extract information of preset fields from the information contained in the log of the application according to the processing strategy of the log of the application and generate a target log. In this way, the technician can maintain the application according to the target log. For example, the preset field can be a field containing error prompt information. The first log collector can extract the field containing the error prompt information in the log to generate a target log, and the technician can find the program vulnerability of the application according to the error prompt information.

[0119] In one implementation, after obtaining the log (original log) generated by the application, the first log collector can read the text information of the original log, extract the information of the preset field (target field information) in the text information according to the preset parsing method, and generate the target log according to the target field information. The preset parsing method can be a regular parsing method, a JSON (JavaScript Object Notation) parsing method, a delimiter parsing method, etc., which is not limited here.

[0120] Based on the same inventive concept, the present invention also provides a log collection system.

[0121] Example 9, see Figure 5 , Figure 5 A structural diagram of a log collection system provided in an embodiment of the present invention includes: multiple host machines (including host machine 5011, host machine 5012, host machine 5013), and control server 502. Embodiment 9 of the present invention is described by taking three host machines as an example, but is not limited thereto. The first host machine can be any one of the host machines 5011, host machine 5012, and host machine 5013.

[0122] The first host machine is used to send a first collection task request to the control server 502 through the first log collector, where the first collection task request includes an application identifier of an application program deployed in each container in the first host machine;

[0123] The control server 502 is used to receive a first collection task request; for each application identifier in the first collection task request, determine whether there is a collection task corresponding to the application identifier in the specified multiple collection tasks; if there is a collection task corresponding to the application identifier, send the collection task corresponding to the application identifier to the first log collector;

[0124] The first host machine is further configured to collect the log of the application program to which the application identifier belongs through the log collector for each application identifier according to the received collection task corresponding to the application identifier.

[0125] That is to say, Figure 5In the embodiment, any host machine can realize log collection for the application corresponding to the collection task through the first log collector based on the method of the embodiment of the present application.

[0126] Example 10, see Figure 6 , Figure 6 A flow chart of a log collection method provided by an embodiment of the present invention.

[0127] Users can edit the collection tasks corresponding to each application according to business needs, and record the collection tasks in the control center (i.e., the control server in this application). The collection tasks include the application identifier of the application and the save path of the log of the application. When the application is manually deployed to the host machine, the host machine can send the container information of the container created when the application is deployed to the control center. The container information includes the application identifier (second application identifier) ​​of the application (second application) deployed in the container. When the control server receives the second application identifier of the second application, it can be determined whether there is a collection task (second collection task) recorded with the second application identifier among the collection tasks recorded locally. If there is a second collection task, it is determined that the log of the second application needs to be collected, then, a log collector is installed in the host machine. After the log collector is successfully installed, the log collector can obtain the second collection task from the control server and collect the running log of the second application according to the second collection task.

[0128] Example 11, see Figure 7 , Figure 7 A schematic diagram of a log collection process for a website provided in an embodiment of the present invention.

[0129] The following takes the ABC website as an example to introduce the website's log collection process.

[0130] Step 1: The user edits the collection task of the ABC website and stores it in the control center (ie, the control server in this application). The collection task can record: application name: ABC website, file path: log / info.log (the storage path of the log generated when the ABC website is running).

[0131] Step 2: Users can use the container management tool to deploy the ABC website in the cluster (preset multiple hosts) managed by the container management tool. For example, deploy the ABC website in container A created by host A.

[0132] Step 3: When container A is created successfully, host A can send the name of ABC website: ABC website, and the address of host A to the control center.

[0133] Step 4: The control center searches the stored collection tasks for a collection task containing the application name "ABC website". If there is a collection task containing the application name "ABC website", the control center can send the installation package and installation instructions of the log collector to host A according to the address of host A, so that host A can install the log collector (log collector A). If there is no collection task containing the application name "ABC website", the control center does not send the installation package and installation instructions of the log collector to host A.

[0134] Step 5: After the log collector A is successfully installed, it obtains the collection task corresponding to the ABC website from the control center. Log collector A collects logs according to the collection task corresponding to the ABC website.

[0135] Step 6: If the business volume of website ABC increases, the container management tool can dynamically expand the capacity of website ABC, that is, deploy part of the business of website ABC in container B created by host B, and website ABC is deployed in container A of host A and container B of host B at the same time. The corresponding host B and the control center can perform the same operations as steps 1 to 5 above.

[0136] Example 12, see Figure 8 , Figure 8 A system block diagram of a log collection system provided by an embodiment of the present invention.

[0137] The log collection system provided by the embodiment of the present invention includes a user interface, a control center (ie, a control server in the present application) and various log collectors.

[0138] The functions of the user interface are: the user can edit and save the collection tasks of the application to be collected through the user interface, and the control center can save the collection tasks.

[0139] The functions of the control center are as follows: when the host creates a container, the control center can receive the information of the application in the container and the address of the host sent by the host. The control center determines whether the host needs to install the log collector based on the information of the application. If the host needs to install the log collector, the control center sends the installation package and installation instructions of the log collector to the host according to the address of the host, so that the host can install the log collector. The control center can send the collection task corresponding to the application deployed by the host to the log collector according to the collection task request sent by the log collector.

[0140] Function of the log collector: obtain the collection tasks corresponding to the applications deployed on the host from the control center, and collect the logs of the corresponding applications according to each collection task.

[0141] Based on the same inventive concept, an embodiment of the present invention further provides a log collection device, which is applied to a control server.

[0142] Example 13, see Fig. 9 , Fig. 9 A structural diagram of a log collection device provided by an embodiment of the present invention, the device includes:

[0143] A receiving module 901 is configured to receive a first collection task request sent by a first log collector in a first host machine; the first collection task request includes an application identifier of an application program deployed in each container in the first host machine;

[0144] A determination module 902 is used to determine, for each application identifier received in the first acquisition task request, whether there is a acquisition task corresponding to the application identifier in the specified multiple acquisition tasks;

[0145] The first sending module 903 is used to send the collection task corresponding to the application identifier to the first log collector if there is a collection task corresponding to the application identifier, so that the first log collector collects the log of the application program to which the application identifier belongs according to the collection task corresponding to the application identifier.

[0146] In the fourteenth embodiment, the device further comprises:

[0147] A second receiving module is used to receive container information of a created target container sent by the first host before receiving the first collection task request sent by the first log collector in the first host; wherein the container information includes an application identifier of a target application deployed in the target container;

[0148] The second judgment module is used to judge whether there is a collection task corresponding to the target application identifier among the specified multiple collection tasks;

[0149] A third judgment module is used to judge whether the log collector is not currently installed in the first host machine if there is a collection task corresponding to the target application identifier;

[0150] The first sending module 903 is further configured to send an installation reminder message to the first host machine if the log collector is not currently installed in the first host machine, so that the first host machine installs the log collector according to the installation reminder message.

[0151] In embodiment 15, the third judgment module is specifically used to judge whether a collection task request sent by the log collector in the first host machine is received within a preset historical time period; if the collection task request sent by the log collector in the first host machine is not received, it is determined that the log collector is not currently installed in the first host machine.

[0152] In the sixteenth embodiment, a collection task corresponding to an application identifier includes: the application identifier, a storage path of a log generated by an application corresponding to the application identifier in a corresponding host machine, and a processing strategy for the log of the application corresponding to the application identifier.

[0153] Based on the same inventive concept, an embodiment of the present invention further provides a log collection device, which is applied to a first host machine.

[0154] Example 17, see Fig.10 , Fig.10 A structural diagram of a log collection device provided by an embodiment of the present invention, the device includes:

[0155] The second sending module 1001 is used to send a first collection task request to the control server through the first log collector, where the first collection task request includes an application identifier of an application program deployed in each container in the first host machine; so that the control server receives the first collection task request, and for each application identifier in the first collection task request, determines whether there is a collection task corresponding to the application identifier in the specified multiple collection tasks, and if there is a collection task corresponding to the application identifier, sends the collection task corresponding to the application identifier to the first log collector;

[0156] The collection module 1002 is used to collect the logs of the application program to which the application identifier belongs through the first log collector for each application identifier according to the received collection task corresponding to the application identifier.

[0157] In the eighteenth embodiment, the collection task corresponding to each application identifier includes: the application identifier, and a storage path of a log generated by an application corresponding to the application identifier in a corresponding host machine;

[0158] The collection module 1002 is specifically configured to obtain, through a first log collector, for each application identifier, the log generated by the application program corresponding to the application identifier from the storage path corresponding to the application identifier.

[0159] In the nineteenth embodiment, the collection task corresponding to each application identifier further includes: a processing strategy for the log of the application program corresponding to the application identifier;

[0160] The device also includes:

[0161] The target log generation module is used to obtain the log generated by the application corresponding to the application identifier from the saving path corresponding to the application identifier through the first log collector for each application identifier, and then, through the first log collector, according to the log processing strategy, extract the information of the preset fields in the obtained log, generate the target log, and save the target log to a preset storage location.

[0162] The embodiment of the present invention further provides an electronic device, such as Fig.11 As shown, it includes a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104.

[0163] Memory 1103, used for storing computer programs;

[0164] The processor 1101 is used to implement the log collection method described in any of the above embodiments when executing the program stored in the memory 1103.

[0165] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0166] The communication interface is used for communication between the above electronic device and other devices.

[0167] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0168] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0169] In another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the log collection method described in any one of the above embodiments is implemented.

[0170] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is executed on a computer, the computer executes the log collection method described in any one of the above embodiments.

[0171] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0172] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0173] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0174] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A log collection method, characterized in that: The method is applied to a control server, and the method comprises: receiving a first collection task request sent by a first log collector in a first host machine; the first collection task request includes an application identifier of an application program deployed in each container in the first host machine; For each application identifier in the first collection task request, determining whether there is a collection task corresponding to the application identifier among the specified multiple collection tasks; If there is a collection task corresponding to the application identifier, the collection task corresponding to the application identifier is sent to the first log collector, so that the first log collector collects the log of the application program to which the application identifier belongs according to the collection task corresponding to the application identifier.

2. The method according to claim 1, characterized in that Before receiving the first collection task request sent by the first log collector in the first host machine, the method further includes: Receive container information of a created target container sent by the first host; wherein the container information includes a target application identifier of a target application deployed in the target container; Determine whether there is a collection task corresponding to the target application identifier among the specified multiple collection tasks; If there is a collection task corresponding to the target application identifier, determining whether a log collector is not currently installed in the first host machine; If the log collector is not currently installed in the first host machine, an installation reminder message is sent to the first host machine, so that the first host machine installs the log collector according to the installation reminder message.

3. The method according to claim 2, characterized in that The determining whether the log collector is not currently installed in the first host machine includes: Determine whether a collection task request sent by the log collector in the first host machine is received within a preset historical time period; If the collection task request sent by the log collector in the first host machine is not received, it is determined that the log collector is not currently installed in the first host machine.

4. The method according to claim 1, characterized in that: The collection task corresponding to an application identifier includes: the application identifier, the storage path of the log generated by the application corresponding to the application identifier in the host machine to which it belongs, and the processing strategy for the log of the application corresponding to the application identifier.

5. The method according to claim 1, characterized in that The method further comprises: Receiving a second collection task request sent by a second log collector in a second host machine; The second collection task request includes the application identifier of the application deployed in each container in the second host; the second host and the first host have the same target application deployed; the target application deployed in the second host is responsible for different business processing from the target application deployed in the first host, and the target application deployed in the second host is obtained by dynamically expanding the capacity based on the target application deployed in the first host; For each application identifier in the second collection task request, determining whether there is a collection task corresponding to the application identifier among the specified multiple collection tasks; If there is a collection task corresponding to the application identifier, the collection task corresponding to the application identifier is sent to the first log collector, so that the second log collector collects the log of the application program to which the application identifier belongs according to the collection task corresponding to the application identifier.

6. A log collection method, characterized in that: The method is applied to a first host machine, and the method includes: Sending a first collection task request to a control server through a first log collector, wherein the first collection task request includes an application identifier of an application program deployed in each container in the first host machine; so that the control server receives the first collection task request, and for each application identifier in the first collection task request, determines whether there is a collection task corresponding to the application identifier among the specified multiple collection tasks, and if there is a collection task corresponding to the application identifier, sends the collection task corresponding to the application identifier to the first log collector; Through the first log collector, for each application identifier, according to the received collection task corresponding to the application identifier, the log of the application program to which the application identifier belongs is collected.

7. The method according to claim 6, characterized in that The collection task corresponding to each application identifier includes: the application identifier, and the storage path of the log generated by the application corresponding to the application identifier in the host machine to which it belongs; The collecting of the logs of the application program to which the application identifier belongs by the first log collector for each application identifier according to the received collection task corresponding to the application identifier includes: Through the first log collector, for each application identifier, the log generated by the application program corresponding to the application identifier is obtained from the storage path corresponding to the application identifier.

8. The method according to claim 7, characterized in that The collection task corresponding to each application identifier further includes: a processing strategy for the log of the application program corresponding to the application identifier; After obtaining, by the first log collector, for each application identifier, the log generated by the application program corresponding to the application identifier from the storage path corresponding to the application identifier, the method further includes: Through the first log collector, according to the processing strategy, information of preset fields in the acquired log is extracted, a target log is generated, and the target log is saved to a preset storage location.

9. A log collection system, characterized in that: The log collection system includes a control server and a first host machine, wherein: The first host machine is used to send a first collection task request to the control server through a first log collector, where the first collection task request includes an application identifier of an application program deployed in each container in the first host machine; The control server is configured to receive the first collection task request; for each application identifier in the first collection task request, determine whether there is a collection task corresponding to the application identifier among the specified multiple collection tasks; if there is a collection task corresponding to the application identifier, send the collection task corresponding to the application identifier to the first log collector; The first host machine is further configured to collect, through the first log collector, for each application identifier and according to a received collection task corresponding to the application identifier, logs of the application program to which the application identifier belongs.

10. A log collection device, characterized in that: The device is applied to a control server, and the device comprises: A receiving module, configured to receive a first collection task request sent by a first log collector in a first host machine; the first collection task request includes an application identifier of an application program deployed in each container in the first host machine; A determination module, configured to determine, for each application identifier in the first acquisition task request, whether there is an acquisition task corresponding to the application identifier in the specified multiple acquisition tasks; The first sending module is used to send the collection task corresponding to the application identifier to the first log collector if there is a collection task corresponding to the application identifier, so that the first log collector collects the log of the application program to which the application identifier belongs according to the collection task corresponding to the application identifier.

11. A log collection device, characterized in that: The device is applied to a first host machine, and the device includes: A second sending module is used to send a first collection task request to the control server through the first log collector, wherein the first collection task request includes an application identifier of an application program deployed in each container in the first host machine; so that the control server receives the first collection task request, and for each application identifier in the first collection task request, determines whether there is a collection task corresponding to the application identifier in the specified multiple collection tasks, and if there is a collection task corresponding to the application identifier, sends the collection task corresponding to the application identifier to the first log collector; The collection module is used to collect the log of the application program to which the application identifier belongs through the first log collector for each application identifier according to the received collection task corresponding to the application identifier.

12. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-5, or 6-8, when executing a program stored in a memory.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1-5, or 6-8 are implemented.

Images