Log collection method and device, program product, cluster and storage medium

Abstract

The embodiment of the invention provides a log collection method and device, a program product, a cluster and a storage medium, relates to the technical field of computers, and can improve the utilization rate of storage resources of management equipment. The method is applied to a management module in a management system, the management module is used for managing at least one virtual machine, and the method comprises the following steps: acquiring first information sent by a first virtual machine in the at least one virtual machine; the first information is used for describing a target log record required by a service in a log generated by the first virtual machine; determining an acquisition interval corresponding to the processing level of the first information; wherein the collection intervals are time intervals and / or position intervals where the log records to be collected are located, and the collection intervals corresponding to different processing levels are different in size; collecting a target log fragment in the collection interval from a log generated by a target virtual machine; the target virtual machine at least comprises a first virtual machine, and the target log fragment at least comprises a target log record.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a log collection method, device, program product, cluster and storage medium. Background Art

[0002] Currently, in order to collect the logs required for the business (such as exception logs) from the logs generated by the virtual machine, the virtual machine needs to transmit and store all the logs generated by itself to a management device (or a management module in a computing device); then, the logs required for the business are manually retrieved and collected from all the logs stored in the management device.

[0003] However, in the above method, the logs required for the business account for a small proportion of the logs stored in the management device, resulting in a large number of invalid log records (i.e., logs not required for the business) in the logs stored in the management device, reducing the utilization rate of the storage resources of the management device, and at the same time reducing the efficiency of collecting the logs required for the business from the large number of logs stored in the management device. Summary of the invention

[0004] The present application provides a log collection method, device, program product, cluster and storage medium, which can improve the utilization rate of storage resources of management equipment.

[0005] In order to achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a log collection method, which is applied to a management module in a management system, and the management module is used to manage at least one virtual machine. The method includes: obtaining first information sent by a first virtual machine in at least one virtual machine; the first information is used to describe a target log record, and the target log record is a log record required for the business in the log generated by the above-mentioned first virtual machine; determining a collection interval corresponding to a processing level of the first information; wherein the collection interval is a time interval and / or a location interval where the log records to be collected are located, and the sizes of the collection intervals corresponding to different processing levels are different; from the log generated by the target virtual machine, collecting the target log fragment within the above-mentioned collection interval; the target virtual machine includes at least the above-mentioned first virtual machine, and the target log fragment includes at least the target log record.

[0007] An embodiment of the present application provides a log collection method, which is applied to a management module in a management system, and the management module is used to manage at least one virtual machine; the method determines the time interval and / or location interval (referred to as: collection interval) where the log records to be collected are located according to the processing level of the first information received from the first virtual machine and used to describe the target log records required for the business, and then collects the target log fragments within the collection interval from the target virtual machines including at least the first virtual machine, and the target log fragments at least include the target log records; since the sizes of the collection intervals corresponding to different processing levels are different, the above method will not collect all the log records generated by the first virtual machine into the management module for storage, thereby avoiding the waste of storage resources corresponding to the management module, and thus improving the utilization rate of the storage resources corresponding to the management module.

[0008] In addition, the above method collects target log fragments of different sizes based on different processing levels, and then the user retrieves relevant log records associated with the target log records from the target log fragments of different sizes; compared with the method of retrieving the relevant log records from the full logs generated by all virtual machines, the efficiency of retrieving the relevant log records is improved.

[0009] In a possible implementation, the processing level is proportional to the size of the acquisition interval.

[0010] In a possible implementation, the processing level includes: the first level or the second level; when the first level is lower than the second level, the collection interval corresponding to the first level includes: the target log record and the log record generated within x time units before the target log record is generated; wherein x is an integer greater than or equal to 0; the collection interval corresponding to the second level includes: the target log record and the log record generated within m time units before the target log record is generated; wherein m is a positive integer greater than x.

[0011] In a possible implementation, the first information includes: an identifier of a target log record, and the determining of the collection interval corresponding to the processing level of the first information includes: determining the collection interval according to the processing level of the first information and the identifier of the target log record.

[0012] In a possible implementation, the above-mentioned determining the collection interval based on the processing level of the first information and the identifier of the target log record includes: determining the collection interval from a first corresponding relationship based on the processing level of the above-mentioned first information and the identifier of the target log record; wherein the first corresponding relationship includes: a corresponding relationship between the processing level and the collection interval.

[0013] In one possible implementation, when the processing level of the above-mentioned first information is higher than the preset level, the target virtual machine includes the first virtual machine, and a virtual machine that has a calling relationship with the first virtual machine; the virtual machine that has a calling relationship with the first virtual machine includes the upstream virtual machine of the above-mentioned first virtual machine, or / and, the downstream virtual machine of the first virtual machine.

[0014] In the above embodiment, when the processing level of the first information is higher than the preset level, target log fragments are respectively collected from the logs generated by the first virtual machine and the upstream virtual machine of the first virtual machine, or / and the downstream virtual machine of the first virtual machine; because the upstream virtual machine and the downstream virtual machine both have a calling relationship with the above-mentioned first virtual machine, the collected target log fragments are more comprehensive, thereby improving the accuracy of the target log fragments.

[0015] In a possible implementation, the method further includes: obtaining topology information; the topology information is used to describe the calling relationship between multiple virtual machines; based on the topology information, determining the upstream virtual machine of the first virtual machine, and / or the downstream virtual machine of the first virtual machine; the upstream virtual machine is a virtual machine among the multiple virtual machines that calls the first virtual machine; the downstream virtual machine is a virtual machine among the multiple virtual machines that is called by the first virtual machine.

[0016] In one possible implementation, the target log record is a log record that satisfies a preset condition, and the preset condition is used to indicate a log record required for the business; the first information includes an identifier of a target sub-condition, and the target sub-condition is a sub-condition satisfied by the target log record among multiple sub-conditions included in the preset condition; before determining the collection interval corresponding to the processing level of the first information, the method further includes: determining the processing level of the first information from a second corresponding relationship based on the identifier of the target sub-condition; wherein the second corresponding relationship includes a corresponding relationship between different sub-conditions and processing levels.

[0017] In one possible implementation, the above-mentioned preset conditions include: a first sub-condition and a second sub-condition; wherein the first sub-condition includes: the current log record is a log record used to indicate a null pointer; the second sub-condition includes: the current log record is a log record used to indicate a request interface timeout.

[0018] In a possible implementation manner, the method further includes: sending a preset condition to the first virtual machine, so that the first virtual machine determines the log record required for the service based on the preset condition.

[0019] In a possible implementation, collecting target log fragments within a collection interval from the logs generated by the target virtual machine includes: sending the collection interval to the target virtual machine so that the target virtual machine collects the target log fragments within the collection interval; and receiving the target log fragments sent by the target virtual machine.

[0020] The embodiment of the present application provides a log collection method, which determines the time interval and / or location interval (referred to as: collection interval) where the log records to be collected are located according to the processing level of the first information received from the first virtual machine for describing the target log records required for the business, and then sends the determined collection interval to the target virtual machine so that the target virtual machine collects the target log fragments within the collection interval; finally, the target log fragments sent by the target virtual machine are received. Since the sizes of the collection intervals corresponding to different processing levels are different, the above method does not collect all the log records generated by the target virtual machine to the management device for storage, thereby avoiding the waste of storage resources in the management device, and thus improving the utilization rate of the storage resources in the management device.

[0021] In one possible implementation, the first information includes a first type identifier, which is used to indicate that the log type of the target log record is the first type. The sending of the collection interval to the target virtual machine includes: sending the collection interval and the first type identifier to the target virtual machine; so that the target virtual machine collects the target log fragment within the collection interval from the first type of log indicated by the first type identifier.

[0022] The above embodiment sends a collection interval and a first type identifier for indicating the log type of the target log record to the target virtual machine through the management device; so that the target virtual machine collects the target log fragments within the collection interval from the first type of log indicated by the first type identifier; it is not necessary to collect the log fragments within the above collection interval from all types of logs, thereby reducing the collected log fragments, and therefore saving storage resources in the management device.

[0023] In a possible implementation manner, the method further includes: storing a correspondence between the first information and the target log segment.

[0024] In the above embodiment, the management device stores the correspondence between the above-mentioned first information and the target log fragment, so that the user can directly obtain the above-mentioned target log fragment based on the first information; the user is not required to retrieve the log fragment corresponding to the above-mentioned first information from multiple different log fragments, thereby narrowing the scope of retrieving related log records of the target log record; therefore, the efficiency of retrieving the related log records is improved.

[0025] In a possible implementation, the target log record includes a record of an abnormal log.

[0026] In a second aspect, an embodiment of the present application provides a log collection method, which is applied to a first virtual machine in at least one virtual machine included in a management system, the management system also including a management module, the management module being used to manage the at least one virtual machine, the method comprising: detecting a target log record that meets a preset condition from a log generated by the first virtual machine; the preset condition is used to indicate a log record required for a business; when the target log record is detected, sending first information describing the target log record to the management module; so that the management module determines a collection interval corresponding to a processing level of the first information; wherein the collection interval is a time interval and / or location interval where the log record to be collected is located, and the size of the collection interval corresponding to different processing levels is different; based on the collection interval sent by the management module, collecting a target log fragment within the collection interval from the log generated by the first virtual machine; and sending the target log fragment to the management module.

[0027] The embodiment of the present application provides a log collection method, which detects target log records that meet preset conditions from the logs generated by the first virtual machine; the preset conditions are used to indicate log records required by the business; when the above target log records are detected, first information describing the target log records is sent to the above management module; so that the management module determines the collection interval corresponding to the processing level of the above first information; wherein the collection interval is the time interval and / or location interval where the log records to be collected are located, and then, based on the collection interval sent by the above management module, the target log fragments within the collection interval are collected from the logs generated by the above first virtual machine; and the target log fragments are sent to the above management module. Since the sizes of the collection intervals corresponding to different processing levels are different, the above method does not collect all the log records generated by the first virtual machine to the management module for storage, thereby avoiding the waste of storage resources corresponding to the management module, and thus improving the utilization rate of the storage resources corresponding to the management module.

[0028] In one possible implementation, the first information includes a first type identifier, which is used to indicate that the log type of the target log record is the first type. Based on the collection interval sent by the management module, the target log fragment within the collection interval is obtained from the log generated by the first virtual machine, including: collecting the target log fragment within the collection interval from the first type of log generated by the first virtual machine.

[0029] In the above embodiment, the first virtual machine collects the target log fragments within the collection interval from the first type of log generated by the above first virtual machine, and it is not necessary to collect the log fragments within the above collection interval from all types of logs, thereby reducing the collected log fragments, and therefore saving the transmission resources between the first virtual machine and the management module.

[0030] In one possible implementation, before sending the first information describing the target log record to the management module, the method also includes: determining the processing level of the first information from a second correspondence relationship based on an identifier of the target sub-condition; the target sub-condition is a sub-condition satisfied by the target log record among multiple sub-conditions included in the preset condition, and the second correspondence relationship includes a correspondence between different sub-conditions and processing levels.

[0031] In a possible implementation, before detecting the target log record that meets the preset condition in the log generated by the first virtual machine, the method further includes: receiving the preset condition sent by the management module.

[0032] In a possible implementation, the first information further includes: an identifier of the target log record, and / or an identifier of the target sub-condition.

[0033] In a third aspect, an embodiment of the present application provides a log collection device, which is deployed in a management module in a management system, and the management module is used to manage at least one virtual machine. The log collection device includes: a transceiver unit and a processing unit; the above-mentioned transceiver unit is used to obtain first information sent by a first virtual machine in at least one virtual machine; the first information is used to describe a target log record, and the target log record is a log record required for the business in a log generated by the first virtual machine; the processing unit is used to determine a collection interval corresponding to a processing level of the first information; wherein the collection interval is a time interval and / or a location interval where the log record to be collected is located, and the size of the collection interval corresponding to different processing levels is different; the transceiver unit is used to collect a target log fragment within the collection interval from the log generated by the target virtual machine; the target virtual machine includes at least the first virtual machine, and the target log fragment includes at least the target log record.

[0034] In a possible implementation, the level of processing is proportional to the size of the collection interval.

[0035] In one possible implementation, the processing level includes: the first level or the second level; when the first level is lower than the second level, the collection interval corresponding to the first level includes: the target log record and the log record generated within x time units before the target log record is generated; wherein x is an integer greater than or equal to 0; the collection interval corresponding to the second level includes: the target log record and the log record generated within m time units before the target log record is generated; wherein m is a positive integer greater than x.

[0036] In a possible implementation, the processing unit is used to determine the collection interval according to the processing level of the first information and the identifier of the target log record; the first information includes: the identifier of the target log record.

[0037] In a possible implementation, the processing unit is specifically configured to determine a collection interval from a first corresponding relationship according to a processing level of the first information and an identifier of a target log record; wherein the first corresponding relationship includes: a corresponding relationship between a processing level and a collection interval.

[0038] In one possible implementation, when the processing level of the first information is higher than a preset level, the target virtual machine includes the first virtual machine, and a virtual machine that has a calling relationship with the first virtual machine; the virtual machines that have a calling relationship with the first virtual machine include the upstream virtual machine of the first virtual machine, or / and, the downstream virtual machine of the first virtual machine.

[0039] In one possible implementation, the transceiver unit is used to obtain topology information; the topology information is used to describe the calling relationship between multiple virtual machines; the processing unit is used to determine the upstream virtual machine of the first virtual machine, or / and, the downstream virtual machine of the first virtual machine based on the topology information; the upstream virtual machine is the virtual machine that calls the first virtual machine among the multiple virtual machines; the downstream virtual machine is the virtual machine that is called by the first virtual machine among the multiple virtual machines.

[0040] In one possible implementation, the processing unit is used to determine the processing level of the first information from the second correspondence based on the identifier of the target sub-condition; wherein the second correspondence includes the correspondence between different sub-conditions and processing levels; the target log record is a log record that satisfies a preset condition, and the preset condition is used to indicate the log record required by the business; the first information includes the identifier of the target sub-condition, and the target sub-condition is a sub-condition satisfied by the target log record among the multiple sub-conditions included in the preset condition.

[0041] In one possible implementation, the preset condition includes: a first sub-condition and a second sub-condition; wherein the first sub-condition includes: the current log record is a log record for indicating a null pointer; the second sub-condition includes: the current log record is a log record for indicating a request interface timeout.

[0042] In a possible implementation, the transceiver unit is used to send a preset condition to the first virtual machine, so that the first virtual machine determines the log record required for the service based on the preset condition.

[0043] In a possible implementation, the transceiver unit is used to send a collection interval to the target virtual machine so that the target virtual machine collects the target log fragment within the collection interval; the transceiver unit is also used to receive the target log fragment sent by the target virtual machine.

[0044] In one possible implementation, the transceiver unit is used to send a collection interval and a first type identifier to a target virtual machine; so that the target virtual machine collects target log fragments within the collection interval from a first type of log indicated by the first type identifier; wherein the first information includes the first type identifier, and the first type identifier is used to indicate that the log type of the target log record is the first type.

[0045] In a possible implementation, the log collection device further includes: a storage unit; the storage unit is used to store the correspondence between the first information and the target log segment.

[0046] In a possible implementation, the target log record includes a record of an abnormal log.

[0047] In a fourth aspect, an embodiment of the present application provides a log collection device, which is deployed in a first virtual machine of at least one virtual machine included in a management system. The management system also includes a management module, which is used to manage at least one virtual machine. The log collection device includes: a processing unit and a transceiver unit; the processing unit is used to detect a target log record that meets a preset condition from a log generated by the first virtual machine; the preset condition is used to indicate a log record required for a business; the transceiver unit is used to send a first information describing the target log record to the management module when the target log record is detected; so that the management module determines a collection interval corresponding to a processing level of the first information; wherein the collection interval is a time interval and / or location interval where the log record to be collected is located, and the size of the collection interval corresponding to different processing levels is different; the processing unit is also used to collect a target log fragment within a collection interval from the log generated by the first virtual machine based on the collection interval sent by the management module; and send the target log fragment to the management module.

[0048] In one possible implementation, the transceiver unit is used to collect target log fragments within a collection interval from a first type of log generated by a first virtual machine; wherein the first information includes a first type identifier, and the first type identifier is used to indicate that the log type of the target log record is the first type.

[0049] In one possible implementation, the processing unit is used to determine the processing level of the first information from a second corresponding relationship based on an identifier of a target sub-condition; the target sub-condition is a sub-condition satisfied by the target log record among multiple sub-conditions included in a preset condition, and the second corresponding relationship includes a correspondence between different sub-conditions and processing levels.

[0050] In a possible implementation, the transceiver unit is used to receive the preset condition sent by the management module.

[0051] In a possible implementation, the first information further includes: an identifier of a target log record, and / or an identifier of a target sub-condition.

[0052] In a fifth aspect, the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in the first aspect or the second aspect and any one of their possible implementations.

[0053] In a sixth aspect, the present application provides a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed on a computing device, the computing device executes the method described in any one of the above-mentioned first aspect or second aspect and their possible implementation methods.

[0054] In a seventh aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in any one of the first aspect or the second aspect and any possible implementation methods thereof.

[0055] It should be understood that the beneficial effects achieved by the technical solutions of the third to seventh aspects of the present application and the corresponding possible implementation methods can be referred to the technical effects of the first or second aspect and their corresponding possible implementation methods mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of a flow chart of an existing log collection method provided in an embodiment of the present application;

[0057] Figure 2 A schematic diagram of a log management system provided in an embodiment of the present application;

[0058] Figure 3 A hardware schematic diagram of a computing device provided in an embodiment of the present application;

[0059] Figure 4 A log collection method process diagram provided in an embodiment of the present application Figure 1 ;

[0060] Figure 5 A log collection method process diagram provided in an embodiment of the present application Figure 2 ;

[0061] Figure 6 A log collection method process diagram provided in an embodiment of the present application Figure 3 ;

[0062] Figure 7 A log collection method process diagram provided in an embodiment of the present application Figure 4 ;

[0063] Figure 8 A schematic diagram of the structure of a log collection device 100 provided in an embodiment of the present application;

[0064] Fig. 9 A schematic diagram of the structure of a log collection device 200 provided in an embodiment of the present application;

[0065] Fig.10 A schematic diagram of a computing device cluster provided in an embodiment of the present application;

[0066] Fig.11 A network connection diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0067] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0068] The terms "first corresponding relationship" and "second corresponding relationship" and the like in the description and claims of the embodiments of the present application are used to distinguish different corresponding relationships rather than to describe a specific order of corresponding relationships.

[0069] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0070] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple cloud systems refers to two or more cloud systems.

[0071] In the embodiment of the present application, the device that generates the log can be a virtual machine, a computing device, or a service deployed on a cloud server. For ease of description, the embodiment of the present application is described by taking the example that the device that generates the log is a virtual machine, and no further elaboration is given later.

[0072] As the business of virtual machines becomes increasingly complex, the amount of logs generated by the virtual machines also increases. How to obtain the logs required for the business from the huge amount of logs is one of the problems to be solved in this field.

[0073] For example, if the business is anomaly detection business / fault analysis business, and the log required by the business is a log fragment associated with the abnormal log record, such as Figure 1 As shown, the log collection method may include: the virtual machines (Virtual Manufacturing, VM)_1 and VM_2 managed by the log management service upload the logs they generate to the log management service in real time through their respective proxy services. The log management service stores the logs uploaded in real time; at the same time, the log management service detects whether there are abnormal log records from the locally stored logs according to preset rules. When the log management service detects an abnormal log record, the log information of the abnormal log record is sent to the alarm service, so that the alarm service generates an alarm information for indicating the abnormal log record according to the log information. Subsequently, after the user (such as: operation and maintenance personnel) obtains the alarm information, the log fragment associated with the abnormal log record is retrieved (i.e.: queried) from the log stored by the log management service, and the log fragment is analyzed to determine the cause of the abnormal log record.

[0074] It should be understood that the log management service described in this application can be replaced by a log management service instance or a log management server or a log management service device or a log management service equipment, etc., and the alarm service can be replaced by an alarm service instance or an alarm server or an alarm service device or an alarm service equipment, etc., without limitation.

[0075] As can be seen from the above, the VM managed by the log management service in the above method needs to upload all logs generated by itself to the log management service in real time, and store the uploaded logs in the log management service. However, since the logs required by the business account for a small proportion of the logs stored by the log management service, there will be a large number of invalid logs (i.e., logs not required by the business) in the logs stored by the log management service, which not only reduces the utilization rate of storage resources in the log management service, but also reduces the efficiency of log collection.

[0076] It should be understood that in the embodiment of the present application, the management module is used to manage at least one virtual machine. When the management module is independently deployed in a computing device, the computing device on which the management module is deployed is a management device. When the management module is deployed in a cloud server, the management module is a log management service provided by the cloud server without any restriction.

[0077] In view of this, an embodiment of the present application provides a log collection method, which is applied to a management module in a management system, and the management module is used to manage at least one virtual machine; the method may include: the management module determines the time interval and / or location interval (referred to as: collection interval) where the log records to be collected are located according to the processing level of the first information received from the first virtual machine and used to describe the target log records required for the business, and then collects the target log fragments within the collection interval from the target virtual machine including at least the first virtual machine, and the target log fragments at least include the target log record; since the sizes of the collection intervals corresponding to different processing levels are different, the above method will not collect all the log records generated by the first virtual machine into the management module for storage, thereby avoiding the waste of storage resources corresponding to the management module and reducing the number of collected logs; therefore, while improving the utilization rate of the storage resources corresponding to the management module, the log collection efficiency is also improved.

[0078] The log collection method provided in the embodiment of the present application is applied to Figure 2 The log management system shown in the figure comprises: a management module, a prompt module and at least one VM; wherein the management module is used to manage at least one VM, such as Figure 2 As shown, the at least one VM may include VM1 and VM2.

[0079] It should be understood that the above-mentioned log management system may include one VM or multiple VMs. The embodiment of the present application takes the log management system including VM1 and VM2 as an example for explanation, and does not limit the number of VMs in the log management system.

[0080] In the present application, VM (ie VM1 and VM2) is used to detect whether there is a target log record (eg, an abnormal log record) required for the business in the log generated by the VM. When the target log record is detected, the VM reports the target log record to the prompt module.

[0081] Optionally, the VM1 and VM2 may include an agent module. Taking VM1 as an example, the agent module on VM1 is used to detect whether the target log record exists in the log generated by VM1, and if the target log record exists, report the target log record to the prompt module.

[0082] In the present application, the prompt module is used to parse the target log record to obtain information (referred to as: first information) used to describe the target log record, where the first information at least includes an identifier of the target log record; and report the first information to the management module.

[0083] It should be understood that the above prompt module can be an independent functional module or a functional module integrated in the VM. The specific embodiment of the present application does not limit the deployment relationship between the above VM and the prompt module. In addition, the naming of the prompt module is not limited and can also be called an alarm module or other named modules.

[0084] In the present application, the management module is used to determine the collection interval including the target log record according to the processing level of the first information, and send the collection interval to the target virtual machine. Among them, the target virtual machine includes the virtual machine (which can be called the first virtual machine) in which the target log record exists in the generated log. The method for determining the processing level of the first information is described in the following embodiment and will not be repeated here.

[0085] Optionally, the log management system further includes: a topology module (not shown in the figure). The topology module manages the topology information of the VM managed by the management module; the management device is also used to obtain the topology information from the topology module, and determine the upstream virtual machine and / or downstream virtual machine that has a call relationship with the first virtual machine based on the topology information.

[0086] Furthermore, the target virtual machine is further configured to collect target log segments within the collection interval after receiving the collection interval, and send the target log segments to the management module for storage.

[0087] It should be understood that the above-mentioned VM1, VM2, prompt module and management module can be functional modules deployed on computing devices, or can be independent computing devices, or can be services or service instances deployed on cloud servers, without limitation. In the case where the above-mentioned VM1, VM2, prompt module and management module are services or service instances deployed on cloud servers, the log management system can be called a cloud management system or a cloud system, the prompt module can be called a prompt service or a prompt service instance, the management module can be called a management service instance or a management service, the topology module can be called a topology service or a topology service instance, etc., without limitation.

[0088] It should be understood that, in the case where VM1, VM2, prompt module and management module can be functional modules deployed on a computing device, the computing device where the above VM1, VM2, prompt module and management module are located can be called a host machine, and the host machines where at least two of VM1, VM2, prompt module and management modules are located can be the same computing device or different computing devices without limitation.

[0089] For example, Figure 2 For example, any host machine among the host machine with the management module deployed, the host machine with the prompt module deployed, the host machine with VM1 deployed, and the host machine with VM2 deployed is a computing device. Figure 3 A schematic diagram of the hardware structure of a computing device, such as Figure 3 As shown, the computing device may include: a processor 301, a memory 302, and a communication interface 303. The processor 301, the memory 302, and the communication interface 303 may be connected to each other via a bus 304, or in other ways.

[0090] The processor 301 includes one or more central processing units (CPUs). The CPU may be a single-core CPU (single-CPU) or a multi-core CPU (multi-CPU). Optionally, the processor 301 may also include a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.

[0091] In the case where the computing device is the host machine of the management module, the processor 301 determines the collection interval where the log records to be collected are located based on the processing level of the first information sent by the prompt module for describing the target log records required for the business; then, the collection interval is sent to the target virtual machine in the VM managed by the management module; and the target log fragments within the collection interval collected by the target virtual machine are stored.

[0092] When the computing device is the host of the prompt module, the processor 301 is used to receive the target log record required for the business sent by the VM managed by the management module from the communication interface 303, and send the first information describing the target log record to the management module through the communication interface 303.

[0093] In the case where the computing device is a host machine of a VM (such as VM1 and VM2), the processor 301 is used to detect whether there is a target log record required for the business in the log generated by the VM, and send the target log record to the prompt module through the communication interface 303. The processor 301 is also used to receive the collection interval sent by the management module from the communication interface 303, collect the target log fragment within the collection interval, and send the target log fragment to the management module through the communication interface 303.

[0094] The memory 302 includes, but is not limited to, a random access memory (RAM), a read only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or an optical memory, a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer. In the embodiment of the present application, the memory 302 can store information such as computer instructions and operating system.

[0095] In a possible implementation, the memory 302 may exist independently of the processor 301. The memory 302 may be connected to the processor 301 via a bus 304 and is used to store data, instructions, or program codes. When the processor 301 calls and executes the instructions or program codes stored in the memory 302, the relevant steps in the data transmission method provided in the embodiment of the present application can be implemented.

[0096] In another possible implementation, the memory 302 may also be integrated with the processor 301 .

[0097] The communication interface 303 may be a transceiver module, which is used to communicate with other devices or communication networks, such as Ethernet, RAN, wireless local area networks (WLAN), etc. The communication interface 303 may receive instructions, messages or data, etc. The transceiver module may be a device such as a transceiver or a transceiver.

[0098] Optionally, the communication interface 303 may also be a transceiver circuit located in the processor 301, for implementing signal input and signal output of the processor 301. The communication interface 303 may be a wired interface (port), such as a fiber distributed data interface (FDDI) or a gigabit Ethernet (GE) interface, or the communication interface 303 may also be a wireless interface.

[0099] The bus 304 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus may also be divided into a serial bus and a parallel bus. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0100] It should be understood that Figure 3 The computing device in the example is only one example of a computing device, which may have Figure 3 More or fewer components shown in the figure may be combined with two or more components, or may have different component configurations. For example, the computing device may also include an intelligent network card, such as a data processing unit (DPU).

[0101] It should be noted that the system architecture and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0102] The present invention provides a log collection method for Figure 2 A management module in which the management module is used to manage at least one virtual machine. Figure 2 The management module and topology module in the same management device are integrated, and Figure 2 In the case where the prompt modules in are respectively integrated into each virtual machine managed by the management module, the method is specifically applied in a management device integrated with a management module, such as Figure 4 The method shown includes: S110-S140.

[0103] S110: The first virtual machine detects that a target log record exists in a log generated by the first virtual machine.

[0104] The first virtual machine may be any virtual machine among at least one virtual machine managed by the management module. Figure 2 VM1 in the

[0105] In the present application, the log generated by a virtual machine may refer to a log used to record event information of the virtual machine during operation; wherein the log is specifically a log record set having at least one log record, and each log record is used to describe an event information of the virtual machine during operation.

[0106] In this application, the target log record can be a log required by the business, where the business includes fault analysis business, fault detection business, and database operation business. In the case where the business is a fault analysis business or a fault detection business, the target log record is an abnormal log record required by the business to indicate an abnormality or fault. For example, when an abnormal event occurs during the execution of action A by the first virtual machine, the first virtual machine will generate an abnormal log record for the abnormal event (i.e., a target log record).

[0107] Optionally, in the present application, a log record required by a business may be indicated by a preset condition, that is, the target log record is a log required by the business, which can be understood as the target log record is a log record that satisfies the preset condition and is determined by the first virtual machine from the log generated by the first virtual machine. The specific implementation of determining the target log record that satisfies the preset condition from the log generated by the first virtual machine is referred to in S310 below, which will not be repeated here.

[0108] It should be understood that the first virtual machine in the present application may execute the above S110 periodically, may execute the above S110 in real time, or may execute the above S110 when there are new log records in the above first virtual machine. The specific embodiment of the present application does not limit the triggering conditions of the above S110.

[0109] S120: The first virtual machine sends first information, and the management module obtains the first information sent by the first virtual machine.

[0110] In the present application, the first information is used to describe the target log record or to indicate the target log record or to indicate that the target log record exists in the log generated by the first virtual machine.

[0111] Optionally, the first information includes: an identifier of the target log record and indication information of the target log record. The identifier of the target log record may include but is not limited to a timestamp or time when the target log record is generated (or referred to as the generation time of the target log record). For example, assuming that the generation time of the target log record is 2024-11-20 16:07:40, the first information includes: 2024-11-20 16:07:40. The indication information of the target log record is information used to describe the target log record.

[0112] In one example, the management module may directly receive the first information sent by the first virtual machine.

[0113] In another example, the management module can obtain the first information sent by the first virtual machine through the prompt module. For example, the first virtual machine sends the first information to the prompt module, the prompt module sends the first information to the management module, and the management module receives the first information from the prompt module.

[0114] S130. The management module determines a collection interval corresponding to a processing level of the first information.

[0115] The processing level of the first information may be the level of the target log record described by the first information in the log record required for the above-mentioned business; wherein the processing level of the first information may be obtained by directly obtaining it from the first information, or the processing level of the first information may be determined based on the first information. Specifically, the embodiment of the present application does not limit the method for obtaining the processing level of the first information.

[0116] In the case of determining the processing level of the first information based on the first information, the implementation method of determining the processing level of the first information refers to the following S340, which will not be repeated here.

[0117] In the present application, the collection interval may include the time interval and / or location interval where the log records to be collected are located. Among them, the collection interval includes the above-mentioned target log records. For example, when an abnormal event occurs in the process of the first virtual machine executing action A, the first virtual machine will generate an abnormal log record for the abnormal event (ie, target log record); and the action A may be the action called by the action B in the process of the first virtual machine executing action B, that is, the reason for generating the abnormal log record may be caused by the action described in the log record before the abnormal log record, so it is necessary to collect the log records before and / or after the abnormal log record, therefore, it is necessary to determine a collection interval including the target log record.

[0118] In one example, taking the time interval in which the collection interval includes the log records to be collected as an example, the collection interval may include: log records generated from the generation time of the target log record to x time units before the generation time of the target log record, and / or log records generated from the generation time of the target log record to y time units after the generation time of the target log record; wherein x and y are both integers greater than or equal to 0. For example, x can be set to 2 minutes, and y can be set to 1 minute. Or both x and y are 0. It should be understood that when both x and y are 0, the collection interval includes the target log record, and when x and / or y are integers greater than 0, the collection interval includes not only the target log record, but also the log records generated before the generation time of the target log record, and / or the log records generated after the generation time of the target log record.

[0119] In another example, taking the collection interval as the location interval of the log records to be collected as an example, the collection interval may include: the target log record to the x log records before the target log record, and / or, the target log record to the y log records after the target log record. It should be understood that in this other example, the virtual machine can sort the log records generated in a continuous time period according to the time of generation to obtain a log segment, in which the log records with an earlier generation time are arranged in front and the log records with a later generation time are arranged in the back, or the log records with a later generation time are arranged in front and the log records with an earlier generation time are arranged in the back.

[0120] In another example, the target log record to the x log records before the target log record can be understood as the position of the target log segment in the log segment and the x log records before the position of the target log record in the log segment. The target log record to the y log records after the target log record can be understood as the position of the target log segment in the log segment and the y log records after the position of the target log record in the log segment.

[0121] It should be understood that for the sake of ease of description, the embodiment of the present application is explained by taking the collection interval as the time interval where the log records to be collected are located as an example, wherein the implementation scheme corresponding to the collection interval being the location interval where the log records to be collected are located can refer to the implementation scheme where the collection interval is the time interval where the log records to be collected are located, and will not be repeated later.

[0122] In the present application, the processing level may include at least one, for example, the processing level may include multiple processing levels of different heights: a first level, a second level, and a third level. Different processing levels correspond to different sizes of acquisition intervals.

[0123] Optionally, the processing level may be directly proportional to or inversely proportional to the size of the acquisition interval. Specifically, the embodiment of the present application does not specifically limit the relationship between the processing level and the size of the acquisition interval.

[0124] Exemplarily, in the case where the processing level is proportional to the size of the above-mentioned collection interval, taking the case where the processing level of the first information includes: the first level or the second level, and the first level is lower than the second level, the collection intervals corresponding to the first level and the second level are as follows:

[0125] The collection interval corresponding to the first level (abbreviated as: first collection interval) includes: the above-mentioned target log record and the log record generated within x time units before the target log record is generated; x is an integer greater than or equal to 0.

[0126] The collection difference corresponding to the second level (abbreviated as: second collection interval) includes: the target log record and the log record generated within m time units before the target log record is generated, where m is an integer greater than x; that is, the range of the time interval indicated by the above-mentioned second collection interval is greater than the range of the time interval indicated by the first collection interval.

[0127] For example, when the processing level is proportional to the size of the above-mentioned collection interval, the corresponding relationship between the processing level and the collection interval is as shown in Table 1 below: 2 processing levels and 2 collection intervals; the 2 processing levels include: the first level and the second level, wherein the second level is higher than the first level; the 2 collection intervals include [t-10 (seconds / s), t] and [t-30s, t], t represents the generation time of the above-mentioned target log record, and "[]" represents a closed interval. Among them, the collection interval corresponding to the first level is [t-10s, t], and the collection interval corresponding to the second level is [t-30s, t]. Assume that the value of t is 2024-11-20 16:07:40, then the collection interval corresponding to the first level is [2024-11-20 16:07:30, 2024-11-2016:07:40], and the collection interval corresponding to the second level is [2024-11-20 16:07:10, 2024-11-20 16:07:40].

[0128] Table 1

[0129] Processing Level Collection interval First level [t-10 (seconds / s), t] Second level [t-30 (seconds / s), t]

[0130] As another example, when the processing level is inversely proportional to the size of the collection interval, when the processing level of the first information includes: the first level or the second level, and the first level is lower than the second level, the collection interval corresponding to the first level is the second collection interval, and the collection interval corresponding to the second level is the first collection interval, that is, when the first level is lower than the second level, the collection interval corresponding to the first level is larger than the collection interval corresponding to the second level.

[0131] The implementation method of the above S130 includes: determining the above collection interval according to the processing level of the first information and the identifier of the target log record. The specific implementation method thereof is referred to the following S350 and will not be repeated here.

[0132] S140: The management module collects target log segments within a collection interval from logs generated by the target virtual machine.

[0133] In the present application, the target log segment at least includes the above-mentioned target log record.

[0134] In the present application, the target virtual machine includes at least a first virtual machine.

[0135] It should be understood that an exception may also occur during the process of one device calling another device; that is, the target log record may also be an exception caused by other devices calling the first virtual machine, and / or an exception caused by the first virtual machine calling other devices.

[0136] Based on this, in one implementation method, when the processing level of the first information is higher than the preset level, the target virtual machine, on the basis of including the above-mentioned first virtual machine, also includes a virtual machine that has a calling relationship with the first virtual machine; wherein the virtual machine that has a calling relationship with the first virtual machine includes: the upstream virtual machine of the first virtual machine, or / and, the downstream virtual machine of the first virtual machine.

[0137] In the present application, the upstream virtual machine is a virtual machine that calls the first virtual machine among at least one virtual machine managed by the management module; the downstream virtual machine is a virtual machine that is called by the first virtual machine among at least one virtual machine managed by the management module.

[0138] Exemplarily, assuming that the preset level is the first level, when the processing level of the first information is higher than the first level, the target virtual machine includes: the first virtual machine, an upstream virtual machine of the first virtual machine, and a downstream virtual machine of the first virtual machine. When the processing level of the first information is lower than or equal to the first level, the target virtual machine is the first virtual machine.

[0139] The implementation method of the above S140 can be that the management module sends the collection interval to the target virtual machine, so that the target virtual machine sends the collected target log fragments to the management module, or the management module actively reads the target log fragments from the target virtual machine. The specific embodiment of the present application does not specifically limit the implementation method of the above S140.

[0140] An embodiment of the present application provides a log collection method, which is applied to a management module in a management system, and the management module is used to manage at least one virtual machine; the method determines the time interval and / or location interval (referred to as: collection interval) where the log records to be collected are located according to the processing level of the first information received from the first virtual machine and used to describe the target log records required for the business, and then collects the target log fragments within the collection interval from the target virtual machine including at least the first virtual machine, and the target log fragments at least include the target log records; since the sizes of the collection intervals corresponding to different processing levels are different, the above method will not collect all the log records generated by the first virtual machine into the management module for storage, thereby avoiding the waste of storage resources corresponding to the management module and reducing the number of logs collected; therefore, while improving the utilization rate of the storage resources corresponding to the management module, the log collection efficiency is also improved.

[0141] In addition, the above method collects target log fragments of different sizes based on different processing levels, and then the user retrieves relevant log records associated with the target log records from the target log fragments of different sizes; compared with the method of retrieving the relevant log records from the full logs generated by all virtual machines, the efficiency of retrieving the relevant log records is improved.

[0142] based on Figure 4 ,like Figure 5 As shown, the embodiment of the present application provides a specific implementation method of S140, which includes: S210-S250.

[0143] S210. The management module determines whether the processing level of the first information is higher than a preset level.

[0144] In a case where the processing level of the first information is lower than or equal to the preset level, the following S220 is performed.

[0145] In the case where the processing level of the first information is higher than the preset level, the following S230 - S250 are performed.

[0146] S220: The management module collects target log segments within a collection interval from logs generated by the first virtual machine.

[0147] It should be understood that, in the present application, S220 is to determine the first virtual machine as the target virtual machine, and then collect the above-mentioned target log fragment from the log generated by the target virtual machine.

[0148] Exemplarily, it is assumed that the processing levels are from the first level (such as the important level) to the third level (such as the prompt level) from high to low; and it is assumed that the above-mentioned preset level is the second level (such as the emergency level); then, when the processing level of the first information is the second level or the third level, since the processing level of the first information is equal to or lower than the preset level; the above-mentioned target log fragment is collected from the log generated by the first virtual machine.

[0149] It should be understood that the implementation method of the above S220 is similar to the implementation method of the above S140. For the specific description of S220, reference can be made to the above related description of S140, which will not be repeated here.

[0150] S230: The management module obtains topology information.

[0151] In this application, topology information is used to describe the calling relationship between multiple virtual machines managed by the management module. The topology information can be embodied in the form of a linked list or a mapping relationship. The embodiment of this application does not limit the embodiment of the topology information.

[0152] The implementation method of S230 may be to directly obtain the topology information from the host device of the management module, or to obtain the topology information from other devices. The specific implementation method of the present application is not limited to the above S230.

[0153] S240: The management module determines an upstream virtual machine of the first virtual machine and a downstream virtual machine of the first virtual machine according to the topology information.

[0154] The implementation of S240 includes: determining the virtual machine that calls the first virtual machine in the topology information as the upstream virtual machine, and determining the virtual machine called by the first virtual machine in the topology information as the downstream virtual machine; its specific implementation refers to the relevant technology and will not be repeated here.

[0155] S250: The management module collects target log segments within a collection interval from the logs generated by the first virtual machine, the logs generated by the upstream virtual machine, and the logs generated by the downstream virtual machine.

[0156] It should be understood that the above S250 is to determine the first virtual machine, the upstream virtual machine and the downstream virtual machine as the target virtual machine, and then respectively collect target log fragments from the days generated by the target virtual machine.

[0157] Exemplarily, based on the example in S220 above, it is assumed that the upstream virtual machine of the first virtual machine is the second virtual machine, and the downstream virtual machine of the first virtual machine is the third virtual machine; and it is further assumed that the processing level of the first information is the first level. Then, since the first level is higher than the preset level, the target log segments within the above collection interval are respectively obtained from the logs generated by the first virtual machine to the third virtual machine.

[0158] It should be understood that the implementation method of the above S250 is similar to the implementation method of the above S140. For the specific description of S250, reference can be made to the above related description of S140, which will not be repeated here.

[0159] In the embodiment of the present application, when the processing level of the first information is higher than the preset level, target log fragments are respectively collected from the logs generated by the first virtual machine and the upstream virtual machine of the first virtual machine, or / and the downstream virtual machine of the first virtual machine; because the upstream virtual machine and the downstream virtual machine both have a calling relationship with the above-mentioned first virtual machine, the collected target log fragments are more comprehensive, thereby improving the accuracy of the target log fragments.

[0160] based on Figure 4 ,like Figure 6 As shown, an embodiment of the present application provides a specific implementation method, which includes: S310-S380.

[0161] S310: The first virtual machine detects a target log record that meets a preset condition from a log generated by the first virtual machine.

[0162] The preset conditions in the present application are used to indicate the log records required for the business; when the log required for the business is an abnormal log record, the preset conditions are used to indicate (or determine) the abnormal log record; when the log required for the business is a log record of database operations, the preset conditions are used to indicate the log records of adding, deleting, modifying and checking the database; the preset conditions are set based on actual business needs, and the embodiments of the present application do not limit the specific content of the preset conditions.

[0163] It should be understood that for ease of description, the embodiment of the present application is explained by taking the log records required by the business as abnormal log records as an example. That is to say, the embodiment of the present application is explained by taking the preset conditions for indicating abnormal log records as an example, and will not be repeated later.

[0164] It should be understood that the preset condition may be sent by the management device to the first virtual machine, or may be pre-configured by the user in the first virtual machine. Specifically, the embodiment of the present application does not limit the source of the preset condition in the first virtual machine.

[0165] The implementation method of the above S310 includes the first virtual machine determining a log record that meets a preset condition from the log generated by the first virtual machine.

[0166] Exemplarily, it is assumed that the preset condition includes: a first sub-condition and a second sub-condition, the first sub-condition includes: the current log record is a log record for indicating a null pointer; the second sub-condition includes: the current log record is a log record for indicating a request interface timeout.

[0167] Then, when the first virtual machine detects log record A, if log record A is a log record used to indicate a null pointer, or if log record A is a log record used to indicate a request interface timeout, log record A is determined as the above-mentioned target log record; if log record A is not a log record used to indicate a null pointer, and log record A is not a log record used to indicate a request interface timeout, the first virtual machine detects the next log record of log record A.

[0168] It should be understood that the multiple sub-conditions in the preset conditions in the present application can specifically be multiple different regular expressions; for example, the first sub-condition is the regular expression "\(error|warn)\("(.+?)\"\s*\+\s*(e|ex|e.getMessage\()\s*\);.*", which is used to indicate the log record of the null pointer.

[0169] S320: The first virtual machine determines whether a target log record is detected.

[0170] When the target log record is not detected in the log generated by the first virtual machine, the above S310 is performed to enable the first virtual machine to detect other log records in the log generated by the virtual machine.

[0171] When a target log record is detected from the log generated by the first virtual machine, the following S330 is performed.

[0172] S330: The first virtual machine sends first information describing a target log record to the management device.

[0173] The first information includes: an identifier of a target log record and an identifier of a target sub-condition; wherein the target sub-condition is a sub-condition satisfied by the target log record among a plurality of sub-conditions included in the above-mentioned preset condition.

[0174] Exemplarily, based on the example of S310 above, when log record A is a log record for indicating a null pointer, the target log record is log record A, and the target sub-condition is the first sub-condition in the preset condition.

[0175] S340: The management device determines a processing level of the first information from the second corresponding relationship based on the identifier of the target sub-condition.

[0176] It should be understood that the management device in the present application is a computing device deployed with a management module.

[0177] The second corresponding relationship in the present application includes the corresponding relationship between different sub-conditions and processing levels.

[0178] The implementation method of S340 in the present application includes: determining the processing level corresponding to the target sub-condition in the second corresponding relationship as the processing level of the first information.

[0179] Exemplarily, it is assumed that the second corresponding relationship includes: 2 sub-condition identifiers and 2 processing levels as shown in Table 2; the identifiers of the 2 sub-conditions include: "sub-condition 1" and "sub-condition 2"; the 2 processing levels include: first level and second level. Based on the example in S330 above, when the identifier of the first sub-condition is "sub-condition 1", the processing level of the first information is the first level; when the identifier of the first sub-condition is "sub-condition 2", the processing level of the first information is the second level.

[0180] Table 2

[0181] Sub-condition identifier Processing Level Subcondition 1 First level Subcondition 2 Second level

[0182] Optionally, in one implementation, S340 is executed by the first virtual machine, and the processing level of the first information is added to the first information; thereby avoiding the problem of high load on the management device caused by the management device determining the processing levels corresponding to the multiple first information after multiple virtual machines send the first information to the management device at the same time, thereby reducing the load pressure on the management device.

[0183] S350: The management device determines a collection interval from a first corresponding relationship according to a processing level of the first information and an identifier of a target log record.

[0184] The first corresponding relationship in the present application includes: a corresponding relationship between a processing level and a collection interval.

[0185] The identifier of the target log record is used to indicate the generation time of the target log record; specifically, the identifier of the target log record may be the generation time of the target log record, or may be a combination of the generation time of the target log record and the unique number of the target log record.

[0186] Exemplarily, assume that the first corresponding relationship includes as shown in Table 3 below: 2 processing levels and 2 collection intervals; the 2 processing levels include: the first level and the second level; the 2 collection intervals include [t-2 (minutes / m), t] and [t-1 (minutes / m), t], where t represents the generation time of the above-mentioned target log record; wherein the collection interval corresponding to the first level is [t-2 (minutes / m), t], and the collection interval corresponding to the second level is [t-1 (minutes / m), t].

[0187] Assume that the value of t is 2024-11-20 16:07; then, when the processing level of the first information is the first level, the collection interval corresponding to the first level is [2024-11-20 16:05, 2024-11-20 16:07]. When the processing level of the first information is the second level, the collection interval corresponding to the first level is [2024-11-20 16:06, 2024-11-20 16:07].

[0188] Table 3

[0189] Processing Level Collection interval First level [t-2 (minutes / m), t] Second level [t-1 (minute / m), t]

[0190] S360: The management device sends a collection interval to the target virtual machine.

[0191] Correspondingly, the target virtual machine receives the collection interval sent by the management device.

[0192] The implementation of the above S360 is as follows:

[0193] When the processing level of the first information is higher than the preset level, the target virtual machine includes: the first virtual machine, the upstream virtual machine of the first virtual machine, and the downstream virtual machine of the first virtual machine; wherein, the method for determining the upstream virtual machine and the downstream virtual machine refers to the above S230-S240 and will not be repeated here.

[0194] Based on this, the implementation of S360 includes: the management device sends the collection interval to the first virtual machine, the upstream virtual machine of the first virtual machine, and the downstream virtual machine of the first virtual machine respectively.

[0195] When the processing level of the first information is lower than or equal to the preset level, the target virtual machine is the first virtual machine.

[0196] Based on this, the implementation method of the above S360 includes: the management device sends the collection interval to the first virtual machine respectively.

[0197] S370: The target virtual machine collects target log segments within the collection interval.

[0198] When the target virtual machine is the first virtual machine, the implementation of S370 is as follows: the first virtual machine collects, from the logs generated by the first virtual machine, log records whose generation time is within the collection interval as the target log fragments.

[0199] When the target virtual machine is the first virtual machine, the upstream virtual machine of the first virtual machine, and the downstream virtual machine of the first virtual machine, the implementation method of the above S370 is: the first virtual machine, the upstream virtual machine of the first virtual machine, and the downstream virtual machine of the first virtual machine respectively collect the logs generated by themselves, and the log records with the generation time within the above collection interval as the above target log fragments.

[0200] S380: The management device receives the target log fragment sent by the target virtual machine.

[0201] In the case where the target virtual machine is the first virtual machine, the above S380 is implemented as follows: the management device receives the target log fragment sent by the first virtual machine.

[0202] When the target virtual machine is the first virtual machine, the upstream virtual machine of the first virtual machine, and the downstream virtual machine of the first virtual machine, the implementation method of the above S380 is: the management device receives the target log fragments sent by the first virtual machine, the upstream virtual machine of the first virtual machine, and the downstream virtual machine of the first virtual machine respectively.

[0203] Optionally, after the above S380, it also includes: the management device stores the correspondence between the above-mentioned first information and the target log fragment, so that the user can directly obtain the above-mentioned target log fragment based on the first information; the user is not required to retrieve the log fragment corresponding to the above-mentioned first information from multiple different log fragments, thereby narrowing the scope of retrieving related log records of the target log record; therefore, the efficiency of retrieving the related log records is improved.

[0204] The embodiment of the present application provides a log collection method, which determines the time interval and / or location interval (referred to as: collection interval) where the log records to be collected are located according to the processing level of the first information received from the first virtual machine for describing the target log records required for the business, and then sends the determined collection interval to the target virtual machine so that the target virtual machine collects the target log fragments within the above collection interval; finally, the target log fragments sent by the target virtual machine are received. Since the sizes of the collection intervals corresponding to different processing levels are different, the above method does not collect all the log records generated by the target virtual machine to the management device for storage, thereby avoiding the waste of storage resources in the management device, and thus improving the utilization rate of the storage resources in the management device.

[0205] based on Figure 6 ,like Figure 7 As shown, an embodiment of the present application provides another implementation method of the above-mentioned S360-S370, and the method includes: S410-S420.

[0206] S410: The management device sends a collection interval and a first type identifier to a target virtual machine.

[0207] The first information in the present application includes: a first type identifier; the first type identifier is used to indicate that the log type of the above-mentioned target log record is the first type; that is, the first type identifier is used to indicate the log type of the log to which the above-mentioned target log record belongs.

[0208] For example, when the first virtual machine detects the target log record from the system log, the first type identifier is used to indicate the log type of the system log.

[0209] S420: The target virtual machine collects a target log segment in the collection interval from among the logs of the first type.

[0210] The implementation of S420 includes: the first virtual machine collects target log segments within a collection interval from the first type of logs generated by the first virtual machine.

[0211] In the case where the target virtual machine further includes: an upstream virtual machine and / or a downstream virtual machine, the above S420 further includes the following:

[0212] The upstream virtual machine collects target log segments within a collection interval from the first type of logs generated by the upstream virtual machine.

[0213] And / or, the downstream virtual machine collects target log segments within a collection interval from the first type of logs generated by the downstream virtual machine.

[0214] In an embodiment of the present application, a collection interval and a first type identifier for indicating the log type of a target log record are sent to a target virtual machine through a management device; so that the target virtual machine collects target log fragments within the collection interval from the first type of log indicated by the first type identifier; and it is not necessary to collect log fragments within the above-mentioned collection interval from all types of logs, thereby reducing the collected log fragments and thus saving storage resources in the management device.

[0215] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, the log collection device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0216] The embodiment of the present application can exemplarily divide the log collection device into functional units according to the above method. For example, the log collection device can include various functional units corresponding to the various functional divisions, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0217] In the case of dividing each functional unit into corresponding functional units, Figure 8 A possible structural diagram of the log collection device 100 involved in the above embodiment is shown; the log collection device 100 is deployed in Figure 2 The management module in the log management system is shown in FIG. Figure 8 As shown, the log collection device 100 includes: a transceiver unit 810 and a processing unit 820 .

[0218] The transceiver unit 810 is used to obtain first information sent by a first virtual machine in at least one virtual machine; for example, execute step S120 in the above method embodiment.

[0219] The processing unit 820 is used to determine the collection interval corresponding to the processing level of the first information; for example, execute step S130 in the above method embodiment.

[0220] The processing unit 820 is further configured to collect target log segments within a collection interval from logs generated by the target virtual machine; for example, to execute step S140 in the above method embodiment.

[0221] Optionally, the level of the above processing is proportional to the size of the collection interval.

[0222] Optionally, when the first level is lower than the second level, the collection interval corresponding to the first level includes: the target log record and the log record generated within x time units before the target log record is generated; wherein x is an integer greater than or equal to 0; the collection interval corresponding to the second level includes: the target log record and the log record generated within m time units before the target log record is generated; wherein m is a positive integer greater than x.

[0223] Optionally, the processing unit 820 is configured to determine a collection interval according to a processing level of the first information and an identifier of a target log record.

[0224] Optionally, the processing unit 820 is specifically configured to determine a collection interval from a first corresponding relationship according to a processing level of the first information and an identifier of a target log record; for example, executing step S350 in the method embodiment.

[0225] Optionally, when the processing level of the first information is higher than a preset level, the target virtual machine includes the first virtual machine, and a virtual machine that has a calling relationship with the first virtual machine; the virtual machines that have a calling relationship with the first virtual machine include the upstream virtual machine of the first virtual machine, or / and, the downstream virtual machine of the first virtual machine.

[0226] Optionally, the transceiver unit 810 is used to obtain topology information; for example, execute step S230 in the above method embodiment.

[0227] The processing unit 820 is configured to determine, according to the topology information, an upstream virtual machine of the first virtual machine, and / or a downstream virtual machine of the first virtual machine; for example, to execute step S240 in the above method embodiment.

[0228] Optionally, the processing unit 820 is configured to determine the processing level of the first information from the second corresponding relationship based on the identifier of the target sub-condition; for example, executing step S340 in the above method embodiment.

[0229] Optionally, the above-mentioned preset conditions include: a first sub-condition and a second sub-condition; wherein, the first sub-condition includes: the current log record is a log record used to indicate a null pointer; the second sub-condition includes: the current log record is a log record used to indicate a request interface timeout.

[0230] Optionally, the transceiver unit 810 is configured to send a preset condition to the first virtual machine.

[0231] Optionally, the transceiver unit 810 is configured to send a collection interval to a target virtual machine; for example, executing step S360 in the above method embodiment.

[0232] The transceiver unit 810 is further configured to receive a target log segment sent by a target virtual machine; for example, to execute step S380 in the above method embodiment.

[0233] Optionally, the transceiver unit 810 is configured to send a collection interval and a first type identifier to a target virtual machine; for example, executing step S410 in the above method embodiment.

[0234] Optionally, the log collection device 100 further includes: a storage unit 830 .

[0235] The storage unit 830 is used to store the correspondence between the first information and the target log segment.

[0236] Optionally, the above target log records include records of exception logs.

[0237] In the case of dividing each functional unit into corresponding functional units, Fig. 9 A possible structural diagram of the log collection device 200 involved in the above embodiment is shown; the log collection device 200 is deployed in Figure 2 The first virtual machine in at least one virtual machine included in the log management system shown. Fig. 9 As shown, the log collection device 200 includes: a processing unit 910 and a transceiver unit 920 .

[0238] The processing unit 910 is used to detect a target log record that meets a preset condition from the log generated by the first virtual machine; for example, executing step S310 in the above method embodiment.

[0239] The transceiver unit 920 is used to send first information describing the target log record to the management module when the target log record is detected; for example, execute step S330 in the above method embodiment.

[0240] The processing unit 910 is further configured to collect target log segments within the collection interval based on the collection interval sent by the management module; for example, execute step S370 in the above method embodiment.

[0241] The transceiver unit 920 is further configured to send the target log segment to the management module; for example, executing step S380 in the above method embodiment.

[0242] Optionally, the processing unit 910 is configured to collect target log segments located within a collection interval in the logs of the first type; for example, executing step S420 in the above method embodiment.

[0243] Optionally, the processing unit 910 is configured to determine a processing level of the first information from the second corresponding relationship based on an identifier of the target sub-condition.

[0244] Optionally, the transceiver unit 920 is used to receive preset conditions sent by the management module.

[0245] Optionally, the first information further includes: an identifier of a target log record, and / or an identifier of a target sub-condition.

[0246] Among them, the transceiver unit 810, the processing unit 820 and the storage unit 830 in the log collection device 100, and the processing unit 910 and the transceiver unit 920 in the log collection device 200 can be implemented by software or by hardware. Exemplarily, the implementation of the processing unit 820 is introduced below by taking the processing unit 820 as an example. Similarly, the implementation of the transceiver unit 810, the storage unit 830, the processing unit 910 and the transceiver unit 920 can refer to the implementation of the processing unit 820.

[0247] As an example of a software functional unit, the processing unit 820 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the processing unit 820 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0248] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0249] As an example of a hardware functional unit, the processing unit 820 may include at least one computing device, such as a server, etc. Alternatively, the processing unit 820 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0250] The multiple computing devices included in the processing unit 820 can be distributed in the same region or in different regions. The multiple computing devices included in the processing unit 820 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the processing unit 820 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0251] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0252] like Fig.10 As shown, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the above log collection method.

[0253] It should be understood that the description of the processors and buses in the above-mentioned multiple computing devices 100 is different from Figure 3 The description of the processor 301 and bus 304 is consistent with that in FIG. Figure 3 The relevant description will not be repeated here.

[0254] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store some instructions for executing the above log collection method. In other words, the combination of one or more computing devices 100 can jointly execute the instructions for executing the above log collection method.

[0255] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the log collection device. That is, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more modules in the transceiver unit 810, the processing unit 820 and the storage unit 830 in the management device or management module. Similarly, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more modules in the processing unit 910 and the transceiver unit 920 in the above-mentioned virtual machine (such as: the first virtual machine).

[0256] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.11 A possible implementation is shown. Fig.11 As shown, two computing devices 100A and 100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of the transceiver unit 810, the processing unit 820, and the storage unit 830. At the same time, the memory 106 in the computing device 100B stores instructions for executing the functions of the processing unit 910 and the transceiver unit 920.

[0257] Fig.11 The connection method between the computing device clusters shown may be based on the fact that the log collection method provided in the present application requires frequent acquisition and transmission of data, and therefore it is considered that the functions implemented by the log collection apparatus 100 and the log collection apparatus 200 are handed over to the computing device 100A for execution.

[0258] It should be understood that Fig.11 The functions of the computing device 100A shown in FIG. 1 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0259] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.10 and Fig.11 The connection mode of the computing device cluster is different in that the memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the log collection method.

[0260] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the log collection method.

[0261] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the log collection method.

[0262] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log collection method, characterized in that: The method is applied to a management module in a management system, the management module is used to manage at least one virtual machine, and the method includes: Acquire first information sent by a first virtual machine among the at least one virtual machine; the first information is used to describe a target log record, and the target log record is a log record required for a business in a log generated by the first virtual machine; Determine a collection interval corresponding to a processing level of the first information; wherein the collection interval is a time interval and / or a location interval where log records to be collected are located, and different processing levels correspond to different sizes of collection intervals; From the log generated by the target virtual machine, target log fragments within the collection interval are collected; the target virtual machine includes at least the first virtual machine, and the target log fragments include at least the target log record.

2. The method according to claim 1, characterized in that The level of the processing is proportional to the size of the acquisition interval.

3. The method according to claim 1 or 2, characterized in that: The processing level includes: the first level or the second level; When the first level is lower than the second level, the collection interval corresponding to the first level includes: the target log record and the log record generated within x time units before the target log record is generated; wherein x is an integer greater than or equal to 0; The collection interval corresponding to the second level includes: the target log record and the log record generated within m time units before the target log record is generated; wherein m is a positive integer greater than x.

4. The method according to any one of claims 1 to 3, characterized in that: The first information includes: an identifier of the target log record, and the determining of a collection interval corresponding to a processing level of the first information includes: The collection interval is determined according to the processing level of the first information and the identifier of the target log record.

5. The method according to claim 4, characterized in that The determining the collection interval according to the processing level of the first information and the identifier of the target log record includes: The collection interval is determined from a first corresponding relationship according to the processing level of the first information and the identifier of the target log record; wherein the first corresponding relationship includes: a corresponding relationship between the processing level and the collection interval.

6. The method according to any one of claims 1 to 5, characterized in that: When the processing level of the first information is higher than a preset level, the target virtual machine includes the first virtual machine and a virtual machine that has a calling relationship with the first virtual machine; the virtual machine that has a calling relationship with the first virtual machine includes the upstream virtual machine of the first virtual machine, or / and, the downstream virtual machine of the first virtual machine.

7. The method according to claim 6, characterized in that The method further comprises: Acquire topology information; the topology information is used to describe the calling relationship between multiple virtual machines; According to the topology information, determine the upstream virtual machine of the first virtual machine, or / and, the downstream virtual machine of the first virtual machine; the upstream virtual machine is the virtual machine that calls the first virtual machine among the multiple virtual machines; the downstream virtual machine is the virtual machine that is called by the first virtual machine among the multiple virtual machines.

8. The method according to any one of claims 1 to 7, characterized in that: The target log record is a log record that satisfies a preset condition, where the preset condition is used to indicate a log record required by the business; the first information includes an identifier of a target sub-condition, where the target sub-condition is a sub-condition satisfied by the target log record among multiple sub-conditions included in the preset condition; Before determining the collection interval corresponding to the processing level of the first information, the method further includes: Based on the identifier of the target sub-condition, the processing level of the first information is determined from the second corresponding relationship; wherein the second corresponding relationship includes a corresponding relationship between different sub-conditions and processing levels.

9. The method according to claim 8, characterized in that The preset condition includes: a first sub-condition and a second sub-condition; wherein the first sub-condition includes: the current log record is a log record for indicating a null pointer; and the second sub-condition includes: the current log record is a log record for indicating a request interface timeout.

10. The method according to claim 8 or 9, characterized in that: The method further comprises: The preset condition is issued to the first virtual machine, so that the first virtual machine determines the log record required for the service based on the preset condition.

11. The method according to any one of claims 1 to 10, characterized in that: The step of collecting target log segments within the collection interval from the logs generated by the target virtual machine includes: Sending the collection interval to the target virtual machine, so that the target virtual machine collects the target log fragment within the collection interval; Receive the target log fragment sent by the target virtual machine.

12. The method according to claim 11, characterized in that The first information includes a first type identifier, where the first type identifier is used to indicate that the log type of the target log record is a first type, and the sending the collection interval to the target virtual machine includes: The collection interval and the first type identifier are sent to the target virtual machine, so that the target virtual machine collects the target log fragment within the collection interval from the first type of log indicated by the first type identifier.

13. The method according to any one of claims 1 to 12, characterized in that: The method further comprises: The correspondence between the first information and the target log segment is stored.

14. The method according to any one of claims 1 to 13, characterized in that: The target log record includes a record of an abnormal log.

15. A log collection method, characterized in that: The method is applied to a first virtual machine in at least one virtual machine included in a management system, the management system further comprising a management module, the management module being used to manage the at least one virtual machine, the method comprising: Detecting target log records that meet preset conditions from the logs generated by the first virtual machine; the preset conditions are used to indicate log records required for the business; When the target log record is detected, first information describing the target log record is sent to the management module; so that the management module determines a collection interval corresponding to a processing level of the first information; wherein the collection interval is a time interval and / or a location interval where the log record to be collected is located, and different collection intervals corresponding to different processing levels have different sizes; Based on the collection interval sent by the management module, target log segments within the collection interval are collected from the logs generated by the first virtual machine; and the target log segments are sent to the management module.

16. The method according to claim 15, characterized in that The first information includes a first type identifier, the first type identifier is used to indicate that the log type of the target log record is a first type, and the acquiring, based on the collection interval sent by the management module, a target log segment within the collection interval from a log generated by the first virtual machine, includes: The target log segments within the collection interval are collected from the first type of logs generated by the first virtual machine.

17. The method according to claim 15 or 16, characterized in that Before sending the first information describing the target log record to the management module, the method further includes: Based on the identification of the target sub-condition, the processing level of the first information is determined from the second corresponding relationship; the target sub-condition is the sub-condition satisfied by the target log record among the multiple sub-conditions included in the preset condition, and the second corresponding relationship includes the corresponding relationship between different sub-conditions and processing levels.

18. The method according to any one of claims 15 to 17, characterized in that: Before detecting a target log record that meets a preset condition in the log generated by the first virtual machine, the method further includes: The preset condition sent by the receiving management module.

19. The method according to any one of claims 15 to 18, characterized in that: The first information also includes: an identifier of the target log record, and / or an identifier of the target sub-condition.

20. A log collection device, characterized in that: The log collection device is deployed in a management module in a management system, and the management module is used to manage at least one virtual machine. The log collection device includes: a transceiver unit and a processing unit; The transceiver unit is used to obtain first information sent by a first virtual machine among the at least one virtual machine; the first information is used to describe a target log record, and the target log record is a log record required for the business in the log generated by the first virtual machine; The processing unit is used to determine a collection interval corresponding to a processing level of the first information; wherein the collection interval is a time interval and / or a location interval where the log records to be collected are located, and the sizes of the collection intervals corresponding to different processing levels are different; The transceiver unit is used to collect target log segments within the collection interval from the logs generated by the target virtual machine; the target virtual machine includes at least the first virtual machine, and the target log segments include at least the target log records.

21. A log collection device, characterized in that: The log collection device is deployed in a first virtual machine of at least one virtual machine included in the management system, the management system further includes a management module, the management module is used to manage the at least one virtual machine, the log collection device includes: a processing unit and a transceiver unit; The processing unit is used to detect a target log record that meets a preset condition from the log generated by the first virtual machine; the preset condition is used to indicate a log record required by the business; The transceiver unit is used to send the first information describing the target log record to the management module when the target log record is detected, so that the management module determines the collection interval corresponding to the processing level of the first information; wherein the collection interval is the time interval and / or location interval where the log record to be collected is located, and the size of the collection interval corresponding to different processing levels is different; The processing unit is further configured to collect target log segments within the collection interval from the logs generated by the first virtual machine based on the collection interval sent by the management module; and send the target log segments to the management module.

22. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 14 or claims 15 to 19.

23. A computer-readable storage medium, characterized in that: Computer instructions are stored, and when the computer instructions are executed on a computing device, the computing device is caused to perform the method according to any one of claims 1 to 14 or claims 15 to 19.

24. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 14 or claims 15 to 19.

Images