Log collection method and system

Abstract

The log collection method and system provided in this specification builds a target Pod, mounts the pre-set target directories in multiple application Pods on the physical machine to the created target Pod, and takes over the writing and collection of log data of all target logs under the target directory. The target Pod writes the log data of all application Pods into a target file, and then writes the target logs in the target file into the storage path corresponding to the application Pod in sequence according to the classification of the application Pod to which they belong, so that random writes to the storage device become sequential writes, greatly improving the write performance.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more particularly to a log collection method and system. Background Art

[0002] Container technology is a virtualization technology that saves more computing resources and is more flexible than virtual machine technology. As a key link of the application, logs can not only fully display the running status of the application, but also help developers to troubleshoot application problems. In the dynamic environment of the container platform, how to quickly and easily automatically discover, write and collect logs during the containerization process of the application is an important aspect of containerized application deployment. In particular, with the development of cloud native technology, the deployment, orchestration and maintenance of application containers are very convenient by utilizing the high availability, scheduling and expansion capabilities of cloud native, which greatly increases the elasticity of application deployment and significantly increases the number of applications deployed on a single physical machine. In the prior art, the logs of each container of the application are stored in files using independent storage processes, and logs are collected using independent log collection plug-ins. When the number of applications and the amount of logs increase significantly, the existing technology solutions are complex in operation and maintenance, have high disk write and read latency, and low user access efficiency, and can no longer meet development needs.

[0003] Therefore, it is necessary to provide a more efficient log collection method and system to improve the log writing efficiency and log collection efficiency of the application, improve the log writing and collection performance, and improve the user experience. Summary of the invention

[0004] This specification provides a more efficient log collection method and system to improve the log writing efficiency and log collection efficiency of an application, improve the log writing and collection performance, and improve the user experience.

[0005] In a first aspect, the present specification provides a log collection method, which is applied to each physical machine, including: building a target Pod, the target Pod including a log collection process; performing directory mounting and resource configuration on the target Pod; sending log write requests of multiple application Pods of the physical machine to the target Pod, the log write request including attributes of the target log and its corresponding log data, the attributes including an application identifier of the application Pod to which the target log belongs; based on the log collection process, uniformly writing the log write requests into a target file cache; and based on the log collection process, classifying the target logs in the target file according to the application identifiers and storing them in the hard disk of the physical machine.

[0006] In some embodiments, the storage path corresponding to the target file is the shared memory of the physical machine.

[0007] In some embodiments, building the target Pod includes: receiving the target Pod deployed based on a daemonset.

[0008] In some embodiments, sending the log write requests of the multiple application Pods of the physical machine to the target Pod includes: sending the log write requests to the target Pod based on a virtual file system technology.

[0009] In some embodiments, the directory mounting and resource configuration of the target Pod includes: based on the virtual file system technology, mounting a preset target directory to the target Pod, and the target log includes the files under the target directory; based on the target Pod, configuring the resources required for each application Pod, including: configuring the listening port of each application Pod; configuring the storage path corresponding to each application Pod in the physical machine; and configuring the storage space size corresponding to each application Pod in the physical machine.

[0010] In some embodiments, configuring the resources required for each application Pod based on the target Pod further includes: configuring a write threshold for each application Pod.

[0011] In some embodiments, the attributes of the target log also include a log name, a log identifier, and the storage path of the target log.

[0012] In some embodiments, storing the target log in the target file in the hard disk of the physical machine according to the classification of the application identifier includes: determining that the size of the target file exceeds a preset value; and storing the target log in the target file in the storage path corresponding to the application identifier in the order of the categories of the application identifier of the target log.

[0013] In some embodiments, the log write requests are uniformly written into the target file cache, including: writing the log write requests into the target file in the form of a key-value pair, the key in the key-value pair including the attributes of the target log, and the value in the key-value pair being the log data of the target log.

[0014] In some embodiments, the log collection method further includes: based on the log collection process, sending the target log in the target file to a remote host according to the application identification classification.

[0015] In a second aspect, the present specification also provides a log collection system, comprising at least one storage medium and at least one processor, wherein the at least one storage medium stores at least one instruction set for log writing; the at least one processor is communicatively connected to the at least one storage medium, wherein when the log collection system is running, the at least one processor reads the at least one instruction set and executes the log collection method described in the first aspect of this specification according to the instructions of the at least one instruction set.

[0016] As can be seen from the above technical solutions, the log collection method and system provided in this specification are used to write and collect logs by constructing a target Pod. The log collection method and system mount the pre-set target directory in multiple application Pods on the physical machine to the created target Pod, and the log data writing and collection of all target logs under the target directory are taken over by the target Pod. The target Pod writes the log data of all application Pods to a target file, and then writes the target logs in the target file to the storage path corresponding to the application Pod in sequence according to the classification of the application Pod to which it belongs, so that the random write to the storage device becomes a sequential write, which greatly improves the write performance. Through the target Pod, the input and output of the application container can be limited and the resources can be isolated, the maximum write capacity of a single container can be controlled, and the blockage of the write to other containers can be prevented. When the remote host collects logs, it does not need to collect from the file, and can directly collect from the target file stored in the memory, thereby reducing the input and output of the disk, reducing the processor consumption, and reducing the collection delay. At the same time, the deployment of the log collection process is independent of the application container, and the user does not need to care about the operation and maintenance of the log collection process, which improves the operation and maintenance efficiency. In addition, the creation and destruction of application containers will be notified to the target Pod by mounting, so that the target Pod has the ability to control the application container, allocate and release resources, and there will be no collection delays and log loss. The log collection method and system provided in this specification greatly improve the efficiency of application log writing and collection, improve log writing and collection performance, and improve user experience in multi-application hybrid deployment, especially in cloud native scenarios.

[0017] Other functions of the log collection method and system provided in this specification will be partially listed in the following description. Based on the description, the contents introduced by the following figures and examples will be obvious to those of ordinary skill in the art. The creative aspects of the log collection method and system provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A diagram showing an application environment of a log collection method and system provided according to an embodiment of this specification;

[0020] Figure 2 A hardware structure diagram of a physical machine provided according to an embodiment of this specification is shown;

[0021] Figure 3 A flow chart of a log collection method provided according to an embodiment of this specification is shown;

[0022] Figure 4 A flowchart of a method for directory mounting and resource configuration provided according to an embodiment of this specification is shown; and

[0023] Figure 5 A flow chart of a method for storing data in a physical machine is shown according to an embodiment of the present specification. DETAILED DESCRIPTION

[0024] The following description provides specific application scenarios and requirements of this specification, with the purpose of enabling those skilled in the art to make and use the contents of this specification. Various local modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but to the widest scope consistent with the claims.

[0025] The terms used herein are only used for the purpose of describing specific example embodiments and are not restrictive. For example, unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an" and "the" may also include plural forms. When used in this specification, the terms "include", "comprise" and / or "contain" mean that the associated integers, steps, operations, elements and / or components exist, but do not exclude the existence of one or more other features, integers, steps, operations, elements, components and / or groups or that other features, integers, steps, operations, elements, components and / or groups may be added in the system / method.

[0026] In view of the following description, these and other features of the present specification, as well as the operation and function of the related elements of the structure, and the economy of the combination and manufacture of the parts can be significantly improved. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0027] The flowcharts used in this specification illustrate the operations implemented by the system according to some embodiments in this specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. On the contrary, the operations may be implemented in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0028] For the convenience of presentation, before we begin the description, we will explain the terms that will be used in the following description as follows:

[0029] Cloud native: A method of building and running applications. It organizes a large number of physical machine resources to provide highly available and scalable service capabilities to facilitate the deployment, orchestration, and maintenance of application containers.

[0030] Physical machine: A physical server, including hardware and systems, such as Linux and Windows. The concept opposite to this in this article is container.

[0031] Container (Linux Container): Through resource isolation technology, the resources of the physical machine (such as processor, memory, disk, etc.) are divided into multiple resource pools, so that an application can use a resource pool independently without being affected by other applications. We call the divided resource pool and the application that uses this resource pool a container.

[0032] Log: In order to record its own status and operation process, the application writes this information into a file. The recorded data is called a log, and the file containing the data is called a log file.

[0033] Virtual File System (VFS): A virtual file operation interface in Linux. It is a distributed file system for network environments and an interface that allows the operating system to use different file system implementations. The Virtual File System (VFS) is an interface layer between the physical file system and the service. It abstracts all the details of each file system in Linux, making different file systems appear the same to the Linux kernel and other processes running in the system. Strictly speaking, VFS is not an actual file system. It only exists in memory and not in any external storage space. VFS is established when the system starts and disappears when the system shuts down.

[0034] Log collection in the prior art is generally divided into two types. One is collection in a container. The other is collection in a physical machine mounted file directory. The collection in the container refers to deploying the collection process as a sidecar together with the application process in the container. The logs in the container are still written to the local storage file of the physical machine, and the collection process reads the file data and sends it to the remote service host. In the method of collection in the container, storage resources are shared between containers on the same physical machine. The storage paths of logs in different containers are different on the disk. When there are many log files, writing log data of different logs is equivalent to randomly writing to different locations of the disk according to the different storage paths of the logs. Although the storage capacity corresponding to the container is allocated, the log writing of all containers must be written through the same disk, and the writing capacity of the same disk is limited. When there are many log files, the log data will be randomly and frequently switched between different storage paths of the disk when writing, which will lead to reduced writing performance. At the same time, due to the competition between the containers for input and output resources, the file writing between the containers affects each other. When a large number of writes are made to a container, the write requests of other containers may be blocked, and resource isolation cannot be achieved. Moreover, the collection process and the application process are deployed together in the container. When upgrading or expanding, the log collection program must be deployed in the container, which will make deployment and operation difficult. At the same time, since it takes time to deploy the collection program after the application container is started, the log collection program will be started later than the application container, which may cause delays or losses in log collection.

[0035] The physical machine mounted file directory collection refers to deploying an independent collection program on each physical machine to collect log files of all containers on the physical machine. Container logs are still written to the local storage file of the physical machine, and directory mounting technology is used to make the file directory visible to the log collection program. However, when there are many log files, the log data will still switch randomly and frequently between different storage paths on the disk when it is written, which will reduce the writing performance.

[0036] Figure 1The following diagram shows an application environment of a log collection system 001 provided according to an embodiment of the present specification. The log collection method provided in the present specification can be applied to Figure 1 In the application environment shown. Figure 1 As shown, the log collection system 001 may include a remote host 200 and a physical machine 400 .

[0037] The developer 002 (also called the user 002) corresponds to the host 200. The host 200 may include an API server and a controller. The physical machine 400 may communicate with the host 200 through a network. The developer 002 may operate the physical machine 400 through the API server. The controller may be used to implement various configurations for the physical machine 400.

[0038] The physical machine 400 may include a hardware device with data information processing functions and a necessary program to drive the hardware device to work. The physical machine 400 may specifically be a computer, a host of a terminal device, etc. The number of the physical machines 400 may be one or more. Figure 1 Only one physical machine 400 is shown. Physical machine 400 may include User 410, Kernel 420 and Device 430. User 410 is a user space for executing the running logic of the application. Kernel 420 is a kernel space configured to control the file system. Device 430 is a storage space for managing storage resources, including hard disk 432.

[0039] The physical machine 400 may include multiple application Pods 440. Application Pods 440 can run on User 410. Application Pods 440 are the smallest deployable unit. An application Pod 440 represents a running workload. An application Pod 440Pod refers to a collection of containers. That is, an application Pod 440 may include one or more containers. Multiple containers share system resources such as networks and jointly provide an application service. Among them, the container is a docker container. Each application Pod 440 has a unique IP address and a unique application identifier (ID). The IP address and application identifier (ID) of each application Pod 440 correspond one-to-one. Multiple containers 442 in an application Pod 440 will always run on the same physical machine 400. The containers in an application Pod 440 share the network and storage, that is, a shared set of Namespaces are implemented to replace the Namespaces of each container to achieve internal communication. Namespace refers to an object, which represents a relatively isolated namespace. Although network storage is shared, CPU and Memory are not. Multiple containers can have their own Cgroup, which means that we can limit the resource (MEM / CPU) usage of the containers in the application Pod440 individually.

[0040] The physical machine 400 may include a log file system 450. The log file system 450 may include a target Pod 452 and a driver 454. The driver 454 may run on the Kernel 420. The target Pod 452 may run on the User 410. The target Pod 452 is deployed based on the daemonset method, which can ensure that there is a copy of the target Pod 452 on each physical machine 400. The target Pod 452 may include a log collection process for writing the logs generated by the container 442 in the application Pod 440 to the hard disk 432 of the physical machine 400, and reading the log data and sending it to the host 200.

[0041] The target Pod 452 may include a target controller 453, a target Handlers 455, and a target Storage 457. The target controller 453 may implement various configurations of the target Pod 452. The target Handlers 455 is a request distribution component that may receive a data request and call different target Storages 457 for executing the data request according to the type of the data request. The data request may be any data request, such as log data writing, log data collection or reading, and the like.

[0042] In some embodiments, the log file system 450 may further include a target file 459. The storage path corresponding to the target file 459 is the shared memory of the physical machine 400. That is, the target file 459 may run on the Kernel 420.

[0043] Figure 2 A hardware structure diagram of a physical machine 400 provided according to an embodiment of this specification is shown. The physical machine 400 can execute the log collection method described in this specification. The log collection method will be described in detail in the following description. The physical machine 400 may include at least one processor 470 and at least one storage medium 480. In some embodiments, the physical machine 400 may also include a communication module 490.

[0044] The storage medium 480 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk, a read-only storage medium (ROM), or a random access storage medium (RAM). The storage medium 480 also includes at least one instruction set stored in the data storage device. The instruction is a computer program code, and the computer program code may include a program, a routine, an object, a component, a data structure, a process, a module, etc. for executing the log collection method provided in this specification.

[0045] At least one processor 470 may be in communication connection with at least one storage medium 480. At least one processor 470 is used to execute the above-mentioned at least one instruction set. When the physical machine 400 is running, at least one processor 470 reads the at least one instruction set and executes the log collection method provided in this specification according to the instructions of the at least one instruction set. The processor 470 may execute all the steps included in the log collection method P100. The processor 470 may be in the form of one or more processors. In some embodiments, the processor 470 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof. For illustration purposes only, only one processor 470 is described in the physical machine 400 in this specification. However, it should be noted that the physical machine 400 in this specification may also include multiple processors 470, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor as described in this specification, or may be performed jointly by multiple processors. For example, if the processor 470 of the physical machine 400 in this specification performs step A and step B, it should be understood that step A and step B may also be performed jointly or separately by two different processors 470 (for example, the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0046] The communication module 490 can be connected to the processor 470 for data communication between the physical machine 400 and the outside world. The communication module 490 can include at least one of a wired communication module and a wireless communication module. The communication method adopted by the wired communication module includes but is not limited to mobile high-definition link technology (HML), universal serial bus (USB), high-definition multimedia interface (HDMI), etc. The communication method adopted by the wireless communication module includes but is not limited to wireless fidelity technology (WiFi), Bluetooth communication technology, low-power Bluetooth communication technology, communication technology based on IEEE802.11s, etc.

[0047] Figure 3A flow chart of a log collection method P100 provided according to an embodiment of the present specification is shown. As described above, the physical machine 400 can execute the log collection method P100 described in the present specification. Specifically, the processor 480 can read the instruction set stored in its local storage medium, and then execute the log collection method P100 described in the present specification according to the provisions of the instruction set. The method P100 can be applied to each physical machine 400 in the log collection system 001. In some embodiments, the method P100 may include:

[0048] S120: Build target Pod452.

[0049] Specifically, step S120 may be: receiving the target Pod 452 deployed based on the daemonset mode. A target Pod 452 may be constructed on each physical machine 400 to write and collect logs generated during the operation of the container 442 in the application Pod 440 on the current physical machine 400. As mentioned above, the target Pod 452 may include a log collection process.

[0050] Specifically, the physical machine 400 can create a corresponding task object based on the build information, that is, the target Pod452. Among them, the build information may include multiple different stages (subtasks) that the target Pod452 needs to execute. The different stages are combined in parallel or serially, that is, the execution order between the subtasks. For example, in this specification, the physical machine 400 can be constructed using the job object of k8s. A job object represents the execution of a build task. That is, multi-stage task construction is achieved by using k8s job scheduling multi-containers. Specifically, the physical machine 400 can create a corresponding container according to each stage in the build information. These containers constitute the target Pod452 and are uniformly managed by a job object.

[0051] In some embodiments, after the target Pod 452 is constructed, the physical machine 400 may also perform directory mounting and resource configuration on the target Pod 452. Figure 3 As shown, the method P100 may further include:

[0052] S130: Perform directory mounting and resource configuration on the target Pod 452.

[0053] The mounting may be to mount the mount directories of different containers in the same application Pod 440 in the same physical machine 400 to a specified directory, thereby sharing the files in the mount directory. Mounting can achieve resource sharing. Mounting the file directory in the application Pod 440 to the target Pod 452 can make the files under the mounted directory visible to the target Pod 452, so that the files under the mounted directory can be shared to the target Pod 452. Figure 4 A flow chart of a method for directory mounting and resource configuration provided according to an embodiment of this specification is shown. Figure 4 Step S130 is shown. Figure 4 As shown, step S130 may include:

[0054] S132: Based on the virtual file system technology, the preset target directory is mounted to the target Pod 452 .

[0055] An application Pod440 may generate a variety of different logs during its operation. Among these logs, there is a log that needs to be permanently stored and output externally, for example, output to the host 200. And some logs do not need to be permanently stored and output. In order to ensure that the logs that need to be permanently stored and the logs that need to be output externally can be normally permanently stored and output, we need to mount the directories of these logs to the target Pod452 so that these logs are visible to the target Pod452. For the convenience of description, we define these logs that need to be permanently stored and output as target logs. The directory of the target log is defined as the target directory. The target log can be a file under the target directory. The target directory is a pre-set directory, such as / home, / logs, and so on.

[0056] As mentioned above, the virtual file system technology can enable interfaces for docking different file systems. Specifically, the physical machine 400 can send the mount request to the driver 454 based on the virtual file system technology. The driver 454 then sends the mount request to the target Pod 452 to mount the target directory to the target Pod 452. When the application Pod 440 is started, the physical machine 400 can mount the target directory to the target Pod 452 based on the virtual file system technology. The physical machine 400 can mount the target directory in each application Pod 440 of multiple application Pods 440 to the target Pod 452. As mentioned above, an application Pod 440 may include one or more containers. The physical machine 400 can mount the target directories of all containers in each application Pod 440 to the target Pod 452.

[0057] like Figure 4 As shown, step S130 may also include:

[0058] S134: configuring the resources required for each application Pod 440 based on the target Pod 452 .

[0059] As mentioned above, in step S132, the physical machine 400 mounts the target directory in the application Pod 440 to the target Pod 452, so that the target directory in the application Pod 440 is visible to the target Pod 452. After receiving the mount request, the target Pod 452 can perform resource configuration on multiple application Pods 440 based on the target Pod 452 to determine the storage resources required for each application Pod 440, such as the storage location and storage space size corresponding to each application Pod 440. Specifically, step S134 may include:

[0060] S134-2: Configure the listening port of each application Pod440.

[0061] In some embodiments, the target Pod452 may also include a monitoring process to monitor the container in the application Pod440 on the physical machine 400 to monitor whether there is a log write request or a log read request in the application Pod440. The log read request may also be called a log acquisition request or a log collection request. The specific monitoring method and the expression form of the monitoring result can be determined according to actual needs, such as real-time monitoring, monitoring according to a preset time interval, etc., and for example, a specific numerical value can also be used to characterize whether there is a log write request or a log read request in the application Pod440. In order to obtain the monitoring result of each application Pod440, the physical machine 400 needs to configure the monitoring port for each application Pod440. Each application Pod440 is configured with a unique monitoring port. As mentioned above, each application Pod440 has a unique IP address and a unique application identifier. The monitoring port, IP address and application identifier of each application Pod440 correspond one to one.

[0062] S134 - 4 : configuring the storage path corresponding to each application Pod 440 in the physical machine 400 .

[0063] The storage path may be the storage location of the target log under the target directory in each application Pod 440 in the hard disk 432 of the physical machine 400. As previously mentioned, storage resources may be shared between different containers in the same application Pod 440. That is, the target log under the target directory between different containers in the same application Pod 440 may be stored in the same location, i.e., under the same storage path. The physical machine 400 may configure the corresponding storage path in the hard disk 432 of the physical machine 440 for each application Pod 440 according to different application Pods 440.

[0064] S134 - 6 : configuring the storage space size corresponding to each application Pod 440 in the physical machine 400 .

[0065] The physical machine 400 can not only configure the storage path corresponding to each application Pod 440 in the hard disk 432 of the physical machine 440 , but also configure the size of the storage space corresponding to each application Pod 440 in the hard disk 432 of the physical machine 400 .

[0066] In some embodiments, step S134 may further include:

[0067] S134 - 8 : Configure the write threshold of each application Pod 440 .

[0068] In some embodiments, the physical machine 400 can also configure the write threshold of each application Pod440 based on the target Pod452 to limit the maximum write capacity of each application Pod440. The configuration of the write threshold can limit the input and output of each application Pod440 and isolate resources, control the maximum write capacity of a single application Pod440, prevent a large number of writes from one application Pod440 from causing blockage and affecting the write of other application Pod440, thereby avoiding log loss or delay. The configuration of the write threshold can avoid multiple application Pod440 from earning multiple input and output resources, so that the target logs of multiple application Pod440 can be written evenly, improve log writing performance, and thus improve log writing efficiency.

[0069] In some embodiments, before step S132, step S130 may further include: performing command configuration on the target Pod 452. Command configuration refers to starting a process in a corresponding container in the target Pod 452 to execute a corresponding task according to a configured command.

[0070] When the target Pod452 completes the directory mounting and resource configuration, it returns success, and the application Pod440 is mounted and started.

[0071] In summary, step S130 can mount the target directories of all application Pod440 to the target Pod452, so that the target logs under the target directories of all application Pod440 are visible to the target Pod452. The target Pod452 can monitor the target directories of all application Pod440. The creation and destruction of each application Pod440 will be notified to the target Pod452 through the directory mounting, so that the target Pod452 has the ability to control the application Pod440, and can allocate and release resources without collection delays and log loss.

[0072] like Figure 3 As shown, the method P100 may further include:

[0073] S140 : Sending log writing requests of multiple application Pods 440 of the physical machine 400 to the target Pod 452 .

[0074] As mentioned above, when the application Pod 440 is running, the target Pod 452 can monitor multiple application Pods 440. The target Pod 452 can monitor the status of each container running in the application Pod 440 in real time, and read the target log information corresponding to each container, that is, the log information under the target directory.

[0075] Each application Pod440 corresponds to a process of an application program. During the operation of the process in the application Pod440, the application Pod440 will generate a log file. The log file records the running status, running time and other information of the process corresponding to the application Pod440. The log file generated by the application Pod440 can be generated in real time. Each application Pod440 can generate multiple log files. The multiple log files may include the target log. When the application Pod440 generates the target log, the physical machine 400 can send the log write request to the target Pod452 based on the virtual file system technology. Specifically, the physical machine 400 can forward the log write request to the target Pod452 through the driver 454 based on the virtual file system technology. The target Pod452 receives the log write request and calls the corresponding target Storage457 interface to perform the write process.

[0076] The log write request may include the properties of the target log and its corresponding log data. The properties of the target log may include the application identifier of the application Pod440 to which the target log belongs. As mentioned above, each application Pod440 has a unique application identifier. In some embodiments, the properties of the target log may also include the log name, log identifier and storage path of the target log. The storage path of the target log may be the storage path corresponding to the application Pod440 to which the target log belongs. The log identifier may be the unique identity identifier corresponding to the target log. The log data may be the content data in the target log.

[0077] like Figure 3 As shown, the method P100 may further include:

[0078] S160: Based on the log collection process, the log writing requests are uniformly written into the target file 459 for cache.

[0079] In order to improve the log writing performance, the target logs generated in all application Pod440 can be first written uniformly into the same file. In this specification, the target logs generated in all application Pod440 can be first written uniformly into the target file 459. Therefore, when the target Pod452 writes the target logs generated by different application Pod440, it first writes the different target logs uniformly into the same target file 459 instead of directly writing the target logs into the corresponding storage path, so as to avoid the target Pod452 frequently switching back and forth between different locations on the disk when writing, thereby changing random writing into sequential writing, increasing the writing speed, improving the writing performance, and thus improving the efficiency of log collection.

[0080] In some embodiments, the log write request is uniformly written into the target file 459 for cache, which may include: writing the log write request into the target file 459 in the form of a key-value pair. In order to improve the data writing capability, the LSM (Log Structure Merge) method may be adopted in the storage layer. The target Pod 452 may organize the log write request into the form of a key-value pair and write it into the target file 459. The key in the key-value pair includes the attributes of the target log, such as the key <file name, inode, offset>. The file name may be a collection of the application identifier of the application Pod 440 to which the target log belongs and the log name of the target log. Inode may be the log identifier of the target log. Offset may be the storage path corresponding to the target log. The value in the key-value pair is a data byte stream of the log data of the target log.

[0081] In some embodiments, the sequential writing of the log write requests into the target file may also be other data storage methods, such as MYSQL, etc.

[0082] like Figure 3 As shown, the method P100 may further include:

[0083] S170: Based on the log collection process, the target log in the target file 459 is classified and stored in the hard disk 432 of the physical machine 400 according to the application identifier.

[0084] Figure 5 A flow chart of a method for storing in a physical machine 400 provided according to an embodiment of the present specification is shown. Figure 5 Corresponding to step S170. Figure 5 As shown, step S170 may include:

[0085] S172: Determine whether the size of the target file 459 exceeds a preset value.

[0086] As mentioned above, in step S160, the target logs are first written uniformly into a target file 459 for caching. The target file 459 may be a file in a memory system. The storage space of the target file 459 is limited. When the size of the target logs in the target file 459 exceeds a preset value, all target logs in the target file 459 may be stored in the hard disk 432 of the physical machine 400.

[0087] The preset value may be any value. The preset value may be obtained based on experience. The preset value may also be obtained based on an experimental method. The preset value may also be obtained based on machine learning. In some embodiments, the size of the preset value may be proportional to the number of application Pods 440 running on the physical machine 400. In some embodiments, the size of the preset value may be determined by the memory size of the physical machine 400.

[0088] like Figure 5 As shown, step S170 may also include:

[0089] S174: storing the target log in the target file 459 into a storage path corresponding to the application identifier in the order of the categories of the application identifier of the target log.

[0090] When the size of the target file 459 exceeds the preset value, the target Pod 452 can classify all the target logs in the target file 459 according to the application identifier of the application Pod 452 to which it belongs, and according to the classification result, the target logs are sequentially stored in the corresponding storage path according to the category of the application identifier. For example, the target logs in the target file 459 belong to N application Pods 440 respectively. Wherein, N is an integer greater than 1. In step S174, the target Pod 452 can first store all the target logs corresponding to the first application Pod 440 in the N application Pods 440 in its corresponding storage path. Then, all the target logs corresponding to the second application Pod 440 in the N application Pods 440 are stored in its corresponding storage path. And so on. The target Pod 452 stores the target logs sequentially according to the category of the application identifier corresponding to the target log, rather than randomly. Therefore, the target Pod 452 can store all the target logs of the next category in the same storage path after all the target logs located in the same storage path are stored in a centralized manner. Therefore, when the target Pod 452 performs permanent storage on the target log, it does not frequently switch between different storage paths, but stores them in a centralized and sequential manner, thereby improving the write performance and write efficiency.

[0091] like Figure 3 As shown, in some embodiments, the method P100 may further include:

[0092] S190: Based on the log collection process, the target log in the target file 459 is sent to the remote host 200 according to the application identification classification.

[0093] In some embodiments, the target Pod 452 may receive a log reading request. The log reading request may be sent by the host 200 to the physical machine 400. The target Pod 452 may send the target log in the target file 459 to the remote host 200 according to the application identification classification based on the log reading request.

[0094] The target Pod452 can classify all target logs in the target file 459 according to the application identifier of the application Pod452 to which they belong, and according to the classification result, send the target logs in the target file 459 to the host 200 according to the category of the application identifier. For example, the target logs in the target file 459 belong to N application Pods 440 respectively. Wherein, N is an integer greater than 1. In step S190, the target Pod452 can send N types of target logs corresponding to the N application Pods 440 to the host 200. Since the target file 459 is located in the memory. Therefore, the target Pod452 can send the target log directly from the memory to the host 200, instead of reading it from the hard disk 432 and then sending it, thereby reducing the input and output of the hard disk, reducing CPU consumption and acquisition delay.

[0095] In some embodiments, the target Pod 452 may send the target log in the target file 459 to the remote host 200 in real time according to the application identification classification. In some embodiments, the target Pod 452 may send the target log in the target file 459 to the remote host 200 according to the application identification classification at regular intervals. In some embodiments, when the size of the target file 459 reaches a preset value, the target Pod 452 may send the target log in the target file 459 to the remote host 200 according to the application identification classification.

[0096] In summary, the log collection method P100 and system 001 provided in this specification, by constructing a target Pod452, are used to write, collect, and perform other operations on the target logs under the target directories of multiple application Pod452. The log collection method P100 and system 001 can mount the pre-set target directories in multiple application Pod440 on the physical machine 400 to the created target Pod452, and the target Pod452 will take over the writing and collection of log data of all target logs under the target directory. The target Pod452 writes the log data of all application Pod440 into a target file 459. When the size of the target file 459 exceeds the preset value, the target logs in the target file 459 are classified according to the application identifier of the application Pod440 to which they belong, and according to the classification result, the target logs are sequentially written into the storage path corresponding to the application Pod440 according to the category of the application identifier, so that the random write to the storage device becomes a sequential write, which greatly improves the write performance. The log collection method P100 and system 001 can limit the input and output of the application Pod440 and isolate resources through the target Pod452, control the maximum write capacity of a single Pod440, and prevent the writing of other Pod440 from being blocked. When the remote host 200 performs log collection, there is no need to read the log data from the hard disk 432, and it can be directly collected from the target file 459 stored in the memory, thereby reducing the input and output of the hard disk 432, reducing processor consumption, and reducing collection delay. At the same time, the deployment of the log collection process is independent of the application Pod440, and the user does not need to care about the operation and maintenance of the log collection process, thereby improving the operation and maintenance efficiency. In addition, the creation and destruction of the application Pod440 will be notified to the target Pod452 by mounting, so that the target Pod452 has the ability to control the application Pod440, and can allocate and release resources, without collection delays and log loss. The log collection method P100 and system 001 provided in this manual greatly improve the efficiency of application log writing and collection, improve log writing and collection performance, and improve user experience in multi-application hybrid deployment, especially in cloud-native scenarios.

[0097] On the other hand, the present specification provides a non-transitory storage medium storing at least one set of executable instructions for log collection. When the executable instructions are executed by the processor, the executable instructions instruct the processor to implement the steps of the log collection method P100 described in the present specification. In some possible implementations, various aspects of the present specification can also be implemented in the form of a program product, which includes a program code. When the program product is run on a computing device (such as a physical machine 400), the program code is used to enable the computing device to execute the steps of the log collection method P100 described in the present specification. The program product for implementing the above method can use a portable compact disk read-only memory (CD-ROM) to include program code and can be run on a computing device. However, the program product of the present specification is not limited to this. In the present specification, the readable storage medium can be any tangible medium containing or storing a program, which can be used by the instruction execution system or used in combination with it. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. Program code for performing the operations of the present specification may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the computing device, partially on the computing device, as a stand-alone software package, partially on the computing device and partially on a remote computing device, or entirely on the remote computing device.

[0098] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0099] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements and modifications to the embodiments. These changes, improvements and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0100] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0101] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. This is also true when the content of each secondary embodiment is less than all the features of a single aforementioned disclosed embodiment.

[0102] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated herein by reference in its entirety for all purposes, except for any prosecution document history related thereto, any equivalent that may be inconsistent or conflicting with this document, or any equivalent prosecution document history that may have a limiting effect on the broadest scope of the claims now or later associated with this document. For example, if there is any inconsistency or conflict between the description, definition, and / or use of a term associated with any of the incorporated materials and the term, description, definition, and / or use associated with this document, the term in this document shall control.

[0103] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. A log collection method, applied to each physical machine, including: Build a target Pod, where the target Pod includes a log collection process; Performing directory mounting and resource configuration on the target Pod, wherein the directory mounting includes: mounting the file directories in the multiple application Pods of the physical machine into the target Pod; Sending log write requests of the multiple application Pods to the target Pod, the log write request including properties of the target log and its corresponding log data, the properties including the application identifier of the application Pod to which the target log belongs; Based on the log collection process, the log writing request is uniformly written into a target file cache; and Based on the storage requirement corresponding to the target file, the target log in the target file is classified and stored in the hard disk of the physical machine according to the application identifier.

2. The log collection method according to claim 1, wherein: The storage path corresponding to the target file is the shared memory of the physical machine.

3. The log collection method according to claim 1, wherein: The target Pod is constructed as follows: Receive the target Pod deployed based on the daemonset mode.

4. The log collection method according to claim 1, wherein: The sending the log writing requests of the multiple application Pods of the physical machine to the target Pod includes: The log write request is sent to the target Pod based on the virtual file system technology.

5. The log collection method according to claim 1, wherein: The target log includes files under the file directory; the directory mounting and resource configuration of the target Pod also includes: Configuring resources required for each application Pod based on the target Pod includes: Configure the listening port of each application Pod; Configuring a storage path corresponding to each application Pod in the physical machine; and The storage space size corresponding to each application Pod in the physical machine is configured.

6. The log collection method according to claim 5, wherein: The configuring the resources required for each application Pod based on the target Pod also includes: Configure the write threshold of each application Pod.

7. The log collection method according to claim 5, wherein: The attributes of the target log also include a log name, a log identifier, and the storage path of the target log.

8. The log collection method according to claim 7, wherein: Based on the storage requirement corresponding to the target file, the target log in the target file is classified and stored in the hard disk of the physical machine according to the application identifier, including: Determining that the size of the target file exceeds a preset value; and According to the order of the categories of the application identifiers of the target logs, the target logs in the target files are stored in the storage paths corresponding to the application identifiers.

9. The log collection method according to claim 1, wherein: The step of writing the log write requests uniformly into a target file cache comprises: The log write request is written into the target file in the form of a key-value pair, where the key in the key-value pair includes the attribute of the target log, and the value in the key-value pair is the log data of the target log.

10. The log collection method according to claim 1, wherein: Also includes: Based on the log collection process, the target log in the target file is sent to a remote host according to the application identification classification.

11. A log collection system, comprising: At least one storage medium storing at least one instruction set for log writing; as well as at least one processor, in communication with the at least one storage medium, Wherein, when the log collection system is running, the at least one processor reads the at least one instruction set and executes the log collection method according to any one of claims 1 to 10 according to the instructions of the at least one instruction set.

Images