Log collection method and device, electronic equipment and medium

Abstract

The application discloses a log collection method and device, electronic equipment and medium, and relates to the technical field of computers.The method comprises the following steps: scanning log files under a parent directory, and determining log files to be collected in the parent directory, wherein the parent directory is used for storing a log directory, the log directory is used for storing log files, and the log directory is named based on a container identifier; and collecting the content in the log files to be collected.The method stores the log directory named based on the container identifier in the parent directory, so that the log files are stored in the same parent directory, and in the case that a new log directory is generated due to container redeployment, the newly generated log directory can be accurately and quickly located, the newly generated log file is quickly located, the coupling between the container and the log collection is reduced, and the log collection tool or the log collection code does not need to be upgraded; and the method does not need to introduce a third-party framework, has low invasiveness, and has high timeliness.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a log collection method, apparatus, electronic device, and medium. Background Technology

[0002] Logs have always been crucial for stable system operation and troubleshooting. In microservice architectures, the large number of containers and their rapidly changing nature make logging increasingly important. However, in container-based microservice architectures, traditional log collection methods struggle to adapt to the dynamic nature of container applications due to the uncertainty of service location, number, and timing. The rule for generating log files in container-based microservice architectures is to create business logs in folders named with unique container identifiers (such as IDs). To collect business logs, the folder must first be identified to find the currently valid logs. However, container IDs change due to container redeployment, requiring upgrades to log collection tools or middleware to locate log files. Upgrading log collection tools or middleware inevitably increases manpower and time costs, and the overall process is cumbersome and prone to errors. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, embodiments of the present invention provide a log collection method, apparatus, electronic device, and medium.

[0004] In a first aspect, embodiments of the present invention provide a log collection method, comprising: scanning log files under a parent directory to determine log files to be collected in the parent directory; the parent directory is used to store log files, and the log directory is named based on a container identifier; and collecting the contents of the log files to be collected.

[0005] In an optional embodiment, scanning the log files in the parent directory to determine the log files to be collected in the parent directory includes: obtaining a local persistent file, the local persistent file being used to record the path information of the collected log files; scanning the log files in the parent directory, and determining the log files to be collected in the parent directory based on the local persistent file; if the content of the log files to be collected has been collected, the method further includes: writing the path information of the collected log files to be collected into the local persistent file.

[0006] In an optional embodiment, writing the path information of the collected log files to be collected into the local persistent file includes: obtaining a doubly linked list; writing the path information of the collected log files to be collected into the doubly linked list; and writing the path information stored in the doubly linked list into the local persistent file, so as to perform local persistent storage of the path information stored in the doubly linked list.

[0007] In an optional embodiment, the method further includes: setting naming rules for log files in the log directory to determine the names of log files according to the naming rules;

[0008] The step of determining the log files to be collected in the parent directory based on the local persistent file includes: determining the log files to be collected in the parent directory based on the names of the log files in the parent directory and the local persistent file.

[0009] In an optional embodiment, setting the naming rules for log files in the log directory to determine the name of the log file according to the naming rules includes: determining the service identifier of the service to which the log file belongs and the time information corresponding to the log file, wherein the time information is used to indicate the generation time of the log file; and generating the name of the log file based on the service identifier and the time information.

[0010] In an optional embodiment, the method further includes configuring the log scan start time and / or the log scan period;

[0011] The step of determining the log files to be collected in the parent directory based on the names of the log files in the parent directory and the local persistent files includes: determining the log files to be collected in the parent directory based on the log scan start time and / or log scan cycle, as well as the names of the log files in the parent directory and the local persistent files.

[0012] In an optional embodiment, determining the log files to be collected in the parent directory based on the log scan start time and / or log scan cycle, the naming rules of the log files, and the local persistent file includes: determining log files in the parent directory whose generation time is between the log scan start time and the current time based on the names of the log files in the parent directory, or determining log files in the parent directory whose generation time is within the log scan cycle based on the names of the log files in the parent directory; selecting the log files whose generation time is between the log scan start time and the current time or whose generation time is within the log scan cycle as target log files; and filtering the log files to be collected from the target log files based on the local persistent file.

[0013] Secondly, embodiments of the present invention provide a log collection device, comprising:

[0014] The scanning module is used to scan the log files under the parent directory and determine the log files to be collected in the parent directory; wherein, the parent directory is used to store log directories, the log directories are used to store log files, and the log directories are named based on container identifiers; the collection module is used to collect the contents of the log files to be collected.

[0015] In an optional embodiment, the scanning module is further configured to: obtain a local persistent file, the local persistent file being used to record the path information of the collected log files; scan the log files in the parent directory, and based on the local persistent file, determine the log files to be collected in the parent directory;

[0016] The device also includes a local persistence module, which is used to write the path information of the collected log file to the local persistence file after the content of the log file to be collected is completed.

[0017] In an optional embodiment, the local persistence module is further configured to: obtain a doubly linked list; write the path information of the log file to be collected after collection to the doubly linked list; and write the path information stored in the doubly linked list to the local persistence file, so as to perform local persistent storage of the path information stored in the doubly linked list.

[0018] In an optional embodiment, the device further includes a configuration module for: setting naming rules for log files in the log directory, so as to determine the name of the log file according to the naming rules;

[0019] The scanning module is also used to: determine the log files to be collected in the parent directory based on the names of the log files in the parent directory and the local persistent files.

[0020] In an optional embodiment, the configuration module is further configured to: determine the service identifier of the service to which the log file belongs and the time information corresponding to the log file, wherein the time information is used to indicate the generation time of the log file; and generate the name of the log file based on the service identifier and the time information.

[0021] In an optional embodiment, the configuration module is further configured to determine the log scan start time and / or log scan cycle;

[0022] The scanning module is also used to: determine the log files to be collected in the parent directory based on the log scan start time and / or log scan cycle, as well as the names of the log files in the parent directory and the local persistent files.

[0023] In an optional embodiment, the scanning module is further configured to: determine, based on the name of the log file in the parent directory, the log file whose generation time is between the start time of the log scan and the current time, or based on the name of the log file in the parent directory, determine the log file whose generation time is within the log scan cycle, and use the log file whose generation time is between the start time of the log scan and the current time or the log file whose generation time is within the log scan cycle as the target log file; and filter out the log file to be collected from the target log file based on the local persistent file.

[0024] Thirdly, embodiments of the present invention also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the log collection method of any of the above embodiments.

[0025] Fourthly, embodiments of the present invention also provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the log collection method of any of the above embodiments.

[0026] One embodiment of the above invention has the following advantages or beneficial effects:

[0027] The log collection method of this invention sets a parent directory to store log directories named based on container identifiers. This ensures that all log files under the same parent directory are stored there. When a new log directory is generated due to container redeployment, the newly generated log directory can be accurately and quickly located, thereby rapidly locating the newly generated log files. This reduces the coupling between the container and log collection. During business upgrades or container upgrades, there is no need to upgrade the log collection tool or code. When collecting log files, the method scans the log files in the parent directory to determine the log files to be collected. Then, the contents of the log files to be collected are read to complete the collection. This method does not require the introduction of third-party frameworks, has low invasiveness, and high efficiency.

[0028] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0029] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0030] Figure 1 A schematic diagram illustrating the main flow of the log collection method according to an embodiment of the present invention is shown;

[0031] Figure 2 A schematic diagram of the log collection method according to an embodiment of the present invention is shown;

[0032] Figure 3 A schematic diagram of the flow of a log collection method according to another embodiment of the present invention is shown;

[0033] Figure 4 A schematic diagram of the log collection device according to an embodiment of the present invention is shown;

[0034] Figure 5 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0035] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0036] Currently, in container-based microservice architectures, log files are stored in folders named with unique container identifiers (such as IDs). Therefore, the location of log files needs to be confirmed using the container's unique identifier (such as ID). For example, in ` / fpath / log / 13sdada / yewu1.log`, `yewu1.log` is the log file, while `13sdada` is the container's unique identifier (such as ID). Only after determining the location of the log file can log collection tools or log collection middleware locate the log file and collect data. However, each time a container starts, a new container ID is generated. Therefore, when a container restarts or is redeployed, the log collection tool (or log collection middleware) or log collection code must be upgraded together to collect log files. The overall process is cumbersome and increases manpower and time costs.

[0037] To address, or at least partially address, the aforementioned technical problems, this invention provides a log collection method. This method uses a parent directory to store log directories named based on container identifiers, ensuring that all log files within the same parent directory are stored there. When the container is redeployed, the new log directory remains in the designated parent directory. Therefore, when a new log directory is generated during container redeployment, the newly generated log directory can be accurately and quickly located, thereby rapidly locating the newly generated log files. This reduces the coupling between the container and log collection, and eliminates the need to upgrade the log collection tool or code during business upgrades or container upgrades. The log collection method of this invention will be described in detail below with reference to the accompanying drawings.

[0038] Figure 1 A schematic diagram illustrating the main flow of the log collection method according to an embodiment of the present invention is shown. Figure 1 As shown, the method includes:

[0039] Step S101: Set a parent directory for storing log directories, which are used to store log files and are named based on container identifiers.

[0040] In this embodiment, the parent directory refers to the directory one level above the log directory named after the container ID, which contains one or more log directories. For example, the parent directory path is / fpath / log / , which contains all log directories named after their container IDs.

[0041] The log directory is used to store log files. It can be understood as a log folder, containing one or more log files. The log directory corresponds to a container and is named based on the container's unique identifier (such as an ID).

[0042] For example, if the current service starts two containers, container a and container b, with container a having the ID ewds and container b having the ID sdfi, then the configured parent directory / fpath / log / will contain two folders named ewds and sdfi, both named after their container IDs. These ewds and sdfi folders constitute the log directories. Log files generated by container a are stored in the ewds folder, and log files generated by container b are stored in the sdfi folder.

[0043] When a service is redeployed, for example, if the redeployed service starts three containers: container a, container b, and container c, and container a's ID is updated to weqq, container b's ID is updated to ddjx, and container c's ID is newly generated as ucjj, then the parent directory / fpath / log / contains five folders, i.e., five log directories: ewds, sdfi, weqq, ddjx, and ucjj. Log files generated by container a are stored in the weqq folder, log files generated by container b are stored in the ddjx folder, and log files generated by container c are stored in the ucjj folder.

[0044] Step S102: Scan the log files in the parent directory to determine the log files to be collected in the parent directory.

[0045] In this embodiment, a scanning cycle can be set to periodically scan the log files in the parent directory. Alternatively, user instructions can be received through a visual interactive interface to scan the log files in the parent directory according to the user's instructions. This invention does not impose any limitations on this.

[0046] When a service starts a container for the first time and scans the log files in the parent directory for the first time, all log files in the parent directory are uncollected log files; that is, all log files in the parent directory are log files to be collected. When a service starts a container for the first time, scans the log files in the parent directory for subsequent times, or the container is redeployed, the parent directory contains both collected log files and log files to be collected. Therefore, embodiments of the present invention can mark or record the collected log files during the collection process to facilitate subsequent differentiation between collected and uncollected log files.

[0047] In optional embodiments, the collected log files can be marked in the following ways: changing the name of the collected log files according to a preset format; or changing the storage format of the log files; or moving the collected log files to a specific folder under the parent directory.

[0048] In other optional embodiments, the names or path information of the collected log files (including the names of the collected log files) can be recorded in a preset data table or file. When scanning log files in the parent directory, the scanned log files are compared with the log files recorded in the data table to determine whether the scanned log files are collected log files. If not, it means that the log file is a log file to be collected.

[0049] Step S103: Collect the contents of the log file to be collected.

[0050] In this step, the content of the log file to be collected, along with the name of the log file, can be delivered as a whole in a producer manner so that consumers can consume it according to their needs.

[0051] The log collection method of this invention sets a parent directory to store log directories named based on container identifiers. This ensures that all log files under the same parent directory are stored there. When a new log directory is generated due to container redeployment, the newly generated log directory can be accurately and quickly located, thereby rapidly locating the newly generated log files. This reduces the coupling between the container and log collection. During business upgrades or container upgrades, there is no need to upgrade the log collection tool or code. When collecting log files, the method scans the log files in the parent directory to determine the log files to be collected. Then, the contents of the log files to be collected are read to complete the collection. This method does not require the introduction of third-party frameworks, has low invasiveness, and high efficiency.

[0052] In an optional embodiment, the method may set a local persistent file to record the path information of the collected log files. This path information is the complete path of a folder named after the container identifier. For example, the path information recorded in the local persistent file is / fpath / log / 13sdada / yewu1.log, where / fpath / log / is the parent directory, 13sdada is the log directory named after the container identifier (such as ID), and yewu1.log is the log file.

[0053] When scanning log files in the parent directory to determine the log files to be collected, the path information of the collected log files recorded in the local persistent file can be used to filter out the log files to be collected, and then the log files to be collected can be read. After reading is complete, the path information of the log files to be collected is written to the local persistent file.

[0054] In an optional embodiment, the log collection method may further include, when writing the path information of the collected log files to a local persistent file:

[0055] Get the doubly linked list;

[0056] Write the path information of the log files to be collected after collection into the doubly linked list;

[0057] The path information stored in the doubly linked list is written to the local persistent file to perform local persistent storage of the path information stored in the doubly linked list.

[0058] Among them, such as Figure 2As shown, a doubly linked list, also called a doubly linked list, is a type of linked list. Each data node in it has two pointers, one pointing to the direct successor and the other to the direct predecessor. Therefore, starting from any node in the doubly linked list, it is easy to access its predecessor and successor nodes. In this embodiment, obtaining the doubly linked list can be either obtaining a pre-created doubly linked list or creating a new one. Specifically, creating the doubly linked list can be done during the first scan. The first scan not only refers to the first scan when the container is first deployed, but also includes the first scan after the container is restarted (the restart reason may be a server crash, causing program abnormalities). Then, during non-first scans, the doubly linked list created during the first scan is obtained. Alternatively, the doubly linked list can be created each time the parent directory is scanned and it is determined that there are log files to be collected, regardless of whether it is the first scan. In an optional embodiment, the number of nodes in the created doubly linked list can be a pre-set fixed value, which is also equal to the number of log files to be collected. For example, if the number of log files to be collected is 60, then a doubly linked list containing 60 nodes is created.

[0059] After obtaining the doubly linked list, the path information of the collected log files to be collected is written into the doubly linked list, which can be written sequentially according to the collection order. Then, the path information stored in the doubly linked list is written to a local persistent file for local persistent storage. Writing the path information of the collected log files to the doubly linked list is an in-memory operation, which is fast, does not affect log collection efficiency, and has high real-time performance. Writing the path information stored in the doubly linked list to the local persistent file allows for a single disk write, saving system resources and avoiding disk fragmentation and disk failures caused by multiple disk writes. The path information stored in the doubly linked list can be written to the local persistent file after all nodes in the doubly linked list have their path information written, or it can be written before the next scan; this invention does not impose any limitations.

[0060] Figure 3 A flowchart illustrating another embodiment of the log collection method of the present invention is shown, as follows: Figure 3 As shown, the method includes:

[0061] Step S301: Set a parent directory for storing log directories, wherein the log directory is used to store log files and is named based on the container identifier;

[0062] Step S302: Set a local persistent file, which is used to record the path information of the collected log files;

[0063] Step S303: Set the naming rules for the log files in the log directory, so as to determine the names of the log files according to the naming rules;

[0064] Step S304: Scan the log files in the parent directory, and determine the log files to be collected in the parent directory based on the names of the log files in the parent directory and the local persistent files;

[0065] Step S305: Create a doubly linked list;

[0066] Step S306: Collect the contents of the log file to be collected;

[0067] Step S307: Write the path information of the collected log files to be collected into the doubly linked list;

[0068] Step S308: Write the path information stored in the doubly linked list into the local persistent file to perform local persistent storage of the path information stored in the doubly linked list.

[0069] The parent directory refers to the directory one level above the log directory named after the container ID. This parent directory contains one or more log directories. For example, the parent directory path is / fpath / log / , which contains all log directories named after the container ID.

[0070] A log directory can be understood as a log folder, which contains one or more log files. The log directory corresponds to a container, is named based on the container's unique identifier (such as an ID), and stores the log files generated by that container.

[0071] Log files can be named according to pre-defined naming rules, which can be flexibly configured according to scenario requirements. For example, a log file name can be generated based on the business identifier of the service to which the log file belongs and the corresponding time information. The time information indicates the creation time of the log file. For instance, if container a's ID is ewds, and the business identifier of the service to which container a belongs is yewu1, the name of the log file generated by container a can include yewu1. If the creation time of the log file is 20220607101011, then the name of the log file is yewu1-20220607101011.log, and the path information of the log file is / fpath / log / ewds / yewu1-20220607101011.log.

[0072] In this embodiment, when scanning the parent directory, log files can be recursively queried. Regular expressions are used to compare the names of the scanned log files with the names of the log files recorded in the local persistent files to filter out the log files to be collected.

[0073] After identifying the log files to be collected, the number of log files is counted, and a doubly linked list is created based on this number. For example, if there are 60 log files to be collected, a doubly linked list with 60 nodes is created. Then, the contents of the log files to be collected are read. After reading, the path information of the log files to be collected is written to the doubly linked list. Finally, the path information in the doubly linked list is written to a local persistent file. This invention does not limit the scope of this method; either the path information in the doubly linked list can be written to the local persistent file after all nodes in the doubly linked list have their path information written, or it can be written to the local persistent file before the next scan.

[0074] In an optional embodiment, the doubly linked list can also be obtained from a previously created doubly linked list. The number of nodes in the obtained doubly linked list may be less than the number of log files to be collected. When writing path information to the doubly linked list, a new node can be added to the end of the doubly linked list and the node at the head can be deleted. When deleting the node at the head, the path information in that node is written to a local persistent file.

[0075] In an optional embodiment, the log collection method can further configure the log scan start time and / or log scan cycle, and determine the log files to be collected in the parent directory based on the log scan start time and / or log scan cycle, as well as the names of the log files in the parent directory and the local persistent files.

[0076] The process of determining the log files to be collected in the parent directory based on the log scan start time and / or log scan cycle, as well as the naming rules of the log files and the local persistent files, may include:

[0077] Based on the name of the log file in the parent directory, determine the log file in the parent directory whose generation time is between the start time of the log scan and the current time, or based on the name of the log file in the parent directory, determine the log file in the parent directory whose generation time is within the log scan cycle, and use the log file whose generation time is between the start time of the log scan and the current time or the log file whose generation time is within the log scan cycle as the target log file;

[0078] Based on local persistent files, the log files to be collected are filtered out from the target log files.

[0079] Specifically, when the current scan is the first scan (the first scan refers not only to the first scan when the container is first deployed, but also to the first scan after the container is restarted; in this case, the local persistent file will contain the path information of previously collected log files), based on the name of the log files in the parent directory, log files whose generation time is between the start time of the log scan and the current time are identified, and these log files are used as target log files; based on the local persistent file, the log files to be collected are filtered from the target log files; when the current scan is not the first scan, based on the name of the log files in the parent directory, log files whose generation time is within the log scan period are identified, and these log files are used as log files to be collected.

[0080] In this embodiment, the name of the log file includes time information, which indicates when the log file was generated. Therefore, based on the name of the log file, it can be determined when the log file was generated, and thus whether the log file is the log file to be collected.

[0081] The log collection method of this invention sets a parent directory to store log directories named based on container identifiers, so that all log files under the log directories are stored in the same parent directory. When a new log directory is generated due to container redeployment, the newly generated log directory can be accurately and quickly located, thereby quickly locating the newly generated log files. This reduces the coupling between the container and log collection. When upgrading business or containers, there is no need to upgrade the log collection tool or log collection code. When collecting log files, the path information of the collected log files recorded in the local persistent file is used to filter out the log files to be collected in the parent directory. Then, the contents of the log files to be collected are read to complete the collection of the log files. The path information of the collected log files is written to a doubly linked list. The writing speed is fast and does not affect the log collection efficiency. It has high real-time performance. Finally, the path information recorded in the doubly linked list is written to the local persistent file for persistent storage. This log collection method does not require the introduction of third-party frameworks, has low intrusion, and high timeliness.

[0082] Figure 4 A schematic diagram of the main modules of the log collection device according to an embodiment of the present invention is shown, such as... Figure 4 As shown, the log collection device 400 includes:

[0083] The scanning module 401 is used to scan the log files under the parent directory and determine the log files to be collected in the parent directory; wherein, the parent directory is used to store the log directory, the log directory is used to store the log files, and the log directory is named based on the container identifier;

[0084] The acquisition module 402 is used to acquire the contents of the log file to be acquired.

[0085] The log collection device of this invention sets a parent directory to store log directories named based on container identifiers. This ensures that all log files under the same parent directory are stored there. When a new log directory is generated due to container redeployment, the device can accurately and quickly locate the newly generated log directory and thus quickly locate the newly generated log files. This reduces the coupling between the container and log collection. During business upgrades or container upgrades, there is no need to upgrade the log collection tool or code. When collecting log files, the device scans the log files in the parent directory to determine the log files to be collected. Then, it reads the contents of the log files to be collected to complete the collection. This method does not require the introduction of third-party frameworks, has low invasiveness, and is highly efficient.

[0086] In an optional embodiment, the scanning module is further configured to: obtain a local persistent file, the local persistent file being used to record the path information of the collected log files; scan the log files in the parent directory, and based on the local persistent file, determine the log files to be collected in the parent directory;

[0087] The device also includes a local persistence module, which is used to write the path information of the collected log file to the local persistence file after the content of the log file to be collected is completed.

[0088] In an optional embodiment, the local persistence module is further configured to:

[0089] Get the doubly linked list;

[0090] Write the path information of the log files to be collected after collection into the doubly linked list;

[0091] The path information stored in the doubly linked list is written to the local persistent file to perform local persistent storage of the path information stored in the doubly linked list.

[0092] In an optional embodiment, the device further includes a configuration module for: setting naming rules for log files in the log directory, so as to determine the names of the log files according to the naming rules;

[0093] The scanning module is also used to: determine the log files to be collected in the parent directory based on the names of the log files in the parent directory and the local persistent files.

[0094] In an optional embodiment, the configuration module is further configured to:

[0095] Determine the service identifier of the service to which the log file belongs and the time information corresponding to the log file, wherein the time information is used to indicate the generation time of the log file;

[0096] The name of the log file is generated based on the service identifier and the time information.

[0097] In an optional embodiment, the configuration module is further configured to determine the log scan start time and / or log scan cycle;

[0098] The scanning module is also used to: determine the log files to be collected in the parent directory based on the log scan start time and / or log scan cycle, as well as the names of the log files in the parent directory and the local persistent files.

[0099] In an optional embodiment, the scanning module is further configured to: determine, based on the name of the log file in the parent directory, the log file whose generation time is between the log scan start time and the current time;

[0100] Alternatively, based on the names of the log files in the parent directory, determine the log files in the parent directory whose generation time was within the log scanning cycle;

[0101] The log file whose generation time is between the log scan start time and the current time will be used as the target log file;

[0102] Alternatively, the log file whose generation time falls within the log scan cycle can be used as the target log file;

[0103] Based on the local persistent file, the log files to be collected are filtered out from the target log files.

[0104] The above-described apparatus can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0105] This invention also provides an electronic device, such as... Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0106] Memory 503 is used to store computer programs;

[0107] When the processor 501 executes the program stored in the memory 503, it performs the following steps: scanning the log files under the parent directory to determine the log files to be collected in the parent directory; the parent directory is used to store the parent directory of the log directory, the log directory is used to store log files, and the log directory is named based on the container identifier; and collecting the contents of the log files to be collected.

[0108] The communication bus 504 mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 504 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.

[0109] Communication interface 502 is used for communication between the aforementioned terminal and other devices.

[0110] The memory 503 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor 501.

[0111] The processor 501 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0112] In another embodiment of the present invention, a computer-readable medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the log collection methods described in the above embodiments.

[0113] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the log collection methods described in the above embodiments.

[0114] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0115] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0116] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0117] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A log collection method, characterized in that, include: Scan the log files in the parent directory to determine the log files to be collected in the parent directory; wherein, the parent directory is used to store the log directory, the log directory is used to store the log files, and the log directory is named based on the container identifier; Collect the contents of the log file to be collected; The step of scanning the log files in the parent directory to determine the log files to be collected in the parent directory includes: Obtain a local persistent file, which is used to record the path information of the collected log files; Scan the log files in the parent directory, and determine the log files to be collected in the parent directory based on the local persistent files; When the content of the log file to be collected is completed, the method further includes: writing the path information of the collected log file to be collected into the local persistent file; When collecting the log files to be collected, the collected log files are marked, including: Change the name of the collected log file according to the preset format; or, change the storage format of the collected log file; or, move the collected log file to a specific folder under the parent directory.

2. The method according to claim 1, characterized in that, The step of writing the path information of the collected log files to the local persistent file includes: Get the doubly linked list; Write the path information of the log files to be collected after collection into the doubly linked list; The path information stored in the doubly linked list is written to the local persistent file to perform local persistent storage of the path information stored in the doubly linked list.

3. The method according to claim 2, characterized in that, The method further includes: setting naming rules for log files in the log directory, so as to determine the name of the log file according to the naming rules; The step of determining the log files to be collected in the parent directory based on the locally persisted files includes: Based on the names of the log files in the parent directory and the local persistent files, the log files to be collected in the parent directory are determined.

4. The method according to claim 3, characterized in that, Setting the naming rules for log files in the log directory, and determining the names of log files according to the naming rules, includes: Determine the service identifier of the service to which the log file belongs and the time information corresponding to the log file, wherein the time information is used to indicate the generation time of the log file; The name of the log file is generated based on the service identifier and the time information.

5. The method according to claim 3, characterized in that, The method also includes configuring the log scan start time and / or log scan cycle; The process of determining the log files to be collected in the parent directory based on the names of the log files in the parent directory and the local persistent files includes: Based on the log scan start time and / or log scan cycle, as well as the names of the log files in the parent directory and the local persistent files, the log files to be collected in the parent directory are determined.

6. The method according to claim 5, characterized in that, The process of determining the log files to be collected in the parent directory based on the log scan start time and / or log scan cycle, the naming rules of the log files, and the local persistent files includes: Based on the name of the log file in the parent directory, determine the log file in the parent directory whose generation time is between the start time of the log scan and the current time, or based on the name of the log file in the parent directory, determine the log file in the parent directory whose generation time is within the log scan cycle, and use the log file whose generation time is between the start time of the log scan and the current time or the log file whose generation time is within the log scan cycle as the target log file; Based on the local persistent file, the log files to be collected are filtered out from the target log files.

7. A log collection device, characterized in that, include: The scanning module is used to scan the log files in the parent directory and determine the log files to be collected in the parent directory; wherein, the parent directory is used to store the log directory, the log directory is used to store the log files, and the log directory is named based on the container identifier; The data acquisition module is used to collect the contents of the log file to be collected; The scanning module is also used to: obtain a local persistent file, which is used to record the path information of the collected log files; scan the log files in the parent directory, and determine the log files to be collected in the parent directory based on the local persistent file; The local persistence module is used to write the path information of the collected log file to the local persistence file after the content of the log file to be collected has been collected. When collecting the log files to be collected, the collected log files are marked, including: Change the name of the collected log file according to the preset format; or, change the storage format of the collected log file; or, move the collected log file to a specific folder under the parent directory.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images