Log data processing method and device, storage medium and electronic terminal

Abstract

The present disclosure relates to the technical field of computer, in particular to a log data processing method and device, a storage medium and an electronic terminal. The method is executed by a log server, and the method comprises: counting target attribute values of each log file; in the case that the target attribute value of a target log file reaches a preset rolling threshold, generating an intermediate file according to the target log file, so that the log data in the intermediate file can be collected by a log collection component; and compressing the intermediate file meeting a compression condition to obtain log data in a compressed format for persistent storage. The method of the present disclosure can perform compression processing only after a period of time of intermediate file generation, and a certain gap period is reserved, so that sufficient time is left for log data collection, and the situation that the log data in the finally generated compressed file is incomplete is avoided.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to a log data processing method, a log data processing device, a storage medium, and an electronic terminal. Background Technology

[0002] Due to limited server storage space, log data is typically compressed to save storage space. Currently, most log collection tools collect log data, but these tools do not support direct collection of zip-format compressed data packets. This results in incomplete log data collection if the log data is compressed before the collection tool has finished collecting it.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this disclosure is to provide a log data processing method, a log data processing device, a storage medium, and an electronic terminal, thereby overcoming, at least to some extent, one or more problems caused by the limitations and defects of related technologies.

[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0006] According to a first aspect of this disclosure, a log data processing method is provided, comprising: statistically analyzing the target attribute values ​​of each log file; generating an intermediate file based on the target log file when the target attribute value of the target log file reaches a preset rolling threshold, so that the log data in the intermediate file can be collected by a log collection component; and compressing the intermediate file that meets the compression conditions to obtain compressed log data for persistent storage.

[0007] In one exemplary embodiment of this disclosure, the step of statistically analyzing the target attribute values ​​of each log file includes: reading preset configuration information in response to a log processing instruction; wherein the preset configuration information includes: a number of specified file types; scanning a target folder according to the preset configuration information, filtering out log files that match each specified file type, and statistically analyzing the target attribute values ​​of the filtered log files.

[0008] In one exemplary embodiment of this disclosure, the method further includes: saving the selected log files to a preset cache area, and performing statistics on the log files in the preset cache area.

[0009] In one exemplary embodiment of this disclosure, generating an intermediate file based on the target log file when the target attribute value of the target log file reaches a preset rolling threshold includes: modifying the target log file into an intermediate file when the target attribute value of the target log file reaches the preset rolling threshold; the method further includes: determining an intermediate file that meets the compression conditions based on the file modification time of each intermediate file and the current time.

[0010] In one exemplary embodiment of this disclosure, modifying the target log file into an intermediate file includes: changing the file format of the target log file to a temporary file format, and adding the file modification time to the file name of the target log file.

[0011] In one exemplary embodiment of this disclosure, determining the intermediate file that meets the compression conditions based on the file modification time of each intermediate file and the current time includes: determining the delayed compression time of each intermediate file based on the file type of each intermediate file; if the difference between the file modification time of an intermediate file and the current time is not less than the delayed compression time of the intermediate file, then the intermediate file is determined as an intermediate file that meets the compression conditions.

[0012] In one exemplary embodiment of this disclosure, generating intermediate files based on the target log file includes: using a first thread to generate intermediate files based on the log file; compressing intermediate files that meet the compression conditions includes: using a second thread to perform a cyclic scan of the directories where each log file is located, determining whether the scanned intermediate files meet the compression conditions, and if so, performing compression processing; the first thread and the second thread are asynchronous threads.

[0013] In one exemplary embodiment of this disclosure, the method further includes: creating a delayed compression task with a delayed compression time based on the intermediate file, and placing the delayed compression task into a delayed queue; the step of compressing the intermediate file that meets the compression conditions includes: compressing the intermediate file corresponding to the delayed compression task that has expired in the delayed queue as an intermediate file that meets the compression conditions.

[0014] According to a second aspect of this disclosure, a log data processing apparatus is provided, comprising:

[0015] The statistics module is used to calculate the target attribute values ​​for each log file.

[0016] The intermediate file generation module is used to generate an intermediate file based on the target log file when the target attribute value of the target log file reaches a preset rolling threshold, so that the log data in the intermediate file can be collected by the log collection component.

[0017] The compressed file generation module is used to compress intermediate files that meet the compression conditions to obtain compressed log data for persistent storage.

[0018] According to a third aspect of this disclosure, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the log data processing method described above.

[0019] According to a fourth aspect of this disclosure, an electronic terminal is provided, comprising:

[0020] Processor; and

[0021] Memory for storing the executable instructions of the processor;

[0022] The processor is configured to execute the above-described log data processing method by executing the executable instructions.

[0023] In one embodiment of the log data processing method provided by this disclosure, log files are statistically analyzed, and when the target attribute value of the log file reaches a preset rolling threshold, an intermediate file is first generated from the log file; and when it is determined that the intermediate file meets the compression conditions, compression processing is performed to obtain compressed log data; thereby, the intermediate file is compressed only after meeting certain compression conditions, avoiding the situation where the log data in the final compressed file is incomplete; and persistent storage of log data can be achieved.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0026] Figure 1 The schematic diagram illustrates a flow chart of a log data processing method according to an exemplary embodiment of the present disclosure;

[0027] Figure 2 This schematic diagram illustrates a method flow chart for statistical analysis of log data according to an exemplary embodiment of this disclosure;

[0028] Figure 3 This schematic diagram illustrates the composition of a log data processing apparatus according to an exemplary embodiment of the present disclosure;

[0029] Figure 4 This schematic diagram illustrates the composition of an electronic device according to an exemplary embodiment of the present disclosure;

[0030] Figure 5 The schematic diagram illustrates a program product in an exemplary embodiment of the present disclosure. Detailed Implementation

[0031] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0032] In related technologies, due to limited server hard drive space, logs are typically compressed into zip format files when defining log rolling. For example, if the log file is app.log and the logs are split daily, then at 12:00, app.log will be compressed into app.2020-11-10.0.zip, and a blank app.log file will be generated to continue writing new logs. Furthermore, existing log collection tools, such as Filebeat, Flume, and Logagent, do not support direct collection of zip format data packets. Also, when using log collection tools, there is often a delay. For example, when the latest log line number of the application has reached n+m lines, the log collection may still be processing the nth line, resulting in the log file being compressed into a zip format file, processing only up to the nth line, while the subsequent m lines of data are not collected. This leads to incomplete logs and may pose risks and financial losses to business problem identification.

[0033] To address the aforementioned issues, this example implementation first provides a log data processing method that avoids incomplete log data collection by setting a specific compression strategy; and it also enables persistent storage of log data. (Reference) Figure 1 As shown, the log data processing method described above may include the following steps:

[0034] Step S11: Calculate the target attribute values ​​for each log file;

[0035] Step S12: When the target attribute value of the target log file reaches the preset rolling threshold, an intermediate file is generated based on the target log file so that the log data in the intermediate file can be collected by the log collection component.

[0036] Step S13: Compress the intermediate files that meet the compression conditions to obtain compressed log data for persistent storage.

[0037] The log data processing method provided in this example implementation has two aspects. First, by statistically analyzing the log files, an intermediate file is generated from the log files when the target attribute value of the log files reaches a preset rolling threshold, making it easier for the intermediate files to be collected. Second, by pre-configuring compression judgment conditions for the intermediate files and monitoring them, compression processing is only performed on the intermediate files after they have been generated for a period of time and when the compression conditions are met. This allows for continuous collection of log data from the intermediate files during that period, ensuring the integrity of the log data in the compressed logs and avoiding incomplete compressed log data.

[0038] The following will describe in more detail each step of the log data processing method in this exemplary embodiment, with reference to the accompanying drawings and embodiments.

[0039] In step S11, the target attribute values ​​of each log file are counted.

[0040] In this example embodiment, the log data processing method described above can be applied to a user terminal or a server; or it can be completed collaboratively by the terminal and the server. For example, the terminal device can compress the generated log data to generate a log file and send it to a log server, which then parses and performs statistics on the received log file from the terminal. Alternatively, the terminal device can collect system or application log data locally and perform statistics on the currently collected log data.

[0041] In this example implementation, refer to Figure 2 As shown, step S11 above may include:

[0042] Step S111: In response to the log processing instruction, read the preset configuration information; wherein, the preset configuration information includes: several specified file types;

[0043] Step S112: Scan the target folder according to the preset configuration information, filter out log files that match each specified file type, and perform statistical analysis on the target attribute values ​​of the filtered log files.

[0044] Specifically, the aforementioned log processing instructions can be system scheduled tasks, user-triggered log processing instructions on the terminal, or log data processing instructions sent from the server to the terminal. These instructions can include configuration information, such as the start time of the log data, application information, file path, file type, etc. After determining the configuration information in the instruction, the specified path can be scanned according to the configuration information, log files of the specified format can be filtered, and the number of filtered log files can be counted.

[0045] For example, log processing instructions can include instructions for processing log data from multiple applications. Each instruction can contain file paths corresponding to different applications, as well as different log file types, and so on. In this case, multiple threads can be created, each executing the log processing task separately, allowing for the scanning, filtering, and statistical analysis of log files corresponding to each application. For instance, the configuration file could contain a file named app.log with the path / data / logs, etc. Furthermore, the configuration information can include the start and / or end times of the log data, allowing filtering of log files within a specified time period.

[0046] In this example implementation, a separate cache area can also be configured on the terminal or server side to write the filtered log files into the preset cache area.

[0047] Specifically, the process can begin by filtering matching log files based on configuration information, then saving the filtered log files to a cache area, where statistical analysis is performed. If multiple applications are simultaneously performing statistical analysis, the cache area can be divided into multiple sub-areas, with statistical analysis performed on the log files within each sub-area. By performing data analysis on log files within the cache area, accurate log file statistics can be achieved. Furthermore, when new log files are found, they can be added to the cache area for subsequent compression processing.

[0048] In step S12, when the target attribute value of the target log file reaches a preset rolling threshold, an intermediate file is generated based on the target log file so that the log data in the intermediate file can be collected by the log collection component.

[0049] In this example implementation, when performing statistics on log files, the statistics can be based on specified attribute values ​​of the log files. For example, the target attribute value could be the number of log files; for instance, when the number of data entries in a file reaches the nth line, it is determined that the rolling threshold has been reached. Alternatively, the rolling threshold can be determined when the number of files reaches a preset value. Alternatively, feature extraction can be performed on the log files, and the rolling threshold can be determined when a specified feature field or content is extracted.

[0050] Alternatively, in some exemplary implementations, the target attribute of the log file may also be the file's creation time.

[0051] In this example implementation, when the target attribute value of the target log file reaches a preset rolling threshold, the target log file is modified into an intermediate file. Specifically, the file format of the target log file can be changed to a temporary file format, and the file modification time can be added to the filename of the target log file.

[0052] For example, when the target attribute value of the target log file reaches a preset rolling threshold, the filename of the target file can be modified. For instance, the log file name can be changed to: filename + timestamp.tmp, thus generating an intermediate file in a specified format. For example, if the target log file is app.log, it can be modified to app.date.index.logtimestamp.tmp. Simultaneously, a new app.log file can be generated for the application. For example, app.log can be modified to app.2010-11-10.0.log.318854238574384.tmp.

[0053] After generating the intermediate files, you can also save them to a specified folder and create the corresponding path.

[0054] In addition, log files can be preprocessed before intermediate files are generated, such as sorting, dividing, and creating indexes.

[0055] In step S13, the intermediate files that meet the compression conditions are compressed to obtain compressed log data for persistent storage.

[0056] In this example implementation, the compression condition described above can be a duration condition. Specifically, when determining the compression condition, the intermediate files that meet the compression condition can be identified based on the file modification time of each intermediate file and the current time.

[0057] In this example implementation, different delay compression times can be pre-configured for different applications, different types of target log files, or different types of intermediate files.

[0058] Specifically, the above method may include: determining the delayed compression time of each intermediate file based on the file type of each intermediate file;

[0059] If the difference between the file modification time of an intermediate file and the current time is not less than the delayed compression time of the intermediate file, then the intermediate file is determined to be an intermediate file that meets the compression conditions.

[0060] For example, the file modification time can be obtained based on the filename of the intermediate file. If the difference between the current time and the file modification time is greater than or equal to the preset delayed compression time, it can be determined that the intermediate file currently meets the preset compression conditions and compression operation can be performed on it.

[0061] Alternatively, in some exemplary embodiments, the above method may further include: creating a delayed compression task with a delayed compression time based on the intermediate file, and placing the delayed compression task into a delayed queue; the step of compressing the intermediate file that meets the compression conditions includes: compressing the intermediate file corresponding to the delayed compression task that has expired in the delayed queue as an intermediate file that meets the compression conditions.

[0062] Specifically, for intermediate files, corresponding delayed compression tasks can be created. Each delayed compression task can be configured with a corresponding delayed compression time based on the application or log file type. Specifically, a delayed queue can be pre-configured. For the currently generated intermediate file, a corresponding delayed compression task can be created and written to this delayed compression task queue. Within this queue, a corresponding timer can be configured for each delayed compression task. Different timers can be configured for delayed compression tasks corresponding to intermediate files generated by different applications. When the timer reaches its predetermined duration, i.e., the delayed compression task has expired, the corresponding intermediate file can be compressed to obtain the compressed log. Alternatively, users can customize the delayed compression time, or define the delay duration based on the characteristics of the log data corresponding to the application. For example, the delayed compression time can be defined as 5 minutes, or 30 seconds, etc.

[0063] In this example implementation, even if the delayed compression task has not reached the specified time, or if the difference between the modification time of the intermediate file and the current time is less than the first time threshold, the task of scanning and filtering log data can still be executed, and new log data can be written to the intermediate file. In this way, log data can be continuously collected during this time gap, ensuring that the log data is collected completely. This, in turn, ensures that the final compressed log file contains complete data content.

[0064] Alternatively, in some exemplary embodiments, the above-mentioned generation of intermediate files based on the target log file may include: using a first thread to generate intermediate files based on the log file; the compression of intermediate files that meet the compression conditions includes: using a second thread to perform a cyclic scan of the directories where each log file is located, determining whether the scanned intermediate files meet the compression conditions, and if so, performing compression processing; the first thread and the second thread are asynchronous threads.

[0065] Specifically, a first thread and a second thread can be pre-configured. The first thread processes the target log file, generating corresponding intermediate files. These intermediate files are not entirely new files; rather, they are modifications of the original target log file. This ensures the file handle remains unchanged, allowing the log collection component to continue collecting logs from the intermediate files. The second thread performs a cyclical scan of the intermediate directories, determining whether the scanned intermediate files meet compression criteria. If so, compression is performed. For example, during the cyclical scan, compression criteria can be configured based on the size of the intermediate file, the amount of log data, or the creation time of the intermediate file.

[0066] For example, when the second thread starts scanning the intermediate file, the application only writes log data up to line n. While generating the intermediate file, the application continues to output log data. When a preset time threshold is reached, the application writes log data up to line n+m, and all of this log data is written to the cache and modified into the intermediate file by the first thread. In this way, the intermediate file containing n+m lines of data can be compressed into a compressed log.

[0067] In this example implementation, for the delay queue, the time difference between the creation time of the intermediate files of each task and the current time is monitored in the queue.

[0068] In other exemplary embodiments of this disclosure, when compressing intermediate files, the intermediate files can also be parsed to extract corresponding feature information, and the corresponding compression algorithm can be determined based on the feature information. For example, the corresponding compression algorithm can be determined based on the data type of the log file, or based on the parsing speed of the log file. Alternatively, the compression algorithm can be determined based on the size of the log file.

[0069] The log data processing method disclosed herein generates a corresponding intermediate file for the target log file and performs delayed compression on the intermediate file instead of immediate compression. This creates a gap between the TMP and ZIP file formats, allowing the log collection program sufficient time to complete log data collection and write the collected data to the intermediate file. This solves the current problem of lost log data. Furthermore, the log compression rate can be achieved at 5%-10%, effectively ensuring disk space utilization during log data compression.

[0070] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may, for example, be executed synchronously or asynchronously in multiple modules.

[0071] Further reference Figure 3 As shown, this example embodiment also provides a log data processing device 30, including: a statistics module 301, an intermediate file generation module 302, and a compressed file generation module 303. Wherein:

[0072] The statistics module 301 can be used to count the target attribute values ​​of each log file.

[0073] The intermediate file generation module 302 can be used to generate an intermediate file based on the target log file when the target attribute value of the target log file reaches a preset rolling threshold, so that the log data in the intermediate file can be collected by the log collection component.

[0074] The compressed file generation module 303 can be used to compress intermediate files that meet the compression conditions to obtain compressed log data for persistent storage.

[0075] In this example embodiment, the statistics module 301 may include: an instruction response unit and a statistics unit. Wherein:

[0076] The instruction response unit can be used to respond to log processing instructions and read preset configuration information; wherein, the preset configuration information includes: several specified file types.

[0077] The statistical unit can be used to scan the target folder according to the preset configuration information, filter out log files that match each specified file type, and perform statistical analysis on the target attribute values ​​of the filtered log files.

[0078] In this example embodiment, the statistics unit can also be used to save the filtered log files to a preset cache area and perform statistics on the log files in the preset cache area.

[0079] In this example embodiment, the intermediate file generation module 302 can be used to modify the target log file into an intermediate file when the target attribute value of the target log file reaches a preset rolling threshold.

[0080] The compressed file generation module 303 can be used to determine the intermediate files that meet the compression conditions based on the file modification time of each intermediate file and the current time.

[0081] In this example embodiment, the intermediate file generation module 302 can be used to modify the file format of the target log file to a temporary file format and add the file modification time to the file name of the target log file.

[0082] In this example embodiment, the compressed file generation module 303 can be used to determine the delayed compression time of each intermediate file according to the file type of each intermediate file; if the difference between the file modification time of an intermediate file and the current time is not less than the delayed compression time of the intermediate file, then the intermediate file is determined to be an intermediate file that meets the compression conditions.

[0083] In this example embodiment, the intermediate file generation module 302 may include: using a first thread to generate an intermediate file based on the log file.

[0084] The compressed file generation module 303 may include using a second thread to perform a cyclic scan of the directories where each log file is located, and determine whether the scanned intermediate files meet the compression conditions. If so, compression processing is performed.

[0085] The first thread and the second thread are asynchronous threads.

[0086] In this example embodiment, the apparatus may further include a delayed compression task configuration module. The delayed compression task configuration module can be used to create a delayed compression task with a delayed compression time based on the intermediate file, and to place the delayed compression task into a delayed queue.

[0087] The compressed file generation module 303 can also be used to compress intermediate files corresponding to expired delayed compression tasks in the delay queue as intermediate files that meet the compression conditions.

[0088] The specific details of each module in the log data processing device 30 described above have been described in detail in the corresponding log data processing method, so they will not be repeated here.

[0089] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0090] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described log data processing method is also provided.

[0091] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0092] The following reference Figure 4 To describe an electronic device 500 according to this embodiment of the present invention. Figure 4 The electronic device 500 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0093] like Figure 4 As shown, the electronic device 500 is manifested in the form of a general-purpose computing device. The components of the electronic device 500 may include, but are not limited to: at least one processing unit 510, at least one storage unit 520, and a bus 530 connecting different system components (including storage unit 520 and processing unit 510).

[0094] The storage unit stores program code that can be executed by the processing unit 510, causing the processing unit 510 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 510 can perform, as follows: Figure 1 The log data processing method shown.

[0095] Storage unit 520 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 5201 and / or cache memory 5202, and may further include a read-only memory (ROM) 5203.

[0096] Storage unit 520 may also include a program / utility 5204 having a set (at least one) program module 5205, such program module 5205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0097] Bus 530 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0098] Electronic device 500 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 500, and / or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 550. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 560. As shown, network adapter 560 communicates with other modules of electronic device 500 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0099] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0100] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.

[0101] refer to Figure 5 As shown, a program product 600 for implementing the above-described method according to an embodiment of the present invention is described. It may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0102] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0103] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0104] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0105] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0106] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0107] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0108] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A log data processing method characterized by comprising: The method is executed by a log server, and the method comprises: counting target attribute values of each log file; generating an intermediate file according to a target log file in a case where a target attribute value of the target log file reaches a preset rolling threshold, so that log data in the intermediate file can be collected by a log collection component; performing compression processing on an intermediate file that meets a compression condition to obtain log data in a compressed format for persistent storage; wherein the generating of the intermediate file according to the target log file in the case where the target attribute value of the target log file reaches the preset rolling threshold comprises: modifying the target log file into an intermediate file when the target attribute value of the target log file reaches the preset rolling threshold; and The method further comprises: determining an intermediate file that meets the compression condition according to a file modification time of each intermediate file and a current time; wherein a delay compression time of each intermediate file is determined according to a file type of each intermediate file; if a difference between the file modification time of an intermediate file and the current time is not less than the delay compression time of the intermediate file, the intermediate file is determined as the intermediate file that meets the compression condition.

2. The log data processing method according to claim 1, characterized by, The counting of the target attribute values of each log file comprises: reading preset configuration information in response to a log processing instruction; wherein the preset configuration information comprises: a plurality of specified file types; scanning a target folder according to the preset configuration information, screening out log files matched with each specified file type, and counting target attribute values of the screened log files.

3. The log data processing method according to claim 2, characterized by, The method further comprises: saving the screened log files to a preset cache area and counting the log files in the preset cache area.

4. The log data processing method of claim 1, wherein, The modifying of the target log file into an intermediate file comprises: modifying a file format of the target log file into a temporary file format, and adding a file modification time in a file name of the target log file.

5. The log data processing method of claim 1, wherein, The generating of the intermediate file according to the target log file comprises: generating an intermediate file according to the log file by using a first thread; The compression processing of the intermediate file that meets the compression condition comprises: performing cyclic scanning on a directory where each log file is located by using a second thread, judging whether the scanned intermediate file meets the compression condition, and performing compression processing if yes; The first thread and the second thread are asynchronous threads.

6. The log data processing method of claim 1, wherein, The method further comprises: creating a delay compression task with a delay compression time based on the intermediate file, and putting the delay compression task into a delay queue; The compression processing of the intermediate file that meets the compression condition comprises: performing compression processing on an intermediate file corresponding to an expired delay compression task in the delay queue as the intermediate file that meets the compression condition.

7. A log data processing apparatus characterized by comprising: Applied to a log server, comprising: a counting module configured to count target attribute values of each log file; an intermediate file generation module configured to generate an intermediate file according to a target log file in a case where a target attribute value of the target log file reaches a preset rolling threshold, so that log data in the intermediate file can be collected by a log collection component; The compression file generation module is configured to compress the intermediate files meeting the compression condition to obtain log data in a compressed format for persistent storage. The intermediate file is generated according to the target log file in a case where a target attribute value of the target log file reaches a preset rolling threshold, and the method comprises the following steps: The target log file is modified into an intermediate file when the target attribute value of the target log file reaches the preset rolling threshold; and The intermediate files meeting the compression condition are determined according to file modification times of the intermediate files and a current time. The delay compression time of each intermediate file is determined according to the file type of the intermediate file. If the difference between the file modification time of an intermediate file and the current time is not less than the delay compression time of the intermediate file, the intermediate file is determined as the intermediate file meeting the compression condition.

8. A storage medium having a computer program stored thereon, the program being executed by a processor to implement the log data processing method according to any one of claims 1 to 6.

9. An electronic terminal, characterized in that comprises: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to execute the log data processing method according to any one of claims 1 to 6 by executing the executable instructions.

Images