A cluster log storage method, device, equipment and medium

Abstract

The present application discloses a cluster log storage method, device, equipment and medium, including: counting the access frequency of log files of each target module in the cluster; determining the compression attribute corresponding to each target module based on the access frequency; wherein the compression attribute includes compression and non-compression; when it is detected that there is a log to be stored in the cluster, determining the compression attribute corresponding to the log to be stored; if the compression attribute is non-compression, storing the log to be stored in a log buffer area, otherwise, compressing the log to be stored to obtain a first compressed log, and storing the first compressed log in a static storage area. It can improve the utilization rate of cluster storage space and log access efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of log processing, and in particular to a cluster log storage method, device, equipment and medium. Background Art

[0002] In the current distributed storage cluster environment, logs are important information in the storage cluster, including various business operation records. In addition, the historical operation status of the cluster and the location of problems during failures need to be analyzed through logs. At the same time, with the increase of time, the log data also grows linearly. Log data occupies a lot of space in data storage, reducing the overall space utilization of the distributed storage cluster. At the same time, due to the large number of logs, log management and retrieval become more difficult. Therefore, solving the problem of massive log data storage and retrieval has an important impact on the cluster. Summary of the invention

[0003] In view of this, the purpose of this application is to provide a cluster log storage method, device, equipment and medium, which can improve the utilization rate of cluster storage space and log access efficiency. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a cluster log storage method, comprising:

[0005] Count the access frequency of the log files of each target module in the cluster;

[0006] Determining a compression attribute corresponding to each of the target modules based on the access frequency; wherein the compression attribute includes compression and non-compression;

[0007] When it is detected that there are logs to be stored in the cluster, the compression attribute corresponding to the logs to be stored is determined;

[0008] If the compression attribute is not compressed, the log to be stored is stored in a log buffer area; otherwise, the log to be stored is compressed to obtain a first compressed log, and the first compressed log is stored in a static storage area.

[0009] Optionally, the access frequency of the log files of each target module in the statistical cluster includes:

[0010] Counting the access frequency of log files of each functional module and each submodule of the functional module;

[0011] Correspondingly, determining the compression attribute corresponding to each of the target modules based on the access frequency includes:

[0012] The compression attribute corresponding to each target module is determined based on the number of accesses; wherein the access frequency of the functional module whose compression attribute is non-compression is higher than the access frequency of the functional module whose compression attribute is compression, and the access frequency of the sub-module whose compression attribute is non-compression is higher than the access frequency of the sub-module whose compression attribute is compression.

[0013] Optionally, determining the compression attribute corresponding to each of the target modules based on the access frequency includes:

[0014] Sorting the access frequencies of the log files of each of the functional modules;

[0015] Filter out a preset number of the functional modules with the highest access frequency to obtain a first module;

[0016] Determine the compression attribute of the first module as non-compression, and determine the compression attribute of the non-first module in the functional module as compression;

[0017] sorting the access frequencies of the log files of all submodules other than the first module in the functional module;

[0018] The compression attribute of the submodules of the first preset proportion with the highest access frequency is determined to be non-compression, and the compression attribute of other submodules other than the first module is determined to be compression.

[0019] Optionally, also include:

[0020] Storing the compression attribute of each target module in a log base library;

[0021] Accordingly, the step of determining the compression attribute corresponding to the log to be stored includes:

[0022] From the log basic library, the compression attribute of the function module to which the log to be stored belongs is searched. If the compression attribute of the function module is not compressed, it is determined that the compression attribute corresponding to the log to be stored is not compressed; if the compression attribute of the function module is compressed, the compression attribute of the submodule to which the log to be stored belongs is searched. If the compression attribute of the submodule is not compressed, it is determined that the compression attribute corresponding to the log to be stored is not compressed; otherwise, it is determined that the compression attribute corresponding to the log to be stored is compressed.

[0023] Optionally, also include:

[0024] When the log to be stored is stored in the log buffer area, the expiration time of the log to be stored is stored in the log basic library;

[0025] Determining a target log from the log buffer area based on the expiration time at a preset time interval or at a specified time;

[0026] Compressing the target log to obtain a second compressed log;

[0027] Migrate the second compressed log to the static storage area.

[0028] Optionally, the determining a target log from the log cache area based on the expiration time includes:

[0029] Determining an expired log from the log buffer area based on the expiration time;

[0030] A specified number of the expired logs are randomly selected to obtain a target log.

[0031] Optionally, also include:

[0032] When the usage ratio of the log buffer area reaches a preset threshold, the access frequencies of the log files of all sub-modules whose compression attribute is not compressed are sorted;

[0033] Compressing the log files of the submodules with the second preset ratio having the least access frequency to obtain a third compressed log;

[0034] Migrate the third compressed log to the static storage area.

[0035] In a second aspect, the present application discloses a cluster log storage device, comprising:

[0036] The access frequency statistics module is used to count the access frequency of the log files of each target module in the cluster;

[0037] A module attribute determination module, used to determine the compression attribute corresponding to each of the target modules based on the access frequency; wherein the compression attribute includes compression and non-compression;

[0038] The log monitoring module to be stored is used to monitor whether there are logs to be stored in the cluster;

[0039] A compression attribute determination module, configured to determine the compression attribute corresponding to the log to be stored when the log to be stored monitoring module detects that there is a log to be stored in the cluster;

[0040] The log storage module to be stored is used for storing the log to be stored in the log buffer area if the compression attribute is not compressed, otherwise, compressing the log to be stored to obtain a first compressed log, and storing the first compressed log in the static storage area.

[0041] In a third aspect, the present application discloses an electronic device, comprising:

[0042] Memory, used to store computer programs;

[0043] The processor is used to execute the computer program to implement the aforementioned cluster log storage method.

[0044] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned cluster log storage method when executed by a processor.

[0045] It can be seen that the present application counts the access frequency of the log files of each target module in the cluster, and determines the compression attribute corresponding to each target module based on the access frequency; wherein the compression attribute includes compression and non-compression. When it is detected that there are logs to be stored in the cluster, the compression attribute corresponding to the logs to be stored is determined. If the compression attribute is non-compression, the logs to be stored are stored in the log cache area. Otherwise, the logs to be stored are compressed to obtain a first compressed log, and the first compressed log is stored in the static storage area. That is, the present application compresses and stores the logs according to the access frequency of the log files of each module, or does not compress them and directly stores them in the log cache area. In this way, on the one hand, some logs are compressed, which can improve the utilization rate of the cluster storage space. On the other hand, the compressed logs and non-compressed logs are stored in different areas. For the non-compressed logs, they can be read directly from the log cache area, which can improve the log access efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0047] Figure 1 A flow chart of a cluster log storage method disclosed in this application;

[0048] Figure 2 A flow chart of a specific cluster log storage method disclosed in this application;

[0049] Figure 3 A schematic diagram of the structure of a cluster log storage device disclosed in this application;

[0050] Figure 4 A schematic diagram of a specific cluster log storage solution disclosed in this application;

[0051] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] In the current distributed storage cluster environment, logs are important information in the storage cluster, including various business operation records. In addition, the historical operation status of the cluster and the location of problems during failures need to be analyzed through logs. At the same time, as time goes by, log data also grows linearly. Log data occupies a large amount of space in data storage, reducing the overall space utilization of the distributed storage cluster. At the same time, due to the large number of logs, it becomes more difficult to manage and retrieve logs. Therefore, solving the problem of massive log data storage and retrieval has an important impact on the cluster. To this end, the present application provides a cluster log storage solution that can improve the utilization of cluster storage space and log access efficiency.

[0054] See also Figure 1 , the embodiment of the present application discloses a cluster log storage method, including:

[0055] Step S11: Count the access frequency of the log file of each target module in the cluster.

[0056] Step S12: Determine the compression attribute corresponding to each of the target modules based on the access frequency; wherein the compression attribute includes compression and non-compression.

[0057] In a specific implementation, the access frequency of the log files of each functional module and each sub-module of the functional module can be counted; the compression attribute corresponding to each target module is determined based on the access number; wherein, the access frequency of the functional module whose compression attribute is non-compression is higher than the access frequency of the functional module whose compression attribute is compression, and the access frequency of the sub-module whose compression attribute is non-compression is higher than the access frequency of the sub-module whose compression attribute is compression.

[0058] The functional module may be an alarm module, a cache module, etc. in the cluster, and each functional module includes multiple submodules. For example, the alarm module includes an alarm submodule, an event submodule, etc.

[0059] Furthermore, in a specific implementation, the access frequency of the log files of each functional module and each submodule of the functional module can be counted regularly, and then the compression attribute corresponding to each target module can be determined based on the access frequency. In this way, the access frequency can be updated in time, and the compression attribute can be modified in time.

[0060] Furthermore, the access frequencies of the log files of the functional modules can be sorted; a preset number of the functional modules with the highest access frequencies are screened out to obtain a first module; the compression attribute of the first module is determined to be non-compressed, and the compression attribute of the non-first modules in the functional modules is determined to be compressed; the access frequencies of the log files of all sub-modules other than the first module in the functional modules are sorted; the compression attribute of a first preset proportion of the sub-modules with the highest access frequencies is determined to be non-compressed, and the compression attribute of other sub-modules other than the first module is determined to be compressed.

[0061] That is, the embodiment of the present application can first sort the access frequency of the log files of each functional module, select a preset number of functional modules with the highest access frequency, determine the compression attribute as compression, and then sort the access frequency of the log files of the sub-modules of the remaining functional modules, and determine the compression attribute of the sub-modules with the highest access frequency of the first preset proportion as compression. In this way, it can avoid the problem that there are log files with high access frequency in the remaining functional modules, but the access efficiency is reduced due to compressed storage.

[0062] The preset number and the first preset ratio can be configured according to actual conditions.

[0063] Step S13: when it is detected that there are logs to be stored in the cluster, the compression attribute corresponding to the logs to be stored is determined.

[0064] In a specific implementation, the compression attribute of each target module can be stored in a log basic library; from the log basic library, the compression attribute of the function module to which the log to be stored belongs is searched; if the compression attribute of the function module is not compressed, the compression attribute corresponding to the log to be stored is determined to be not compressed; if the compression attribute of the function module is compressed, the compression attribute of the submodule to which the log to be stored belongs is searched; if the compression attribute of the submodule is not compressed, the compression attribute corresponding to the log to be stored is determined to be not compressed; otherwise, the compression attribute corresponding to the log to be stored is determined to be compressed.

[0065] Furthermore, in a specific implementation, the log storage tasks issued by the cluster can be monitored. When a log storage task is detected, it indicates that there are logs to be stored in the cluster. The compression attribute corresponding to the log to be stored corresponding to the log storage task is determined and stored.

[0066] In addition, a globally unique identification number can be generated for the corresponding log storage task to facilitate task processing.

[0067] Step S14: if the compression attribute is not compressed, the log to be stored is stored in the log buffer area; otherwise, the log to be stored is compressed to obtain a first compressed log, and the first compressed log is stored in the static storage area.

[0068] It can be seen that the embodiment of the present application counts the access frequency of the log files of each target module in the cluster, and determines the compression attribute corresponding to each target module based on the access frequency; wherein, the compression attribute includes compression and non-compression. When it is detected that there are logs to be stored in the cluster, the compression attribute corresponding to the logs to be stored is determined. If the compression attribute is non-compression, the logs to be stored are stored in the log buffer area. Otherwise, the logs to be stored are compressed to obtain a first compressed log, and the first compressed log is stored in the static storage area. That is, the present application compresses and stores the logs according to the access frequency of the log files of each module, or does not compress them and directly stores them in the log buffer area. In this way, on the one hand, some logs are compressed, which can improve the utilization rate of the cluster storage space. On the other hand, the compressed logs and non-compressed logs are stored in different areas. For the non-compressed logs, they can be read directly from the log buffer area, which can improve the log access efficiency.

[0069] See also Figure 2 As shown, the embodiment of the present application discloses a cluster log storage method, including:

[0070] Step S21: Count the access frequency of the log file of each target module in the cluster.

[0071] Step S22: Determine the compression attribute corresponding to each of the target modules based on the access frequency; wherein the compression attribute includes compression and non-compression.

[0072] Step S23: when it is detected that there are logs to be stored in the cluster, the compression attribute corresponding to the logs to be stored is determined.

[0073] Step S24: if the compression attribute is not compressed, the log to be stored is stored in the log buffer area; otherwise, the log to be stored is compressed to obtain a first compressed log, and the first compressed log is stored in the static storage area.

[0074] Step S25: When the log to be stored is stored in the log buffer area, the expiration time of the log to be stored is stored in the log basic library.

[0075] Step S26: determining a target log from the log buffer area based on the expiration time at a preset time interval or at a specified time.

[0076] For example, once a day, once a week, at a specified time every day, on a specified day of the month, etc.

[0077] In a specific implementation, expired logs may be determined from the log buffer area based on the expiration time; and a specified number of the expired logs may be randomly selected to obtain a target log.

[0078] Step S27: compress the target log to obtain a second compressed log.

[0079] Step S28: Migrate the second compressed log to the static storage area.

[0080] It is understandable that the embodiments of the present application can process expired log information, and in order to reduce the pressure of processing expired logs, the expired logs can be randomly sampled to select target logs at preset time intervals or at specified times, and then compressed and stored.

[0081] In addition, in a specific implementation, this embodiment can sort the access frequencies of log files of all sub-modules whose compression attribute is not compressed when the usage ratio of the log cache area reaches a preset threshold; compress the log files of the sub-modules with the second preset ratio with the least access frequency to obtain a third compressed log; and migrate the third compressed log to the static storage area.

[0082] It is understandable that since the log storage capacity usage increases linearly, when the capacity usage ratio reaches the threshold, the log files with the second preset ratio ranked at the bottom are forced to be packaged according to the access frequency sorting to release storage resources, where the percentage of released resources and the second preset ratio can be configured.

[0083] Furthermore, when the present embodiment obtains the log reading request issued by the cluster, the compression attribute and location information of the corresponding module of the corresponding log are read from the log basic library. If the compression attribute is compression, the log is read from the static storage area according to the location information, decompressed and returned, and the log and log access record are stored in the log basic library. If the compression attribute is not compressed, the log is read from the log buffer area according to the location information, returned directly, and the expiration time of the log is updated.

[0084] See also Figure 3 As shown, the embodiment of the present application discloses a cluster log storage device, including:

[0085] The access frequency statistics module 11 is used to count the access frequency of the log files of each target module in the cluster;

[0086] A module attribute determination module 12, configured to determine a compression attribute corresponding to each of the target modules based on the access frequency; wherein the compression attribute includes compression and non-compression;

[0087] The log monitoring module 13 is used to monitor whether there are logs to be stored in the cluster;

[0088] The compression attribute determination module 14 is used to determine the compression attribute corresponding to the log to be stored when the log to be stored monitoring module detects that there is a log to be stored in the cluster;

[0089] The log storage module 15 is used to store the log to be stored in the log buffer area if the compression attribute is not compressed, otherwise, compress the log to be stored to obtain a first compressed log, and store the first compressed log in the static storage area.

[0090] It can be seen that the embodiment of the present application counts the access frequency of the log files of each target module in the cluster, and determines the compression attribute corresponding to each target module based on the access frequency; wherein, the compression attribute includes compression and non-compression. When it is detected that there are logs to be stored in the cluster, the compression attribute corresponding to the logs to be stored is determined. If the compression attribute is non-compression, the logs to be stored are stored in the log buffer area. Otherwise, the logs to be stored are compressed to obtain a first compressed log, and the first compressed log is stored in the static storage area. That is, the present application compresses and stores the logs according to the access frequency of the log files of each module, or does not compress them and directly stores them in the log buffer area. In this way, on the one hand, some logs are compressed, which can improve the utilization rate of the cluster storage space. On the other hand, the compressed logs and non-compressed logs are stored in different areas. For the non-compressed logs, they can be read directly from the log buffer area, which can improve the log access efficiency.

[0091] The access frequency statistics module 11 is specifically used to count the access frequencies of log files of each functional module and each submodule of the functional module;

[0092] Accordingly, the module attribute determination module 12 is specifically used for:

[0093] The compression attribute corresponding to each target module is determined based on the number of accesses; wherein the access frequency of the functional module whose compression attribute is non-compression is higher than the access frequency of the functional module whose compression attribute is compression, and the access frequency of the sub-module whose compression attribute is non-compression is higher than the access frequency of the sub-module whose compression attribute is compression.

[0094] In a specific implementation, the module attribute determination module 12 is specifically used to:

[0095] Sorting the access frequencies of the log files of each of the functional modules;

[0096] Filter out a preset number of the functional modules with the highest access frequency to obtain a first module;

[0097] Determine the compression attribute of the first module as non-compression, and determine the compression attribute of the non-first module in the functional module as compression;

[0098] sorting the access frequencies of the log files of all submodules other than the first module in the functional module;

[0099] The compression attribute of the submodules of the first preset proportion with the highest access frequency is determined to be non-compression, and the compression attribute of other submodules other than the first module is determined to be compression.

[0100] The device also includes a compression attribute storage module, which is used to:

[0101] Storing the compression attribute of each target module in a log base library;

[0102] Accordingly, the compression attribute determination module 14 is specifically configured to:

[0103] From the log basic library, the compression attribute of the function module to which the log to be stored belongs is searched. If the compression attribute of the function module is not compressed, it is determined that the compression attribute corresponding to the log to be stored is not compressed; if the compression attribute of the function module is compressed, the compression attribute of the submodule to which the log to be stored belongs is searched. If the compression attribute of the submodule is not compressed, it is determined that the compression attribute corresponding to the log to be stored is not compressed; otherwise, it is determined that the compression attribute corresponding to the log to be stored is compressed.

[0104] The device also includes:

[0105] An expiration time storage module, used for storing the expiration time of the log to be stored in the log basic library when the log to be stored is stored in the log buffer area;

[0106] A target log determination module, configured to determine a target log from the log buffer area based on the expiration time at a preset time interval or at a specified time;

[0107] A log compression module, used for compressing the target log to obtain a second compressed log;

[0108] The first log migration module is used to migrate the second compressed log to the static storage area.

[0109] Furthermore, the target log determination module is specifically configured to determine expired logs from the log buffer area based on the expiration time; and randomly select a specified number of the expired logs to obtain the target logs.

[0110] Furthermore, the device also includes a second log migration module, which is used to sort the access frequency of log files of all sub-modules whose compression attribute is not compressed when the usage ratio of the log cache area reaches a preset threshold; compress the log files of the sub-modules with the second preset ratio with the least access frequency to obtain a third compressed log; and migrate the third compressed log to the static storage area.

[0111] For example, see Figure 4As shown, the embodiment of the present application discloses a specific schematic diagram of a cluster log storage solution. In a specific implementation, the aforementioned modules can be integrated to deploy a log task monitoring module, a log task management module, a log access module, a log update module, a log data statistics module, a log basic library, etc. Among them, 1) the log task monitoring module is used to provide the cluster with log task monitoring and log task issuance functions. When the cluster issues a log storage task, the task is sent to the log task management module for processing, and a globally unique identification number is generated for the log task. 2) The log task management module manages the issued log storage tasks, determines whether compression is required and obtains storage location information according to the information in the log basic library when storing logs, and then stores it in the log storage server. If compression is required, it is compressed and stored in the static storage area and the location information is recorded. If compression is not required, it is stored in the log buffer area, and the location information stored in the log buffer area and the configuration expiration time are recorded in the log basic library. When reading the log, it is read according to the location information of the log, and according to the compression attribute, the log data is directly returned to the user after decompression or without decompression. 3) Log access module: This module receives log reading requests sent by the cluster, obtains the target log through the log task management module, and stores the log access record in the log basic library after the log access is completed. If the file is accessed in the log cache area, the expiration time of the log file is refreshed at the same time. 4) Log data statistics module: This module supports frequency statistics and sorting of log file accesses of each module. For the n (configurable) most frequently used functional modules, the newly generated logs are directly placed in the log cache area. For other functional modules, the newly generated logs of the top m% (configurable) sub-modules with the highest frequency are placed in the log cache area. In order to reduce the performance pressure caused by real-time updates in large-scale clusters, log statistics are analyzed and sorted, and the compression attributes of the corresponding modules are updated using a timed update strategy (configurable). The frequency statistics of the log files of the log module are modified in real time. 5) Log update module, which processes the log information that has reached the expiration time. In order to reduce the pressure on the system to process expired log files, this module adopts a timing strategy and a periodic strategy. The timing strategy: the system periodically processes expired files by random sampling. The periodic strategy: periodically batch processes expired logs and migrates expired logs from the log buffer area for packaged storage. In addition, the log storage capacity occupancy increases linearly. When the capacity usage ratio reaches the threshold, the log update module forces the packaging of the log files of the bottom k% (configurable) sub-modules according to the access frequency sorting, and releases storage resources (the percentage of released resources is configurable). 6) Log basic library: stores log access records and the compression properties of each module of the log, as well as the expiration time information of the log file.Through the above modules, the access frequency of log data is counted, the log data is divided into high and low usage frequencies, the low-frequency log data is packaged and stored, and the high-frequency logs are stored in the log cache area, which improves the utilization of storage space and log access efficiency. The use of automated methods to regularly and regularly process expired logs improves the cluster storage utilization and avoids manual migration. In this way, the distributed storage cluster can optimize the storage of log files to a certain extent, reduce the space occupied by logs, and at the same time improve the retrieval efficiency of logs, and improve the data reliability and stability of the cluster.

[0112] See also Figure 5 As shown, an embodiment of the present application discloses an electronic device 20, including a processor 21 and a memory 22; wherein the memory 22 is used to store a computer program; the processor 21 is used to execute the computer program, and the cluster log storage method disclosed in the above embodiment.

[0113] For the specific process of the above cluster log storage method, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described in detail here.

[0114] Furthermore, the memory 22 as a carrier for storing resources may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0115] In addition, the electronic device 20 also includes a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26; wherein the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input / output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0116] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the cluster log storage method disclosed in the aforementioned embodiment.

[0117] For the specific process of the above cluster log storage method, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described in detail here.

[0118] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0119] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0120] The above is a detailed introduction to a cluster log storage method, device, equipment and medium provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A cluster log storage method, characterized in that: include: Count the access frequency of the log files of each target module in the cluster; The target module is a functional module, each functional module includes multiple sub-modules; Determine the compression attribute corresponding to each of the target modules based on the access frequency: first sort the access frequency of the log files of each functional module, select a preset number of functional modules with the highest access frequency, and determine the compression attribute as non-compression, then sort the access frequency of the log files of the sub-modules of the remaining functional modules, and determine the compression attribute of the sub-modules with the highest access frequency of a first preset proportion as non-compression; wherein the compression attribute includes compression and non-compression; When it is detected that there are logs to be stored in the cluster, the compression attribute corresponding to the logs to be stored is determined; If the compression attribute is not compressed, the log to be stored is stored in the log buffer area; otherwise, the log to be stored is compressed to obtain a first compressed log, and the first compressed log is stored in the static storage area; The compression attribute corresponding to the log to be stored is determined as follows: Storing the compression attribute of each target module in a log base library; From the log basic library, the compression attribute of the function module to which the log to be stored belongs is searched. If the compression attribute of the function module is not compressed, it is determined that the compression attribute corresponding to the log to be stored is not compressed; if the compression attribute of the function module is compressed, the compression attribute of the submodule to which the log to be stored belongs is searched. If the compression attribute of the submodule is not compressed, it is determined that the compression attribute corresponding to the log to be stored is not compressed; otherwise, it is determined that the compression attribute corresponding to the log to be stored is compressed.

2. The cluster log storage method according to claim 1, characterized in that: The access frequency of the log files of each target module in the statistical cluster includes: Counting the access frequency of log files of each functional module and each submodule of the functional module; Correspondingly, determining the compression attribute corresponding to each of the target modules based on the access frequency includes: The compression attribute corresponding to each target module is determined based on the access frequency; wherein the access frequency of the functional module whose compression attribute is non-compression is higher than the access frequency of the functional module whose compression attribute is compression, and the access frequency of the sub-module whose compression attribute is non-compression is higher than the access frequency of the sub-module whose compression attribute is compression.

3. The cluster log storage method according to claim 2, characterized in that: The determining, based on the access frequency, the compression attribute corresponding to each of the target modules comprises: Sorting the access frequencies of the log files of each of the functional modules; Filter out a preset number of the functional modules with the highest access frequency to obtain a first module; Determine the compression attribute of the first module as non-compression, and determine the compression attribute of the non-first module in the functional module as compression; sorting the access frequencies of the log files of all submodules other than the first module in the functional module; The compression attribute of the submodules of the first preset proportion with the highest access frequency is determined to be non-compression, and the compression attribute of other submodules other than the first module is determined to be compression.

4. The cluster log storage method according to claim 1, characterized in that: Also includes: When the log to be stored is stored in the log buffer area, the expiration time of the log to be stored is stored in the log basic library; Determining a target log from the log buffer area based on the expiration time at a preset time interval or at a specified time; Compressing the target log to obtain a second compressed log; Migrate the second compressed log to the static storage area.

5. The cluster log storage method according to claim 4, characterized in that: The determining a target log from the log buffer area based on the expiration time includes: Determining an expired log from the log buffer area based on the expiration time; A specified number of the expired logs are randomly selected to obtain a target log.

6. The cluster log storage method according to claim 2, characterized in that: Also includes: When the usage ratio of the log buffer area reaches a preset threshold, the access frequencies of the log files of all sub-modules whose compression attribute is not compressed are sorted; Compressing the log files of the submodules with the second preset ratio having the least access frequency to obtain a third compressed log; Migrate the third compressed log to the static storage area.

7. A cluster log storage device, characterized in that: include: The access frequency statistics module is used to count the access frequency of the log files of each target module in the cluster; The target module is a functional module, each functional module includes multiple sub-modules; The module attribute determination module is used to determine the compression attribute corresponding to each of the target modules based on the access frequency: firstly, the access frequency of the log files of each functional module is sorted, a preset number of functional modules with the highest access frequency are selected, and the compression attribute is determined as non-compression, and then the access frequency of the log files of the sub-modules of the remaining functional modules is sorted, and the compression attribute of the sub-modules with the highest access frequency of the first preset proportion is determined as non-compression; wherein, the compression attribute includes compression and non-compression; The log monitoring module to be stored is used to monitor whether there are logs to be stored in the cluster; A compression attribute determination module, configured to determine the compression attribute corresponding to the log to be stored when the log to be stored monitoring module detects that there is a log to be stored in the cluster; A log storage module to be stored, for storing the log to be stored in a log buffer area if the compression attribute is not compressed, otherwise, compressing the log to be stored to obtain a first compressed log, and storing the first compressed log in a static storage area; A compression attribute storage module, used for storing the compression attribute of each target module in a log basic library; Wherein, the compression attribute determination module is further used for: From the log basic library, the compression attribute of the function module to which the log to be stored belongs is searched. If the compression attribute of the function module is not compressed, it is determined that the compression attribute corresponding to the log to be stored is not compressed; if the compression attribute of the function module is compressed, the compression attribute of the submodule to which the log to be stored belongs is searched. If the compression attribute of the submodule is not compressed, it is determined that the compression attribute corresponding to the log to be stored is not compressed; otherwise, it is determined that the compression attribute corresponding to the log to be stored is compressed.

8. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the cluster log storage method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the cluster log storage method according to any one of claims 1 to 6.

Images