Log compression method and device, electronic equipment and storage medium

Abstract

The invention discloses a log compression method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining original log data, carrying out the preprocessing of the original log data, and analyzing the importance degree of each field in the log data through employing a feature selection algorithm; according to the importance degree, key log features in the log data are extracted, and dimension reduction processing is conducted on the extracted log features through a dimension reduction algorithm to obtain compressed log data; and storing the compressed log data in a database in a binary form. According to the technical scheme, the data dimension in the log compression process can be remarkably reduced, the storage capacity occupied by the compressed data is reduced, meanwhile, important information in the log data is reserved, and therefore the log analysis efficiency and the troubleshooting efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a log compression method, device, electronic equipment and storage medium. Background Art

[0002] In current large-scale distributed systems, the recording of log data is the key to system debugging, monitoring, and user behavior analysis. However, as the scale of the system expands and the complexity of the business increases, the amount of log data increases dramatically, and storage and processing become increasingly difficult. In order to ensure system performance, log data often needs to be compressed and stored, but traditional compression methods do not target the unique structure and characteristics of logs, and often cannot achieve efficient storage and retain key data.

[0003] Traditional log compression schemes mostly use lossless compression algorithms, such as ZIP and GZIP, but these algorithms are not optimized for the characteristics of logs, resulting in low compression ratio and efficiency; secondly, all fields of the logs are stored in traditional schemes, without selecting the most critical fields for troubleshooting, monitoring and analysis, resulting in redundant data occupying a large amount of storage space, and being unable to flexibly handle different types of log data, lacking the ability to dynamically adjust compression strategies, and unable to adapt to the characteristics of different log types; finally, the log decompression process of traditional compression schemes is relatively slow, and it is difficult to selectively restore some fields, which increases the complexity of the system. Summary of the invention

[0004] The present invention provides a log compression method, device, electronic device and storage medium, which can significantly reduce the data dimension in the log compression process, reduce the storage capacity occupied by compressed data, and retain important information in the log data, thereby improving the efficiency of log analysis and troubleshooting.

[0005] According to one aspect of the present invention, a log compression method is provided, the method comprising:

[0006] Obtaining original log data, preprocessing the original log data, and then using a feature selection algorithm to analyze the importance of each field in the log data;

[0007] Extracting key log features from the log data according to the importance, and performing dimensionality reduction processing on the extracted log features using a dimensionality reduction algorithm to obtain compressed log data;

[0008] The compressed log data is stored in a database in binary form.

[0009] Optionally, preprocessing the original log data includes:

[0010] Cleaning the original log data according to the preset log filtering rules;

[0011] Using regular expressions, cleaning up noise data in the original log data;

[0012] According to the structural information of the original log data, the original log data is subjected to feature marking and encoding processing;

[0013] The numerical features in the original log data are standardized.

[0014] Optionally, a feature selection algorithm is used to analyze the importance of each field in the log data, including:

[0015] The Lasso regression algorithm is used to analyze the regression coefficients corresponding to each field in the log data;

[0016] The importance of each field in the log data is determined according to the regression coefficient.

[0017] Optionally, a dimensionality reduction algorithm is used to reduce the dimensionality of the extracted log features to obtain compressed log data, including:

[0018] Determine, according to the extracted log features, a covariance matrix corresponding to the log features, and variance contribution rates of each dimension corresponding to the covariance matrix;

[0019] A dimensionality reduction algorithm is used to map the log features to a low-dimensional space according to the covariance matrix and the variance contribution rate to obtain compressed log data.

[0020] Optionally, storing the compressed log data in a database in binary form includes:

[0021] According to the time series corresponding to the compressed log data, the compressed log data is stored in blocks;

[0022] The feature mapping relationship between the compressed log data and the original log data, and the dimension reduction parameters corresponding to the dimension reduction algorithm are stored in a database.

[0023] Optionally, after storing the compressed log data in binary form in the database, the method further includes:

[0024] In response to the log restoration request, the compressed log data is restored to original log data according to the compressed log data and pre-stored dimensionality reduction parameters.

[0025] Optionally, after storing the compressed log data in binary form in the database, the method further includes:

[0026] According to the preset cycle, the dimension reduction parameters corresponding to the dimension reduction algorithm are dynamically updated according to different log data types and feature distribution results, so as to perform dimension reduction processing on key log features in subsequent log data according to the updated dimension reduction parameters.

[0027] According to another aspect of the present invention, a log compression device is provided, the device comprising:

[0028] A log preprocessing module is used to obtain raw log data, preprocess the raw log data, and then use a feature selection algorithm to analyze the importance of each field in the log data;

[0029] A dimensionality reduction processing module, used to extract key log features from the log data according to the importance, and perform dimensionality reduction processing on the extracted log features using a dimensionality reduction algorithm to obtain compressed log data;

[0030] The log storage module is used to store the compressed log data in a database in binary form.

[0031] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0032] at least one processor; and

[0033] a memory communicatively connected to the at least one processor; wherein,

[0034] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log compression method described in any embodiment of the present invention.

[0035] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log compression method described in any embodiment of the present invention when executed.

[0036] The technical solution provided by the embodiment of the present invention obtains original log data, pre-processes the original log data, then uses a feature selection algorithm to analyze the importance of each field in the log data, extracts key log features in the log data according to the importance, and uses a dimensionality reduction algorithm to perform dimensionality reduction processing on the extracted log features to obtain compressed log data. The technical means of storing the compressed log data in a database in binary form can significantly reduce the data dimension in the log compression process, reduce the storage capacity occupied by compressed data, and retain important information in the log data, thereby improving log analysis efficiency and troubleshooting efficiency.

[0037] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 is a flow chart of a log compression method provided according to an embodiment of the present invention;

[0040] Figure 2 is a flow chart of another log compression method provided according to an embodiment of the present invention;

[0041] Figure 3 is a structural schematic diagram of a log compression device provided according to an embodiment of the present invention;

[0042] Figure 4 It is a structural schematic diagram of an electronic device for implementing the log compression method of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0044] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0045] Figure 1 This is a flow chart of a log compression method provided by an embodiment of the present invention. This embodiment is applicable to the case of compressing large-scale log data in a distributed system. The method can be executed by a log compression device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0046] Step 110: Acquire original log data, pre-process the original log data, and then use a feature selection algorithm to analyze the importance of each field in the log data.

[0047] In this embodiment, the original log data may be log data to be compressed. After obtaining the original log data, the original log data may be preprocessed (for example, cleaning, denoising, removing redundant data, etc.) to ensure that the preprocessed log data is suitable for subsequent dimensionality reduction processing.

[0048] In this step, after preprocessing the original log data, a feature selection algorithm can be used to analyze the importance of each field in the log data according to the type of log data and user needs, and discard unnecessary redundant fields. For example, for user login logs, the "operation type" and "timestamp" fields can be used as important fields, and the "user IP address" or "device ID" fields can be discarded.

[0049] Step 120: extract key log features from the log data according to the importance, and use a dimensionality reduction algorithm to perform dimensionality reduction processing on the extracted log features to obtain compressed log data.

[0050] In this step, field values ​​with high importance can be extracted as key log features. The extracted key log features are in the form of a two-dimensional matrix, where each row represents a piece of log data and each column represents the features of the corresponding log data.

[0051] In a specific embodiment, it is assumed that the original log data is as shown in Table 1:

[0052] Table 1

[0053]

[0054] The key log features extracted by the above method can be shown in Table 2:

[0055] Table 2

[0056] Timestamp Operation Type User ID Operation Results 2023-09-12 10:00 Log in user01 success 2023-09-12 10:05 quit user01 success

[0057] In this step, after extracting the key log features in the log data, a dimensionality reduction algorithm can be used to reduce the dimensionality of the key log features to obtain compressed log data. Optionally, the dimensionality reduction algorithm may include principal component analysis (PCA), linear discriminant analysis (LDA), t-distributed stochastic neighbor embedding (t-SNE) and other dimensionality reduction algorithms, which are not limited in this embodiment.

[0058] In a specific embodiment, assuming that the extracted key log features include four dimensions (i.e., "timestamp", "operation type", "user ID", and "operation result"), after the key log features are reduced in dimension using the above-mentioned dimensionality reduction algorithm, compressed log data as shown in Table 3 can be obtained:

[0059] Table 3

[0060] Principal component 1 Principal component 2 0.95 0.12 0.87 0.22

[0061] The advantage of this setting is that by using a dimensionality reduction algorithm to compress 4-dimensional data into 2-dimensional data, each data point is represented only by "principal component 1" and "principal component 2", which can reduce the amount of data while retaining the key information of the log data.

[0062] Step 130: Store the compressed log data in a database in binary form.

[0063] The technical solution provided by the embodiment of the present invention obtains original log data, pre-processes the original log data, then uses a feature selection algorithm to analyze the importance of each field in the log data, extracts key log features in the log data according to the importance, and uses a dimensionality reduction algorithm to perform dimensionality reduction processing on the extracted log features to obtain compressed log data. The technical means of storing the compressed log data in a database in binary form can significantly reduce the data dimension in the log compression process, reduce the storage capacity occupied by compressed data, and retain important information in the log data, thereby improving log analysis efficiency and troubleshooting efficiency.

[0064] Figure 2 A flowchart of another log compression method provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method includes:

[0065] Step 210: Obtain original log data, pre-process the original log data, and then use the Lasso regression algorithm to analyze the regression coefficient corresponding to each field in the log data.

[0066] In one implementation of the present embodiment, the raw log data is preprocessed, including: cleaning the raw log data according to preset log filtering rules; using regular expressions to clean up noise data in the raw log data; feature marking and encoding the raw log data based on structural information of the raw log data; and standardizing numerical features in the raw log data.

[0067] In a specific embodiment, the original log data can be cleaned by using preset filtering rules (such as removing invalid data, abnormal records or duplicate records). Assume that the original log data is as shown in Table 4:

[0068] Table 4

[0069]

[0070] Among them, there are two completely duplicate log data in Table 4, so one of the redundant log data can be deleted and only the other log data is retained.

[0071] In another specific embodiment, regular expressions and preset rules may be used to remove noise data in the original log data. For example, log data with invalid characters as shown in Table 5 may be removed:

[0072] Table 5

[0073]

[0074] In another specific embodiment, feature marking and encoding can be performed according to the structure of the original log data. For example, for user operation logs, label encoding can be used to encode the operation types "login" and "logout" as 1 and 2 respectively, and the operation result "success" as 1, so that the data is suitable for subsequent processing. Taking the data in Table 2 as an example, the encoded data can be shown in Table 6:

[0075] Table 6

[0076]

[0077]

[0078] In another specific embodiment, the numerical features in the original log data can be standardized so that each feature has a similar distribution to facilitate the processing of the dimensionality reduction algorithm. Assuming that the field "operation duration" of the operation log is unevenly distributed, the field can be standardized by z-score so that its mean is 0 and its variance is 1. For example, assuming that the original log data is as shown in Table 7, the data in Table 8 can be obtained after standardization:

[0079] Table 7

[0080] Timestamp User ID Operation time 2023-09-12 10:00 user01 10 2023-09-12 10:05 user01 20

[0081] Table 8

[0082] Timestamp User ID Operation time 2023-09-12 10:00 user01 -0.707 2023-09-12 10:05 user01 0.707

[0083] In this embodiment, after the original log data is preprocessed, a Lasso regression algorithm may be used to analyze the regression coefficient corresponding to each field in the log data.

[0084] Among them, the Lasso (Least Absolute Shrinkage and Selection Operator) regression algorithm is a linear regression method for feature selection. It limits the size of the regression coefficient in the model by introducing the L1 regularization term, so that the coefficients of some unimportant features shrink to 0, thereby achieving the purpose of feature selection. In the scenario of user login logs, assume that we have multiple features that can be used to analyze user behavior, including but not limited to the following fields:

[0085] Timestamp: records the time of user operation;

[0086] Operation type (operation_type): login, logout, modify information, etc.;

[0087] User ID (user_id): uniquely identifies the user;

[0088] Operation result (operation_result): success or failure;

[0089] IP address (IP_address): records the source IP address of user operations;

[0090] Device ID (device_id): the device ID operated by the user;

[0091] The goal of the Lasso regression algorithm is to select the most predictive features for user behavior by analyzing these fields and remove redundant information, such as IP addresses or device IDs. In order to perform feature selection, the features of the user login log can be encoded and Lasso regression can be applied. Assuming that the goal is to predict whether a user login operation is successful (operation_result is the target variable), we hope to determine which features are more important for predicting whether the login is successful or not through feature selection.

[0092] In a specific embodiment, it is assumed that the original log data is as shown in Table 9:

[0093] Table 9

[0094]

[0095] In order to perform regression analysis, the non-numerical features in Table 9 can be encoded. For example, login is coded as 1, and logout is coded as 2; success is coded as 1, and failure is coded as 0; user ID, IP address, and device ID can be directly label encoded (Label Encoding), and the encoded data is shown in Table 10:

[0096] Table 10

[0097]

[0098]

[0099] The goal of Lasso regression is to find the features that have the greatest impact on the operation result (operation_result) by adjusting the regularization strength. Assuming that we perform Lasso regression on the above data, the feature and regression coefficient mapping table obtained can be shown in Table 11:

[0100] Table 11

[0101] feature Regression coefficient Timestamp 0.1 Operation Type 0.5 User ID 0 IP address 0 Device ID 0

[0102] Step 220: Determine the importance of each field in the log data according to the regression coefficient.

[0103] In this embodiment, the regression coefficient of the operation type (operation_type) is relatively large (0.5), indicating that it is an important factor affecting the success of the login operation. For example, the login operation is more likely to fail than the logout operation. The regression coefficient of the timestamp (timestamp) is 0.1. Although it is not as important as the operation type, it still contributes to the result to a certain extent. For example, the probability of successful login in different time periods may be different. The coefficients of user ID, IP address, and device ID are 0, indicating that the influence of these features on the prediction of the login operation results can be ignored. According to the results of the Lasso regression analysis, these features will be considered redundant features and eliminated.

[0104] Step 230: extract key log features from the log data according to the importance, and determine the covariance matrix corresponding to the log features and the variance contribution rate of each dimension corresponding to the covariance matrix according to the extracted log features.

[0105] In this embodiment, through the feature selection of the Lasso regression algorithm, unimportant features (IP address and device ID) can be discarded, and only "operation type", "timestamp" and "operation result" are retained as key log features, as shown in Table 12:

[0106] Table 12

[0107] Timestamp Operation Type Operation Results 2023-09-12 10:00 1 1 2023-09-12 10:05 1 0 2023-09-12 10:10 2 1 2023-09-12 10:15 1 1

[0108] The advantage of this setting is that it can effectively select features in user login logs that have a greater impact on the prediction results and discard irrelevant or redundant features, thereby improving the compression efficiency of massive log data.

[0109] Step 240: Use a dimensionality reduction algorithm to map the log features to a low-dimensional space according to the covariance matrix and the variance contribution rate to obtain compressed log data.

[0110] In this embodiment, optionally, a PCA algorithm can be used to map log features to a low-dimensional space to obtain compressed log data. The purpose of the PCA algorithm is to retain the most informative part of the data by compressing the dimensions. Specifically, the PCA algorithm will find a new coordinate system (principal component) so that the data loses as little original information as possible in the new coordinate system.

[0111] Taking the 4-dimensional data in Table 2 as an example, the PCA algorithm will analyze the correlation between the 4-dimensional data, find out which changes in the dimensions can best explain the differences in the entire data set, and then generate new coordinate axes (i.e., principal components). Assume that the PCA algorithm generates 2 principal components, which means that the original 4-dimensional data is compressed to 2 dimensions. After compression, each log data is no longer represented by 4 features, but by the values ​​of 2 principal components (as shown in Table 3). Among them, "principal component 1" and "principal component 2" are new coordinate axes, representing the data after PCA dimensionality reduction. These principal components are not simple mappings of the original features, but combinations of the original 4 features. The PCA algorithm ensures that the first principal component (principal component 1) retains as much data variance as possible (i.e., the main information of the data), and the second principal component (principal component 2) retains the remaining secondary information.

[0112] Taking the data in Table 3 as an example, principal component 1 (0.95, 0.87) represents the most important change information in the data, which may contain the main changes in the operation type and timestamp. Principal component 2 (0.12, 0.22) represents minor changes, which may be differences between user IDs or operation results. The reduced "principal component 1" and "principal component 2" may seem unrelated to the original "timestamp", "operation type", etc., but in fact they are linear combinations of the original features and represent the main information of the data. By reducing the dimension, the storage space of the compressed data is saved while retaining the most useful log information.

[0113] Step 250: Store the compressed log data in a database in binary form.

[0114] In one implementation of the present embodiment, the compressed log data is stored in a database in binary form, including: storing the compressed log data in blocks according to the time series corresponding to the compressed log data; storing the feature mapping relationship between the compressed log data and the original log data, and the dimensionality reduction parameters corresponding to the dimensionality reduction algorithm in the database.

[0115] After the compressed log data is stored in blocks, each block of data is compressed separately and marked with a time or operation label to facilitate future retrieval. At the same time, metadata (including the mapping relationship of the original features and the dimensionality reduction parameters) can also be stored in each block to facilitate reverse operations during restoration.

[0116] Step 260: In response to the log restoration request, restore the compressed log data to original log data according to the compressed log data and pre-stored dimensionality reduction parameters.

[0117] In this embodiment, during the log restoration process, the inverse operation of the PCA algorithm can be used to restore the principal component data to a dimension close to the original dimension. For example, assuming that the stored covariance matrix and eigenvector matrix are used, the restored data is shown in Table 13:

[0118] Table 13

[0119] Timestamp User ID Operation Type Operation Results 2023-09-12 10:00 user01 1 1 2023-09-12 10:05 user01 2 1

[0120] For missing fields, you can use interpolation or default values ​​to complete them. For example, assuming that the "Device ID" field is discarded during the dimensionality reduction process, you can use the default device ID to fill in the data in Table 13 during restoration, and get the data shown in Table 14:

[0121] Table 14

[0122] Timestamp User ID Operation Type Operation Results Device ID 2023-09-12 10:00 user01 1 1 device01 2023-09-12 10:05 user01 2 1 device01

[0123] Step 270: dynamically update the dimension reduction parameters corresponding to the dimension reduction algorithm according to different log data types and feature distribution results according to a preset period, so as to perform dimension reduction processing on key log features in subsequent log data according to the updated dimension reduction parameters.

[0124] In this embodiment, each type of log data can be periodically analyzed to evaluate the effectiveness of existing compression methods, and the parameters of the dimensionality reduction algorithm (such as the number of principal components of PCA or the classification criteria of LDA) can be automatically adjusted through a machine learning model to ensure a balance between compression rate and information retention.

[0125] The advantage of this setting is that the compression effect of log data can be optimized by dynamically adjusting the dimensionality reduction algorithm parameters.

[0126] The technical solution provided by the embodiment of the present invention obtains original log data, pre-processes the original log data, and then uses the Lasso regression algorithm to analyze the regression coefficient corresponding to each field in the log data, determines the importance of each field in the log data according to the regression coefficient, extracts the key log features in the log data according to the importance, determines the covariance matrix corresponding to the log features and the variance contribution rate of each dimension corresponding to the covariance matrix according to the extracted log features, uses a dimensionality reduction algorithm, maps the log features to a low-dimensional space according to the covariance matrix and the variance contribution rate, obtains compressed log data, stores the compressed log data in a database in binary form, restores the compressed log data to the original log data according to the compressed log data and the pre-stored dimensionality reduction parameters in response to a log restoration request, and dynamically updates the dimensionality reduction parameters corresponding to the dimensionality reduction algorithm according to different log data types and feature distribution results according to a preset period. The technical means can significantly reduce the data dimension in the log compression process, reduce the storage capacity occupied by the compressed data, and retain important information in the log data, thereby improving the log analysis efficiency and troubleshooting efficiency.

[0127] Figure 3 A structural diagram of a log compression device provided by an embodiment of the present invention, wherein the device is applied to an electronic device, such as Figure 3 As shown, the device includes: a log preprocessing module 310, a dimensionality reduction processing module 320 and a log storage module 330.

[0128] The log preprocessing module 310 is used to obtain the original log data, preprocess the original log data, and then use a feature selection algorithm to analyze the importance of each field in the log data;

[0129] A dimension reduction processing module 320 is used to extract key log features from the log data according to the importance, and perform dimension reduction processing on the extracted log features using a dimension reduction algorithm to obtain compressed log data;

[0130] The log storage module 330 is used to store the compressed log data in a database in binary form.

[0131] The technical solution provided by the embodiment of the present invention obtains original log data, pre-processes the original log data, then uses a feature selection algorithm to analyze the importance of each field in the log data, extracts key log features in the log data according to the importance, and uses a dimensionality reduction algorithm to perform dimensionality reduction processing on the extracted log features to obtain compressed log data. The technical means of storing the compressed log data in a database in binary form can significantly reduce the data dimension in the log compression process, reduce the storage capacity occupied by compressed data, and retain important information in the log data, thereby improving log analysis efficiency and troubleshooting efficiency.

[0132] Based on the above embodiment, the log preprocessing module 310 includes:

[0133] A data cleaning unit, used to clean the original log data according to a preset log filtering rule;

[0134] A noise cleaning unit, used to clean the noise data in the original log data using a regular expression;

[0135] An encoding processing unit, used for performing feature marking and encoding processing on the original log data according to the structural information of the original log data;

[0136] A standardization processing unit, used for performing standardization processing on the numerical features in the original log data;

[0137] The regression analysis unit is used to analyze the regression coefficients corresponding to each field in the log data using the Lasso regression algorithm;

[0138] The importance determination unit is used to determine the importance of each field in the log data according to the regression coefficient.

[0139] The dimension reduction processing module 320 includes:

[0140] A covariance matrix determination unit, used to determine the covariance matrix corresponding to the log feature and the variance contribution rate of each dimension corresponding to the covariance matrix according to the extracted log feature;

[0141] The feature mapping unit is used to adopt a dimensionality reduction algorithm to map the log features to a low-dimensional space according to the covariance matrix and the variance contribution rate to obtain compressed log data.

[0142] The log storage module 330 includes:

[0143] A block storage unit, used for storing the compressed log data in blocks according to the time series corresponding to the compressed log data;

[0144] The metadata storage unit is used to store the feature mapping relationship between the compressed log data and the original log data, and the dimension reduction parameters corresponding to the dimension reduction algorithm in a database.

[0145] The device also includes:

[0146] A log restoration module, configured to, in response to a log restoration request, restore the compressed log data to original log data according to the compressed log data and pre-stored dimensionality reduction parameters;

[0147] The parameter updating module is used to dynamically update the dimension reduction parameters corresponding to the dimension reduction algorithm according to different log data types and feature distribution results at a preset period, so as to perform dimension reduction processing on key log features in subsequent log data according to the updated dimension reduction parameters.

[0148] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not fully described in the embodiments of the present invention, please refer to the methods provided by all the above embodiments of the present invention.

[0149] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0150] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0151] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0152] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a log compression method.

[0153] In some embodiments, the log compression method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the log compression method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the log compression method in any other appropriate manner (e.g., by means of firmware).

[0154] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0156] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0157] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0158] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0159] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0160] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0161] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log compression method, characterized in that: The method comprises: Obtaining original log data, preprocessing the original log data, and then using a feature selection algorithm to analyze the importance of each field in the log data; Extracting key log features from the log data according to the importance, and performing dimensionality reduction processing on the extracted log features using a dimensionality reduction algorithm to obtain compressed log data; The compressed log data is stored in a database in binary form.

2. The method according to claim 1, characterized in that Preprocessing the original log data includes: Cleaning the original log data according to the preset log filtering rules; Using regular expressions, cleaning up noise data in the original log data; According to the structural information of the original log data, the original log data is subjected to feature marking and encoding processing; The numerical features in the original log data are standardized.

3. The method according to claim 1, characterized in that The feature selection algorithm is used to analyze the importance of each field in the log data, including: The Lasso regression algorithm is used to analyze the regression coefficients corresponding to each field in the log data; The importance of each field in the log data is determined according to the regression coefficient.

4. The method according to claim 1, characterized in that The extracted log features are reduced in dimension using a dimensionality reduction algorithm to obtain compressed log data, including: Determine, according to the extracted log features, a covariance matrix corresponding to the log features, and variance contribution rates of each dimension corresponding to the covariance matrix; A dimensionality reduction algorithm is used to map the log features to a low-dimensional space according to the covariance matrix and the variance contribution rate to obtain compressed log data.

5. The method according to claim 1, characterized in that: Storing the compressed log data in a database in binary form includes: According to the time series corresponding to the compressed log data, the compressed log data is stored in blocks; The feature mapping relationship between the compressed log data and the original log data, and the dimension reduction parameters corresponding to the dimension reduction algorithm are stored in a database.

6. The method according to claim 5, characterized in that After storing the compressed log data in a database in binary form, the method further includes: In response to the log restoration request, the compressed log data is restored to original log data according to the compressed log data and pre-stored dimensionality reduction parameters.

7. The method according to claim 1, characterized in that After storing the compressed log data in a database in binary form, the method further includes: According to the preset cycle, the dimension reduction parameters corresponding to the dimension reduction algorithm are dynamically updated according to different log data types and feature distribution results, so as to perform dimension reduction processing on key log features in subsequent log data according to the updated dimension reduction parameters.

8. A log compression device, characterized in that: The device comprises: A log preprocessing module is used to obtain raw log data, preprocess the raw log data, and then use a feature selection algorithm to analyze the importance of each field in the log data; A dimensionality reduction processing module, used to extract key log features from the log data according to the importance, and perform dimensionality reduction processing on the extracted log features using a dimensionality reduction algorithm to obtain compressed log data; The log storage module is used to store the compressed log data in a database in binary form.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log compression method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log compression method according to any one of claims 1 to 7 when executed.

Images