Log data storage method and device, equipment and medium

Abstract

The invention discloses a log data storage method and device, equipment and a medium. The log data storage method comprises the following steps: acquiring log access frequency sample data; determining a log access frequency prediction model according to the log access frequency sample data and the time sequence pre-training model; determining the predicted access frequency of the target log based on the log access frequency prediction model and the log data with the to-be-predicted access frequency; and determining a storage format and a storage device type of the target log according to the predicted access frequency of the target log, and storing the target log according to the storage format and the storage device type of the target log. According to the technical scheme, the storage efficiency can be improved, the storage cost can be reduced, and meanwhile the system performance is optimized.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a log data storage method, device, equipment and medium. Background Art

[0002] With the development of big data technology, the amount of log data has shown explosive growth, and the access frequencies of different types of log data vary greatly. Refined storage management has become an urgent problem to be solved. However, traditional static storage solutions usually adopt static strategies. All log data is stored according to fixed strategies and cannot be dynamically adjusted according to actual access needs, resulting in a waste of storage resources. It is difficult to adapt to the needs of modern systems for low-cost and efficient storage. Summary of the invention

[0003] The present invention provides a log data storage method, device, equipment and medium to solve the problems of low storage efficiency and high storage cost in traditional log storage.

[0004] According to one aspect of the present invention, a log data storage method is provided, comprising:

[0005] Get log access frequency sample data;

[0006] Determine the log access frequency prediction model based on the log access frequency sample data and the time series pre-training model;

[0007] Determine the predicted access frequency of the target log based on the log access frequency prediction model and the access frequency to be predicted log data;

[0008] The access frequency is predicted according to the target log, the storage format and storage device type of the target log are determined, and the target log is stored according to the storage format and storage device type of the target log.

[0009] According to another aspect of the present invention, there is provided a log data storage device, comprising:

[0010] A data acquisition module is used to obtain log access frequency sample data;

[0011] A prediction model determination module is used to determine a log access frequency prediction model based on log access frequency sample data and a time series pre-training model;

[0012] A log access frequency prediction module is used to determine the predicted access frequency of the target log based on the log access frequency prediction model and the access frequency log data to be predicted;

[0013] The data storage module is used to predict the access frequency according to the target log, determine the storage format and storage device type of the target log, and store the target log according to the storage format and storage device type of the target log.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log data storage method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log data storage method described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention obtains log access frequency sample data, thereby determining the log access frequency prediction model according to the log access frequency sample data and the time series pre-training model, and determining the target log predicted access frequency based on the log access frequency prediction model and the access frequency log data to be predicted, and then determining the storage format and storage device type of the target log according to the predicted access frequency of the target log, and storing the target log according to the storage format and storage device type of the target log. In this solution, by predicting the log access frequency, timely predicting whether the log is frequently accessed, and then allocating the appropriate storage format and storage device type to the log based on the predicted access frequency, that is, dynamically adjusting the log storage strategy through the prediction result of the log access frequency, ensuring that the logs with high frequency access are stored in high-speed storage devices, and the logs with low frequency access are stored in low-cost low-speed storage devices, solving the problems of low storage efficiency and high storage cost in traditional log storage, improving storage efficiency, reducing storage cost, and optimizing system performance.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flow chart of a log data storage method provided in Embodiment 1 of the present invention;

[0023] Figure 2 A flow chart of a log data storage method provided in Embodiment 2 of the present invention;

[0024] Figure 3 A structural diagram of a log data storage device provided in Embodiment 3 of the present invention;

[0025] Figure 4 A schematic diagram of the structure of an electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Embodiment 1

[0029] Figure 1This is a flow chart of a log data storage method provided in the first embodiment of the present invention. This embodiment is applicable to the case of efficiently storing log data. The method can be executed by a log data storage device. The log data storage device can be implemented in the form of hardware and / or software. The log data storage device can be configured in an electronic device. The electronic device can be a computer or a server, etc. Figure 1 As shown, the method includes:

[0030] Step 110: Obtain log access frequency sample data.

[0031] The log access frequency sample data may be data obtained by extracting access frequency features from the collected log data according to the log type. The log type may include login type, payment type, etc. The log access frequency sample data may include but is not limited to log time, log type, and log access frequency, etc.

[0032] Step 120: Determine a log access frequency prediction model based on the log access frequency sample data and the time series pre-training model.

[0033] Among them, the time series pre-training model can be a model that can analyze the input using the time series. Exemplarily, the time series pre-training model can include but is not limited to LSTM (Long short-term memory). The log access frequency prediction model can be a model obtained by training the time series pre-training model based on the log access frequency sample data. The log access frequency prediction model can be used to predict the number of log accesses.

[0034] In the embodiment of the present invention, the time series pre-training model can be trained using the log access frequency sample data, and the model that meets the accuracy requirements after training is used as the log access frequency prediction model.

[0035] In a specific example, the LSTM model can be used as a time series pre-training model. The input of the time series pre-training model is usually a sliding window, which uses the access frequency of the past n days of logs as input to predict the access frequency of the next day. For example, assuming n = 7, the access frequency of the past 7 days is used as input to predict the number of visits on the 8th day. The input format that the log access frequency prediction model can accept can be (number of samples, time step, feature dimension), where the feature dimension is 1, that is, the feature of access frequency. The number of samples can be the number of samples used in this model training. The time step is the number of days of the prediction time. The LSTM model consists of the following key components: input layer, LSTM layer, fully connected layer, and output layer. The input layer size is (N,1), that is, the access frequency data of the past N days. The LSTM layer can have multiple units (for example, 64 units) for processing input data. The fully connected layer outputs a single value, which represents the predicted access frequency of the predicted day (for example, write 1, that is, predict the access frequency of the next day, write 7, that is, output the predicted value of the next 7 days).

[0036] Step 130: Determine the predicted access frequency of the target log based on the log access frequency prediction model and the access frequency to be predicted log data.

[0037] The access frequency to be predicted log data may be log data for which access frequency prediction is required. The target log predicted access frequency may be a log access frequency outputted by analyzing the access frequency to be predicted log data based on a log access frequency prediction model.

[0038] In an embodiment of the present invention, the access frequency log data to be predicted can be converted into a time series format, and the data after the format conversion can be input into a log access frequency prediction model to predict the access frequency of the corresponding type of log of the access frequency log data to be predicted in the future through the log access frequency prediction model, that is, to obtain the predicted access frequency of the target log.

[0039] Optionally, the log access frequency prediction model predicts the access frequency of logs of a corresponding type of the log data with access frequency to be predicted in one or more days in the future.

[0040] Step 140: predict the access frequency of the target log, determine the storage format and storage device type of the target log, and store the target log according to the storage format and storage device type of the target log.

[0041] Among them, the target log may be a log corresponding to the predicted access frequency of the target log, for predicting the access frequency. The storage format may be the storage method of the target log. Optionally, the storage format may include uncompressed storage and compressed storage (compression may be performed using compression algorithms such as LZ4 and Gzip). The storage device type may be the type of device that stores log data. The storage device type may include high-speed storage devices, conventional storage devices, and low-speed storage devices. Exemplarily, high-speed storage devices may include, but are not limited to, SSDs (Solid State Disks). Conventional storage devices may include, but are not limited to, ordinary hard disks. Low-speed storage devices may include, but are not limited to, HDDs (Hard Disk Drives) or cloud storage devices, etc.

[0042] In an embodiment of the present invention, the access frequency can be predicted based on the target log to determine whether the target log is frequently accessed, thereby determining the storage format and storage device type of the target log based on the storage strategy corresponding to the access frequency of the target log, and storing the target log according to the storage format and storage device type of the target log.

[0043] Optionally, when the target log has a high access frequency, a high-speed storage strategy is used (i.e., the target log is not compressed and stored in a high-speed storage device); when the target log has a normal access frequency, a normal storage strategy is used (i.e., the target log is not compressed and stored in a regular storage device); when the target log has a low access frequency or gradually decreases, a low-speed storage strategy is used (i.e., the target log is compressed and stored in a low-speed storage device). The high and low access frequency classification can be set by the user.

[0044] For example, if the target logs from 2024-09-10 to 2024-09-12 are predicted to have a high access frequency, they are stored in high-speed storage such as SSD to ensure efficient reading performance. If the target logs from 2024-09-13 to 2024-09-16 are predicted to have a low access frequency, they are migrated to HDD or cloud storage devices for compressed storage to save storage costs.

[0045] For example, select the LZ4 compression algorithm for data compression. Assuming that the target log is 50 GB before compression and the compression ratio is 3:1, the space occupied by the compressed data is 16.67 GB. The compressed log data can be migrated to the HDD for long-term storage.

[0046] Through this solution, log storage can be dynamically adjusted based on log access requirements. That is, it is predicted that the target log access frequency will be low or gradually decrease in the future, and it will be migrated from the high-speed storage device to the low-cost low-speed storage device; conversely, the frequently accessed target log will be stored on the high-speed storage device first to avoid the waste of storage resources. Cold data can be automatically migrated to the low-cost low-speed storage device to reduce storage costs.

[0047] The technical solution of the embodiment of the present invention obtains log access frequency sample data, thereby determining the log access frequency prediction model according to the log access frequency sample data and the time series pre-training model, and determining the target log predicted access frequency based on the log access frequency prediction model and the access frequency log data to be predicted, and then determining the storage format and storage device type of the target log according to the predicted access frequency of the target log, and storing the target log according to the storage format and storage device type of the target log. In this solution, by predicting the log access frequency, timely predicting whether the log is frequently accessed, and then allocating the appropriate storage format and storage device type to the log based on the predicted access frequency, that is, dynamically adjusting the log storage strategy through the prediction result of the log access frequency, ensuring that the logs with high frequency access are stored in high-speed storage devices, and the logs with low frequency access are stored in low-cost low-speed storage devices, solving the problems of low storage efficiency and high storage cost in traditional log storage, improving storage efficiency, reducing storage cost, and optimizing system performance.

[0048] Embodiment 2

[0049] Figure 2 This is a flow chart of a log data storage method provided in the second embodiment of the present invention. This embodiment is specific based on the above embodiment and provides a specific optional implementation method for determining the predicted access frequency of the target log based on the log access frequency prediction model and the access frequency log data to be predicted. Figure 2 As shown, the method includes:

[0050] Step 210: Obtain log access frequency sample data.

[0051] In an optional embodiment of the present invention, obtaining log access frequency sample data may include: obtaining target historical log data; performing data preprocessing on the target historical log data to obtain preprocessed historical log data, and counting the log access frequency of the preprocessed historical log data by log type to obtain log access frequency sample data.

[0052] The target historical log data may be log data of a period of time in the past. The embodiment of the present invention does not limit the format and source of the target historical log data. The pre-processed historical log data may be data obtained by processing the target historical log data based on a preset data processing rule.

[0053] In an embodiment of the present invention, target historical log data can be obtained, and then data preprocessing is performed on the target historical log data based on preset data processing rules to obtain preprocessed historical log data, ensuring that the preprocessed historical log data is in a unified data format, and according to the log type and the preset data statistical period, the access frequency of various types of logs in the preprocessed historical log data is counted to obtain log access frequency sample data.

[0054] For example, assume that the following target historical log data is collected:

[0055] [2024-09-10 10:00:01] User A successfully logged in (source: auth-service, request ID: 001);

[0056] [2024-09-10 10:00:02] User A successfully logged in (source: auth-service, request ID: 001);

[0057] [2024-09-10 10:01:00] User B's payment failed (source: payment-service, request ID: 002);

[0058] [2024-09-10 10:02:00] User C successfully queried the order (source: order-service, request ID: 003);

[0059] [2024-09-10 10:02:05] User B's payment is successful (source: payment-service, request ID: 004);

[0060] The above-mentioned target historical log data is deduplicated, duplicate entries are deleted and key information is retained, and then the logs are organized into a unified format through standardized operations to obtain the following preprocessed historical log data:

[0061] [2024-09-10 10:00:01] User A successfully logged in (auth-service, request ID: 001);

[0062] [2024-09-10 10:01:00] User B payment failed (payment-service, request ID: 002);

[0063] [2024-09-10 10:02:00] User C successfully queried the order (order-service, request ID: 003);

[0064] [2024-09-10 10:02:05] User B's payment is successful (payment-service, request ID: 004).

[0065] In a specific example, based on different log types (such as login logs, payment logs, etc.), the time series information of pre-processed historical log data is modeled, and the access frequency of each type of log in the past period of time is counted. The final access frequency sample data of the payment type log can be referred to as follows:

[0066] Date: 2024-08-10, number of successful payment log accesses: 320;

[0067] Date: 2024-08-11, number of successful payment log accesses: 350;

[0068] Date: 2024-08-12, number of successful payment log accesses: 330;

[0069] ……;

[0070] Date: 2024-09-08, number of successful payment log accesses: 380;

[0071] Date: 2024-09-09, number of accesses to the payment success log: 400.

[0072] Step 220: Determine a log access frequency prediction model based on the log access frequency sample data and the time series pre-training model.

[0073] In an optional embodiment of the present invention, determining a log access frequency prediction model based on log access frequency sample data and a time series pre-trained model may include: converting the log access frequency sample data into a time series format according to the log type to obtain a time series training sample; and using the time series training sample to train a time series pre-trained model to obtain a log access frequency prediction model.

[0074] The time series training samples may be sample data obtained by converting log access frequency sample data into a time series format.

[0075] In an embodiment of the present invention, the log access frequency sample data can be converted into a time series format for each type of log according to the log type to obtain a time series training sample, and then the time series training sample can be used to train and optimize the iterative time series pre-training model to obtain a log access frequency prediction model.

[0076] In an optional embodiment of the present invention, using time series training samples to train a time series pre-training model to obtain a log access frequency prediction model can include: dividing the time series training samples according to a preset sample division ratio to obtain a time series training set and a time series verification set; training the time series pre-training model based on the time series training set, and adjusting the weight parameters of the time series pre-training model based on back propagation and gradient descent algorithms to obtain a time series model to be generalized; optimizing the time series model to be generalized based on the time series verification set and the loss function to obtain a log access frequency prediction model.

[0077] Among them, the preset sample division ratio can be a pre-set division ratio for the time series training samples, and the specific division ratio value can be set by the user. The time series training set can be a sample set in the time series training samples used to train the time series pre-training model. The time series validation set can be a sample set used to evaluate the generalization ability of the time series pre-training model. The time series model to be generalized can be a neural network obtained after the time series pre-training model completes the model generalization. Optionally, the loss function can specifically include but is not limited to the mean square error loss function and the cross entropy loss function.

[0078] In an embodiment of the present invention, a preset sample division ratio configured by a user can be obtained first, and then the time series training samples can be divided based on the preset sample division ratio to obtain a time series training set and a time series verification set, so as to train a time series pre-training model using the time series training set. During the training process, the time series pre-training model adjusts the weight parameters of the network in the time series pre-training model through back propagation and gradient descent algorithms to obtain a time series model to be generalized. The time series verification set is further used to generalize the time series model to be generalized, and the loss function is used to evaluate the generalized time series model to be generalized, and iterative optimization is performed to improve the model accuracy to generate a log access frequency prediction model.

[0079] Step 230: Based on the current model incremental training data set, incremental training is performed on the log access frequency prediction model to obtain a current incremental training prediction model.

[0080] The current model incremental training data set may be log access frequency data collected based on a model update cycle, and the current incremental training prediction model may be a log access frequency prediction model after incremental training is completed.

[0081] In an embodiment of the present invention, based on the model update cycle, the current model incremental training data set can be obtained, and then the current model incremental training data set can be added to the log access frequency sample data, and the log access frequency prediction model can be incrementally trained to obtain the current incremental training prediction model. By continuously improving the prediction ability of the model, it is ensured that its prediction results can keep up with the changes in the log access pattern to ensure the accuracy of the model prediction.

[0082] For example, if the model update cycle is one day, then every day, the log access data of the previous day (i.e., the current model incremental training data set) is added to the log access frequency sample data, and the model is incrementally trained. The model update cycle can also be three days or one week, etc., for users to adjust.

[0083] Step 240: Determine the predicted access frequency of the target log according to the access frequency to be predicted log data and the current incremental training prediction model.

[0084] In an embodiment of the present invention, the access frequency log data to be predicted is converted into a time series format, and the data after the format conversion is input into the current incremental training prediction model, so as to predict the access frequency of the corresponding type of log of the access frequency log data to be predicted in the future through the current incremental training prediction model, that is, to obtain the predicted access frequency of the target log.

[0085] Step 250: predict the access frequency of the target log, determine the storage format and storage device type of the target log, and store the target log according to the storage format and storage device type of the target log.

[0086] In an optional embodiment of the present invention, determining the storage format and storage device type of the target log based on the predicted access frequency of the target log may include: determining the access frequency exceeding limit time window based on the predicted access frequency of the target log, the lower limit threshold of the access frequency, and the upper limit threshold of the access frequency; if the access frequency exceeding limit type of the target log is the first exceeding limit type within the access frequency exceeding limit time window, the storage format of the target log is non-compressed storage, and the storage device type is a high-speed storage type; if the access frequency exceeding limit type of the target log is the second exceeding limit type within the access frequency exceeding limit time window, the storage format of the target log is compressed storage, and the storage device type is a low-speed storage type.

[0087] Among them, the lower limit threshold of access frequency can be a preset minimum comparison number of access frequency. The upper limit threshold of access frequency can be a preset maximum comparison number of access frequency. The access frequency over-limit time window can be a time period when the target log predicts that the access frequency is lower than the lower limit threshold of access frequency, or higher than the upper limit threshold of access frequency. The first over-limit type can be a type that exceeds the upper limit threshold of access frequency. The second over-limit type can be a type that is lower than the lower limit threshold of access frequency.

[0088] In an embodiment of the present invention, the predicted access frequency of the target log can be compared with the lower threshold of the access frequency and the upper threshold of the access frequency, respectively, and the time when the predicted access frequency of the target log is greater than or equal to the upper threshold of the access frequency, or the time when the predicted access frequency of the target log is less than or equal to the lower threshold of the access frequency, is determined to obtain an access frequency over-limit time window. When the access frequency over-limit type of the target log is the first over-limit type in the access frequency over-limit time window, based on the preset log storage policy, it can be determined that the storage format of the target log is non-compressed storage, and the storage device type is a high-speed storage type. When the access frequency over-limit type of the target log is the second over-limit type in the access frequency over-limit time window, based on the preset log storage policy, it can be determined that the storage format of the target log is compressed storage, and the storage device type is a low-speed storage type.

[0089] In an optional embodiment of the present invention, after storing the target log according to the storage format and storage device type of the target log, it may also include: when the access frequency limit type is the second limit type and it is monitored that the target log access frequency continues to decrease, according to the log life cycle management rules, the target log is automatically deleted or archived.

[0090] The log lifecycle management rules may be log lifecycle management rules pre-set by the customer.

[0091] In the embodiment of the present invention, if the target log's access frequency over-limit type is the second over-limit type in the access frequency over-limit time window, it indicates that the target log has a low access frequency, and the target log access frequency is further monitored. When it is detected that the target log access frequency continues to decrease (the continuous decrease can be defined by the user, such as a decrease for three or five consecutive days), the target log is automatically deleted or archived according to the log life cycle management rules. This solution, through the log life cycle management rules, gets rid of the drawbacks of the log life cycle relying on manual intervention, reduces management complexity, and reduces the occurrence of data management errors.

[0092] For example, when it is monitored that the access frequency of logs from 2024-09-13 to 2024-09-16 continues to decrease and it is predicted that the logs will hardly be accessed in the next 30 days, these logs can be archived to cloud storage and scheduled to be automatically deleted after 90 days.

[0093] The technical solution of the embodiment of the present invention obtains log access frequency sample data, thereby determining the log access frequency prediction model according to the log access frequency sample data and the time series pre-training model, further incrementally training the log access frequency prediction model based on the current model incremental training data set, obtaining the current incremental training prediction model, and determining the target log predicted access frequency according to the access frequency to be predicted log data and the current incremental training prediction model, so as to determine the storage format and storage device type of the target log according to the predicted access frequency of the target log, and store the target log according to the storage format and storage device type of the target log. In this solution, by predicting the log access frequency, timely predicting whether the log is frequently accessed, so as to allocate the appropriate storage format and storage device type to the log based on the predicted access frequency, that is, dynamically adjusting the log storage strategy through the prediction result of the log access frequency, ensuring that the logs with high frequency access are stored in the high-speed storage device, and the logs with low frequency access are stored in the low-cost low-speed storage device, solving the problems of low storage efficiency and high storage cost in traditional log storage, improving storage efficiency, reducing storage cost, and optimizing system performance.

[0094] Embodiment 3

[0095] Figure 3 This is a structural diagram of a log data storage device provided in Embodiment 3 of the present invention.

[0096] like Figure 3 As shown, the device comprises:

[0097] The data acquisition module 310 is used to acquire log access frequency sample data;

[0098] A prediction model determination module 320 is used to determine a log access frequency prediction model based on log access frequency sample data and a time series pre-training model;

[0099] The log access frequency prediction module 330 is used to determine the predicted access frequency of the target log based on the log access frequency prediction model and the access frequency to be predicted log data;

[0100] The data storage module 340 is used to predict the access frequency according to the target log, determine the storage format and storage device type of the target log, and store the target log according to the storage format and storage device type of the target log.

[0101] The technical solution of the embodiment of the present invention obtains log access frequency sample data, thereby determining the log access frequency prediction model according to the log access frequency sample data and the time series pre-training model, and determining the target log predicted access frequency based on the log access frequency prediction model and the access frequency log data to be predicted, and then determining the storage format and storage device type of the target log according to the predicted access frequency of the target log, and storing the target log according to the storage format and storage device type of the target log. In this solution, by predicting the log access frequency, timely predicting whether the log is frequently accessed, and then allocating the appropriate storage format and storage device type to the log based on the predicted access frequency, that is, dynamically adjusting the log storage strategy through the prediction result of the log access frequency, ensuring that the logs with high frequency access are stored in high-speed storage devices, and the logs with low frequency access are stored in low-cost low-speed storage devices, solving the problems of low storage efficiency and high storage cost in traditional log storage, improving storage efficiency, reducing storage cost, and optimizing system performance.

[0102] Optionally, the data acquisition module 310 is specifically used to acquire target historical log data; perform data preprocessing on the target historical log data to obtain preprocessed historical log data, and count the log access frequency of the preprocessed historical log data by log type to obtain the log access frequency sample data.

[0103] Optionally, the prediction model determination module 320 is specifically used to convert the log access frequency sample data into a time series format according to the log type to obtain a time series training sample; and use the time series training sample to train the time series pre-training model to obtain the log access frequency prediction model.

[0104] Optionally, the log access frequency prediction module 330 is specifically used to perform incremental training on the log access frequency prediction model based on the current model incremental training data set to obtain a current incremental training prediction model; and determine the target log predicted access frequency according to the access frequency log data to be predicted and the current incremental training prediction model.

[0105] Optionally, the data storage module 340 is specifically used to determine the access frequency exceeding limit time window based on the predicted access frequency of the target log, the lower limit threshold of the access frequency, and the upper limit threshold of the access frequency; if the access frequency exceeding limit type of the target log is the first exceeding limit type in the access frequency exceeding limit time window, the storage format of the target log is non-compressed storage, and the storage device type is a high-speed storage type; if the access frequency exceeding limit type of the target log is the second exceeding limit type in the access frequency exceeding limit time window, the storage format of the target log is compressed storage, and the storage device type is a low-speed storage type.

[0106] Optionally, the prediction model determination module 320 is used to divide the time series training samples according to a preset sample division ratio to obtain a time series training set and a time series verification set; based on the time series training set, the time series pre-training model is trained, and based on the back propagation and gradient descent algorithm, the weight parameters of the time series pre-training model are adjusted to obtain the time series model to be generalized; based on the time series verification set and the loss function, the time series model to be generalized is optimized to obtain the log access frequency prediction model.

[0107] Optionally, the log data storage device also includes a log lifecycle management module, which is used to automatically delete or archive the target log according to the log lifecycle management rules when the access frequency limit type is the second limit type and the access frequency of the target log is monitored to continue to decrease.

[0108] The log data storage device provided in the embodiment of the present invention can execute the log data storage method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0109] Embodiment 4

[0110] Figure 4 A schematic diagram of an electronic device that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0111] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0112] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0113] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 performs the various methods and processes described above, such as a log data storage method.

[0114] In some embodiments, the log data storage method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the log data storage method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the log data storage method in any other appropriate manner (e.g., by means of firmware).

[0115] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0117] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0119] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0120] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of traditional physical hosts and VPS servers, which are difficult to manage and have weak business scalability.

[0121] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0122] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log data storage method, characterized in that: include: Get log access frequency sample data; Determine the log access frequency prediction model based on the log access frequency sample data and the time series pre-training model; Determine the predicted access frequency of the target log based on the log access frequency prediction model and the access frequency to be predicted log data; The storage format and storage device type of the target log are determined according to the predicted access frequency of the target log, and the target log is stored according to the storage format and storage device type of the target log.

2. The method according to claim 1, characterized in that The obtaining of log access frequency sample data includes: Get the target historical log data; The target historical log data is preprocessed to obtain preprocessed historical log data, and the log access frequency of the preprocessed historical log data is counted according to log type to obtain the log access frequency sample data.

3. The method according to claim 1, characterized in that Determining the log access frequency prediction model according to the log access frequency sample data and the time series pre-training model includes: According to the log type, the log access frequency sample data is converted into a time series format to obtain a time series training sample; The time series training samples are used to train the time series pre-training model to obtain the log access frequency prediction model.

4. The method according to claim 1, characterized in that The step of determining the predicted access frequency of the target log based on the log access frequency prediction model and the access frequency to be predicted log data includes: Based on the current model incremental training data set, incremental training is performed on the log access frequency prediction model to obtain a current incremental training prediction model; The predicted access frequency of the target log is determined according to the access frequency to be predicted log data and the current incremental training prediction model.

5. The method according to claim 1, characterized in that The step of predicting the access frequency of the target log and determining the storage format and storage device type of the target log includes: Determine the access frequency exceeding the limit time window according to the predicted access frequency of the target log, the lower limit threshold of the access frequency, and the upper limit threshold of the access frequency; If the access frequency over-limit type of the target log is the first over-limit type in the access frequency over-limit time window, the storage format of the target log is non-compressed storage, and the storage device type is a high-speed storage type; If the access frequency exceeding limit type of the target log is the second exceeding limit type in the access frequency exceeding limit time window, the storage format of the target log is compressed storage, and the storage device type is a low-speed storage type.

6. The method according to claim 3, characterized in that The using the time series training samples to train the time series pre-training model to obtain the log access frequency prediction model includes: Dividing the time series training samples according to a preset sample division ratio to obtain a time series training set and a time series verification set; Based on the time series training set, the time series pre-training model is trained, and based on the back propagation and gradient descent algorithm, the weight parameters of the time series pre-training model are adjusted to obtain the time series model to be generalized; Based on the time series validation set and the loss function, the time series model to be generalized is optimized to obtain the log access frequency prediction model.

7. The method according to claim 5, characterized in that After storing the target log according to the storage format and storage device type of the target log, the method further includes: When the access frequency over-limit type is the second over-limit type and when it is monitored that the access frequency of the target log continues to decrease, the target log is automatically deleted or archived according to the log lifecycle management rule.

8. A log data storage device, characterized in that: include: A data acquisition module is used to obtain log access frequency sample data; A prediction model determination module is used to determine a log access frequency prediction model based on log access frequency sample data and a time series pre-training model; A log access frequency prediction module, used to determine the predicted access frequency of the target log based on the log access frequency prediction model and the access frequency to be predicted log data; The data storage module is used to predict the access frequency according to the target log, determine the storage format and storage device type of the target log, and store the target log according to the storage format and storage device type of the target log.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log data storage method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log data storage method according to any one of claims 1 to 7 when executed.

Images