A database-based log processing method, apparatus, and device.

Abstract

This article belongs to the financial field, specifically the field of log processing. It provides a database-based log processing method, apparatus, and device. The method includes: receiving log files; determining the type information of the log files according to a preset classification model, and storing the log files in a designated folder corresponding to the type information; generating index path information for the log files based on the log files and the designated folder; and updating the database's search engine based on the index path information. This method enables optimized processing of log files in a database, facilitating log statistics and extraction, and improving the efficiency of log file management.

Description

Technical Field

[0001] This article belongs to the field of database technology, specifically relating to a database-based log processing method, apparatus, and device. Background Technology

[0002] Log files are one of the primary methods currently used to record system operation status and events. For example, user actions, system status, and exception information can be recorded in log files. Based on these records, developers can effectively receive technical operational assurance and a basis for improvement.

[0003] However, with the continuous development of internet technology and network environment, more and more terminals are accessing the internet. Therefore, most network service systems need to operate 24 / 7, generating a large amount of data during operation, including numerous log files. Due to the sheer number of log files, they often contain and interfere with each other, and many contain irrelevant content. Traditional log management methods involve sending system logs periodically or in real-time, achieving centralized log transmission, but without analyzing and organizing the logs, resulting in low efficiency in log extraction and management. Summary of the Invention

[0004] In view of the above-mentioned problems in the prior art, the purpose of this paper is to provide a database-based log processing method, apparatus and device that can improve the efficiency of log management.

[0005] To solve the above-mentioned technical problems, the specific technical solution presented in this paper is as follows:

[0006] On the one hand, this paper provides a database-based log processing method, which includes:

[0007] Receive log files;

[0008] Based on a preset classification model, the type information of the log file is determined, and the log file is stored in a designated folder corresponding to the type information;

[0009] Generate the index path information of the log file based on the log file and the specified folder;

[0010] Update the search engine of the database based on the index path information.

[0011] Furthermore, the process after receiving the log file also includes:

[0012] The log files are stored in a first folder, which includes at least one designated folder;

[0013] The log file is preprocessed, including at least virus scanning and log compression.

[0014] Further, storing the log file in the first folder includes:

[0015] Determine whether the size of the log file is greater than a first preset threshold;

[0016] If so, the log file is divided according to the preset segmentation rules so that the size of each sub-log file after segmentation is less than the preset threshold.

[0017] Each of the split sub-log files is sent to the first folder, and the sub-log files are merged according to the preset merging rules to form the original log file.

[0018] Furthermore, the preset classification model is obtained through the following steps:

[0019] Retrieve historical log files;

[0020] Based on the historical log files, calculate the similarity between any two historical log files;

[0021] Based on the similarity and preset clustering rules, the historical log files are clustered to obtain multiple sets of historical log files;

[0022] According to the preset annotation rules, each set of historical log files is annotated to determine the annotation information of each historical log file;

[0023] The historical log files and their corresponding annotation information are fed into the initial classification model for training, resulting in a pre-trained classification model.

[0024] Alternatively, the initial classification model can be a KNN classifier or an SVM classifier.

[0025] Furthermore, after storing the log file in the designated folder corresponding to the type information, the process further includes:

[0026] The log file is segmented into words to obtain the data information of the log file;

[0027] Determine whether the data is invalid or improperly formatted;

[0028] When invalid data exists, the invalid data should be cleaned up.

[0029] When there is irregular data, the irregular data is converted according to the preset conversion rules.

[0030] Furthermore, after storing the log file in the designated folder corresponding to the type information, the process further includes:

[0031] The log files in the specified folder are compressed according to a specified period.

[0032] Furthermore, after storing the log file in the designated folder corresponding to the type information, the process further includes:

[0033] Get the total size of log files in each specified folder;

[0034] Determine whether the total size of the log files exceeds a preset second threshold;

[0035] If so, then the specified folders whose total size of the log files exceeds the second preset threshold are identified, and the specified folders are compressed according to the preset compression rules until the total size of the log files in all specified folders is lower than the second preset threshold.

[0036] Further, generating the index path information of the log file based on the log file and the specified folder includes:

[0037] Obtain the configuration file information of the specified folder and the location information of the log file in the specified folder;

[0038] Based on the configuration file information and the location information, determine the archive file location information of the log file;

[0039] Based on the archive file location information, the index path information of the log file is generated, whereby the index path information is the log capture path of the log file.

[0040] Furthermore, the method also includes:

[0041] Retrieve log query information within a preset time period;

[0042] Based on the log query information, determine the log files whose query count meets the preset conditions;

[0043] Adjust the location information of the log file in the specified folder to obtain the index path information of the adjusted log file;

[0044] The search engine of the database is updated based on the adjusted index path information of the log file.

[0045] On the other hand, this paper also provides a database-based log processing device, the device comprising:

[0046] The log file receiving module is used to receive log files;

[0047] The log file processing module is used to determine the type information of the log file according to a preset classification model, and store the log file in a specified folder corresponding to the type information;

[0048] The index path information generation module is used to generate the index path information of the log file based on the log file and the specified folder;

[0049] The search engine update module is used to update the search engine of the database based on the index path information.

[0050] On the other hand, this document also provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described above.

[0051] Finally, this document also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0052] By adopting the above technical solution, the database-based log processing method, apparatus, and device described in this paper classify the received log files according to a preset classification model, thereby realizing the classified storage of log files. Then, based on the stored log files, corresponding index path information is generated, realizing the updating of the database search engine. This paper can optimize the processing of log files in the database, facilitate log statistics and extraction, and improve the management efficiency of log files.

[0053] To make the above and other objects, features and advantages of this document more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments or prior art described herein, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this article. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 The diagram illustrates the steps of the database-based log processing method provided in the embodiments of this article;

[0056] Figure 2 This document illustrates the log file processing steps in an embodiment of the invention.

[0057] Figure 3 This document illustrates a schematic diagram of the steps for determining the preset classification model in an embodiment of the invention.

[0058] Figure 4 This document illustrates a schematic diagram of the specified folder processing steps in an embodiment of the invention.

[0059] Figure 5 This document illustrates the log file optimization steps in an embodiment of the invention.

[0060] Figure 6 A schematic diagram of the log processing system structure provided in the embodiments of this article is shown;

[0061] Figure 7 A schematic diagram of the structure of the database-based log processing device provided in the embodiments of this article is shown;

[0062] Figure 8 A schematic diagram of the computer device structure provided in the embodiments of this article is shown.

[0063] Explanation of symbols in the attached drawings:

[0064] 100. Log file receiving module;

[0065] 200. Log file processing module;

[0066] 300. Index path information generation module;

[0067] 400. Search engine update module;

[0068] 802. Computer equipment;

[0069] 804, Processor;

[0070] 806. Memory;

[0071] 808. Drive mechanism;

[0072] 810. Input / Output Module;

[0073] 812. Input devices;

[0074] 814. Output devices;

[0075] 816. Presentation equipment;

[0076] 818. Graphical User Interface;

[0077] 820. Network interface;

[0078] 822. Communication link;

[0079] 824. Communication bus. Detailed Implementation

[0080] The technical solutions in the embodiments described below will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments described herein, and not all of the embodiments. Based on the embodiments described herein, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this document.

[0081] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0082] With the continuous development of internet technology and network environment, more and more terminals are connected to the internet, resulting in a large number of log files generated during system operation. Due to the excessive number of log files, different log files may contain and affect each other, and these log files also contain a lot of irrelevant content. Traditional log management methods involve sending system logs periodically or in real time, achieving centralized transmission of logs, but without analyzing and organizing the logs, resulting in low efficiency in log extraction and management.

[0083] To address the aforementioned issues, this paper provides a database-based log processing method that can improve log management efficiency. Figure 1 This is a schematic diagram illustrating the steps of a database-based log processing method provided in the embodiments of this document. This specification provides the operational steps of the method described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or device products, the methods shown in the embodiments or accompanying drawings can be executed sequentially or in parallel. Specifically, as shown... Figure 1 As shown, the method may include:

[0084] S101: Receive log file;

[0085] S102: Determine the type information of the log file according to the preset classification model, and store the log file in the specified folder corresponding to the type information;

[0086] S103: Generate the index path information of the log file based on the log file and the specified folder;

[0087] S104: Update the search engine of the database according to the index path information.

[0088] This manual can be understood as follows: by classifying the received log files in real time using a preset classification model, the manual stores them in the corresponding designated folders, thereby achieving rapid classification of different log files. Based on the classified log files and their designated folders, the manual generates the index path information of the log files and updates the search engine, thereby improving the efficiency of log file management and enabling rapid processing based on the storage location of the logs during log extraction and processing.

[0089] The log file can be information recorded during system operation. Taking a financial system as an example, user operation information, system operation status, and abnormal information in the banking operation platform are recorded in the form of log files and stored in a database (such as a MySQL database) to facilitate subsequent extraction and analysis of the log files, thereby resolving the corresponding abnormal information or optimizing the system operation.

[0090] The designated folder can be the storage space for different log files obtained from different classification rules within the database. By storing log files of the same type in the same designated folder, preliminary processing of log files is achieved, which facilitates log file statistics and further optimization, reduces the operating pressure on the server, and improves the efficiency of data statistics and optimization.

[0091] In this embodiment of the specification, after receiving the log file, the method further includes:

[0092] The log files are stored in a first folder, which includes at least one designated folder;

[0093] The log file is preprocessed, including at least virus scanning and log compression.

[0094] The first folder can be a database space for storing the log files. The log files are preprocessed within this database space, such as virus scanning and log compression. Optionally, a virus scanning script can be set to scan the initial log files for viruses. If a virus is found in the initial log file, it will be rejected or deleted, thus preventing it from entering the instruction folder where the files are stored, which can protect the database. Alternatively, a log compression script can be set to compress the initial log files, reducing the memory space occupied by the initial log files, providing more space for the specified folder, and reducing the operating pressure.

[0095] Furthermore, the virus scanning script can also be set at the file receiving port of the database, which can perform virus scanning on log files that are about to enter the database from outside the database, thereby ensuring that the log files entering the database are safe and reliable, and further improving the protection of the database.

[0096] In a further embodiment, the method further includes:

[0097] Receive log recovery commands from users;

[0098] Based on the log recovery command, determine the location information and recovery location information of the log to be recovered;

[0099] Based on the location information of the log to be recovered, the log to be recovered is extracted and added to the recovery location information.

[0100] This means that a log recovery script can also be set up in this manual to recover deleted logs. For example, since the storage time of log files is fixed, when the validity period expires, the log file does not need to be saved. Therefore, expired log files can be deleted (for example, by using a polling method to delete), or the user can actively delete log files. The deleted log files will be saved to a preset folder (i.e., the trash). By actively triggering the log recovery script by the user, the deleted files can be recovered, improving the efficiency of data management.

[0101] In the embodiments described in this specification, such as Figure 2 As shown, storing the log file in the first folder includes:

[0102] S201: Determine whether the size of the log file is greater than a first preset threshold;

[0103] S202: If so, the log file is divided according to the preset segmentation rules so that the size of each sub-log file after segmentation is less than the preset threshold.

[0104] S203: Send each of the split sub-log files to the first folder, and merge the sub-log files into the original log file according to the preset merging rules.

[0105] In the process of transmitting the log file to the database, to improve the security and reliability of data transmission, large files can be avoided from being transmitted in a single transmission, reducing packet loss or data corruption during transmission and ensuring the integrity of data transmission. Optionally, when the log file to be transmitted is large, it can be segmented to ensure that the size of each individual file transmitted does not exceed the first preset threshold. A segmenter can be used for segmentation. The first preset threshold is set according to actual conditions, such as the data transmission speed, and is not limited in this specification.

[0106] The preset segmentation rule and the preset merging rule correspond to each other. For example, the log files can be divided equally so that the resulting sub-log files are all the same size and less than the first preset threshold. This ensures that the original log file can be obtained after merging the multiple sub-log files. Log merging is achieved in the first folder, thereby ensuring that the log files can be completely entered into the specified folder and ensuring data integrity. The preset segmentation rule and the preset merging rule are not limited in this specification.

[0107] In the embodiments described in this specification, such as Figure 3 As shown, the preset classification model is obtained through the following steps:

[0108] S301: Retrieve historical log files;

[0109] S302: Calculate the similarity between any two historical log files based on the historical log files;

[0110] S303: Based on the similarity and preset clustering rules, the historical log files are clustered to obtain multiple sets of historical log files;

[0111] S304: According to the preset annotation rules, each set of historical log files is annotated to determine the annotation information of each historical log file;

[0112] S305: Input the historical log file and its corresponding annotation information into the initial classification model for training to obtain the trained preset classification model.

[0113] This specification can be understood as follows: the embodiments of this specification perform clustering and partitioning of historical log files, thereby realizing the partitioning of storage space in the database for different log file types. Furthermore, the partitioned historical log files are trained using machine learning to obtain a preset classification model for automatic partitioning of log file types, thereby improving the efficiency of subsequent log files entering the database for classification.

[0114] Optionally, the similarity between two historical log files is first calculated. This similarity can be calculated using methods such as cosine similarity or Euclidean distance. Since log files are generally stored in text format, the text vectors of the historical log files can be obtained first, and the similarity between any two historical log files can be calculated based on these text vectors. Obtaining text vectors is a common process in Natural Language Processing (NLP), and the specific calculation process is not limited in the embodiments described in this specification. Based on the similarity between different historical log files, clustering is performed using preset clustering rules to obtain a set of historical log files with multiple cluster centers. These preset clustering rules may include, but are not limited to, similarity magnitude, classification granularity, keywords, log type (according to size, business attributes, functional data, etc.), and compression degree.

[0115] After clustering is completed, multiple second folders (i.e., specified folders) can be set in the first folder. Each second folder stores a set of historical log files, thereby realizing the classification, processing and storage of log files.

[0116] The preset labeling rules can be rules for labeling according to log file type. Specifically, it involves determining the log file type in each historical log file set (according to size, business attributes, functional data, etc.); determining the probability distribution of each log file type in each historical log file set based on the log file type; and determining the type with the highest probability as the labeling attribute of the current historical log file based on the probability distribution.

[0117] Other labeling methods may be used in some other embodiments, which are not limited in the embodiments described in this specification.

[0118] It should be noted that after labeling each set of historical log files, all historical log files in that set are standardized accordingly. This ensures that during model training, the output type of each historical log file in the same set is consistent, thus guaranteeing the reliability and accuracy of clustering and classification.

[0119] Alternatively, the initial classification model may be a KNN classifier or an SVM classifier. In some other embodiments, other classification models may also be used, which are not limited in the embodiments described in this specification.

[0120] In a further embodiment, in order to ensure the accuracy and real-time performance of the preset classification model, the preset classification model can be optimized according to a specified time period. For example, new log files can be obtained at the end of each day or at the end of each weekend, and the model parameters can be adjusted according to the new log files to optimize the preset classification model, thereby improving the timeliness of the preset classification model.

[0121] In this embodiment of the specification, after storing the log file in the designated folder corresponding to the type information, the method further includes:

[0122] The log file is segmented into words to obtain the data information of the log file;

[0123] Determine whether the data is invalid or improperly formatted;

[0124] When invalid data exists, the invalid data should be cleaned up.

[0125] When there is irregular data, the irregular data is converted according to the preset conversion rules.

[0126] This step can be understood as a further optimization of the log files in the specified folder. The invalid data can be meaningless data. Since different text positions in the log file represent different meanings, the meaning of data at different positions can be obtained by determining the text type and format of the log file in advance and by word segmentation. If data appears in other positions, it indicates that the data at that position is invalid and can be cleared. Accordingly, by using the pre-determined log format, it is determined whether the format of the log file is correct or whether the format of some data is correct (i.e., whether it is regular). If it is incorrect (i.e., irregular data), the irregular data is converted according to the pre-determined log format to obtain regular data.

[0127] In a further embodiment, after storing the log file in the designated folder corresponding to the type information, the method further includes:

[0128] The log files in the specified folder are compressed according to a specified period.

[0129] In this step, log files in each specified folder can be compressed using a polling method. The specified period can be one day, one week, or one month. By compressing the log files in the specified folders, the usable space of the data in the specified folders can be further increased, the server load can be reduced, and the efficiency of data statistics and optimization can be improved. The method and extent of compression are not limited in the embodiments of this specification.

[0130] In the embodiments described in this specification, such as Figure 4 As shown, after storing the log file in the designated folder corresponding to the type information, the method further includes:

[0131] S401: Get the total size of log files in each specified folder;

[0132] S402: Determine whether the total size of the log files exceeds a preset second preset threshold;

[0133] S403: If so, determine the specified folder whose total size of the log files exceeds the second preset threshold, and compress the specified folder according to the preset compression rules until the total size of the log files in all specified folders is lower than the second preset threshold.

[0134] This can be understood as determining whether to compress each specified folder by measuring the amount of space it has used, thereby ensuring high utilization efficiency for each specified folder. The theoretical storage space of each specified folder is the same; therefore, to ensure effective operation of each specified folder, the usable space in each folder should be maintained at a certain level. This ensures that subsequent log files can be stored smoothly. Therefore, a second preset threshold is set as a warning value for compressing the specified folder. The second preset threshold is set according to actual conditions and is not limited in this embodiment.

[0135] The preset compression rules can be set based on the difference between the total size of the log files and the second preset threshold. For example, different difference gradients can be set, each corresponding to a different degree of compression. For instance, a first difference gradient, a second difference gradient, and a third difference gradient can be set. The first difference gradient is 0-50M, the second difference gradient is 50M-100M, and the third difference gradient is above 100M. The first difference gradient is used for the first difference gradient, the second difference gradient is used for the second difference gradient, and the third difference gradient is used for the third difference gradient. The larger the difference gradient, the higher the degree of compression (the higher the degree of compression, the greater the system consumption and the greater the computational resources consumed). This can improve the efficiency of compression. For specified folders with smaller difference gradients, the compression speed can be improved, reducing system performance consumption.

[0136] In a further embodiment, the theoretical storage space of the multiple designated folders in the first folder may be the same or different. Therefore, setting a threshold is not conducive to unified management and optimization. Instead, a usage rate threshold can be set. By calculating the usage rate of each designated folder and comparing it with the usage rate threshold, when the usage rate exceeds the threshold, the designated folder can be compressed to ensure that its usage rate is below the threshold. The compression method is the same as the steps described above and is not limited in the embodiments of this specification.

[0137] In this embodiment of the specification, generating the index path information of the log file based on the log file and the specified folder includes:

[0138] Obtain the configuration file information of the specified folder and the location information of the log file in the specified folder;

[0139] Based on the configuration file information and the location information, determine the archive file location information of the log file;

[0140] Based on the archive file location information, the index path information of the log file is generated, whereby the index path information is the log capture path of the log file.

[0141] Therefore, by determining the index path information of newly added log files, the database search engine can be updated in a timely manner, thus facilitating the querying of the log files. The configuration file information can be the configuration information of the specified folder in the database, and the location information can be obtained according to the storage method of the specified folder, such as storing sequentially according to storage time. Therefore, the archive file location information of the log files can be obtained through the configuration file information and the location information. The archive file location information is the specific information of the log files in the database, and thus the index path information of the log files can be obtained through the archive file location information.

[0142] After updating the search engine of the database, the corresponding log files can be queried as needed, such as reading and viewing logs by keywords. Furthermore, in order to improve the efficiency of log file information recording, the access time, log file name, username and other information can be recorded when a user accesses a log file. Optionally, an information recording script can be set up to record information to achieve automated statistics and recording.

[0143] In a further embodiment, a log index table can be established based on the updated search engine. For example, different log index keywords can be set according to log clustering types, and each keyword corresponds to the corresponding path information, namely a Uniform Resource Locator (URL). The viewing strategy is to set up a log viewer on the web system. The viewer categorizes and views log files according to their type, corresponding to the log index table in the database. Based on the log URL reference information in the index table, real-time log file information can be obtained. Users can select the corresponding time and log type to view the log information stored in the specified folder.

[0144] To further improve the efficiency of querying and analyzing log files, alternatively, such as Figure 5 As shown, the method further includes:

[0145] S501: Obtain log query information within a preset time period;

[0146] S502: Based on the log query information, determine the log files whose query count meets the preset conditions;

[0147] S503: Adjust the location information of the log file in the specified folder to obtain the index path information of the adjusted log file;

[0148] S504: Update the search engine of the database according to the adjusted log file index path information.

[0149] This specification can be understood as follows: by statistically analyzing log query data, the embodiments of this specification determine the log files that are queried more frequently, and then adjust the location information of the corresponding log files to make the query speed of the adjusted log files faster. Therefore, in this step, adjusting the location information of the log files in the specified folder can optimize the query path of the log files and improve the query speed of the log files.

[0150] The preset time period can be set according to the actual situation. For example, the path information of the log file can be optimized once a week. In actual operation, the information recording script can be set to collect and analyze the log query information in real time, and the log optimization script can be used to optimize the log according to the preset time period and the query data of the log statistics script.

[0151] As mentioned above, the storage time of log files is fixed. When the validity period expires, the log file does not need to be saved. Therefore, expired log files can be deleted (for example, by using a polling method to delete them), or the user can actively delete the log files. Thus, each specified folder can also be configured with a corresponding log deletion script to achieve the deletion of the corresponding log files.

[0152] This specification provides a database-based log processing method. By performing multi-layered optimization processing (virus scanning, compression) on log files entering the database, and combining this with a preset classification model to divide the log files into designated folders, the method achieves classified storage of log files, reducing the load on the server. Then, by combining statistical analysis during the log query process, the method optimizes the log file query path and improves the efficiency of data statistics and optimization.

[0153] Based on the methods provided above, embodiments of this specification also provide a database-based log processing system, which runs on the database, such as... Figure 6As shown, a main folder is created in the system, and virus scanning script, log compression script, log classification script, log recovery script, and information recording script are set up in the main folder. The working order of each script is: virus scanning script to log compression script to log classification script. This performs preliminary processing and classification of the raw log files to facilitate better statistics, optimization, and analysis of the log files. When a user accesses a log file, the access time, log file name, and whether file information is extracted are saved to the information recording script so that the administrator can statistically analyze the access data and rearrange the files with the most accesses for faster access by users. Multiple subfolders are created in the main folder (e.g., subfolder 1, subfolder 2, subfolder 3, ..., subfolder N), and log statistics script, log optimization script, and log deletion script are set up in the subfolders. This allows for data statistics and optimization of files of the same type, which can reduce the server load and improve the efficiency of data statistics and optimization.

[0154] Based on the same inventive concept, embodiments of this specification also provide a database-based log processing device, such as... Figure 7 As shown, the device includes:

[0155] Log file receiving module 100 is used to receive log files;

[0156] The log file processing module 200 is used to determine the type information of the log file according to a preset classification model, and store the log file in a specified folder corresponding to the type information;

[0157] The index path information generation module 300 is used to generate the index path information of the log file based on the log file and the specified folder;

[0158] The search engine update module 400 is used to update the search engine of the database based on the index path information.

[0159] The beneficial effects obtained by the above-described device are the same as those obtained by the above-described method, and will not be described in detail in the embodiments of this specification.

[0160] It should be noted that the database-based log processing method and apparatus provided in the embodiments of this specification can be used in the financial field for log processing, and can also be used in any field other than the financial field. The application field of the database-based log processing method and apparatus provided in the embodiments of this specification is not limited.

[0161] like Figure 8As shown in the embodiments herein, a computer device is provided. This computer device can be the database-based log processing apparatus provided above, executing the methods described above. The computer device 802 may include one or more processors 804, such as one or more central processing units (CPUs), each of which can implement one or more hardware threads. The computer device 802 may also include any memory 806 for storing information of any kind, such as code, settings, data, etc. Non-limitingly, for example, the memory 806 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 802. In one case, when the processor 804 executes associated instructions stored in any memory or combination of memories, the computer device 802 can perform any operation of the associated instructions. The computer device 802 also includes one or more drive mechanisms 808 for interacting with any memory, such as hard disk drive mechanisms, optical disk drive mechanisms, etc.

[0162] Computer device 802 may also include an input / output module 810 (I / O) for receiving various inputs (via input device 812) and providing various outputs (via output device 814). A specific output mechanism may include a presentation device 816 and an associated graphical user interface (GUI) 818. In other embodiments, the input / output module 810 (I / O), input device 812, and output device 814 may be omitted, and the device may function solely as a computer device within a network. Computer device 802 may also include one or more network interfaces 820 for exchanging data with other devices via one or more communication links 822. One or more communication buses 824 couple the components described above together.

[0163] Communication link 822 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 822 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0164] Corresponding to Figures 1-5 In addition to the methods described above, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described methods.

[0165] This embodiment also provides a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to perform the following: Figures 1 to 5 The method shown.

[0166] It should be understood that in the various embodiments of this document, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0167] It should also be understood that, in the embodiments herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship.

[0168] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0169] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0170] In the embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described herein, depending on actual needs.

[0172] Furthermore, the functional units in the various embodiments of this document can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0173] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this paper, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this paper. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] This document uses specific embodiments to illustrate the principles and implementation methods of this document. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this document. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this document. Therefore, the content of this specification should not be construed as a limitation of this document.

Claims

1. A database-based log processing method, characterized in that, The method includes: Receive log files; Based on a preset classification model, the category information of the log file is determined, and the log file is stored in a designated folder corresponding to the category information; Generate the index path information of the log file based on the log file and the specified folder; Update the search engine of the database based on the index path information; The preset classification model is obtained through the following steps: Retrieve historical log files; Based on the historical log files, calculate the similarity between any two historical log files; Based on the similarity and preset clustering rules, the historical log files are clustered to obtain multiple sets of historical log files. The preset clustering rules include one of the following: classification granularity, log type, and compression degree. According to the preset annotation rules, each set of historical log files is annotated to determine the annotation information of each historical log file; The historical log files and their corresponding annotation information are fed into the initial classification model for training to obtain the trained preset classification model. The method further includes: Retrieve log query information within a preset time period; Based on the log query information, determine the log files whose query count meets the preset conditions; The location information of the log file in the specified folder is adjusted to obtain the index path information of the adjusted log file; wherein, the index path information of the adjusted log file includes the query path after the log file path length is optimized. The search engine of the database is updated based on the adjusted index path information of the log file.

2. The method according to claim 1, characterized in that, The process after receiving the log file also includes: The log files are stored in a first folder, which includes at least one designated folder; The log file is preprocessed, including at least virus scanning and log compression.

3. The method according to claim 2, characterized in that, The step of storing the log file in the first folder includes: Determine whether the size of the log file is greater than a first preset threshold; If so, the log file is divided according to the preset segmentation rules so that the size of each sub-log file after segmentation is less than the preset threshold. Each of the split sub-log files is sent to the first folder, and the sub-log files are merged according to the preset merging rules to form the original log file.

4. The method according to claim 1, characterized in that, The initial classification model is either a KNN classifier or an SVM classifier.

5. The method according to claim 1, characterized in that, After storing the log file in the specified folder corresponding to the type information, the process further includes: The log file is segmented into words to obtain the data information of the log file; Determine whether the data is invalid or improperly formatted; When invalid data exists, the invalid data should be cleaned up. When there is irregular data, the irregular data is converted according to the preset conversion rules.

6. The method according to claim 1, characterized in that, After storing the log file in the designated folder corresponding to the type of information, the process further includes: The log files in the specified folder are compressed according to a specified period.

7. The method according to claim 1, characterized in that, After storing the log file in the specified folder corresponding to the type information, the process further includes: Get the total size of log files in each specified folder; Determine whether the total size of the log files exceeds a preset second threshold; If so, then the specified folders whose total size of the log files exceeds the second preset threshold are identified, and the specified folders are compressed according to the preset compression rules until the total size of the log files in all specified folders is lower than the second preset threshold.

8. The method according to claim 1, characterized in that, The step of generating the index path information of the log file based on the log file and the specified folder includes: Obtain the configuration file information of the specified folder and the location information of the log file in the specified folder; Based on the configuration file information and the location information, determine the archive file location information of the log file; Based on the archive file location information, the index path information of the log file is generated, whereby the index path information is the log capture path of the log file.

9. A database-based log processing device, characterized in that, The device includes: The log file receiving module is used to receive log files; The log file processing module is used to determine the category information of the log file according to a preset classification model, and store the log file in a specified folder corresponding to the category information; The index path information generation module is used to generate the index path information of the log file based on the log file and the specified folder; The search engine update module is used to update the search engine of the database based on the index path information; The preset classification model is obtained through the following steps: Retrieve historical log files; Based on the historical log files, calculate the similarity between any two historical log files; Based on the similarity and preset clustering rules, the historical log files are clustered to obtain multiple sets of historical log files. The preset clustering rules include one of the following: classification granularity, log type, and compression degree. According to the preset annotation rules, each set of historical log files is annotated to determine the annotation information of each historical log file; The historical log files and their corresponding annotation information are fed into the initial classification model for training to obtain the trained preset classification model. The search engine update module is also used for: Retrieve log query information within a preset time period; Based on the log query information, determine the log files whose query count meets the preset conditions; The location information of the log file in the specified folder is adjusted to obtain the index path information of the adjusted log file; wherein, the index path information of the adjusted log file includes the query path after the log file path length is optimized. The search engine of the database is updated based on the adjusted index path information of the log file.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 8.

Images