Efficient log analysis and filtering management system and method
By introducing configuration management module, log management module, log analysis module and real-time log viewing module into the log analysis tool, modularization of logs and real-time filtering is realized, and existing tools are solved in multi-module log management and real-time log analysis. The shortcomings of the system improve analysis efficiency and system maintainability.
Patent Information
- Application Number
- CN202411512966.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing log analysis tools are insufficient in multi-module log management and real-time log analysis, and cannot effectively distinguish logs from different functional modules, resulting in inefficient analysis and lack of unified configuration management and real-time log viewing functions.
An efficient log analysis and filtering management system was designed, including configuration management module, log management module, log analysis module, log database and real-time log viewing module. The classification storage and efficient filtering of logs are realized through modular division and association marking, supporting Multi-platform adaptation and real-time log viewing.
It realizes unified management and efficient filtering of multi-module logs, improves log analysis efficiency, meets the needs of multi-platform and real-time debugging, and improves the maintainability of the system.
Smart Images

Figure CN119025490A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of log analysis and transition, and in particular, relates to an efficient log analysis and filtering management system and method. Background Art
[0002] Log analysis is an important part of software development and maintenance, especially in complex systems, where logs generated by various modules need to be effectively managed and analyzed in order to quickly locate problems and fix them. However, existing log analysis tools have many deficiencies in log management and analysis, and it is difficult to meet the needs of multi-module and multi-category log management.
[0003] Currently, commonly used log analysis tools such as TextAnalysisTool.NET have basic log search and filtering functions, but have the following shortcomings: 1. Lack of log categories: Existing tools are usually unable to classify logs in detail. For example, they are unable to distinguish logs of different functional modules (such as OTA, streaming, live broadcast, etc.), resulting in low analysis efficiency.
[0004] 2. Independent configuration management: Each module needs to be configured separately, and the logs of different modules cannot be managed and displayed uniformly, which increases the complexity of management and maintenance.
[0005] 3. Inability to filter and highlight efficiently: Existing tools cannot quickly filter and highlight logs based on log categories, resulting in inefficient log analysis.
[0006] 4. Lack of real-time log viewing function: Most tools do not support real-time serial port log viewing and filtering, and cannot meet the demand for real-time logs during development and debugging.
[0007] These problems make existing log analysis tools inadequate in multi-module log management and real-time log analysis, making it difficult to meet actual application needs. Summary of the invention
[0008] The purpose of this application is to provide an efficient log analysis and filtering management system and method to solve the problem of fewer existing log categories and low management efficiency.
[0009] The technical solution of the present application is: an efficient log analysis and filtering management system, comprising a configuration management module, a log management module, a log analysis module, a log database and a real-time log viewing module; the configuration management module is used to manage, classify and update configuration files; the log analysis module is used to find and transmit configuration files through the configuration management module; the log database is used to classify and store log files; the real-time log viewing module is used to retrieve corresponding log files and pass them to the user end; the configuration management module comprises a configuration receiving unit, a configuration classification unit and a configuration storage unit; the configuration receiving module is used to receive configuration files uploaded by users of user ends of multiple platforms and pass them to the configuration classification unit; the configuration classification unit is used to parse the configuration files and modularize the configuration files according to the parsed content; the configuration storage unit classifies and stores the configuration files according to the modularization results.
[0010] Preferably, the specific method for modular division is: define different configuration modules respectively, and place them in the configuration storage module, each configuration module is a different function type, and set corresponding function fields for each configuration module; after receiving a configuration file, pair the configuration files respectively according to the function fields, and determine whether there are corresponding function fields in the configuration file. If so, store the configuration file in the corresponding configuration module; if not, pair the function fields of the next configuration module until all function fields are paired, and finally form the configuration information.
[0011] Preferably, a secondary submodule, a tertiary submodule or more submodules are provided in each configuration module; each submodule is provided with a corresponding matching field, and the configuration files that have completed the previous level of matching are matched through the matching field.
[0012] Preferably, it also includes a log management module, which includes a log receiving unit, a log allocation unit and a daily configuration association unit; the log receiving unit is used to collect the log files after configuration from each configuration module, the log allocation unit is used to classify and manage the logs, obtain log classification data, and then store the log classification data in the log database; the daily configuration association unit is used to associate the classified log files with the corresponding configuration files.
[0013] Preferably, the specific method for the log allocation unit to classify and manage logs is: define log categories, each log category defines a category character, after the log allocation unit receives the corresponding log file, it matches the log file in turn through the category character, after the pairing is completed, obtains the log pairing data, and stores the log pairing data in the log database according to the log classification data.
[0014] Preferably, the method for associating configuration files is: setting an association tag, and the specific method for setting the association tag is: the association tag defines a prefix and a suffix, the log files corresponding to the same configuration file have the same prefix and different suffixes; the log files corresponding to different configuration files have different prefixes; when the configuration file is stored, the prefix of the association tag is set for it; after receiving a new log file, its associated configuration file is determined by the prefix tag, and then the corresponding suffix is set according to the order in which it is associated with the configuration file, and so on, until the association of all log files is completed.
[0015] Preferably, the log analysis module includes a data receiving unit and a log retrieval unit, the data receiving unit can receive the configuration information and log classification data of the configuration management module; the log retrieval unit is provided with a plurality of keywords, each keyword has a one-to-one correspondence with a function field of a configuration file, and the keyword is triggered by the user end, when the user of the user end inputs or selects one or more keywords, the log analysis module first matches the keywords with the function field and the matching field, and then filters the configuration file until the corresponding configuration file is found, and finally calls out all associated log files through the associated tags.
[0016] As a specific implementation method, an efficient log analysis and filtering management method includes: being able to execute a log processing process, a configuration management process and a real-time log viewing process; when executing the log processing process: after a user end uploads a configuration file, a configuration management module receives the configuration file to classify and store the configuration file, and uploads the configuration information of the configuration file to a log management module; the log management module collects log files respectively according to the configuration information of the configuration file; then the log management module classifies the collected log files, and the classified log files are stored in a log database according to the classification results; the classified log classification data is transmitted to a log analysis module; when a user views the required log through the configuration module, the corresponding configuration module is clicked on the screen to filter the configuration file, and then for the found configuration file, the corresponding log file is called out through the configuration file to highlight the log file; finally, the highlighted log file is displayed on the user end.
[0017] Preferably, when executing the configuration management process: when the user end uploads a new configuration file, the configuration management module receives and parses the configuration file, updates the configuration information according to the parsing result and applies it to the corresponding configuration module; then the updated configuration information is notified to the log management module, the log management module updates the log classification data according to the updated configuration information, and the log management module applies the new configuration information according to the log separation data.
[0018] Preferably, when executing the real-time log viewing process: when the user queries any log, the serial port is opened to access the log management module in real time, the log management module retrieves the required log file from the log database, and passes it to the real-time log viewing module, the real-time log viewing module filters the log according to the specified configuration module and highlights the required log file, and passes the display result to the user end in real time.
[0019] The advantages of the efficient log analysis and filtering management system and method of this application are as follows: 1. Multi-platform adapted log control mechanism: Provides a multi-platform adapted log control mechanism, supporting cross-platform and multi-module log management.
[0020] 2. Unified configuration management: realize unified management and update of log configuration to improve the maintainability of the system.
[0021] 3. Efficient log filtering and highlighting: Supports fast filtering and highlighting of logs to improve the efficiency of log analysis.
[0022] 4. Real-time log viewing and filtering: Supports real-time serial port log viewing and specified module log filtering to meet real-time debugging needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solution provided by the present application, the following is a brief introduction to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present application.
[0024] Figure 1 This is a schematic diagram of the overall structure of this application; Figure 2 A diagram of the configuration management process for this application.
[0025] In the figure, 1. Configuration management module; 2. Log management module; 3. Log analysis module; 4. Log database; 5. Real-time log viewing module; 6. Log display interface. DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] An efficient log analysis and filtering management system, such as Figure 1, including a configuration management module 1, a log management module 2, a log analysis module 3, a log database 4 and a real-time log viewing module 5. The configuration management module 1 is used to manage, classify and update configuration files; the log analysis module 3 is used to find and transmit configuration files through the configuration management module 1; the log database 4 is used to classify and store log files; the real-time log viewing module 5 is used to retrieve corresponding log files and transmit them to the user end.
[0028] The configuration management module 1, the log management module 2, the log analysis module 3, the log database 4 and the real-time log viewing module 5 are independent of each other and are completely decoupled from each other.
[0029] The configuration management module 1 includes a configuration receiving unit, a configuration classification unit and a configuration storage unit. The configuration receiving module is used to receive configuration files uploaded by users of the user terminals of multiple platforms and pass them to the configuration classification unit; the configuration classification unit is used to parse the configuration files and divide the configuration files into modules according to the parsed contents; the configuration storage unit classifies and stores the configuration files according to the modular division results.
[0030] The specific method of modularizing the configuration file by the configuration classification unit is as follows: different configuration modules are defined respectively, and placed in the configuration storage module, each configuration module is a different function type, including live broadcast, streaming or OTA, etc., and corresponding function fields are set for each configuration module; after receiving a configuration file, the configuration files are paired according to the function fields to determine whether there are corresponding function fields in the configuration file. If so, the configuration file is stored in the corresponding configuration module; if not, the function fields of the next configuration module are paired until all function fields are paired and the configuration information is finally formed.
[0031] If all function fields are paired but still cannot be matched with the corresponding configuration module, a new configuration module is created and the configuration file is placed in the new configuration module.
[0032] Preferably, the log analysis module 3 and the real-time log viewing module 5 are both provided with a log display interface 6, which can display the filtered logs normally and in a highlighted manner.
[0033] Preferably, in order to enable faster searching, a secondary submodule, a tertiary submodule or more submodules are set in each configuration module. For example, in the configuration module of the live broadcast class, secondary submodules such as games, dancing, sports, and learning can be set according to different live broadcast types. At the same time, the secondary submodules can be further subdivided to form tertiary submodules, or more submodules. Each submodule is provided with a corresponding matching field, and the matching field is used to match the configuration files that have completed the previous level of matching. For example, the matching field of the game class only matches the configuration files that have been configured with the function fields of the live broadcast class to ensure the matching efficiency. Through this design, a more detailed functional division is achieved to improve the efficiency of subsequent searches.
[0034] The log management module 2 includes a log receiving unit, a log allocation unit and a daily configuration association unit. The log receiving unit is used to collect the log files after configuration from each configuration module, the log allocation unit is used to classify and manage the logs, obtain log classification data, and then store the log classification data in the log database 4, so that the logs are clearer and more orderly; the daily configuration association unit is used to associate the classified log files with the corresponding configuration files, so that after the configuration files are found, the corresponding log files can be retrieved.
[0035] The specific method for the log allocation unit to classify and manage logs is: define log categories, log categories include security logs, application logs, system logs, etc.; each log category is defined with category characters, and after the log allocation unit receives the corresponding log file, it matches the log file in turn through the category characters. After the pairing is completed, the log pairing data is obtained, and the log pairing data is stored in the log database 4 according to the log classification data.
[0036] Since each log file is associated with a configuration file, and each configuration file corresponds to multiple different types of logs, an associated tag is set for different types of logs in the same configuration file to facilitate extraction. The tag can be a digital tag, a character tag, or a combination of a number and a character.
[0037] The specific setting method of the association tag is as follows: the association tag defines the prefix and suffix. The log files corresponding to the same configuration file have the same prefix and different suffixes; the log files corresponding to different configuration files have different prefixes, and the suffixes can be the same or different. When the configuration file is stored, the prefix of the association tag is set for it; after receiving a new log file, the configuration file associated with it is determined by the prefix tag, and then the corresponding suffix is set according to the order in which it is associated with the configuration file, such as setting the suffix of the first associated log file to 01, the suffix of the second associated log file to 02, and so on, so as to complete the association of all log files.
[0038] The log analysis module 3 includes a data receiving unit and a log retrieval unit. The data receiving unit can receive the configuration information and log classification data of the configuration management module 1. The log retrieval unit is provided with a number of keywords, each keyword has a one-to-one correspondence with a function field of a configuration file, and the keyword is triggered by the user end. When the user of the user end enters or selects one or more keywords, the log analysis module 3 first matches the function field and the matching field through the keyword, and then filters the configuration file until the corresponding configuration file is found, and finally calls out all the associated log files through the associated tags. In this way, all relevant configuration files and log files can be found by searching for keywords, and the found log files can be transmitted. There are generally multiple log files found through this design.
[0039] The real-time log viewing module 5 includes a log filtering module and a highlight display module. The log filtering module retrieves the real-time information of the log file of the log analysis module 3 through the serial port, and performs real-time log filtering of the specified configuration module according to user needs until the log file required by the user is found; the highlight display module is used to adjust the brightness of the filtered log file to form a highlight display.
[0040] Through the above design, the advantages are as follows: 1. Multi-platform adapted log control mechanism: Provides a multi-platform adapted log control mechanism, supporting cross-platform and multi-module log management.
[0041] 2. Unified configuration management: realize unified management and update of log configuration to improve the maintainability of the system.
[0042] 3. Efficient log filtering and highlighting: Supports fast filtering and highlighting of logs to improve the efficiency of log analysis.
[0043] 4. Real-time log viewing and filtering: Supports real-time serial port log viewing and specified module log filtering to meet real-time debugging needs.
[0044] As a specific implementation method, it also includes an efficient log analysis and filtering management method, including: being able to execute a log processing flow, a configuration management flow and a real-time log viewing flow, and the log processing flow and the configuration management flow run synchronously.
[0045] When executing the log processing flow: After the user uploads the configuration file, the configuration management module 1 receives the configuration file, classifies and stores the configuration file, and uploads the configuration information of the configuration file to the log management module 2; the log management module 2 collects log files according to the configuration information of the configuration file; then the log management module 2 classifies the collected log files, and the classified log files are stored in the log database 4 according to the classification results; the classified log data after classification is passed to the log analysis module 3; when the user views the required log through the configuration module, the configuration file is filtered by clicking the corresponding configuration module on the screen, and then the configuration file is found, and the corresponding log file is called out through the configuration file to highlight the log file; finally, the highlighted log file is displayed on the user side. This ensures the orderly management and rapid analysis of the logs.
[0046] Preferably, multiple clients can access the configuration management module 1 at the same time to perform log processing.
[0047] like Figure 2 , when executing the configuration management process: When the user uploads a new configuration file, the configuration management module 1 receives and parses the configuration file, updates the configuration information according to the parsing result and applies it to the corresponding configuration module; then notifies the updated configuration information to the log management module 2, which updates the log classification data according to the updated configuration information, and applies the new configuration information according to the log separation data. This achieves unified management and update of log configuration.
[0048] When executing the real-time log viewing process: When the user queries any log, the serial port is opened to access the log management module 2 in real time. The log management module 2 retrieves the required log file from the log database 4 and passes it to the real-time log viewing module 5. The real-time log viewing module 5 filters the logs according to the specified configuration module and highlights the required log files, and transmits the display results to the user end in real time, thereby ensuring that the user can monitor and debug the logs in real time.
[0049] Finally, it should be noted that: the drawings of the embodiments disclosed in the present invention only involve structures related to the embodiments disclosed in the present invention, and other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of the present invention can be combined with each other; Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An efficient log analysis and filtering management system, characterized by: The system comprises a configuration management module (1), a log analysis module (3), a log database (4) and a real-time log viewing module (5); the configuration management module (1) is used for managing, classifying and updating configuration files; the log analysis module (3) is used for searching and transmitting configuration files through the configuration management module (1); the log database (4) is used for classifying and storing log files; the real-time log viewing module (5) is used for retrieving corresponding log files and transmitting them to a user end; the configuration management module (1) comprises a configuration receiving unit, a configuration classification unit and a configuration storage unit; the configuration receiving module is used for receiving configuration files uploaded by users of user ends of multiple platforms and transmitting them to the configuration classification unit; the configuration classification unit is used for parsing the configuration files and modularizing the configuration files according to the parsed contents; the configuration storage unit classifies and stores the configuration files according to the modularization results.
2. The efficient log analysis and filtering management system according to claim 1, characterized in that: The specific method for modular division is: define different configuration modules respectively and place them in the configuration storage module, each configuration module is a different function type, and set corresponding function fields for each configuration module; after receiving a configuration file, pair the configuration files according to the function fields to determine whether there are corresponding function fields in the configuration file. If so, store the configuration file in the corresponding configuration module; if not, pair the function fields of the next configuration module until all function fields are paired and the configuration information is finally formed.
3. The efficient log analysis and filtering management system according to claim 2, characterized in that: Second-level submodules, third-level submodules or more submodules are set in each configuration module; each submodule is provided with a corresponding matching field, and the configuration files that have completed the previous level matching are matched through the matching field.
4. The efficient log analysis and filtering management system according to claim 2, characterized in that: It also includes a log management module (2), the log management module (2) including a log receiving unit, a log allocation unit and a daily configuration association unit; the log receiving unit is used to collect the log files after configuration is completed from each configuration module; the log allocation unit is used to classify and manage the logs to obtain log classification data, and then store the log classification data in a log database (4); the daily configuration association unit is used to associate the classified log files with the corresponding configuration files.
5. The efficient log analysis and filtering management system according to claim 4, characterized in that: The specific method of the log allocation unit for classifying and managing logs is as follows: defining log categories, defining category characters for each log category, and after the log allocation unit receives the corresponding log file, pairing the category characters with the log file in sequence, and after the pairing is completed, obtaining log pairing data, and storing the log pairing data in the log database (4) according to the log classification data.
6. The efficient log analysis and filtering management system according to claim 5, characterized in that: The method for associating configuration files is: setting association tags. The specific method for setting association tags is: the association tags define prefixes and suffixes. The log files corresponding to the same configuration file have the same prefix and different suffixes; the log files corresponding to different configuration files have different prefixes; when the configuration file is stored, the prefix of the association tag is set for it; after receiving a new log file, its associated configuration file is determined by the prefix tag, and then the corresponding suffix is set according to the order in which it is associated with the configuration file, and so on, until all log files are associated.
7. The efficient log analysis and filtering management system according to claim 4, characterized in that: The log analysis module (3) comprises a data receiving unit and a log retrieval unit. The data receiving unit is capable of receiving configuration information and log classification data of the configuration management module (1). The log retrieval unit is provided with a plurality of keywords, each keyword having a one-to-one correspondence with a function field of a configuration file, and the keyword is triggered by a user end. When a user of the user end inputs or selects one or more keywords, the log analysis module (3) first matches the keyword with the function field and the matching field, and then filters the configuration file until a corresponding configuration file is found, and finally calls out all associated log files through associated tags.
8. An efficient log analysis and filtering management method, using the efficient log analysis and filtering management system according to any one of claims 1 to 6, characterized in that: include: Ability to execute log processing flow, configuration management flow and real-time log viewing flow; When executing the log processing flow: after the user end uploads the configuration file, the configuration management module (1) receives the configuration file to classify and store the configuration file, and uploads the configuration information of the configuration file to the log management module (2); the log management module (2) collects log files respectively according to the configuration information of the configuration file; then the log management module (2) classifies the collected log files, and the classified log files are stored in the log database (4) according to the classification results; the classified log data after classification is transmitted to the log analysis module (3); when the user views the required log through the configuration module, the corresponding configuration module is clicked on the screen to filter the configuration file, and then for the found configuration file, the corresponding log file is called out through the configuration file to highlight the log file; finally, the highlighted log file is displayed on the user end.
9. The efficient log analysis and filtering management method according to claim 8, characterized in that: When executing the configuration management process: after the user uploads a new configuration file, the configuration management module (1) receives the configuration file and parses the configuration file, updates the configuration information according to the parsing result and applies it to the corresponding configuration module; then the updated configuration information is notified to the log management module (2), the log management module (2) updates the log classification data according to the updated configuration information, and the log management module (2) applies the new configuration information according to the log separation data.
10. The efficient log analysis and filtering management method according to claim 8, characterized in that: When executing the real-time log viewing process: when the user queries any log, the serial port is opened to access the log management module (2) in real time, the log management module (2) retrieves the required log file from the log database (4) and transmits it to the real-time log viewing module (5), the real-time log viewing module (5) performs log filtering according to the specified configuration module and highlights the required log file, and transmits the display result to the user end in real time.
Citation Information
Patent Citations
Method and device for collecting logs
CN103944973A
Log processing method and device, terminal equipment and storage medium
CN108038231A
Log processing method, device, terminal device and storage medium
CN108108284A
A log processing method based on a message queue and a search engine and related equipment
CN109815201A
Log management method, log management device, processor and log platform
CN116226067A
