A Log Optimization System and Method Based on Big Data

Abstract

This application provides a log optimization system and method based on big data, relating to the field of log management technology. The method includes: acquiring historical reference log data from a similar type of storage database; simplifying and numbering the historical log content information to form a log number retrieval table and a historical log content library; combining the log number retrieval table with the historical log content information to form a historical log content record library; acquiring historical log processing information and simplifying and numbering the historical log processing information to form a log processing retrieval table and a historical log processing content library; combining the log processing retrieval table to form a historical log record and processing library; and acquiring real-time log data and recording it in the historical log record and processing library. This method can reasonably optimize log data to reduce its storage space occupancy and make log data recording more efficient.

Description

Technical Field

[0001] This application relates to the field of log management technology, and more specifically, to a log optimization method based on big data. Background Technology

[0002] During operation, the application monitors its own performance and records various events in real time. This serves two purposes: firstly, it allows for data tracing of events and understanding of the program's specific execution; secondly, it generates operational analysis data, providing data support for program optimization and development.

[0003] However, current program logging mainly involves directly describing program events, and the resulting log data is not organized or simplified. As the number of log records increases, the storage space occupied by the log records also increases, which becomes a burden on the application, greatly reducing the efficiency of program operation and wasting a lot of storage space.

[0004] Therefore, designing a log optimization system and method based on big data to reasonably optimize log data, thereby reducing its storage space occupancy and making log data recording more efficient, is an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a log optimization method based on big data. By acquiring log data from similar storage databases, the log content is rationally numbered, thereby simplifying the amount of log records, significantly reducing storage requirements, and effectively saving data storage space. For log content, a time-based content record library is formed based on recording time. Simultaneously, the numbered log content is statistically organized to create a numbered index table and content library for retrieval. This ensures complete preservation of log content information, providing a simple and effective recording format while also rationally organizing and summarizing the recorded content, facilitating more reasonable and effective basic data for log content analysis. Furthermore, log processing information is also numbered based on the time dimension and mapped to the numbers of log content in the content record library, forming complete log record data. For subsequent real-time log records, numbering is performed based on the newly established numbering model, ensuring real-time optimization of log records. Since log optimization is based on big data, the real-time acquired log records primarily involve repetitive comparison and confirmation of numbering, thus greatly improving log recording efficiency.

[0006] The purpose of this application is also to provide a log optimization system based on big data, which optimizes log numbering and efficiently records real-time log data through different functional units, thereby improving log recording while ensuring the integrity of log data.

[0007] Firstly, this application provides a log optimization method based on big data, including: acquiring historical log data from similar storage databases to form historical reference log data; establishing a simplified numbering model based on log content; simplifying the numbering of historical log content information in the historical reference log data to form a log number retrieval table and a historical log content library; combining the log number retrieval table with the historical log content information based on the historical log time information in the historical reference log data to form a historical log content record library; acquiring historical log processing information from the historical reference log data; establishing a simplified numbering model based on processing information; simplifying the numbering of historical log processing information to form a log processing retrieval table and a historical log processing content library; combining the log processing retrieval table with the historical log processing time information in the historical reference log data to enrich the historical log content record library with processing information to form a historical log record and processing library; acquiring real-time log data and recording it in the historical log record and processing library.

[0008] In this application, the method acquires log data from similar storage databases, rationally numbers the log content, thereby simplifying the amount of log records, significantly reducing storage requirements, and effectively saving data storage space. For log content, on the one hand, a time-based content record library is formed based on recording time; on the other hand, the numbered log content is rationally statistically organized to form a numbered index table and content library for retrieval. This ensures complete preservation of log content information, providing a simple and effective recording format while also rationally organizing and summarizing the recorded content, facilitating more reasonable and effective basic data for log content analysis. Furthermore, log processing information is also numbered based on the time dimension and correlated with the numbers of log content in the content record library, forming complete log record data. For subsequent real-time log records, numbering is based on the newly established numbering model, ensuring real-time optimization of log records. Since log optimization is based on big data, the real-time acquired log records primarily involve repetitive comparison and confirmation of numbering, thus greatly improving log recording efficiency.

[0009] One possible implementation involves acquiring historical log data from similar storage databases to form historical reference log data, and establishing a simplified numbering model based on log content. This involves simplifying and numbering the historical log content information in the historical reference log data to create a log number retrieval table and a historical log content library. This includes: acquiring historical log content information and fragmenting each log entry, dividing it into variables and invariants to form initial unit log content groups; performing logical analysis on all initial unit log content groups based on record objects to form result unit log content groups; combining all result unit log content groups to form a result unit log content set; extracting invariants from all result unit log content groups in the result unit log content set, performing a quantity percentage analysis on the invariants, and arranging the invariants sequentially based on the quantity percentage analysis results to form a historical log content library; numbering the invariants according to their arrangement in the historical log content library, and determining the corresponding variable values ​​for different numbered invariants based on the results of the logical analysis based on record objects to form a log number retrieval table.

[0010] In this application, when optimizing log content, considering that log content is usually quite verbose, to more concisely reflect the recorded events, the log content is first fragmented, removing redundant record descriptions, such as separators between different objects and distinguishing characters between different codes. This can reduce the storage size of the subsequent log content library to some extent. The fragmented log content is then grouped based on event objects to form initial unit log content groups. The initial unit log content groups are then merged according to their invariants, and the merged unit log content groups are then organized and summarized to form a historical log content library. Here, the invariants are numbered to form a log number retrieval table mainly because invariants are the most basic fixed parameters in event records, accurately expressing the specific event situation. Variables, on the other hand, only describe the degree of invariants; therefore, using invariants as the basic parameters for numbering retains the record of the specific event situation. Furthermore, the initial unit log content groups are determined based on the different record objects within each log record. It is known that at a certain recording point in time, different objects may be recorded simultaneously and stored together to form the log record content at that recording point in time. Since the invariants and variables reflected by the different recording objects are also completely different, and there is no absolute correlation between these invariants and variables, dividing the log content based on the recording objects to form the initial unit log content can reasonably classify highly correlated parameters, which facilitates subsequent data analysis and numbered retrieval.

[0011] As one possible implementation, logical analysis based on record objects is performed on all initial unit log content groups to form result unit log content groups. This includes: performing the following logical analysis on the initial unit log content groups, and merging different initial unit log content groups that simultaneously meet the following logical analysis conditions: the non-variable record objects in different initial unit log content groups are the same; the events recorded by non-variables in different initial unit log content groups are the same; the variables corresponding to non-variables in different initial unit log content groups reflect the same event state; the merged unit log content groups and the initial unit log content groups that do not need to be merged are determined as result unit log content groups.

[0012] In this application, to ensure that no identical records appear in the resulting historical log content repository, a logical judgment and analysis based on objects is performed after obtaining the initial unit log content group. Records describing the same object at the same time within the initial unit log content group are merged. This may further optimize the data in the historical log content repository and reduce its storage footprint. The object-based logical analysis mainly considers three aspects: determining whether the objects are identical, determining whether the recorded events are identical, and whether variables reflecting the degree of invariance can be considered identical. Especially for the determination of variable similarity, the focus is on variables that may differ numerically but reflect the same characteristics and corresponding processed information. This also facilitates better matching when corresponding log processing content later, reducing the complexity of matching log content with log processing information.

[0013] One possible implementation involves numbering the invariants in the historical log content library according to their order, and determining the corresponding variable values ​​for different invariant numbers based on the logical analysis results of the record objects, thus forming a log number retrieval table. This includes: traversing the total number of invariants in the historical log content library, determining the number of digits based on the total number, and numbering the invariants sequentially, with the numbering being non-consecutive; for the resulting log content group formed by merging, determining the variables in the multiple initial log content groups before merging as the selection options for the variables corresponding to the invariants in the resulting log content group; and combining the invariants, the variables corresponding to the invariants, and the numbers corresponding to the invariants to form the log number retrieval table.

[0014] In this application, the log number retrieval table is sorted based on the proportion of non-variables within the total number. When numbering after sorting, considering the possibility of new log content being added later, the numbering is non-contiguous, which also helps to distinguish different log content in terms of numbering. Of course, to further improve the data rationality of the log number retrieval table, it is also possible to segment the numbering based on the record object. That is, different number segments are determined as the numbers for log content generated on different objects, and different existing log content on the corresponding object is assigned numbers from the number segments. This object-based numbering allows for simple and quick identification of the object recorded by the log content based on the number, further optimizing the log number retrieval table and providing good numbering guidance for data analysis.

[0015] As one possible implementation, combining a log number retrieval table, historical log content information is numbered and combined based on the historical log time information in the historical reference log data to form a historical log content record library. This includes: obtaining historical log time information, determining the result unit log content group contained at each log record time point, and determining the number combination information at each log record time point in chronological order using the log number retrieval table. Where t is the sequence number of the log record time point determined according to the time order, and t is a non-zero natural number, A n B represents the nth invariant number recorded at the log recording time point. n This represents the value of the variable corresponding to the nth non-variable at the log recording time point; all numbered combinations are arranged in chronological order to form a historical log content record library.

[0016] In this application, the log number retrieval table and the historical log content library are a basic numbering database for log content based on historical log data. All recorded historical logs can be found in the historical log content library, and their corresponding numbers are determined according to the log number retrieval table. This forms the data foundation for log records. Based on the historical log time information, the log record content at each log record time point is expressed by numbering according to the historical log content library and the log number retrieval table, forming log record data in the time dimension, i.e., forming the historical log content record library. Since the variables corresponding to non-variables may change at each log record time point, the variables are expressed using numerical data, thus forming a complete optimized data record. Compared to the original content records, this significantly reduces the excessive storage space occupied by repetitive records of the same content, simplifies the recording method, further reduces the storage space occupied by log records, and also reasonably optimizes the log records by merging and dividing the log content based on objects, which is beneficial for data analysis of the log content while completely preserving the initial log data information.

[0017] One possible implementation involves acquiring historical log processing information from historical reference log data and establishing a simplified numbering model based on the processing information. This simplified numbering of historical log processing information forms a log processing retrieval table and a historical log processing content library. The process includes: acquiring historical log processing information and identifying unit processing operations within the historical log processing information to form initial unit processing log entries; analyzing the processing effects of all initial unit processing log entries to form result unit processing log entries; combining all result unit processing log entries to form a result unit processing log entry set; analyzing the quantity proportion of all result unit processing log entries in the result unit processing log entry set and arranging the result unit processing log entries according to the quantity proportion analysis results to form a historical log processing content library; and numbering the result unit processing log entries according to the order in the historical log processing content library to form a log processing retrieval table.

[0018] In this application, the log content for recording issues also records corresponding processing information. Therefore, to ensure complete and reasonable optimization of this type of log content, this application also rationally numbers the historical log processing information and matches it with the log content in the historical log content recording library to form a complete column of log content and corresponding log processing information numbers. The numbering of historical log processing information is also merged into similar categories to reduce the storage footprint of historical log processing information. Of course, since the unit log processing entries for historical log processing information are matched with log content, the division of unit log processing entries is based on the division of log content groups, which facilitates subsequent matching of unit log processing entries with log content groups. After forming the resulting unit log processing entries, organizing the resulting unit log processing entries forms a historical log processing content library and a log processing retrieval table corresponding to the historical log processing information. This provides a data foundation for matching the numbering of processing information with log content.

[0019] As one possible implementation, by combining a log processing retrieval table and enriching the historical log content record library with historical log processing time information from historical reference log data, a historical log record and processing library is formed. This includes: determining the result unit used to process log entries at each log record time point based on historical log processing time information; and enriching each number combination information by combining the log processing retrieval table and the number combination information corresponding to each log record time point in the historical log content record library, thus forming log record number information. ,in, This represents the log entry number of the result unit corresponding to the nth non-variable and variable at the log recording time point; the log entry numbers are arranged in chronological order to form a historical log record and processing database.

[0020] In this application, after the unit log processing entries are numbered, a number correspondence is established with the corresponding log content to enrich the historical log content record library, forming a complete historical log record and processing library.

[0021] One possible implementation involves acquiring real-time log data and recording it in a historical log recording and processing database. This includes: acquiring real-time log content information from the real-time log data and comparing it with the historical log content database. If the historical log content database contains non-variables and variables identical to the real-time log content, then: determining the ID information of the real-time log content information in a log ID retrieval table based on the identical non-variables and variables in the historical log content database; and recording the real-time log content information in the historical log recording and processing database according to the real-time log time information based on the ID information.

[0022] In this application, for real-time acquired log data, after optimizing log recording based on big data, it is necessary to record it using the same optimization method to ensure the consistency of log recording format. Here, assuming the log content of the real-time acquired log data is existing in the big data database, the corresponding number of the real-time log content can be determined by comparing and confirming it in the historical log content database. Then, the real-time log database is recorded in the historical log recording and processing database according to the time dimension.

[0023] One possible implementation involves acquiring real-time log data and recording it in a historical log recording and processing database. This includes: acquiring real-time log content information from the real-time log data and comparing it with the historical log content database. If no non-variables or variables identical to the real-time log content exist in the historical log content database, then: adding the new non-variables and variables from the real-time log content information to the historical log content database and adding them to the log number retrieval table to obtain the number information of the real-time log content information; acquiring real-time log processing information from the real-time log data, incorporating it into the historical log processing content database, and adding a number to the log processing retrieval table for the real-time log processing information; and recording the real-time log processing information in the historical log recording and processing database according to the real-time log time information, combining the number of the real-time log content information and the corresponding number of the real-time log processing information.

[0024] In this application, real-time log data may include instances where the log content is entirely new. In such cases, the real-time log content needs to be added to the historical log content database, and a new ID is assigned to each log entry, which is then recorded in a log ID retrieval table. Based on the new ID of the real-time log content, it is added to the historical log content database in chronological order. It is understandable that new log content often corresponds to entirely new log processing information; therefore, this log processing information is also added to the historical log processing content database, forming a log processing retrieval table. Furthermore, the IDs corresponding to the real-time log processing information are used to enrich the historical log records and processing database.

[0025] Secondly, this application provides a log optimization system based on big data, employing the log optimization method based on big data from the first aspect. The system includes a data acquisition unit for acquiring historical log data from a similar storage database to form historical reference log data; a log optimization unit for optimizing and analyzing the historical reference logs from the data acquisition unit, forming a log number retrieval table, a historical log content library, a historical log content record library, a log processing retrieval table, a historical log processing content library, and a historical log record and processing library; and a real-time acquisition unit for acquiring real-time log data and providing it to the log optimization unit for storage and analysis of log optimization.

[0026] In this application, different functional units are used to optimize the numbering of logs and efficiently record real-time log data, thereby reasonably optimizing and improving log recording while ensuring the integrity of log data.

[0027] The beneficial effects of the log optimization system and method based on big data provided by this invention are as follows:

[0028] This method acquires log data from similar storage databases, rationally numbers the log content, thereby simplifying the amount of log records, significantly reducing storage space, and effectively saving data storage capacity. For log content, a time-based content record library is created based on recording time. Simultaneously, the numbered log content is statistically organized to form a numbered index table and content library for retrieval. This ensures complete preservation of log record information, providing a simple and effective recording format while rationally organizing and summarizing the recorded content, facilitating more reasonable and effective basic data for log content analysis. Furthermore, log processing information is also numbered based on the time dimension and mapped to the numbers of log content in the content record library, forming complete log record data. For subsequent real-time log records, numbering is based on the newly established numbering model, ensuring real-time optimization of log records. Since log optimization is based on big data, the real-time acquired log records primarily involve repetitive comparison and confirmation of numbering, thus greatly improving log recording efficiency.

[0029] This model optimizes log numbering and efficiently records real-time log data through different functional units, thereby improving log recording while ensuring the integrity of log data. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart illustrating the steps of a log optimization method based on big data provided in this application embodiment. Detailed Implementation

[0032] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0033] During operation, the application monitors its own performance and records various events in real time. This serves two purposes: firstly, it allows for data tracing of events and understanding of the program's specific execution; secondly, it generates operational analysis data, providing data support for program optimization and development.

[0034] However, current program logging mainly involves directly describing program events, and the resulting log data is not organized or simplified. As the number of log records increases, the storage space occupied by the log records also increases, which becomes a burden on the application, greatly reducing the efficiency of program operation and wasting a lot of storage space.

[0035] refer to Figure 1This application provides a log optimization method based on big data. This method acquires log data from similar storage databases, rationally numbers the log content, thereby simplifying the amount of log data and significantly reducing storage space. For log content, a time-based content record library is formed based on recording time. Simultaneously, the numbered log content is statistically organized to form a numbered index table and content library for retrieval. This ensures complete preservation of log content information, providing a simple and effective recording format while rationally organizing and summarizing the recorded content, facilitating more reasonable and effective basic data for log content analysis. Furthermore, log processing information is also numbered based on the time dimension and mapped to the numbers in the content record library, forming complete log record data. For subsequent real-time log records, numbering is performed based on the newly established numbering model, ensuring real-time optimization of log records. Since log optimization is based on big data, the real-time acquired log records primarily involve repetitive comparison and confirmation of numbering, thus greatly improving log recording efficiency.

[0036] Log optimization methods based on big data include the following main steps:

[0037] S1: Obtain historical log data from databases of the same type to form historical reference log data, and establish a simplified numbering model based on log content. Simplify the numbering of historical log content information in the historical reference log data to form a log number retrieval table and a historical log content library.

[0038] This step includes: acquiring historical log content information, fragmenting each log entry and dividing it into variables and non-variables to form initial unit log content groups; performing logical analysis based on record objects on all initial unit log content groups to form result unit log content groups, and combining all result unit log content groups to form a result unit log content set; extracting non-variables from all result unit log content groups in the result unit log content set, performing quantity percentage analysis on the non-variables, and arranging the non-variables in order according to the quantity percentage analysis results to form a historical log content library; numbering the non-variables according to the order of arrangement in the historical log content library, and determining the corresponding variable values ​​for different numbered non-variables based on the results of the logical analysis based on record objects to form a log number retrieval table.

[0039] When optimizing log content, considering its typically complex nature, the log content is first fragmented to concisely reflect the recorded events. Redundant descriptions, such as separators between different objects and distinguishing characters between different codes, are removed. This reduces the storage size of the resulting log content library. The fragmented log content is then grouped based on event objects to form initial unit log content groups. These initial unit log content groups are then merged according to their invariants, and the merged unit log content groups are organized and summarized to form the historical log content library. The invariants are numbered to create a log number retrieval table because invariants are the most basic fixed parameters in event records, accurately expressing specific event details. Variables, on the other hand, only describe the degree of invariants; therefore, using invariants as the basis for numbering preserves the record of specific event details. Furthermore, the initial unit log content groups are determined based on the different record objects within each log entry. It is known that at a given recording point in time, different objects may be recorded simultaneously and stored together to form the log entry content for that point in time. Since the nonvariables and variables reflected by the different recorded objects are completely different, and there is no absolute correlation between these nonvariables and variables, dividing the log content based on the recorded objects can reasonably classify highly correlated parameters, which will facilitate subsequent data analysis and number retrieval.

[0040] The process involves performing logical analysis on all initial unit log content groups based on the record objects to form result unit log content groups. This includes: performing the following logical analysis on the initial unit log content groups, and merging different initial unit log content groups that simultaneously meet the following logical analysis conditions: the non-variable record objects in different initial unit log content groups are the same; the events recorded by non-variables in different initial unit log content groups are the same; the variables corresponding to non-variables in different initial unit log content groups reflect the same event state; and determining the merged unit log content groups and the initial unit log content groups that do not need to be merged as result unit log content groups.

[0041] To ensure that no duplicate records appear in the resulting historical log content repository, object-based logical analysis is performed after obtaining the initial unit log content group. Records describing the same object at the same time are merged within the initial unit log content group. This approach can further optimize the data in the historical log content repository and reduce its storage footprint. The object-based logical analysis considers three main aspects: whether the objects are identical, whether the recorded events are identical, and whether variables reflecting non-variable aspects can be considered identical. Particular emphasis is placed on determining variable similarity, where variables may differ numerically but reflect the same characteristics and corresponding processed information. This facilitates better matching when mapping log processing content, reducing the complexity of matching log content with log processing information.

[0042] The log numbers are assigned based on the order of the invariants in the historical log content library. Based on the logical analysis results of the record objects, the corresponding variable values ​​for each invariant with a different number are determined, forming a log number retrieval table. This process includes: traversing the total number of invariants in the historical log content library, determining the number of digits based on the total number, and assigning non-consecutive numbers to the invariants; for the merged log content group, identifying the variables from the initial log content groups before merging as the selection options for the invariants in the resulting log content group; and combining the invariants, their corresponding variables, and their corresponding numbers to form the log number retrieval table.

[0043] For the sorting of the log number retrieval table, this application sorts based on the proportion of non-variables within the total. When numbering after sorting, considering the possibility of new log content being added later, the numbering is non-contiguous, which also helps to distinguish different log content in terms of numbering. Of course, to further improve the data rationality of the log number retrieval table, it is also possible to segment the numbering based on the record object. That is, different number segments are determined as the numbers for log content generated on different objects, and different existing log content on the corresponding object is assigned numbers from the number segments. This object-based numbering allows for simple and quick identification of the object recorded by the log content based on the number, further optimizing the log number retrieval table and providing good numbering guidance for data analysis.

[0044] S2: Combining the log number retrieval table, the historical log content information is numbered and combined based on the historical log time information in the historical reference log data to form a historical log content record library.

[0045] By combining the log number retrieval table, historical log content information is numbered and combined based on the historical log time information in the historical reference log data to form a historical log content record library. This includes: obtaining historical log time information, determining the result unit log content group contained at each log record time point, and determining the number combination information at each log record time point in chronological order using the log number retrieval table. Where t is the sequence number of the log record time point determined according to the time order, and t is a non-zero natural number, A n B represents the nth invariant number recorded at the log recording time point. n This represents the value of the variable corresponding to the nth non-variable at the log recording time point; all numbered combinations are arranged in chronological order to form a historical log content record library.

[0046] The log number lookup table and historical log content repository are a basic numbered database for log content based on historical log data. All recorded historical logs can be found in the historical log content repository, and their corresponding numbers are determined according to the log number lookup table. This forms the data foundation for log records. Based on the historical log time information, the log record content at each log record time point is expressed by numbering according to the historical log content repository and the log number lookup table, forming log record data in the time dimension, i.e., forming the historical log content record repository. Since the variables corresponding to non-variables may change at each log record time point, the variables are expressed using numerical data, thus forming a complete and optimized data record. Compared to the original content records, this significantly reduces the excessive storage space occupied by repetitive records of the same content, simplifies the recording method, further reduces the storage space occupied by log records, and also reasonably optimizes the log records by merging and dividing log content based on objects, which is beneficial for data analysis of log content while completely preserving the initial log data information.

[0047] S3: Obtain historical log processing information from historical reference log data, establish a simplified numbering model based on processing information, simplify the numbering of historical log processing information, and form a log processing retrieval table and a historical log processing content library.

[0048] Historical log processing information is obtained from historical reference log data, and a simplified numbering model based on the processing information is established. This simplified numbering of historical log processing information forms a log processing retrieval table and a historical log processing content library. The process includes: obtaining historical log processing information and identifying unit processing operations within the information to form initial unit processing log entries; analyzing the processing effects of all initial unit processing log entries to form result unit processing log entries; combining all result unit processing log entries to form a result unit processing log entry set; analyzing the proportion of all result unit processing log entries in the result unit processing log entry set and arranging them according to the proportion analysis results to form a historical log processing content library; and numbering the result unit processing log entries in the historical log processing content library according to their order to form a log processing retrieval table.

[0049] For log entries recording issues, corresponding processing information is also recorded. Therefore, to ensure complete and reasonable optimization of this type of log content, this application also rationally numbers the historical log processing information and matches it with the log content in the historical log content recording library to form a complete column of log content and corresponding log processing information numbers. The numbering of historical log processing information is also merged into similar categories to reduce the storage footprint of historical log processing information. Of course, since the unit log processing entries for historical log processing information are matched with log content, the division of unit log processing entries is based on the division of log content groups, which facilitates subsequent matching of unit log processing entries with log content groups. After forming the resulting unit log processing entries, organizing the resulting unit log processing entries forms a historical log processing content library and a log processing retrieval table corresponding to the historical log processing information. This provides a data foundation for matching the numbering of processing information with log content.

[0050] S4: Combine the log processing retrieval table with the historical log processing time information in the historical reference log data to enrich the historical log content record library with processing information, forming a historical log record and processing library.

[0051] By combining the log processing retrieval table and based on the historical log processing time information in the historical reference log data, the historical log content record library is enriched with processing information to form a historical log record and processing library. This includes: determining the result unit used to process the log entry at each log record time point based on the historical log processing time information; and enriching each number combination information by combining the log processing retrieval table and the number combination information corresponding to each log record time point in the historical log content record library to form log record number information. ,in, This represents the log entry number of the result unit corresponding to the nth non-variable and variable at the log recording time point; the log entry numbers are arranged in chronological order to form a historical log record and processing database.

[0052] After the unit log processing entries are numbered, a number correspondence is established with the corresponding log content to enrich the historical log content record library, forming a complete historical log record and processing library.

[0053] It should be noted that for historical log records and processing databases, there may be parallel processing scheme information in the result unit processing log entries corresponding to individual invariants and variables. Therefore, when sequentially numbering these parallel processing scheme information, the impact on invariants and variable combinations needs to be considered. Here, a method for determining the numbering order of parallel processing scheme information is provided:

[0054] First, the variable range corresponding to the non-variable is obtained based on the historical log content record library, forming a variable range set D; the range in the variable range set D is analyzed for overlap, and the range with the highest overlap is determined as the direct variable range E=[e1, e2] corresponding to the non-variable, where e1 represents the lower boundary value of the range with the highest overlap, and e2 represents the upper boundary value of the range with the highest overlap.

[0055] For invariants whose range is entirely within the direct variable range E, they are numbered in descending order of their values ​​according to the following formula:

[0056] , where e x and e y Let T represent the range of variables corresponding to the invariant, and let T be the time taken to continuously observe the invariant after it has been acquired, until it is no longer recorded or returns to normal. In other words, the larger the proportion of the variable range relative to the direct variable range E, the smaller T is, and the better the processing effect of the solution.

[0057] For nonvariables whose range does not entirely fall within the direct variable range E, they are numbered in descending order of their values ​​according to the following formula:

[0058] , where e z This refers to the overlap between the range of the non-variable and the range of the direct variable. In other words, the larger the proportion of the variable's range falling within the direct variable range E, the smaller T is, and the better the processing effect of the solution.

[0059] For nonvariables whose range is not at all within the direct variable range E, they are numbered in descending order of their values ​​according to the following formula:

[0060] , where represents the range of the variable corresponding to the non-variable. That is, the wider the variable range, the smaller T, and the better the processing effect of the solution.

[0061] Of course, after determining the numbering order of the variable ranges for different situations, the variable ranges with priority in the direct variable range E are numbered sequentially, while those variable ranges not at all in the direct variable range E are numbered last. This sequential numbering also provides a reasonable reference for the subsequent selection of the order of processing schemes corresponding to the variables.

[0062] S5: Obtain real-time log data and record it in the historical log recording and processing database based on the real-time log data.

[0063] There are two ways to process real-time log data:

[0064] The first method involves acquiring real-time log data and recording it in the historical log recording and processing database. This includes: acquiring real-time log content information from the real-time log data and comparing it with the historical log content database. If the historical log content database contains non-variables and variables identical to the real-time log content, then: determining the ID information of the real-time log content information in the log ID retrieval table based on the identical non-variables and variables in the historical log content database; and recording the real-time log content information in the historical log recording and processing database according to the real-time log time information based on the ID information.

[0065] For real-time log data, after optimizing log recording based on big data, it needs to be recorded using the same optimization method to ensure the consistency of log recording format. Here, we assume the log content of the real-time log data is already present in the big data database. Therefore, by comparing and confirming with the historical log content database, the corresponding log content number of the real-time log data can be determined. Then, the real-time log database is recorded in the historical log recording and processing database according to the time dimension.

[0066] The second type:

[0067] The process involves acquiring real-time log data and recording it in the historical log recording and processing database. This includes: acquiring real-time log content information from the real-time log data and comparing it with the historical log content database. If no non-variables or variables identical to the real-time log content exist in the historical log content database, then: the new non-variables and variables from the real-time log content information are added to the historical log content database, and the new non-variables and variables are added to the log number retrieval table to obtain the number information of the real-time log content information; acquiring real-time log processing information from the real-time log data, incorporating the real-time log processing information into the historical log processing content database, and adding the number of the real-time log processing information to the log processing retrieval table; and recording the real-time log content information number and the corresponding real-time log processing information number in the historical log recording and processing database according to the real-time log time information.

[0068] Of course, real-time log data may also contain entirely new events. In such cases, the real-time log content needs to be added to the historical log content repository, and a new ID should be assigned to each log entry, recorded in a log ID retrieval table. Based on the new ID of the real-time log content, it is added to the historical log content record repository in chronological order. It's understandable that new log content often corresponds to entirely new log processing information; therefore, this processing information is also added to the historical log processing content repository, forming a log processing retrieval table. Furthermore, the IDs corresponding to the real-time log processing information are used to enrich the historical log records and processing repository.

[0069] This application also provides a log optimization system based on big data, employing the log optimization method based on big data provided in this application. The system includes a data acquisition unit for acquiring historical log data from a similar type of storage database to form historical reference log data; a log optimization unit for optimizing and analyzing the historical reference logs from the data acquisition unit, forming a log number retrieval table, a historical log content library, a historical log content record library, a log processing retrieval table, a historical log processing content library, and a historical log record and processing library; and a real-time acquisition unit for acquiring real-time log data and providing it to the log optimization unit for storage and analysis of log optimization.

[0070] This method optimizes log numbering and efficiently records real-time log data through different functional units, thereby improving log recording while ensuring the integrity of log data.

[0071] In summary, the beneficial effects of the big data-based log optimization system and method provided in this application are as follows:

[0072] This method acquires log data from similar storage databases, rationally numbers the log content, thereby simplifying the amount of log records, significantly reducing storage space, and effectively saving data storage capacity. For log content, a time-based content record library is created based on recording time. Simultaneously, the numbered log content is statistically organized to form a numbered index table and content library for retrieval. This ensures complete preservation of log record information, providing a simple and effective recording format while rationally organizing and summarizing the recorded content, facilitating more reasonable and effective basic data for log content analysis. Furthermore, log processing information is also numbered based on the time dimension and mapped to the numbers of log content in the content record library, forming complete log record data. For subsequent real-time log records, numbering is based on the newly established numbering model, ensuring real-time optimization of log records. Since log optimization is based on big data, the real-time acquired log records primarily involve repetitive comparison and confirmation of numbering, thus greatly improving log recording efficiency.

[0073] This model optimizes log numbering and efficiently records real-time log data through different functional units, thereby improving log recording while ensuring the integrity of log data.

[0074] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0075] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0076] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0077] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0078] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0079] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0080] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0081] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A log optimization method based on big data, characterized in that, include: Historical log data from similar storage databases is obtained to form historical reference log data. A simplified numbering model based on log content is established, and the historical log content information in the historical reference log data is simplified and numbered to form a log number retrieval table and a historical log content library. Combining the log number retrieval table, the historical log content information is numbered and combined based on the historical log time information in the historical reference log data to form a historical log content record library; Obtain historical log processing information from the historical reference log data, establish a simplified numbering model based on the processing information, simplify the numbering of the historical log processing information, and form a log processing retrieval table and a historical log processing content library. Combining the log processing retrieval table and based on the historical log processing time information in the historical reference log data, the historical log content record library is enriched with processing information to form a historical log record and processing library. Acquire real-time log data and record it in the historical log record and processing database based on the real-time log data; This involves acquiring historical log data from similar storage databases to form historical reference log data, establishing a simplified numbering model based on log content, and simplifying the numbering of historical log content information in the historical reference log data to form a log number retrieval table and a historical log content library, including: The historical log content information is obtained, and each log entry in the historical log content information is fragmented and divided into variables and non-variables to form an initial unit log content group. Perform logical analysis based on the record object on all the initial unit log content groups to form result unit log content groups, and combine all the result unit log content groups to form a result unit log content group set; Extract the nonvariates from all the log content groups of the result unit in the result unit log content group set, perform a quantity percentage analysis on the nonvariates, and arrange the nonvariates in order according to the quantity percentage analysis results to form the historical log content library; The log numbers are assigned according to the order of the nonvariables in the historical log content library, and the variable values ​​corresponding to the nonvariables with different numbers are determined based on the results of the logical analysis based on the record objects, thus forming the log number retrieval table. Perform logical analysis based on record objects on all the initial unit log content groups to form the result unit log content groups, including: The initial unit log content group is subjected to the following logical analysis, and different initial unit log content groups that simultaneously meet the following logical analysis conditions are merged: The non-variable record objects in different initial unit log content groups are the same; The events recorded in the nonvariable records in different initial unit log content groups are the same; The variables corresponding to the non-variables in different initial unit log content groups reflect the same event state; The merged unit log content group and the initial unit log content group that does not need to be merged are determined as the result unit log content group; The log numbers are assigned according to the order of the invariants in the historical log content library, and the corresponding variable values ​​under different invariant numbers are determined based on the results of the logical analysis based on the record objects, forming the log number retrieval table, including: Iterate through the total number of nonvariables in the historical log content library, determine the number of digits for the numbering based on the total number, and number the nonvariables sequentially, with the numbering being non-consecutive. For the resulting unit log content group formed by merging, the variables in the multiple initial unit log content groups before merging are determined as the selection items for the variables corresponding to the non-variables in the resulting unit log content group; The log number retrieval table is formed by combining the nonvariable, the variable corresponding to the nonvariable, and the number corresponding to the nonvariable; Combining the aforementioned log number retrieval table, the historical log content information is numbered and combined based on the historical log time information in the historical reference log data to form a historical log content record library, including: Obtain the historical log time information, determine the result unit log content group contained at each log recording time point, and determine the number combination information at each log recording time point in chronological order according to the log number retrieval table: Where t is the sequence number of the log record time point determined according to the time order, and t is a non-zero natural number, A n B represents the nth nonvariable number recorded at the time point of the log record. n This represents the value of the variable corresponding to the nth non-variable at the time point of the log record; Arrange all the numbered combinations in chronological order to form the historical log content record library; Obtain historical log processing information from the historical reference log data, and establish a simplified numbering model based on the processing information. Simplify the numbering of the historical log processing information to form a log processing retrieval table and a historical log processing content library, including: The historical log processing information is obtained, and the unit processing operation is identified in the historical log processing information to form an initial unit processing log entry; Analyze all the initial unit processing log entries based on the processing effect to form result unit processing log entries, and combine all the result unit processing log entries to form a result unit processing log entry set; The number proportion of all the result unit processing logs in the result unit processing log set is analyzed, and the result unit processing logs are arranged according to the number proportion analysis results to form the historical log processing content library; The log processing retrieval table is formed by numbering the log entries processed by the result units in the historical log processing content library according to their order. Combining the aforementioned log processing retrieval table and based on the historical log processing time information in the historical reference log data, the historical log content record library is enriched with processing information to form a historical log record and processing library, including: Based on the historical log processing time information, the result unit used to process the log entry at each log recording time point is determined. Then, combining the log processing retrieval table and the number combination information corresponding to each log recording time point in the historical log content record library, each number combination information is enriched to form log record number information: ,in, This indicates the number of the log entry processed by the result unit corresponding to the nth non-variable and variable at the log recording time point; The log record numbers are arranged in chronological order to form the historical log record and processing database.

2. The log optimization method based on big data according to claim 1, characterized in that, The step of acquiring real-time log data and recording it in the historical log record and processing database includes: Obtain the real-time log content information from the real-time log data, and compare the real-time log content information with the historical log content database. If the historical log content database contains non-variables and variables identical to the real-time log content, then: Based on the same nonvariates and variables in the historical log content library, the number information of the real-time log content information is determined on the log number retrieval table; Based on the numbering information of the real-time log content, the logs are recorded in the historical log recording and processing database according to the real-time log time information.

3. The log optimization method based on big data according to claim 2, characterized in that, The step of acquiring real-time log data and recording it in the historical log record and processing database includes: Obtain the real-time log content information from the real-time log data, and compare the real-time log content information with the historical log content database. If there are no non-variables or variables identical to the real-time log content in the historical log content database, then: Add the new non-variables and variables from the real-time log content information to the historical log content library, and add the new non-variables and variables to the log number retrieval table to obtain the number information of the real-time log content information; Obtain real-time log processing information from the real-time log data, incorporate the real-time log processing information into the historical log processing content library, and add the number of the real-time log processing information to the log processing retrieval table. Based on the number of the real-time log content information and the corresponding number of the real-time log processing information, the logs are recorded in the historical log record and processing database according to the real-time log time information.

4. A log optimization system based on big data, employing the log optimization method based on big data as described in any one of claims 1-3, characterized in that, include: The data acquisition unit is used to acquire historical log data from similar storage databases and form historical reference log data. The log optimization unit is used to optimize and analyze the historical reference logs in the data acquisition unit, and respectively form a log number retrieval table, a historical log content library, a historical log content record library, a log processing retrieval table, a historical log processing content library, and a historical log record and processing library. The real-time acquisition unit is used to collect real-time log data and provide the real-time log data to the log optimization unit for log optimization, storage, and analysis.

Images