Log filing method and device and processor

Abstract

The invention discloses a log archiving method and device and a processor. The invention discloses a log archiving method and device and a processor. Whether unarchived original logs exist in a storage system or not is checked, and if the unarchived original logs exist, the storage time of the logs is checked. And when the storage time reaches or exceeds an archiving threshold value, archiving the log into a first-level archiving statistical table. And if no unarchived log exists, checking whether the total data amount of the archived log in each stage of archiving statistical table reaches a threshold value or not. And further archiving the archived logs in the archived statistical table of which the total log data amount reaches the threshold value to a next-level statistical table. After archiving, the original logs and the logs in the current archiving statistical table are deleted, and the data volume is reduced after each time of archiving. According to the method, space occupation is reduced through hierarchical storage and automatic archiving, data redundancy is avoided by dynamically adjusting the archiving table, storage management is simplified, and the log processing difficulty is reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a log archiving method, device and processor. Background Art

[0002] The service gateway call log records the information generated by the service gateway in the process of forwarding client requests to backend microservices, and receiving and returning responses. These logs are essential for monitoring, troubleshooting, performance analysis, and security auditing. However, for businesses with a large number of interface calls, the call logs accumulated over a period of time may become huge. This creates two difficulties for the business system: on the one hand, the storage space of the business system is limited, so it is difficult to retain the service gateway call logs related to the business system for a long time; on the other hand, storing a large number of logs in the business system will interfere with the analysis of the real-time call status of the business system, making it difficult to process and analyze these logs. Summary of the invention

[0003] Based on the above problems, the present application provides a log archiving method, device and processor, the purpose of which is to reduce the storage space occupied by the service gateway call log in the business system, while reducing the difficulty of log processing and analysis.

[0004] The embodiments of the present application disclose the following technical solutions:

[0005] A log archiving method, the method comprising:

[0006] Query whether there are unarchived original logs in the storage system; the storage system includes a plurality of archive statistics tables with descending levels; each archive statistics table is respectively set with an archive data volume threshold;

[0007] If it exists, query the total storage time of the original log that has been stored in the storage system. When the storage time of the original log is greater than or equal to the archiving time threshold, archive the original log to obtain an archive log, and store the archive log in the first-level archiving statistics table; the first-level archiving statistics table is the highest-level archiving statistics table in the storage system;

[0008] If it does not exist, query whether the total amount of archived log data in the archived statistical tables at all levels is greater than or equal to the corresponding archived data volume threshold, and further archive all archived logs in the archived statistical tables whose total amount of archived log data is greater than or equal to the corresponding archived data volume threshold, and store them all in the archived statistical tables of the next level;

[0009] Among them, after the original log is archived to the first-level archive statistics table, the original log in the storage system is deleted; when the archive log in the current archive statistics table is archived to the next-level archive statistics table, the archive log in the current archive statistics table is deleted; the data volume of the log after each archiving is less than the data volume of the log before archiving.

[0010] In a possible implementation, the log archiving process includes:

[0011] Determine one or more key information in the log to be archived and the information key level corresponding to the key information; the log to be archived is a log in the statistical table to be archived; the statistical table to be archived is an archive statistical table whose total data amount of the archived log is greater than or equal to its corresponding archive data amount threshold; each level of the archive statistical table corresponds to one or more information key levels; as the level decreases, the number of information key levels corresponding to each archive statistical table decreases in sequence;

[0012] Based on the information key level corresponding to the next level archiving statistical table of the to-be-archived statistical table, determining one or more archiving information from the one or more key information;

[0013] The archive information is combined based on the semantic logic of the log to be archived to obtain an archive log, and the archive log is stored in an archive statistical table at a lower level of the statistical table to be archived.

[0014] In a possible implementation, the method further includes:

[0015] Based on the business information corresponding to the original log, searching whether the original log records a key event in the business information; the key event is determined based on a business tag of the business information;

[0016] If yes, create a copy of the original log, record the copy of the original log as an important log, and record the important log in an important log area of ​​the storage system;

[0017] Otherwise, the original logs are summarized into summary logs, and the summary logs are recorded in a summary log area in the storage system;

[0018] Among them, the important log, the original log corresponding to the important log and the archived log archived based on the original log of the important log have the same unique identifier; the summary log, the original log of the summary log and the archived log archived based on the original log of the summary log have the same unique identifier.

[0019] In a possible implementation, the archiving statistics table at each level includes multiple types of archiving statistics tables;

[0020] The method further comprises:

[0021] Get various business types of logs and set multiple archive statistics table levels;

[0022] At each level of the archive statistics table, multiple types of archive statistics tables are constructed based on the various business types.

[0023] In a possible implementation, the method includes:

[0024] The original log is backed up to a disk in a text-sending format.

[0025] In a possible implementation, the method includes:

[0026] Obtain a target log query request; the target log query request includes a target business type and a target time range;

[0027] Determine one or more target archiving statistical tables from archiving statistical tables at all levels in the storage system based on the target business type;

[0028] Based on the target time range, querying the archive log corresponding to the target log query request from the one or more target archive statistics tables as the target log;

[0029] The important log corresponding to the unique identifier of the target log is queried, and the target log is supplemented based on the important log to obtain an output log.

[0030] In a possible implementation, querying the important log corresponding to the unique identifier of the target log, and supplementing the target log based on the important log to obtain an output log includes:

[0031] Querying whether there is an important log corresponding to the unique identifier of the target log in the important log area;

[0032] If it exists, supplement the target log based on the important log to obtain an output log;

[0033] If it does not exist, the original log corresponding to the unique identifier of the target log is read from the disk as the important log, and the target log is supplemented based on the important log to obtain an output log.

[0034] In a possible implementation, the method further includes:

[0035] In response to the important log query request of the output log, if the important log corresponding to the unique identifier of the target log exists in the important log area or the disk, then display the important log corresponding to the unique identifier of the target log; otherwise, do not respond; and / or,

[0036] In response to the summary log query request of the output log, if the summary log corresponding to the unique identifier of the target log exists in the summary log area, the summary log corresponding to the unique identifier of the target log is displayed; otherwise, no response is given.

[0037] A log archiving device, the device comprising:

[0038] The first query unit is used to query whether there are unarchived original logs in the storage system; the storage system includes a plurality of archive statistics tables with descending levels; each archive statistics table is respectively set with an archive data volume threshold;

[0039] The second query unit is used to query the total storage time of the original log that has been stored in the storage system if there are unarchived original logs in the storage system. When the storage time of the original log is greater than or equal to the archiving time threshold, the original log is archived to obtain an archived log, and the archived log is stored in a first-level archiving statistical table; the first-level archiving statistical table is the highest-level archiving statistical table in the storage system;

[0040] A third query unit, for querying whether the total amount of archived log data in the archived statistical tables at all levels is greater than or equal to the corresponding archived data amount threshold if there is no unarchived original log in the storage system;

[0041] The archiving processing unit is used to further archive all the archive logs in the archiving statistical table whose total amount of archive log data is greater than or equal to the corresponding archive data amount threshold, and store all of them in the next-level archiving statistical table;

[0042] Among them, after the original log is archived to the first-level archive statistics table, the original log in the storage system is deleted; when the archive log in the current archive statistics table is archived to the next-level archive statistics table, the archive log in the current archive statistics table is deleted; the data volume of the log after each archiving is less than the data volume of the log before archiving.

[0043] A processor is used to run a computer program, and when the program is run, the log archiving method as described above is executed.

[0044] Compared with the prior art, this application has the following beneficial effects:

[0045] The embodiment of the present application queries whether there are unarchived original logs in the storage system. If there are, the storage time of the original log is queried, and when the storage time of the original log is greater than or equal to the archiving time threshold, the original log is archived to obtain an archive log, and stored in a first-level archiving statistical table. If not, the total amount of data of the archive logs in the archiving statistical tables at all levels is queried to see whether it is greater than or equal to the corresponding archive data volume threshold. All archive logs in the archiving statistical table whose total amount of data of the archive logs is greater than or equal to the corresponding archive data volume threshold are further archived and all stored in the next-level archiving statistical table. After the original log is archived, the original log in the storage system is deleted, and when the archive log in the current archive statistical table is archived to the next-level archiving statistical table, the archive log in the current archive statistical table also needs to be deleted, and the data volume of the log after each archiving is less than the data volume of the log before archiving. The present application significantly reduces the space occupation of the storage system by storing the original logs in different archiving statistical tables in a hierarchical manner. At the same time, multiple descending levels of archiving statistical tables are provided, and each table corresponds to a different data volume threshold. In this way, it is possible to automatically determine whether the archived logs need to be further archived based on the archived data volume threshold, ensuring the dynamic adjustment of the archived statistics table and avoiding the waste of space caused by excessive accumulation of log data in the archived statistics table. In addition, after each archiving operation, the old logs will be cleared, avoiding data redundancy and simplifying storage management, thereby reducing the difficulty of log processing and analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 A flow chart of an implementation method of a log archiving method provided in an embodiment of the present application;

[0048] Figure 2 A flowchart of an implementation method of a method for obtaining important logs and summary logs provided in an embodiment of the present application;

[0049] Figure 3 A flowchart of an implementation method of a log archiving processing method provided in an embodiment of the present application;

[0050] Figure 4 A flowchart of an implementation method of a log query method provided in an embodiment of the present application;

[0051] Figure 5A schematic diagram of the structure of a log archiving device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0053] As described above, the service gateway call log records the information generated by the service gateway in the process of forwarding client requests to backend microservices and receiving and returning responses. These logs are essential for monitoring, troubleshooting, performance analysis, and security auditing. The following are some key meanings and contents of the service gateway call log:

[0054] Request information recording: The log records detailed information of each request passing through the gateway, including but not limited to request time, request method (GET, POST, etc.), request Uniform Resource Locator (URL), Hypertext Transfer Protocol (HTTP) header, request body, client Internet Protocol (IP) address, etc. This helps to track the complete life cycle of a single request;

[0055] Response information recording: In addition to request information, the log also records the response details of the backend service, such as response status code, response body, response time, etc. This is very important for monitoring the health status and performance indicators of the service;

[0056] Errors and exceptions: When an error or exception occurs during request processing, the log will record the error information, stack trace, and any related error codes or messages in detail, which is crucial for quickly locating and resolving the problem;

[0057] Performance indicators: Recording request processing time and response time can help identify performance bottlenecks and provide a basis for optimizing services;

[0058] Traffic control information: If the gateway implements traffic control policies, such as current limiting and circuit breaking, the log will record the events that trigger these policies to help monitor system stability and prevent service overload.

[0059] However, for businesses with huge volumes of interface calls, log data will accumulate quickly, leading to two major problems: first, the storage space of the business system is limited, making it difficult to preserve a large number of logs for a long time; second, a large number of logs will interfere with the system's real-time analysis, increasing the difficulty of processing and analysis.

[0060] Based on this, the embodiment of the present application provides a log archiving method, device and processor, by querying whether there are unarchived original logs in the storage system; if there are, check the storage time of the original log, if the time reaches the archiving threshold, archive it to the first-level archiving statistical table, and delete the original log; if there are no unarchived logs, check the total data of the logs in the archiving statistical tables at all levels, if the threshold is reached, the logs in the archiving statistical table that reach the threshold are further archived to the next level table, and the logs in the current table are deleted. After the archiving operation, the amount of data in each level of the archiving statistical table is reduced. The present application reduces the space occupied by the storage system and optimizes the storage resources. Only the original logs that meet the archiving time threshold are archived, which ensures the timeliness and accuracy of log archiving, and avoids redundant processing of all logs. At the same time, the hierarchical archiving mechanism (from the first level to the next level) makes the archiving processing of log data more systematic and efficient, and can better manage a large amount of log data. In addition, the archived logs are automatically deleted and archived to the next level table, which reduces manual intervention, improves the archiving efficiency, and further reduces the difficulty of log processing and analysis.

[0061] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] See also Figure 1 As shown in FIG. 1 , this figure is a flow chart of a log archiving method provided in an embodiment of the present application, such as Figure 1 As shown, the method may include S101-S103:

[0064] S101: Query whether there are unarchived original logs in the storage system.

[0065] The storage system will check whether there are unarchived original logs in the storage system within a fixed time period or at a fixed time point. In the storage system, there are multiple archive statistics tables of different levels, and these tables are gradually reduced according to the level. Each archive statistics table sets a specific archive data volume threshold to trigger the corresponding archive operation. This means that the storage system automatically manages and archives logs according to different levels of data volume thresholds, so as to effectively control the log storage and retention policy.

[0066] Optionally, the storage system may be, but is not limited to, a search engine. A search engine is a tool for searching information on the Internet or in a computer system. It can find and present relevant information by keywords, phrases, or questions.

[0067] Optionally, the fixed time period may be, but is not limited to, 24 hours or other time periods. The present application does not impose any specific restrictions on the range of the fixed time period, and the user may adjust the range of the fixed time period according to actual needs.

[0068] Optionally, the fixed time point may be but is not limited to early morning every day or other fixed time points. This application does not impose specific restrictions on the fixed time points, and users may adjust the fixed time points according to actual needs.

[0069] In a possible implementation, the method includes: backing up the original log in a text-sending format to a hard disk.

[0070] Back up the original log files to the hard disk in the form of text files. This process involves converting the original log data into text format and saving these text files to the computer's hard drive or an external hard drive so that the log information can be easily accessed, viewed, and restored when needed. This backup method helps protect important log data and ensure data reliability and accessibility.

[0071] A hard disk is a device used to store data in a computer, also known as a hard disk drive. It usually consists of a rotating platter (disk) and a moving read-write head, and data is stored and read through the magnetic surface on these platters. Hard disks are used for long-term storage of data such as files, applications, and operating systems, and are widely used in personal computers, servers, and other computing devices.

[0072] In one possible implementation, each level of the archiving statistics table includes multiple types of archiving statistics tables.

[0073] Each archive statistics table level contains archive statistics tables of different types or categories. In other words, each archive statistics table level can be divided into multiple subcategories or types of archive statistics tables for more detailed management and organization of archive data. This structural design can help the system to classify, store and back up log data more effectively, and ensure that appropriate archive strategies are implemented for different categories of data.

[0074] In one possible implementation, the method further includes:

[0075] Various business types of logs are obtained, and multiple archive statistics table levels are set. Then, under each of the archive statistics table levels, multiple types of archive statistics tables are constructed based on the various business types.

[0076] Specifically, first, identify and obtain different business types in the log data. Then, for these different business types, the system will set up multiple archive statistics tables at different levels. This allows for flexible management and archiving of log data based on the different characteristics and needs of the business types. Next, for each set archive statistics table level, at this level, the system will build multiple types of archive statistics tables based on different business types. This means that each archive statistics table level will contain specific archive statistics for different business types, thereby achieving more detailed and precise log data management and archiving control.

[0077] See also Figure 2 , Figure 2 A flowchart of an implementation method of a method for obtaining important logs and summary logs provided in an embodiment of the present application can be implemented through steps A1-A3:

[0078] A1: Based on the business information corresponding to the original log, search whether the original log records key events in the business information.

[0079] Based on the business information involved in the original log, it is possible to find out whether the original log contains key events related to the business information. This process involves using the tags of the business information to determine what kind of events are considered key events. In other words, the system will look for log records in the original log that record important events in the business information based on the business tags and the corresponding business information. This helps to quickly identify and extract information that is critical to a specific business process or activity, and may trigger further actions or decisions.

[0080] If the retrieved original log records the key events in the business information, proceed to step A2.

[0081] A2: Create a copy of the original log, record the copy of the original log as an important log, and record the important log in an important log area of ​​the storage system.

[0082] When a key event in business information is recorded in the original log, the system will create a copy of the original log. This copy is marked as an important log to indicate that it contains important business information. This important log is then stored in an area dedicated to storing important logs. This ensures that the log data corresponding to important business events can be specially backed up and managed for further analysis, tracking or auditing. Such operations help maintain data integrity and reliability, and make the processing of important events more efficient and secure.

[0083] Business information refers to data related to a specific business process or activity. On the WeChat platform, business information may include users’ chat records, videos, pictures, transfer records, etc.

[0084] Business tags for business information are tags used to classify or label business information to determine its importance or other characteristics. For example, transfer records may be labeled as "financial transactions" and chat records may be labeled as "user interactions."

[0085] Key events are events that are considered very important in business information and are usually determined based on business tags.

[0086] Exemplarily, it is assumed that the system is processing the original log of the WeChat platform, and the original log of the WeChat platform includes the following two kinds of original logs: chat record log and transfer record log.

[0087] The chat log includes text messages, pictures or videos sent by users, so the business label of the chat log is "user interaction". The transfer log records the user's financial transactions, including the transfer amount, time, transaction parties, etc., so the business label of the transfer log is marked as "financial transaction". The original log corresponding to "user interaction" is a non-critical event, and the original log corresponding to "financial transaction" is a critical event. At this point, it can be determined that the original log such as the transfer log in the WeChat platform records the key events in the business information, so a copy of the original log such as the transfer log is created and recorded as an important log.

[0088] It should also be noted that important logs, the original logs corresponding to important logs, and the archived logs based on the original logs of important logs have the same unique identifier. This helps to ensure the consistency and integrity of data and facilitates tracking and retrieval. In this way, no matter what stage the logs are in, they can be associated and managed through unique identifiers.

[0089] If the key event in the business information is not recorded in the retrieved original log, proceed to step A3.

[0090] A3: Aggregate the original logs into summary logs, and record the summary logs in a summary log area in the storage system.

[0091] If no important events or data (such as key business activities) are found in the original logs, these original logs will be processed and summarized into more concise summary logs. Then, these summary logs will be stored in a specific area (summary log area) in the system, which is specially used to store this summarized information. This can save storage space and improve management efficiency.

[0092] It should be noted that summary logs play an important role in data management and analysis. Their main functions include:

[0093] Save storage space: Summary logs reduce the amount of data storage required by compressing and summarizing information. This is especially important when processing large amounts of raw log data, because raw logs are usually very detailed and take up a lot of storage space;

[0094] Improve data processing efficiency: Summarized information is easy to query and analyze quickly. Summary logs usually contain key statistics and summary information, which allows users to quickly get an overall overview without having to process a large amount of detailed data;

[0095] Simplified data management: Summary logs help organize and manage log data. By summarizing data into a concise format, management complexity is reduced, especially when dealing with large volumes of logs.

[0096] Support quick decision-making: Summary logs provide an overview of business activities, allowing decision makers to quickly understand the status of the system or the overall trend of the business and make timely decisions;

[0097] Support for long-term storage and archiving: Summary logs can be used for long-term storage and archiving, which helps provide a high-level view without having to access all detailed original log records when you need to review or review historical data.

[0098] In summary, summary logs help manage and analyze data by providing a condensed, summarized view while reducing storage requirements and improving processing efficiency.

[0099] It should also be noted that the summary log, the original log of the summary log, and the archived log based on the original log of the summary log have the same unique identifier, which can ensure their consistency and association in the system. Whether it is original data, summary information or archived records, they can be tracked and verified through this unique identifier to ensure data integrity and accuracy.

[0100] If there are unarchived original logs in the storage system, the process goes to step S102.

[0101] S102: querying the total storage time of the original log that has been stored in the storage system, and when the storage time of the original log is greater than or equal to the archiving time threshold, archiving the original log to obtain an archived log, and storing the archived log in a first-level archiving statistics table.

[0102] If there are unarchived original logs in the storage system, the storage system will query the total time these logs have been stored in the system. If the storage time of these original logs exceeds or is equal to the set archiving time threshold, the storage system will archive these original logs, generate archive logs, and save the archive logs to the highest level first-level archiving statistics table. The first-level archiving statistics table mentioned here is the highest level table in the storage system, which is used to store important data that has been archived. Once the original log is successfully archived to the first-level archiving statistics table, the storage system will delete the copy of the original log in the storage system to save space and maintain data neatness.

[0103] If there is no unarchived original log in the storage system, the process proceeds to step S103.

[0104] S103 queries whether the total amount of archived log data in the archived statistical tables at all levels is greater than or equal to the corresponding archived data amount threshold, and further archives all archived logs in the archived statistical tables whose total amount of archived log data is greater than or equal to their corresponding archived data amount threshold, and stores them all in the next level of archived statistical tables.

[0105] If there are no unarchived original logs in the storage system, the system will check whether the total amount of archived logs in the archive statistics table at each level reaches or exceeds the archive data volume threshold set for each level. For all archive logs in the archive statistics table whose total amount of data meets or exceeds the threshold, the system will perform further archiving processing and store all these archive logs in the next-level archive statistics table. When archiving the archive logs in the current archive statistics table to the next-level archive statistics table, the system will delete the archive logs in the current archive statistics table. This helps keep the system clean and avoid data redundancy. After each archiving operation, the data volume of the newly generated archive log will be smaller than the log data volume before archiving. This ensures that the data is gradually reduced during the continuous archiving process, which is conducive to the effective use and management of system resources.

[0106] See also Figure 3 , Figure 3 A flowchart of an implementation method of a log archiving processing method provided in an embodiment of the present application can be implemented through steps B1-B3:

[0107] B1: Determine one or more key information in the log to be archived and the information criticality level corresponding to the key information.

[0108] In the log to be archived, one or more key information is identified, and the information key level associated with these information is identified. This helps to identify the most important data content in the log. The log to be archived refers to the log in the statistics table to be archived, and the statistics table to be archived refers to the archive statistics table whose total data of the archive log is greater than or equal to its corresponding archive data volume threshold. Each archive statistics table level is associated with one or more information key levels. As the level decreases, the number of information key levels corresponding to each archive statistics table will also decrease accordingly. This structure helps to classify and manage archive data according to different levels of importance.

[0109] Example 1, assuming that the log to be archived is: "Xiao Ming has 4 RMB notes of 100 yuan, 3 RMB notes of 50 yuan, and 2 RMB notes of 30 yuan". In this log to be archived, "RMB, 100 yuan, 50 yuan, 30 yuan, 4 notes, 3 notes, 2 notes, and Xiao Ming" are all key information, the key level of "RMB" is level 1, the key level of "100 yuan, 50 yuan, 30 yuan" is level 2, the key level of "4 notes, 3 notes, 2 notes" is level 3, and the key level of "Xiao Ming" is level 4. Therefore, we can know that the key levels of information in the log to be archived are level 1, level 2, level 3, and level 4.

[0110] B2: Based on the information key level corresponding to the next level archiving statistical table of the to-be-archived statistical table, determine one or more archiving information from the one or more key information.

[0111] According to the information criticality level corresponding to the next-level archive statistics table of the to-be-archived statistics table, one or more information contents to be archived are determined from one or more key information in the to-be-archived log. During the log data processing, the system selects appropriate key information according to the information criticality level specified in the next-level archive statistics table, and then stores this information as archive data in the corresponding archive location. This helps ensure effective management and optimized storage of log data.

[0112] Example 2, continuing with Example 1, assume that the information key levels corresponding to the next-level archiving statistical table of the statistical table to be archived are level one, level two, and level three. Therefore, it can be determined from "RMB, 100 yuan, 50 yuan, 30 yuan, 4 sheets, 3 sheets, 2 sheets, and Xiao Ming" that "RMB, 100 yuan, 50 yuan, 30 yuan, 4 sheets, 3 sheets, and 2 sheets" is the archived information.

[0113] B3: combining the archive information based on the semantic logic of the log to be archived to obtain an archive log, and storing the archive log in an archive statistical table at a lower level of the statistical table to be archived.

[0114] According to the semantic logic of the log to be archived, the previously determined archive information is combined to generate the final archive log. By applying semantic logic, the system can integrate the archive information according to the relevance and meaning between different information to form a complete archive record. Then, this archive log will be stored in the archive statistics table of the next level of the archive statistics table for further archive processing and management. This process helps ensure the integrity and accuracy of the archive data and enables the system to effectively store and retrieve these archive records.

[0115] Example 3, taking Example 2 as an example to continue the explanation, based on the semantic logic of the log to be archived "Xiao Ming has 4 RMB notes of 100 yuan, 3 RMB notes of 50 yuan, and 2 RMB notes of 30 yuan", the archived information "RMB, 100 yuan, 50 yuan, 30 yuan, 4 notes, 3 notes, and 2 notes" are combined to obtain an archived log such as "RMB: 100 yuan × 4, 50 yuan × 3, 30 yuan × 2".

[0116] See also Figure 4 , Figure 4 A flowchart of an implementation method of a log query method provided in an embodiment of the present application, which can be implemented through steps C1-C4:

[0117] C1: Obtain a target log query request; the target log query request includes a target business type and a target time range.

[0118] The system first obtains the user's log query request, which contains two key parameters: the required business type (such as the average response time of all interfaces) and the query time range (for example, the last 34 days).

[0119] C2: Determine one or more target archiving statistical tables from archiving statistical tables at various levels in the storage system based on the target business type.

[0120] From the archive statistics tables at different levels in the storage system, select one or more archive statistics tables that meet the target based on the target business type. Specifically, based on the target business type, select from multiple archive statistics tables those that best support the target business type.

[0121] C3: Based on the target time range, query the archive log corresponding to the target log query request from the one or more target archive statistics tables as the target log.

[0122] The specified time range is used to retrieve log data that matches the time range from the previously selected target archive statistics table. In simple terms, it is to search the target archive table for logs that match the time range to meet the query request requirements.

[0123] For example, if a user wants to view the average response time of all interfaces in the past 34 days, then first he needs to determine one or more target archive statistics tables from all archive statistics tables of the storage system based on the business type of "average response time of the interface". Of the interface information for these 34 days, the summary information for the first 30 days is stored in the first-level archive statistics table, and the summary information for the last 4 days is stored in the second-level archive statistics table. At this time, if you want to calculate the average response time of all interface calls in the past 34 days, you need to use the archive logs corresponding to the target log query requests in the first-level archive statistics table and the second-level archive statistics table as the target logs.

[0124] C4: querying the important log corresponding to the unique identifier of the target log, and supplementing the target log based on the important log to obtain an output log.

[0125] Find the important log corresponding to the target log unique identifier. If the important log is found, use it to supplement the target log to generate the final output log.

[0126] "Supplementing the target log based on the important log to obtain the output log" means integrating the additional, critical detailed information (important log) into the existing main log data (target log) to generate a final log (output log) containing complete and accurate records. This process is to enhance and improve the content of the target log through the important log.

[0127] In a possible implementation, querying the important log corresponding to the unique identifier of the target log, and supplementing the target log based on the important log to obtain an output log includes:

[0128] Check whether there is an important log corresponding to the unique identifier of the target log in the important log area. If so, supplement the target log based on the important log to obtain an output log; otherwise, read the original log corresponding to the unique identifier of the target log from the disk as the important log, and supplement the target log based on the important log to obtain an output log.

[0129] Specifically, first search for the log corresponding to the unique identifier of the target log in the "Important Log Area". If an important log corresponding to the unique identifier of the target log is found in the "Important Log Area", this log is used to supplement the target log and generate an output log. If the corresponding important log is not found, read the original log that matches the unique identifier of the target log from the disk and use it as an important log to supplement the target log, and finally obtain the output log.

[0130] In a possible implementation, the method further includes:

[0131] In response to the important log query request of the output log, if the important log corresponding to the unique identifier of the target log exists in the important log area or the disk, the important log corresponding to the unique identifier of the target log is displayed; otherwise, no response is given.

[0132] When receiving an important log request for querying output logs, if an important log corresponding to the target log unique identifier is found in the important log area or on disk, the log is displayed; if not found, no response is given.

[0133] In a possible implementation, the method further includes:

[0134] In response to the summary log query request of the output log, if the summary log corresponding to the unique identifier of the target log exists in the summary log area, the summary log corresponding to the unique identifier of the target log is displayed; otherwise, no response is given.

[0135] When receiving a summary log request for querying output logs, if a summary log corresponding to the target log unique identifier is found in the summary log area, the log is displayed; if not found, no response is given.

[0136] Based on the description of S101-S103, the embodiment of the present application checks whether there are unarchived original logs in the storage system. If there are unarchived logs, the storage time of these logs is checked. If the storage time of the logs reaches or exceeds the preset archiving time threshold, these original logs are archived and stored in the first-level archiving statistical table. If there are no unarchived logs, the total data volume of the archived logs in the archiving statistical tables at all levels is checked to see whether it has reached the respective thresholds. If the total amount of archived log data in a certain archiving statistical table reaches or exceeds its corresponding threshold, these archived logs are further archived and stored in the next-level archiving statistical table. After the original logs are archived, these original logs need to be deleted from the storage system, and when the archived logs are moved to the next-level archiving statistical table, these logs also need to be deleted from the current archiving statistical table. After each archiving operation, the data volume of the archived logs should be less than the data volume before archiving. The present application can significantly reduce the space occupied by the storage system by storing the original logs in different archiving statistical tables in a hierarchical manner. At the same time, by setting multiple archive statistics tables with different data volume thresholds, it is possible to automatically determine whether further archiving is needed based on the data volume threshold, thereby dynamically adjusting the archive statistics table to avoid excessive accumulation of log data and waste of space. In addition, after archiving, old logs will be cleaned up to reduce data redundancy, simplify storage management, and reduce the difficulty of log processing and analysis.

[0137] Based on the log archiving method provided by the above method embodiment, the embodiment of the present application also provides a log archiving device, which will be described below in conjunction with the accompanying drawings.

[0138] See also Figure 5 As shown in FIG. 1 , this figure is a schematic diagram of the structure of a log archiving device provided in an embodiment of the present application. Figure 5 As shown, the log archiving device includes:

[0139] The first query unit 501 is used to query whether there are unarchived original logs in the storage system; the storage system includes a plurality of archive statistics tables with descending levels; each archive statistics table is respectively set with an archive data volume threshold;

[0140] The second query unit 502 is used to query the total storage time of the original log that has been stored in the storage system if there are unarchived original logs in the storage system. When the storage time of the original log is greater than or equal to the archiving time threshold, the original log is archived to obtain an archived log, and the archived log is stored in a first-level archiving statistical table; the first-level archiving statistical table is the highest-level archiving statistical table in the storage system;

[0141] The third query unit 503 is used to query whether the total amount of archived log data in the archive statistics tables at each level is greater than or equal to the corresponding archived data amount threshold if there is no unarchived original log in the storage system;

[0142] The archiving processing unit 504 is used to further archive all the archive logs in the archiving statistical table whose total amount of archive log data is greater than or equal to the corresponding archive data amount threshold, and store all of them in the next-level archiving statistical table;

[0143] Among them, after the original log is archived to the first-level archive statistics table, the original log in the storage system is deleted; when the archive log in the current archive statistics table is archived to the next-level archive statistics table, the archive log in the current archive statistics table is deleted; the data volume of the log after each archiving is less than the data volume of the log before archiving.

[0144] In a possible implementation manner, the device further includes:

[0145] The first determination unit is used to determine one or more key information in the log to be archived and the information key level corresponding to the key information; the log to be archived is a log in the statistical table to be archived; the statistical table to be archived is an archived statistical table whose total data amount of the archived log is greater than or equal to its corresponding archived data amount threshold; each level of the archived statistical table corresponds to one or more information key levels; as the level decreases, the number of information key levels corresponding to each archived statistical table decreases in sequence;

[0146] A second determining unit is used to determine one or more archived information from the one or more key information based on the information key level corresponding to the next level archived statistical table of the to-be-archived statistical table;

[0147] The archiving information combining unit is used to combine the archiving information based on the semantic logic of the log to be archived to obtain the archiving log, and store the archiving log in the archiving statistical table at the next level of the statistical table to be archived.

[0148] In a possible implementation manner, the device further includes:

[0149] A retrieval unit, configured to retrieve, based on the business information corresponding to the original log, whether a key event in the business information is recorded in the original log; the key event is determined based on a business tag of the business information;

[0150] An important log acquisition unit, for creating a copy of the original log if the original log records a key event in the business information, recording the copy of the original log as an important log, and recording the important log in an important log area in the storage system;

[0151] a summary log acquisition unit, for aggregating the original log into a summary log if the key event in the business information is not recorded in the original log, and recording the summary log in a summary log area in the storage system;

[0152] Among them, the important log, the original log corresponding to the important log and the archived log archived based on the original log of the important log have the same unique identifier; the summary log, the original log of the summary log and the archived log archived based on the original log of the summary log have the same unique identifier.

[0153] In a possible implementation, the archiving statistics table at each level includes multiple types of archiving statistics tables;

[0154] In a possible implementation manner, the device further includes:

[0155] The acquisition setting unit is used to obtain various business types of logs and set multiple archive statistics table levels;

[0156] The archiving statistics table construction unit is used to construct multiple types of archiving statistics tables based on the various business types at each archiving statistics table level.

[0157] In a possible implementation manner, the device further includes:

[0158] The backup unit is used to back up the original log to a disk in a text-sending format.

[0159] In a possible implementation manner, the device further includes:

[0160] A target log query request unit, used to obtain a target log query request; the target log query request includes a target business type and a target time range;

[0161] A third determining unit, configured to determine one or more target archiving statistical tables from archiving statistical tables at all levels in the storage system based on the target business type;

[0162] A fourth query unit, configured to query, based on the target time range, from the one or more target archive statistics tables for an archive log corresponding to the target log query request as a target log;

[0163] The first log supplement unit is used to query the important log corresponding to the unique identifier of the target log, and supplement the target log based on the important log to obtain an output log.

[0164] In a possible implementation, the first log supplement unit specifically includes:

[0165] An important log query unit, used for querying whether there is an important log corresponding to the unique identifier of the target log in the important log area;

[0166] a second log supplement unit, configured to supplement the target log based on the important log to obtain an output log if an important log corresponding to the unique identifier of the target log exists in the important log area;

[0167] The third log supplement unit is used to read the original log corresponding to the unique identifier of the target log from the disk as the important log if the important log corresponding to the unique identifier of the target log does not exist in the important log area, and supplement the target log based on the important log to obtain an output log.

[0168] In a possible implementation manner, the device further includes:

[0169] A first display unit, in response to an important log query request of the output log, is used to display the important log corresponding to the unique identifier of the target log if the important log area or the disk has an important log corresponding to the unique identifier of the target log; otherwise, no response; and / or,

[0170] The second display unit is used to display the summary log corresponding to the unique identifier of the target log in response to the summary log query request of the output log if the summary log area contains the summary log corresponding to the unique identifier of the target log; otherwise, no response.

[0171] In addition, an embodiment of the present application further provides a processor for running a computer program, and when the program is running, the log archiving method as described above is executed.

[0172] The embodiment of the present application first checks whether there are unarchived original logs in the storage system. If there are, the storage duration of these original logs is queried. If the storage time reaches the archiving time threshold, it is archived in the first-level archiving statistical table, and the original log is deleted from the storage system. If there are no unarchived original logs, the total amount of archived logs in the archiving statistical tables at all levels is checked to see whether it reaches the archived data volume threshold. If it reaches, these logs are further archived in the next-level archiving statistical table, and the archived logs are deleted from the current archiving statistical table. After each archiving process, the data volume of the archived logs will be reduced.

[0173] This application effectively reduces the space occupied by the storage system by storing the original logs in different archive statistics tables. Multiple descending levels of archive statistics tables are set, and each table corresponds to a different data volume threshold. This can automatically determine whether the archive logs need further processing, thereby dynamically adjusting the archive statistics table to prevent excessive accumulation of log data in the table and avoid wasting space. In addition, old logs will be cleaned up after each archiving to reduce data redundancy, simplify storage management, and reduce the complexity of log processing and analysis.

[0174] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the unit described as a separate component may or may not be physically separated, and the component prompted as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0175] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A log archiving method, characterized in that: The method comprises: Query whether there are unarchived original logs in the storage system; the storage system includes a plurality of archive statistics tables with descending levels; each archive statistics table is respectively set with an archive data volume threshold; If it exists, query the total storage time of the original log that has been stored in the storage system. When the storage time of the original log is greater than or equal to the archiving time threshold, archive the original log to obtain an archive log, and store the archive log in the first-level archiving statistics table; the first-level archiving statistics table is the highest-level archiving statistics table in the storage system; If it does not exist, query whether the total amount of archived log data in the archived statistical tables at all levels is greater than or equal to the corresponding archived data volume threshold, and further archive all archived logs in the archived statistical tables whose total amount of archived log data is greater than or equal to the corresponding archived data volume threshold, and store them all in the archived statistical tables of the next level; Among them, after the original log is archived to the first-level archive statistics table, the original log in the storage system is deleted; when the archive log in the current archive statistics table is archived to the next-level archive statistics table, the archive log in the current archive statistics table is deleted; the data volume of the log after each archiving is less than the data volume of the log before archiving.

2. The method according to claim 1, characterized in that The log archiving process includes: Determine one or more key information in the log to be archived and the information key level corresponding to the key information; the log to be archived is a log in the statistical table to be archived; the statistical table to be archived is an archive statistical table whose total data amount of the archived log is greater than or equal to its corresponding archive data amount threshold; each level of the archive statistical table corresponds to one or more information key levels; as the level decreases, the number of information key levels corresponding to each archive statistical table decreases in sequence; Based on the information key level corresponding to the next level archiving statistical table of the to-be-archived statistical table, determining one or more archiving information from the one or more key information; The archive information is combined based on the semantic logic of the log to be archived to obtain an archive log, and the archive log is stored in an archive statistical table at a lower level of the statistical table to be archived.

3. The method according to claim 1, characterized in that: The method further comprises: Based on the business information corresponding to the original log, searching whether the original log records a key event in the business information; the key event is determined based on a business tag of the business information; If yes, create a copy of the original log, record the copy of the original log as an important log, and record the important log in an important log area of ​​the storage system; Otherwise, the original logs are summarized into summary logs, and the summary logs are recorded in a summary log area in the storage system; Among them, the important log, the original log corresponding to the important log and the archived log archived based on the original log of the important log have the same unique identifier; the summary log, the original log of the summary log and the archived log archived based on the original log of the summary log have the same unique identifier.

4. The method according to claim 1, characterized in that Each level of the archive statistics table includes multiple types of archive statistics tables; The method further comprises: Get various business types of logs and set multiple archive statistics table levels; At each level of the archive statistics table, multiple types of archive statistics tables are constructed based on the various business types.

5. The method according to claim 1, characterized in that The method comprises: The original log is backed up to a disk in a text-sending format.

6. The method according to claim 3, characterized in that The method comprises: Obtain a target log query request; the target log query request includes a target business type and a target time range; Determine one or more target archiving statistical tables from archiving statistical tables at all levels in the storage system based on the target business type; Based on the target time range, querying the archive log corresponding to the target log query request from the one or more target archive statistics tables as the target log; The important log corresponding to the unique identifier of the target log is queried, and the target log is supplemented based on the important log to obtain an output log.

7. The method according to claim 6, characterized in that The querying of the important log corresponding to the unique identifier of the target log, and supplementing the target log based on the important log to obtain an output log, includes: Querying whether there is an important log corresponding to the unique identifier of the target log in the important log area; If it exists, supplement the target log based on the important log to obtain an output log; If it does not exist, the original log corresponding to the unique identifier of the target log is read from the disk as the important log, and the target log is supplemented based on the important log to obtain an output log.

8. The method according to claim 7, characterized in that The method further comprises: In response to the important log query request of the output log, if the important log corresponding to the unique identifier of the target log exists in the important log area or the disk, then display the important log corresponding to the unique identifier of the target log; otherwise, do not respond; and / or, In response to the summary log query request of the output log, if the summary log corresponding to the unique identifier of the target log exists in the summary log area, the summary log corresponding to the unique identifier of the target log is displayed; otherwise, no response is given.

9. A log archiving device, characterized in that: The device comprises: The first query unit is used to query whether there are unarchived original logs in the storage system; the storage system includes a plurality of archive statistics tables with descending levels; each archive statistics table is respectively set with an archive data volume threshold; The second query unit is used to query the total storage time of the original log that has been stored in the storage system if there are unarchived original logs in the storage system. When the storage time of the original log is greater than or equal to the archiving time threshold, the original log is archived to obtain an archived log, and the archived log is stored in a first-level archiving statistical table; the first-level archiving statistical table is the highest-level archiving statistical table in the storage system; A third query unit, for querying whether the total amount of archived log data in the archived statistical tables at all levels is greater than or equal to the corresponding archived data amount threshold if there is no unarchived original log in the storage system; The archiving processing unit is used to further archive all the archive logs in the archiving statistical table whose total amount of archive log data is greater than or equal to the corresponding archive data amount threshold, and store all of them in the next-level archiving statistical table; Among them, after the original log is archived to the first-level archive statistics table, the original log in the storage system is deleted; when the archive log in the current archive statistics table is archived to the next-level archive statistics table, the archive log in the current archive statistics table is deleted; the data volume of the log after each archiving is less than the data volume of the log before archiving.

10. A processor, characterized in that: Used to run a computer program, which executes the log archiving method according to any one of claims 1 to 8 when the program is run.

Images