Log processing method and apparatus, electronic device, and medium
By identifying the target fields of transaction events to generate log data and storing it in the database, the problems of log chaos and storage space occupation in high-concurrency transaction environments are solved, achieving lightweight and efficient retrieval of the log system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA MOBILE GROUP SICHUAN
- Filing Date
- 2025-01-21
- Publication Date
- 2026-07-21
AI Technical Summary
In high-concurrency transaction environments, the system logs are messy and consume a lot of storage space, making it difficult to achieve lightweight operation.
By identifying target fields for transaction events, such as thread identifier, separator identifier, business transaction number, and application identifier, log data is generated and the target data is stored in the database, reducing unnecessary identifier records.
This approach achieves a lightweight log system, improves log retrieval efficiency, reduces storage space usage, and avoids chaotic log logic relationships.
Smart Images
Figure CN122432235A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a log processing method, apparatus, electronic device, and medium. Background Technology
[0002] In traditional business scenarios, especially in the production environment of the telecommunications industry, subsystems handle massive daily transaction volumes, often reaching tens of millions or more. This high-concurrency transaction environment leads to highly complex system logs, with logs generated by different machines and different services at the same time being intertwined and overlapping, making log organization and analysis extremely difficult.
[0003] To address the issue of disorganized logs, related technologies have added machine, process, and thread identifiers to the logs to categorize logs from different sources and types and to establish contextual relationships. However, while these methods have solved the problem of disorganized log logic to some extent, recording machine, process, and thread identifiers in each log entry consumes a significant amount of additional storage space, making it difficult to achieve a lightweight logging system and increasing the system's burden. Summary of the Invention
[0004] This disclosure provides a log processing method, apparatus, electronic device, and medium to solve problems in related technologies. It determines the target fields corresponding to each transaction event and generates log data corresponding to each transaction event based on these target fields and preset log output rules. Then, the target data in the log data is stored in a database, enabling subsequent quick retrieval of log data corresponding to each transaction event using this target data. This avoids the problem of chaotic log logic relationships while reducing storage space usage by storing only the target data in the database, thus achieving a lightweight log system.
[0005] The first aspect of this disclosure provides a log processing method, which includes: determining a target field corresponding to a transaction event; obtaining log data corresponding to the transaction event based on the target field and preset log output rules; extracting target data from the log data and storing the target data in a database to query the log data corresponding to the transaction event using the target data.
[0006] In some embodiments of this disclosure, the target field includes at least one of the following: the thread identifier, separator identifier, business transaction number, application identifier, and transaction time corresponding to the transaction event. The application identifier consists of the host number and business number corresponding to the transaction event.
[0007] In some embodiments of this disclosure, log data corresponding to a transaction event is obtained based on a target field and a preset log output rule. Then, the method further includes storing the log data into log files corresponding to the transaction event, wherein the log files are generated at preset time intervals.
[0008] In some embodiments of this disclosure, the target data includes the business transaction number, application identifier, and transaction time corresponding to the transaction event. The target data in the log data is stored in a database. Then, the method includes: determining the business transaction number to be retrieved; obtaining the application identifier and transaction time corresponding to the business transaction number from the database based on the business transaction number; establishing a connection with the host corresponding to the transaction event based on the host number and business number in the application identifier, and obtaining the log file corresponding to the transaction event from the host corresponding to the transaction event; determining the log data corresponding to the transaction event from the log file corresponding to the transaction event, and using the log data corresponding to the transaction event as the query result.
[0009] In some embodiments of this disclosure, determining the log data corresponding to a transaction event from the log file corresponding to the transaction event includes: determining the target line where the business transaction number is located in the log file corresponding to the transaction event; extracting the thread identifier of the target line according to a preset log output rule; querying the log data in the log file according to the thread identifier; and when a separator identifier is found, using the log data before the separator identifier as the log data corresponding to the transaction event.
[0010] In some embodiments of this disclosure, determining the business transaction number to be retrieved includes: obtaining an input time range and a first transaction number, the first transaction number corresponding to at least one transaction event; connecting all hosts based on the time range and the first transaction number, and accessing log files within the time range; querying log data in the log files, determining at least one thread identifier in the target line where the first transaction number is located, and storing the at least one thread identifier in a dictionary; sequentially querying log data in the log files based on the at least one thread identifier, and deleting the at least one thread identifier from the dictionary when a separator is found, and using the log data before the separator as the log data corresponding to the first transaction number; determining whether a complete retrieval signal has been received, and if a complete retrieval signal has been received, extracting at least one business transaction number from the log data corresponding to the first transaction number based on the complete retrieval signal; storing the at least one business transaction number in a first list, and using each business transaction number in the first list as the business transaction number to be retrieved.
[0011] In some embodiments of this disclosure, after determining whether a complete retrieval signal has been received, the method includes: if a complete retrieval signal has not been received, then using the log data corresponding to the first transaction number as the query result.
[0012] In some embodiments of this disclosure, log data in a log file is queried sequentially based on at least one thread identifier. When a separator identifier is found, at least one thread identifier is deleted from the dictionary, and the log data before the separator identifier is used as the log data corresponding to the first transaction number. This includes: querying log data in a log file based on a first thread identifier among at least one thread identifier; deleting the first thread identifier from the dictionary when a separator identifier is found, and using the log data before the separator identifier as the first log data; querying log data in a log file based on a second thread identifier among at least one thread identifier; deleting the second thread identifier from the dictionary when a separator identifier is found, and using the log data before the separator identifier as the second log data; and summarizing the first log data and the second log data into the log data corresponding to the first transaction number.
[0013] A second aspect of this disclosure provides a log processing apparatus, the apparatus comprising:
[0014] The determination unit is used to determine the target field corresponding to a transaction event;
[0015] The generation unit is used to obtain log data corresponding to the transaction event based on the target field and preset log output rules;
[0016] The retrieval unit is used to store the target data in the log data into the database, so as to use the target data to query the log data corresponding to the transaction event.
[0017] A third aspect of this disclosure provides an electronic device including: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, performs the method described in the first aspect of this disclosure.
[0018] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the first aspect of this disclosure.
[0019] In summary, the log processing method proposed in this disclosure involves determining the target field corresponding to a transaction event; obtaining log data corresponding to the transaction event based on the target field and preset log output rules; extracting the target data from the log data; and storing the target data in a database to query the log data corresponding to the transaction event. This approach avoids the problem of chaotic log logic relationships, reduces storage space usage by storing only the target data in the database, and achieves a lightweight log system. Furthermore, querying log data using the target data further improves log retrieval efficiency.
[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.
[0022] Figure 1 A flowchart of a log processing method provided in an embodiment of this disclosure;
[0023] Figure 2 A flowchart of a log processing method provided in an embodiment of this disclosure;
[0024] Figure 3 A flowchart of a log processing method provided in an embodiment of this disclosure;
[0025] Figure 4 A specific log processing system provided in this disclosure embodiment;
[0026] Figure 5 A schematic diagram illustrating the business process of a specific log recording module provided in this embodiment of the disclosure;
[0027] Figure 6 A schematic diagram illustrating the business process of a specific log retrieval module provided in this embodiment of the disclosure;
[0028] Figure 7 This is a schematic diagram of the structure of a log processing device provided in an embodiment of the present disclosure;
[0029] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0030] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments are described below with reference to the accompanying drawings.
[0031] In traditional business scenarios, the daily transaction volume of telecommunications industry subsystems in production environments often reaches tens of millions or more. Due to the high transaction concurrency, the system logs are interspersed with logs generated by different machines and different businesses at the same time.
[0032] Currently, logging methods in related technologies must be based on the machine identifier, process identifier, and thread identifier recorded in the logs in order to classify various logs and perform contextualization.
[0033] However, when performing log retrieval, if machine, process, and thread identifiers are not added to the logs, the log logic becomes chaotic, making analysis and organization difficult and hindering readability. Furthermore, existing logging methods record application, process, and thread identifiers in each log entry, and these various business identifiers consume additional storage space, making it difficult to achieve a lightweight logging system.
[0034] Due to the characteristics of distributed systems, logs are generated by multiple system hosts and business processes, resulting in a large volume of log content, complex types, and overlapping generation times. To categorize and manage logs from various hosts and business processes, and to handle the massive, high-concurrency log recording demands, there is an urgent need to find a more efficient and lightweight log processing method.
[0035] To address the aforementioned issues, the log processing method disclosed herein determines the target field corresponding to each transaction event; based on the target field and preset log output rules, it obtains the log data corresponding to each transaction event; it extracts the target data from the log data and stores the target data in a database. This allows users to quickly determine the log file on the host where the log is located by reading the target data stored in the database when querying relevant logs, thus improving retrieval efficiency. Furthermore, this disclosure utilizes the target field to generate the log data corresponding to each transaction event, avoiding the problem of chaotic log logic relationships, reducing storage space usage, and achieving a lightweight log system.
[0036] The log processing method provided in this application will be described in detail below with reference to the accompanying drawings.
[0037] Figure 1 This is a flowchart illustrating a log processing method provided in an embodiment of this disclosure. Figure 1 As shown, the log processing method includes steps 101-103.
[0038] Step 101: Determine the target field corresponding to the transaction event.
[0039] In embodiments of this disclosure, the target field refers to a field that is required when processing the log of transaction events.
[0040] The target fields include at least one of the following: the thread identifier, separator identifier, business transaction number, application identifier, and transaction time corresponding to the transaction event. The application identifier consists of the host number and business number corresponding to the transaction event.
[0041] The target fields can also include log level, timestamp, transaction time, and custom content.
[0042] The transaction number is a unique and non-repeating identifier for a transaction.
[0043] The thread identifier is the thread number (thread_id) in a multi-threaded program, which is the sequence number of the thread currently being used by the application server at the current transaction time. In this disclosure, each transaction occupies one thread, thus obtaining a thread identifier. When the transaction ends, the thread resource is reclaimed, and the thread identifier can be reused by new transactions. It should be noted that the number of threads that the application server in this disclosure can manage generally does not exceed 400, so the value of the thread identifier can be set to be distributed between 1 and 400.
[0044] The delimiter is a customer-defined keyword recorded in the log output to indicate the end of the current transaction event. For example, the delimiter could be `trans_end`. This disclosure can utilize the delimiter to record each log output to distinguish each transaction event.
[0045] The log level refers to the level of the output log, which can include: DEBUG, INFO, ERROR, etc. The timestamp is the time when the current transaction event is output in the current log line, including the date and time (the timestamp in this disclosure can be accurate to the second). The transaction time refers to the time when the current transaction event occurred, including the date and time (the transaction time in this disclosure can also be accurate to the second). In this disclosure, the transaction time can be included in custom content. Custom content in this disclosure refers to the content of the log output, which is determined according to business needs and is not limited in this embodiment.
[0046] The application identifier is the identifier for the business host.
[0047] This disclosure allows for the numbering of hosts (01, 02, ..., n) and the labeling of services (001, 002, ..., m). Based on the host and service corresponding to each transaction event, an application identifier corresponding to the transaction event is generated. That is, the application identifier of each transaction event consists of the host number and the service number.
[0048] This disclosure predetermines target fields for each transaction event and simplifies log recording identification by using only simple thread identifiers and separators to fully obtain the logs for each transaction event.
[0049] Step 102: Based on the target field and the preset log output rules, obtain the log data corresponding to the transaction event.
[0050] In the embodiments of this disclosure, log data for transaction events is generated based on the target fields determined in step 101 and preset log output rules. The preset log output rules may include log output format, log output content, etc.
[0051] Specifically, this disclosure allows setting the log output format, that is, outputting the thread identifier to each line of the log. The log output format can be set as: <log level><timestamp><thread identifier>custom content.
[0052] Specifically, the log output content of this disclosure can be configured such that each transaction event requires a separator identifier; the log input parameters need to be set with a business transaction number (wherein, if the log input parameters of the current transaction event itself include a business transaction number, then it is not necessary to set the business transaction number again); the log output parameters set with the business transaction number and the application identifier, the application identifier integrating two key data points: host and business. This disclosure reduces the recording of machine identifiers and thread identifiers in the line log by designing the application identifier as the log output parameter.
[0053] Once the log data corresponding to a transaction event is obtained, this disclosure allows the log data to be stored in a log file corresponding to the transaction event. The log file is generated at a preset time interval, which can be one hour, but can be set according to actual circumstances and is not limited in this embodiment. Specifically, this disclosure can extract the transaction time corresponding to the current transaction event from the log data value, and store the log data according to the transaction time corresponding to the current transaction event in the log file corresponding to that transaction time.
[0054] Step 103: Extract the target data from the log data and store the target data in the database so as to use the target data to query the log data corresponding to the transaction event.
[0055] In embodiments of this disclosure, the target data includes the business transaction number, application identifier, and transaction time corresponding to the transaction event. This disclosure can extract the target data from the log data generated by the current transaction event and store the target data in a database, enabling the target data to be used as an index for querying log data. For example, when querying log data for a specific transaction event, the relevant log data can be quickly located using the business transaction number.
[0056] In one optional embodiment of this disclosure, taking transaction number 13438121409 as an example for a transaction event (paying phone bill), the steps for storing the target data in the database are as follows:
[0057] Determine the target field of the transaction event, that is, set the input parameters in the target field to include at least the record information containing the business transaction number;
[0058] The system outputs custom content (<log level>, <timestamp>, <thread identifier>) according to preset log output rules, and writes a separator at the end to obtain the log data.
[0059] In this example, the business transaction number is 202405010717333195351382523064, the thread number is 88, and the separator is "End do request". The resulting log data is as follows:
[0060] <info><2024-05-01 07:17:33.363><88>[BossServ][…… <transinfo> <transido> 202405010717333195351382523064< / transido> <transidotime> 20240501071733< / transidotime> < / transinfo> ]
[0061] ……
[0062] <info><2024-05-01 07:17:33.485><88>[BossServ]…… <transidh> 28013002320240501071733279655< / transidh> ]
[0063] <debug><2024-05-01 07:17:33.485> <88> [BossServ]End do request
[0064] After obtaining the above log data, this disclosure can write the target data (i.e., business transaction number, application identifier, and transaction time) in the log data into the database.
[0065] The application identifier is incorporated into the preset number of digits of the log output parameters (the pipeline returned as required by the interface specification). According to the preset number of digits, the application identifier is extracted from the log output parameters in the log data in this disclosure. For example, if the preset number of digits is 4-9, the log output parameters returned in this disclosure are 28013002320240501071733279655, and the application identifier is 130023.
[0066] Table 1 shows the target data stored in the database:
[0067] property value The only flowing water 202405010717333195351382523064 Application Identifier 130023 Trading hours 2024-05-01 07:17:33
[0068] Table 1
[0069] In summary, the log processing method proposed in this disclosure involves: determining the target field corresponding to a transaction event; obtaining log data corresponding to the transaction event based on the target field and preset log output rules; extracting target data from the log data and storing the target data in a database; and then using the target data to query the log data corresponding to the transaction event. This achieves efficient processing and storage of log data for transaction events, enabling rapid subsequent querying and analysis. While avoiding the problem of chaotic log logic relationships, storing only the target data in the database reduces storage space usage, achieving a lightweight log system. Furthermore, using the target data to query log data further improves log retrieval efficiency.
[0070] As one possible implementation, such as Figure 2 The flowchart shown illustrates a log processing method. Based on the above embodiment, the target data in the log data is stored in a database. The subsequent specific process includes the following steps:
[0071] Step 201: Determine the business transaction number to be retrieved, and based on the business transaction number, retrieve the application identifier and transaction time corresponding to the business transaction number from the database.
[0072] In embodiments of this disclosure, the business transaction number to be retrieved can be determined. The business transaction number to be retrieved can be manually entered or determined based on relevant information entered manually.
[0073] In this disclosure, since the database stores the business transaction number, application identifier, and transaction time, this disclosure can retrieve the application identifier and transaction time from the database record based on the input business transaction number.
[0074] For example, taking the target data stored in the database as Table 1, this disclosure can find the application identifier 130023 and the transaction time 2024-05-01 07:17:33 based on the business transaction number 202405010717333195351382523064.
[0075] Step 202: Based on the host number and service number in the application identifier, establish a connection with the host corresponding to the transaction event, and obtain the log file corresponding to the transaction event from the host corresponding to the transaction event.
[0076] In the embodiments of this disclosure, since the application identifier in this disclosure consists of a host number and a service number, this disclosure can connect to the host corresponding to the transaction time through the service number and host number recorded in the application identifier, find the corresponding log file according to the transaction time, and then perform log data query.
[0077] For example, taking application identifier 130023 as an example, this disclosure can identify the host with host number 0023 and business number 13, and find the log file 2024050107 based on the transaction time 2024-05-01 07:17:33.
[0078] Step 203: Determine the log data corresponding to the transaction event from the log file corresponding to the transaction event, and use the log data corresponding to the transaction event as the query result.
[0079] In embodiments of this disclosure, the target line containing the business transaction number in the log file corresponding to the transaction event can be determined; the thread identifier of the target line can be extracted according to preset log output rules; the log data in the log file can be queried based on the thread identifier; and when a separator identifier is found, the log data before the separator identifier is used as the log data corresponding to the transaction event.
[0080] For example, this disclosure can find the thread identifier 88 based on the target line where 202405010717333195351382523064 is located.
[0081] Search based on thread identifier 88 and print the log line containing the thread identifier until a separator is encountered, then exit the search.
[0082] Starting from the target line containing business transaction number 202405010717333195351382523064, log data is queried, including the target line containing thread identifier 88, until the End do request separator is encountered. At this point, the log data is output as the query result.
[0083] In summary, this disclosure directly locates the host and log file containing the log data by using the target data recorded in the database, and can stop immediately after obtaining the complete log data in the log file, reducing the number of traversals and improving retrieval efficiency.
[0084] As one possible implementation, such as Figure 3 The flowchart shown illustrates a log processing method. Based on the above embodiment, the specific process of determining the business transaction number to be retrieved includes the following steps:
[0085] Step 301: Obtain the input time range and the first transaction number, where the first transaction number corresponds to at least one transaction event.
[0086] In the embodiments of this disclosure, if it is necessary to extract multiple log data based on the first transaction number, then the time range for retrieving the log data needs to be provided.
[0087] For example, this disclosure uses the business transaction records of number 13438121409 in April 2024 to query log data. To simplify the log content, it is assumed that the customer only made 3 payment actions in April, which triggered 3 transaction events.
[0088] This disclosure allows for the acquisition of a manually input time range and a first transaction number. The time range input is: 202404, and the first transaction number is 13438121409.
[0089] Step 302: Connect all hosts based on the time range and the first transaction number, and access the log files within the time range.
[0090] In embodiments of this disclosure, after obtaining the time range and the first transaction number, the program of this disclosure concurrently connects to all hosts and accesses the log files contained within the time range.
[0091] Step 303: Query the log data in the log file, determine at least one thread identifier in the target line where the first transaction number is located, and store at least one thread identifier in a dictionary.
[0092] In some embodiments of this disclosure, when querying log data in the log file, the thread identifier of the target line where the first transaction number is located is obtained, and the thread identifier is stored in a dictionary (DIC, Dictionary, is a data structure in Python, which is essentially a collection of key and value and their corresponding relationship).
[0093] For example, the thread identifier written to the dictionary could be: 188, 116, 125.
[0094] Step 304: Based on at least one thread identifier, sequentially query the log data in the log file. When a separator identifier is found, delete at least one thread identifier from the dictionary and use the log data before the separator identifier as the log data corresponding to the first transaction number.
[0095] In some embodiments of this disclosure, log data in a log file is queried based on a first thread identifier among at least one thread identifier. When a separator identifier is found, the first thread identifier is deleted from the dictionary, and the log data before the separator identifier is used as the first log data. Log data in a log file is queried based on a second thread identifier among at least one thread identifier. When a separator identifier is found, the second thread identifier is deleted from the dictionary, and the log data before the separator identifier is used as the second log data. The first log data and the second log data are aggregated into log data corresponding to the first transaction number.
[0096] For example, the thread identifiers written to the dictionary could be 188, 116, and 125. First, 188 is used as the first thread identifier. Based on this first thread identifier, the log data in the log file is queried until a separator is encountered. At the separator, the first thread identifier 188 is deleted from the dictionary. Then, 116 and 125 are used as the second thread identifiers, and the process for the first thread identifier is repeated to obtain the log data corresponding to each thread identifier. The log data corresponding to each thread identifier is then summarized and used as the log data corresponding to the first transaction number, which is then output.
[0097] Step 305: Determine whether a complete retrieval signal has been received. If a complete retrieval signal has been received, extract at least one business transaction number from the log data corresponding to the first transaction number based on the complete retrieval signal.
[0098] Step 306: Store at least one business transaction number in a first list, and use each business transaction number in the first list as the business transaction number to be retrieved.
[0099] In some embodiments of this disclosure, when extracting multiple log data entries based on the first transaction number, it is also necessary to determine the retrieval method. The retrieval method includes two modes: direct log output and complete log output.
[0100] The log direct output mode can be directly output by retrieving the first transaction number. That is, if a complete retrieval signal is not received, the log data corresponding to the first transaction number will be used as the query result and the query result will be output.
[0101] The complete log output mode can be achieved by using steps 201 to 203 to determine the log data corresponding to each transaction event in at least one transaction event based on the determined business transaction number with search function in the first list, and then summarizing the log data corresponding to each transaction event into the query results.
[0102] Specifically, this disclosure can obtain the application identifier and transaction time corresponding to each business transaction number from the database based on each business transaction number in the first list; establish a connection with the host corresponding to each transaction event based on the host number and business number in the application identifier, and obtain the log file corresponding to each transaction event from the host corresponding to each transaction event; and determine the log data corresponding to each transaction event from the log file corresponding to each transaction event.
[0103] Specifically, the process involves determining the log data corresponding to each transaction event from the log file corresponding to each transaction event. This includes: determining the target line containing the first business transaction number in the first log file corresponding to the first transaction event in at least one transaction event; extracting the thread identifier of the target line according to the preset log output rules; querying the log data in the first log file based on the thread identifier; and when a separator identifier is found, using the log data before the separator identifier as the log data corresponding to the first transaction event.
[0104] In summary, the log processing method disclosed herein can determine the business transaction number to be retrieved by inputting the first transaction number, and then determine the log data based on the business transaction number according to the retrieval method, thereby further improving the log query efficiency.
[0105] based on Figures 1 to 3 In this embodiment, the present disclosure provides a specific log processing system, such as Figure 4 As shown.
[0106] In embodiments of this disclosure, reference is made to Figure 4 The log processing system consists of two parts: a log recording module and a log retrieval module. The log recording module is responsible for the collection and storage of log data, including data preparation and log recording. The log retrieval module is responsible for single or batch log data queries, and includes three parts: providing direct output of query results based on the first transaction number (keyword search), extracting business transaction numbers (unique transaction number extraction), and retrieving complete transaction log data (complete transaction log retrieval).
[0107] like Figure 5 As shown, this disclosure provides a schematic diagram of the business process of a specific log recording module. (Refer to...) Figure 5 When logging, this disclosure can determine the target fields for each transaction event. Specifically, the input parameters for each transaction event include the business transaction number (i.e., a unique transaction ID), each transaction event occupies a thread identifier, and the output parameters for each transaction event include the business transaction number, application identifier, and transaction time. Based on these target fields, log data corresponding to each transaction event can be generated. Simultaneously with log data generation, target data—namely, the business transaction number, application identifier, and transaction time—can be extracted from the log data and stored in the database `dp` to enable subsequent querying of log data using the target data.
[0108] like Figure 6 As shown, this disclosure provides a schematic diagram of the business process of a specific log retrieval module. (Refer to...) Figure 6 The log retrieval module includes a keyword retrieval module, a unique transaction log extraction module, and a complete transaction log retrieval module. The log retrieval module supports single or batch queries, with batch retrieval divided into two modes: direct log output and complete log output. Specifically, this disclosure can obtain the input time range and the first transaction number; connect to all hosts based on the time range and the first transaction number, and access log files within the time range; query log data in the log files, determine at least one thread identifier in the target line containing the first transaction number, and store at least one thread identifier in a dictionary; query log data in the log files sequentially based on at least one thread identifier, and when a separator is found, delete at least one thread identifier from the dictionary, and use the log data before the separator as the log data corresponding to the first transaction number; determine whether a complete retrieval signal has been received; if a complete retrieval signal is received, extract at least one business transaction number from the log data corresponding to the first transaction number based on the complete retrieval signal, store at least one business transaction number in a first list (i.e., transaction slip), determine the log data corresponding to each transaction event in at least one transaction event using each business transaction number in the first list, and summarize the log data corresponding to each transaction event into a query result. If a complete retrieval signal is not received, the log data corresponding to the first transaction number will be used as the query result.
[0109] Corresponding to the methods provided in the above embodiments, this disclosure also provides a log processing apparatus. Since the apparatus provided in this disclosure corresponds to the methods provided in the above embodiments, the implementation of the methods is also applicable to the apparatus provided in this embodiment, and will not be described in detail in this embodiment.
[0110] Figure 7 This is a schematic diagram of the structure of a log processing device 700 provided in an embodiment of this disclosure. Figure 7 As shown, the log processing device includes:
[0111] Determining unit 710 is used to determine the target field corresponding to the transaction event;
[0112] The generation unit 720 is used to obtain log data corresponding to the transaction event based on the target field and preset log output rules;
[0113] The retrieval unit 730 is used to store the target data in the log data into the database, so as to use the target data to query the log data corresponding to the transaction event.
[0114] In some embodiments of this disclosure, the target field includes at least one of the following: the thread identifier, separator identifier, business transaction number, application identifier, and transaction time corresponding to the transaction event. The application identifier consists of the host number and business number corresponding to the transaction event.
[0115] In some embodiments of this disclosure, the generation unit 720 is further configured to: obtain log data corresponding to a transaction event based on a target field and a preset log output rule, and then store the log data into a log file corresponding to the transaction event, wherein the log file is generated at a preset time interval.
[0116] In some embodiments of this disclosure, the target data includes the business transaction number, application identifier, and transaction time corresponding to the transaction event. The retrieval unit 730 is further configured to: store the target data in the log data into a database, then determine the business transaction number to be retrieved; based on the business transaction number, retrieve the application identifier and transaction time corresponding to the business transaction number from the database; establish a connection with the host corresponding to the transaction event based on the host number and business number in the application identifier, and retrieve the log file corresponding to the transaction event from the host corresponding to the transaction event; determine the log data corresponding to the transaction event from the log file corresponding to the transaction event, and use the log data corresponding to the transaction event as the query result.
[0117] In some embodiments of this disclosure, the retrieval unit 730 is further configured to: determine the target line in the log file corresponding to the transaction event where the business transaction number is located; extract the thread identifier of the target line according to the preset log output rules; query the log data in the log file according to the thread identifier; and when the separator identifier is found, use the log data before the separator identifier as the log data corresponding to the transaction event.
[0118] In some embodiments of this disclosure, the retrieval unit 730 is further configured to: obtain an input time range and a first transaction number, the first transaction number corresponding to at least one transaction event; connect to all hosts based on the time range and the first transaction number, and access log files within the time range; query log data in the log files, determine at least one thread identifier in the target line where the first transaction number is located, and store the at least one thread identifier in a dictionary; query log data in the log files sequentially based on the at least one thread identifier, and when a separator identifier is found, delete the at least one thread identifier from the dictionary, and use the log data before the separator identifier as the log data corresponding to the first transaction number; determine whether a complete retrieval signal has been received, and if a complete retrieval signal has been received, extract at least one business transaction number from the log data corresponding to the first transaction number based on the complete retrieval signal; store the at least one business transaction number in a first list, and use each business transaction number in the first list as the business transaction number to be retrieved.
[0119] In some embodiments of this disclosure, the retrieval unit 730 is further configured to: if a complete retrieval signal is not received, use the log data corresponding to the first transaction number as the query result.
[0120] In some embodiments of this disclosure, the retrieval unit 730 is further configured to: query log data in a log file based on a first thread identifier among at least one thread identifier; when a separator identifier is found, delete the first thread identifier from the dictionary and use the log data before the separator identifier as the first log data; query log data in a log file based on a second thread identifier among at least one thread identifier; when a separator identifier is found, delete the second thread identifier from the dictionary and use the log data before the separator identifier as the second log data; and summarize the first log data and the second log data into log data corresponding to the first transaction number.
[0121] The methods and apparatus provided in the embodiments of this application have been described above. To implement the functions of the methods provided in the embodiments of this application, the electronic device may include a hardware structure and software modules, and may implement the above functions in the form of a hardware structure, software modules, or a hardware structure plus software modules. One of the above functions may be executed in the form of a hardware structure, software modules, or a hardware structure plus software modules.
[0122] Figure 8 This is a block diagram illustrating an electronic device 800 for implementing the above-described log processing method, according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, computer, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.
[0123] Reference Figure 8 The electronic device 800 may include a communication interface 801, capable of interacting with other devices; a processor 802, connected to the communication interface 801 to interact with other devices, used to execute the methods provided by one or more of the above-described technical solutions when running a computer program; and a memory 803, on which the computer program is stored. Specifically, the specific processing procedure of the processor 802 can refer to the log processing method described in the above embodiments of this disclosure.
[0124] Of course, in practical applications, the various components in electronic device 800 are coupled together through bus system 804. It can be understood that bus system 804 is used to realize the connection and communication between these components. In addition to a data bus, bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 8 The general labeled all buses as Bus System 804.
[0125] The memory 803 in this embodiment is used to store various types of data to support the operation of the electronic device 800. Examples of such data include any computer program used to operate on the electronic device 800.
[0126] The methods disclosed in the embodiments of this application can be applied to processor 802, or implemented by processor 802. Processor 802 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 802 or by instructions in the form of software. The processor 802 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 802 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 803. Processor 802 reads the information in memory 803 and combines its hardware to complete the steps of the aforementioned method.
[0127] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.
[0128] Embodiments of this disclosure also propose a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the log processing method described in the above embodiments of this disclosure.
[0129] Embodiments of this disclosure also provide a computer program product, including a computer program that is executed by a processor using the log processing method described in the above embodiments of this disclosure.
[0130] Embodiments of this disclosure also propose a chip including one or more interface circuits and one or more processors; the interface circuits are used to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory, which, when executed by the processor, cause the electronic device to perform the log processing method described in the above embodiments of this disclosure.
[0131] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0132] In the description of this specification, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with an embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0133] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.
[0134] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (control method), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic device, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0135] It should be understood that various parts of the embodiments of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0136] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0137] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.
[0138] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.< / debug> < / info> < / info>
Claims
1. A log processing method, characterized in that, The method includes: Determine the target field corresponding to the transaction event; Based on the target field and the preset log output rules, the log data corresponding to the transaction event is obtained; Extract the target data from the log data and store the target data in the database so as to use the target data to query the log data corresponding to the transaction event.
2. The method according to claim 1, characterized in that, The target field includes at least one of the following: The transaction event is defined by its thread identifier, separator identifier, business transaction number, application identifier, and transaction time. The application identifier is composed of the host number and business number corresponding to the transaction event.
3. The method according to claim 1, characterized in that, Based on the target field and preset log output rules, the method obtains the log data corresponding to the transaction event. Afterwards, the method further includes: The log data is stored in the log file corresponding to the transaction event, and the log file is generated at a preset time interval.
4. The method according to claim 1, characterized in that, The target data includes the business transaction number, application identifier, and transaction time corresponding to the transaction event. The method involves storing the target data from the log data into a database, followed by: Determine the business transaction number to be retrieved, and based on the business transaction number, retrieve the application identifier and transaction time corresponding to the business transaction number from the database; Based on the host number and service number in the application identifier, a connection is established with the host corresponding to the transaction event, and the log file corresponding to the transaction event is obtained from the host corresponding to the transaction event; The log data corresponding to the transaction event is determined from the log file corresponding to the transaction event, and the log data corresponding to the transaction event is used as the query result.
5. The method according to claim 4, characterized in that, Determining the log data corresponding to the transaction event from the log file corresponding to the transaction event includes: Determine the target line in the log file corresponding to the transaction event where the business transaction number is located; Extract the thread identifier of the target line according to the preset log output rules; The log data in the log file is queried based on the thread identifier. When a separator identifier is found, the log data before the separator identifier is used as the log data corresponding to the transaction event.
6. The method according to claim 4, characterized in that, The process of determining the business transaction number to be retrieved includes: Obtain the input time range and the first transaction number, where the first transaction number corresponds to at least one transaction event; Connect to all hosts based on the time range and the first transaction number, and access log files within the time range; Query the log data in the log file to determine at least one thread identifier in the target line where the first transaction number is located, and store the at least one thread identifier in a dictionary; Based on the at least one thread identifier, the log data in the log file is queried sequentially. When a separator identifier is found, at least one thread identifier is deleted from the dictionary, and the log data before the separator identifier is used as the log data corresponding to the first transaction number. Determine whether a complete retrieval signal has been received. If a complete retrieval signal has been received, extract at least one business transaction number from the log data corresponding to the first transaction number based on the complete retrieval signal. The at least one business transaction number is stored in a first list, and each business transaction number in the first list is used as the business transaction number to be retrieved.
7. The method according to claim 6, characterized in that, After determining whether a complete retrieval signal has been received, the method includes: If no complete retrieval signal is received, the log data corresponding to the first transaction number will be used as the query result.
8. The method according to claim 6, characterized in that, The step of sequentially querying the log data in the log file based on the at least one thread identifier, and upon finding a separator identifier, deleting at least one thread identifier from the dictionary, and using the log data before the separator identifier as the log data corresponding to the first transaction number, includes: Based on the first thread identifier in the at least one thread identifier, query the log data in the log file. When a separator identifier is found, delete the first thread identifier in the dictionary and take the log data before the separator identifier as the first log data. Based on the second thread identifier in the at least one thread identifier, query the log data in the log file. When a separator identifier is found, delete the second thread identifier in the dictionary and use the log data before the separator identifier as the second log data. The first log data and the second log data are aggregated into the log data corresponding to the first transaction number.
9. A log processing device, characterized in that, The device includes: The determination unit is used to determine the target field corresponding to a transaction event; The generation unit is used to obtain the log data corresponding to the transaction event based on the target field and the preset log output rules; The retrieval unit is used to store the target data in the log data into a database, so as to use the target data to query the log data corresponding to the transaction event.
10. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. Wherein, when the processor is used to run the computer program, it performs the method of any one of claims 1 to 8.
11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 8.