A log data processing method, device, equipment and medium for distributed system

Abstract

The present invention discloses a log data processing method, device, equipment and medium of a distributed system, and relates to the field of data processing technology. The method comprises: sending a log index request to each server in the distributed system to obtain incremental log index data returned by the server; obtaining a log query request from a user; the log query request carries a unique tracking number; based on the unique tracking number in the log query request, sending a log extraction request carrying the same unique tracking number as the log query request to the corresponding server to obtain log fragments returned by at least one server; merging the log fragments by timestamp to obtain integrated log data, and sending the integrated log data to the user. The present invention solves the deficiencies of existing log management technologies in terms of resource consumption, deployment and maintenance complexity, query efficiency and scalability, and provides a more efficient, flexible and cost-effective log management solution for distributed systems.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a log data processing method, device, equipment and medium for a distributed system. Background Art

[0002] In software development, distributed systems have become the norm. A typical distributed system consists of multiple services distributed on different physical or virtual machines to jointly complete complex business processes.

[0003] Distributed architecture improves the scalability, availability, and flexibility of the system, but it also brings challenges in monitoring and troubleshooting. Log management of distributed systems is one of the key challenges. Since log data is scattered across multiple nodes, log collection, query, and analysis become complicated. For example, distributed systems use a global trace identifier (GTID) plus a mapped diagnostic context (MDC) to record the processing process in a log file. However, when it is necessary to query all log data for a specific global trace identifier, the process becomes extremely complicated. Users must log in to each server and use commands such as grep to search multiple log files. This process is inefficient.

[0004] Therefore, how to improve the log data processing solution that can solve the log data collection and query problems in a distributed environment is a technical problem that the industry urgently needs to solve. Summary of the invention

[0005] In view of this, the embodiments of the present invention provide a log data processing method, apparatus, device and medium for a distributed system, so as to solve the problem of complex log data collection and query in a distributed environment.

[0006] According to a first aspect, an embodiment of the present invention provides a log data processing method for a distributed system, the method being applied to an application end, the method comprising:

[0007] Sending a log index request to each server in the distributed system to obtain incremental log index data returned by the server; after receiving the log index request, the server is used to obtain the incremental log index data generated between the current log index request and the previous adjacent log index request, and send the incremental log index data to the application end;

[0008] The incremental log index data includes: data generation time, the server to which it belongs, a unique identifier of the log file, a unique tracking number, a log file name, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file;

[0009] Obtaining a log query request from a user; the log query request carries the unique tracking number;

[0010] Based on the unique tracking number in the log query request, a log extraction request carrying the unique tracking number identical to the log query request is sent to the corresponding server to obtain a log fragment returned by at least one of the servers; after receiving the log extraction request, the server is used to filter out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain the log fragment, and send the log fragment to the application;

[0011] The log segments are merged according to timestamps to obtain integrated log data, and the integrated log data is sent to the user.

[0012] In combination with the first aspect, in a first implementation of the first aspect, before sending a log extraction request carrying the same unique tracking number as the log query request to the corresponding server based on the unique tracking number in the log query request, and obtaining a log fragment returned by at least one of the servers, the method further includes:

[0013] Determine whether the unique tracking number in the log query request is recorded and the server corresponding to the incremental log index data containing the unique tracking number.

[0014] In combination with the first aspect, in a second implementation of the first aspect, merging the log segments by timestamp to obtain integrated log data, and sending the integrated log data to a user specifically includes:

[0015] Obtain the timestamp of the log fragment and add an integration tag to each line of the log;

[0016] All log fragments returned by the server are merged in the order of timestamps to obtain the integrated log data, and the integrated log data is sent to the user.

[0017] According to a second aspect, an embodiment of the present invention further provides a log data processing method for a distributed system, the method being applied to a server, the method comprising:

[0018] Receive log indexing requests from the application and traverse all activity logs;

[0019] Obtaining incremental log index data generated between the current log index request and the previous adjacent log index request in the activity log, and sending the incremental log index data to the application end;

[0020] The incremental log index data includes: data generation time, the server to which it belongs, a unique identifier of the log file, a unique tracking number, a log file name, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file;

[0021] Receive a log extraction request carrying the unique tracking number from an application end, filter out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain log fragments, and send the log fragments to the application end; the application end is used to parse the unique tracking number in the user's log query request, determine the server end corresponding to the incremental log index data containing the unique tracking number, and send a log extraction request carrying the unique tracking number that is the same as the log query request to the corresponding server end.

[0022] In combination with the second aspect, in a first implementation of the second aspect, obtaining the incremental log index data generated between the current log index request and the previous adjacent log index request in the activity log specifically includes:

[0023] Determine a first log to be processed in the activity log based on the archiving status of the activity log;

[0024] Determine the end offset position of the last adjacent log index request scanning the log file, and use the first to-be-processed log after the end offset position as incremental log data;

[0025] Find the first and last occurrence positions of each unique tracking number in the incremental log data, and use the first occurrence position of the unique tracking number as the first offset of the unique tracking number and the last occurrence position of the unique tracking number as the second offset of the unique tracking number;

[0026] Generate incremental log index data based on each of the unique tracking numbers, the first offset of the unique tracking number, the second offset, and basic information of the incremental log data;

[0027] The basic information includes: data generation time, the server to which it belongs, a unique identifier of the log file, and the log file name.

[0028] In combination with the second aspect, in a first implementation manner of the second aspect, the receiving a log extraction request carrying the unique tracking number from the application end, filtering out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset of the unique tracking number, and the second offset, obtaining the log fragment, and sending the log fragment to the application end, specifically includes:

[0029] Based on the archiving status of the activity log, extracting a second log to be processed from the activity log;

[0030] Locate the pointer in the second log to be processed to the position corresponding to the first offset of the unique tracking number, and determine the correctness of the second log to be processed based on the position;

[0031] When it is determined that the second log to be processed is correct, the pointer in the second log to be processed is located at the positions corresponding to the first offset and the second offset of the unique tracking number, and all contents containing the unique tracking number are filtered out from the second log to be processed between the two positions to obtain the log fragment.

[0032] According to a third aspect, an embodiment of the present invention further provides a log data processing device for a distributed system, the device being applied to an application end, and the device comprising:

[0033] The log index module is used to send log index requests to each server in the distributed system to obtain incremental log index data returned by the server; after receiving the log index request, the server obtains the incremental log index data generated between the current log index request and the previous adjacent log index request, and sends the incremental log index data to the application;

[0034] The incremental log index data includes: data generation time, the server to which it belongs, a unique identifier of the log file, a unique tracking number, a log file name, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file;

[0035] A log query module, used to obtain a log query request from a user; the log query request carries the unique tracking number;

[0036] A log extraction module, configured to send a log extraction request carrying the same unique tracking number as the log query request to the corresponding server based on the unique tracking number in the log query request, and obtain a log fragment returned by at least one of the servers; after receiving the log extraction request, the server is configured to filter out all contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain the log fragment, and send the log fragment to the application;

[0037] The fragment integration module is used to merge the log fragments according to timestamps to obtain integrated log data, and send the integrated log data to the user.

[0038] According to a fourth aspect, an embodiment of the present invention further provides a log data processing device for a distributed system, the device being applied to a server, and the device comprising:

[0039] The log traversal module is used to receive log index requests from the application side and traverse all activity logs;

[0040] An incremental scanning module, used to obtain incremental log index data generated between the current log index request and the previous adjacent log index request in the activity log, and send the incremental log index data to the application end;

[0041] The incremental log index data includes: data generation time, the server to which it belongs, a unique identifier of the log file, a unique tracking number, a log file name, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file;

[0042] A fragment sending module is used to receive a log extraction request carrying the unique tracking number from an application end, filter out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain log fragments, and send the log fragments to the application end; the application end is used to parse the unique tracking number in the user's log query request, determine the server end corresponding to the incremental log index data containing the unique tracking number, and send a log extraction request carrying the unique tracking number that is the same as the log query request to the corresponding server end.

[0043] According to the fifth aspect, an embodiment of the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the log data processing method of the distributed system as described in any one of the above are implemented.

[0044] According to the sixth aspect, an embodiment of the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the log data processing method of the distributed system as described in any one of the above are implemented.

[0045] According to the seventh aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the log data processing method of the distributed system as described in any one of the above items.

[0046] The log data processing method, device, equipment and medium of the distributed system of the present invention perform incremental log indexing, wherein the incremental log index data contains information of a unique tracking number, a first offset and a second offset, wherein the first offset is the position of the first appearance of the unique tracking number in the log file, and the second offset is the position of the last appearance of the unique tracking number in the log file, that is, the first offset is the starting position of the corresponding unique tracking number, and the second offset is the ending position of the corresponding unique tracking number. By only recording the first and last appearance positions of each global tracking number in the log file, a lightweight log index database is formed, which not only reduces the demand for storage resources, simplifies the deployment and maintenance process, reduces the dependence on professional knowledge, and improves the manageability of the distributed system, but also improves the efficiency and real-time performance of log query, realizes rapid positioning and indexing of log data, and the incremental log indexing and log extraction method improve the scalability of the distributed system, so that the system can more flexibly cope with the increase in the amount of log data, maintain efficient log processing capabilities, solve the deficiencies of existing log management technologies in terms of resource consumption, deployment and maintenance complexity, query efficiency and scalability, and provide a more efficient, flexible and cost-effective log management solution for distributed systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0048] Figure 1 One of the flow diagrams of the log data processing method of the distributed system provided by the present invention applied to the application end is shown;

[0049] Figure 2The second flow chart of the log data processing method of the distributed system provided by the present invention is applied to the application end;

[0050] Figure 3 A schematic diagram showing a flow chart of the log data processing method of a distributed system provided by the present invention applied to a server;

[0051] Figure 4 A schematic diagram showing the structure of the log data processing device of the distributed system provided by the present invention applied to the application end;

[0052] Figure 5 A schematic diagram showing the structure of the log data processing device of the distributed system provided by the present invention applied to the server;

[0053] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0055] In software development, distributed systems have become the norm. A typical distributed system consists of multiple services distributed on different physical or virtual machines to jointly complete complex business processes. Distributed architecture improves the scalability, availability, and flexibility of the system, but it also brings challenges in monitoring and troubleshooting. Log management of distributed systems is one of the key challenges. Since log data is scattered on multiple nodes, log collection, query, and analysis become complicated. For example, a distributed system uses GTID plus MDC to record the processing process to a log file, but when all log data for a specific GTID needs to be queried, the process becomes extremely complicated. Users must log in to each server and use commands such as grep to search multiple log files, which is inefficient.

[0056] Among them, MDC is an auxiliary means of logging, which is used to track and distinguish log records in multi-threaded applications in the log system. MDC can provide a structured context information for logs, making log records in concurrent environments clearer and easier to track. In multi-threaded applications, since multiple threads may run at the same time and perform different tasks, it is difficult to distinguish which log entries belong to which thread or task in the log file. MDC solves this problem by assigning a unique identifier (such as a tracking number, transaction ID, etc.) to each thread or task and attaching this identifier to all log entries generated by the thread. In this way, even in a highly concurrent environment, it is easy to track the log information of a specific thread or task.

[0057] GTID is an identifier used to uniquely identify and track the entire request or transaction process in a distributed system. In scenarios such as microservice architecture, distributed database, and distributed transaction management, GTID has become an important means to ensure data consistency, troubleshooting, performance monitoring, and log tracking.

[0058] In order to solve the above problems of log management, a variety of log management solutions have emerged, such as ELK (Elasticsearch, Logstash, Kibana) stack, Fluentd, etc. These solutions provide real-time collection, storage, search and visualization of log data through centralized log management. However, these solutions usually require a lot of resource investment, especially in large-scale distributed environments, and are more complicated in configuration and maintenance.

[0059] For example, the ELK stack is one of the more popular log management solutions. Elasticsearch is an open source distributed search engine that provides data collection, analysis, and storage functions; Logstash provides log collection, analysis, and filtering, and supports large amounts of data acquisition. It comes with input, filter syntax (grok), and output; Kibana provides a friendly Web interface for log analysis. After the data is stored in ES, it can be added, deleted, modified, and checked on the Kibana page, and data can be interacted with, and various dimensional tables and graphics can be generated. In the ELK stack solution, the workflow is: Logstash is used to collect and preprocess logs, Elasticsearch is used to store and retrieve logs, and Kibana is used to visualize and analyze logs. However, ELK also has problems such as high resource consumption, complex deployment and maintenance, etc.

[0060] Fluentd is another popular open source data collector for unified log layers. Compared with the ELK stack, Fluentd provides a more lightweight solution. It can be used with Elasticsearch and other storage backends to provide flexible log collection and processing capabilities. Although Fluentd has a rich plugin ecosystem, performance tuning and resource management are still issues when processing large amounts of log data.

[0061] It can be seen that existing log management solutions, such as ELK solutions, require a large amount of storage and computing resources to process and store log data, which may lead to excessive costs for resource-limited environments; secondly, complex configuration and maintenance requirements make it difficult for existing log management solutions to be quickly deployed and maintained, especially in large-scale distributed systems, where management and optimization of these systems require expertise and a lot of time; although existing technologies provide powerful log search and analysis capabilities, the query efficiency may be unsatisfactory when processing massive log data, especially for scenarios that require real-time response, the real-time performance is insufficient; finally, as the scale of the system expands, the scalability of existing log management solutions has always been a problem, especially when the amount of log data increases dramatically, and maintaining the performance and stability of the system requires additional work.

[0062] In order to solve the above problems, a log data processing method for a distributed system is provided in this specification, which aims to reduce resource consumption, simplify the deployment and maintenance process of distributed system log data management, and improve the query efficiency and real-time performance of log data. The log data processing method for a distributed system provided in this specification can be applied to electronic devices with log data processing functions. The electronic devices may include notebooks, desktop computers, smart phones, smart wearable devices (virtual reality glasses, smart watches, etc.), tablet computers, etc. Of course, the log data processing method for a distributed system provided in this specification can also be applied to applications running in the above-mentioned electronic devices. For example, the log data processing method for a distributed system can be applied to a browser with a log data processing function, or it can be applied to log data processing software. Figure 1 FIG. 1 is a flow chart of a log data processing method of a distributed system according to an embodiment of the present invention. Figure 1 As shown, the method is applied to the application side, and the method may include the following steps:

[0063] S10. Send a log index request to each server in the distributed system to obtain incremental log index data returned by each server. In this embodiment, the server is each application server such as a server, and the server is used to obtain the incremental log index data generated between the current log index request and the previous adjacent log index request after receiving the above-mentioned log index request, and send the incremental log index data to the application.

[0064] The incremental log index data includes: data generation time, server to which it belongs, unique identifier of the log file, unique tracking number, log file name, first offset and second offset, the first offset is the first appearance position of the unique tracking number in the log file, the second offset is the last appearance position of the unique tracking number in the log file, that is, the first offset is the starting position of the corresponding unique tracking number, and the second offset is the ending position of the corresponding unique tracking number. It should be noted that in this embodiment, the unique tracking number adopts the global tracking number.

[0065] The data generation time is used to record the generation time of the incremental log index data; the server (server ID) to which it belongs identifies the application server to which the incremental log index data belongs; the unique identifier of the log file (log file ID) is used to determine whether it is the same log file in subsequent query extraction; the log file name includes the current log file name and the next log file name. The current log file name indicates the name of the currently active log file, and the next log file name records the name of the archived file if the currently active log file is archived.

[0066] Therefore, the method performs an incremental log indexing service of logs, wherein the log indexing request may be generated periodically or based on a specific trigger condition, for example, when a user actively performs a corresponding indexing, a log indexing request is also generated. In this embodiment, the method for generating the log indexing request is not limited.

[0067] Therefore, the log index request is responsible for collecting the index information of incremental logs regularly or based on specific trigger conditions, and storing this index information in the database. In addition, based on the log index request, the application side provides an external log extraction interface for retrieving and extracting log data according to demand.

[0068] Specifically, the application side can be pre-configured by the user with two interfaces to achieve efficient management and retrieval of log data in the distributed system:

[0069] Incremental log indexing interface: It is specifically used to incrementally collect the global tracking number and its corresponding start and end offset information in the log file, which is then reported to the log indexing service for further processing and indexing.

[0070] Log extraction interface: Based on the global tracking number, the start and end offset information of the global tracking number, etc. (which may also include the file name) provided by the user, the receiving server filters and returns the corresponding log fragments, and then integrates the log fragments to obtain a complete log file for the user to view.

[0071] In this embodiment, after obtaining the incremental log index data returned by the server, the incremental log index data will also be converted into a format that can be stored in the database. The converted data still includes fields such as the data generation time, the server to which it belongs, the unique identifier of the log file, the unique tracking number, the log file name, the first offset, and the second offset. These fields indicate that the tracking number has appeared in the specified service, the specified log file (including the file name and file ID), and the specified position segment.

[0072] Through such a setting, the application side can regularly obtain the incremental log data of each application server in the distributed system and store it in the database for subsequent query and analysis. In addition, for log files that have not been completely scanned within the scheduled time period but have been archived, they can also be completely scanned and processed.

[0073] S20: Obtain a log query request from the user, wherein the log query request carries a unique tracking number specified by the user.

[0074] S40. Based on the unique tracking number in the log query request, a log extraction request carrying the same unique tracking number as the log query request is sent to the corresponding server to obtain a log fragment returned by at least one server. In this embodiment, after receiving the log extraction request, the server is used to filter out all the contents of the unique tracking number in the log file based on the unique tracking number specified by the user, the first offset and the second offset corresponding to the unique tracking number, obtain the log fragment, and send the log fragment to the application.

[0075] S50, merge the log fragments by timestamp to obtain integrated log data, and send the integrated log data to the user. Since the running order of each application server in the distributed system is not necessarily consistent, the log fragments will first be sorted according to the size of the timestamp, and then the corresponding merging and integration operations will be performed. The final integrated log data example is as follows: [

[0077] {"Log file name": $logfile, "Service name": $appname, "Log content": "xxxx", "Log timestamp": $timestamp}, ... ]

[0079] With this setup, the application can effectively extract and merge the log records of the user-specified global tracking number and return them to the user in chronological order. In addition, it can also effectively handle the situation where the file being scanned has been archived when it is actually extracted.

[0080] In this embodiment, the database is a data storage center that records the mapping relationship between the global tracking number and the service node, file name, start and end offset. These index information plays a key role in extracting log data, allowing users to quickly locate and retrieve the required log information.

[0081] The log data processing method of the distributed system of the present invention sends a log index request to each server in the distributed system to obtain incremental log index data returned by each server, that is, to perform incremental log indexing, wherein the incremental log index data contains information of a unique tracking number, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file, that is, the first offset is the starting position of the corresponding unique tracking number, and the second offset is the ending position of the corresponding unique tracking number, and then sends a log extraction request carrying the same unique tracking number as the user's log query request to the corresponding server, obtains log fragments returned by at least one server, and then merges the log fragments according to timestamps to obtain an integrated log data. The system can collect and integrate log data and send it to users. By only recording the first and last occurrence of each global tracking number in the log file, a lightweight log index database is formed. This not only reduces the demand for storage resources, simplifies the deployment and maintenance process, reduces dependence on professional knowledge, and improves the manageability of distributed systems, but also improves the efficiency and real-time performance of log queries, and realizes rapid positioning and indexing of log data. The incremental log index and log extraction method improve the scalability of distributed systems, allowing the system to more flexibly respond to the increase in log data volume and maintain efficient log processing capabilities. It solves the shortcomings of existing log management technologies in resource consumption, deployment and maintenance complexity, query efficiency, and scalability, and provides a more efficient, flexible, and cost-effective log management solution for distributed systems.

[0082] Combine the following Figure 2 The log data processing method of the distributed system of the present invention is described, and the method also includes:

[0083] S30, determine whether the unique tracking number in the log query request is recorded and the server corresponding to the incremental log index data containing the unique tracking number. After receiving the log query extraction request with the global tracking number specified by the user, the application side queries all records of the global tracking number in the database. If no record is found, it can return information such as "no such record" to the user; if a record is found, multiple records of the same application and the same file ID are merged.

[0084] In this embodiment, step S50 specifically includes:

[0085] S51 , obtaining the timestamp of the log segment, and adding an integration tag to each line of the log, wherein the integration tag may include relevant information such as the file name, service name, etc.

[0086] S52: merge all log fragments returned by the server in the order of timestamps to obtain integrated log data, and send the integrated log data to the user. That is, sort the log fragments returned by the server in order of timestamps, and then perform corresponding merging and integration operations.

[0087] In order to solve the above problems, a log data processing method for a distributed system is provided in this specification, which aims to reduce resource consumption, simplify the deployment and maintenance process of distributed system log data management, and improve the query efficiency and real-time performance of log data. The log data processing method for a distributed system provided in this specification can be applied to electronic devices with log data processing functions. The electronic devices may include notebooks, desktop computers, smart phones, smart wearable devices (virtual reality glasses, smart watches, etc.), tablet computers, etc. Of course, the log data processing method for a distributed system provided in this specification can also be applied to applications running in the above-mentioned electronic devices. For example, the log data processing method for a distributed system can be applied to a browser with a log data processing function, or it can be applied to log data processing software. Figure 3 FIG. 1 is a flow chart of a log data processing method of a distributed system according to an embodiment of the present invention. Figure 3 As shown, the method is applied to the server, and the method may include the following steps:

[0088] S60: Receive a log index request from the application end, and traverse all activity logs.

[0089] S70: Obtain incremental log index data generated between the current log index request and the previous adjacent log index request in the activity log, and send the incremental log index data to the application end.

[0090] The incremental log index data includes: data generation time, server to which it belongs, unique identifier of the log file, unique tracking number, log file name, first offset and second offset, the first offset is the first appearance position of the unique tracking number in the log file, the second offset is the last appearance position of the unique tracking number in the log file, that is, the first offset is the starting position of the corresponding unique tracking number, and the second offset is the ending position of the corresponding unique tracking number. It should be noted that in this embodiment, the unique tracking number adopts the global tracking number.

[0091] The data generation time is used to record the generation time of the incremental log index data; the server (server ID) to which it belongs identifies the application server to which the incremental log index data belongs; the unique identifier of the log file (log file ID) is used to determine whether it is the same log file in subsequent query extraction; the log file name includes the current log file name and the next log file name. The current log file name indicates the name of the currently active log file, and the next log file name records the name of the archived file if the currently active log file is archived.

[0092] S80. Receive a log extraction request carrying a unique tracking number from an application end, filter out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain log fragments, and send the log fragments to the application end. In this embodiment, the application end is used to parse the unique tracking number in the user's log query request, determine the server end corresponding to the incremental log index data containing the unique tracking number, and send a log extraction request carrying the same unique tracking number as the log query request to the corresponding server end.

[0093] The log data processing method of the distributed system of the present invention obtains the incremental log index data generated between the current log index request and the adjacent previous log index request in the activity log, and sends the incremental log index data to the application end, that is, performs incremental log indexing, and the incremental log index data contains information about the unique tracking number, the first offset and the second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file, that is, the first offset is the starting position of the corresponding unique tracking number, and the second offset is the ending position of the corresponding unique tracking number. Afterwards, when the user performs a query and extraction of the log file, based on the unique tracking number, the first offset and the second offset of the unique tracking number, all the contents of the unique tracking number in the log file are filtered out. Get the log fragments and send them to the application end. By only recording the first and last occurrences of each global tracking number in the log file, a lightweight log index database is formed. This not only reduces the demand for storage resources, simplifies the deployment and maintenance process, reduces dependence on professional knowledge, and improves the manageability of distributed systems, but also improves the efficiency and real-time performance of log queries, and achieves rapid positioning and indexing of log data. The incremental log index and log extraction method improve the scalability of the distributed system, allowing the system to more flexibly respond to the increase in the amount of log data, maintain efficient log processing capabilities, and solve the shortcomings of existing log management technologies in resource consumption, deployment and maintenance complexity, query efficiency, and scalability, providing a more efficient, flexible, and cost-effective log management solution for distributed systems.

[0094] In this embodiment, step S70 specifically includes:

[0095] S71. Based on the archiving status of the activity log, determine the first log to be processed in the activity log. Check whether the last processed log file has been archived (compare the last_fileId of the last activity log with the fileId of the current activity log). If it has been archived, the files to be processed currently include the archived file (logfile_next) and the current activity file (logfile); otherwise, only the current activity file (logfile) is processed.

[0096] S72. Determine the end offset position of the log file scanned by the last adjacent log index request, and use the first log to be processed after the end offset position as the incremental log data, that is, extract the offset position last_offset of the log file scanned last time, and scan the remaining file content from this position as the incremental scan range.

[0097] S73. Find the first and last occurrence positions of each unique tracking number in the incremental log data, and use the first occurrence position of the unique tracking number as the first offset of the unique tracking number and the last occurrence position of the unique tracking number as the second offset of the unique tracking number.

[0098] S74, based on each unique tracking number, the first offset of the unique tracking number, the second offset and the basic information of the incremental log data, generate incremental log index data. The basic information includes: data generation time, the server to which it belongs, the unique identifier of the log file and the log file name. After obtaining this information, the information is integrated and sent to the application end.

[0099] In this embodiment, step S80 specifically includes:

[0100] S81, extracting a second log to be processed from the activity log based on the archiving status of the activity log;

[0101] S82, locate the position corresponding to the first offset of the unique tracking number from the pointer in the second log to be processed, and determine the correctness of the second log to be processed based on the position. That is, determine whether it is consistent with the expected log file based on the global tracking number. If it is not consistent, it can return information such as "extraction failed: log file does not match, may be manually modified".

[0102] S83, when it is determined that the second log to be processed is correct, locate the first offset and the second offset of the unique tracking number respectively corresponding to the pointer in the second log to be processed, filter out all the contents containing the unique tracking number from the second log to be processed between the two positions, and obtain a log fragment. From the position [offset0, offset1], filter out all the contents that meet the log record rule, that is, all the contents of the tracking number in this log file, and return them to the log index service.

[0103] It should be noted that if the current active log file does not match the log extraction request of the user, it is considered that the log file has been archived and needs to be extracted from the archived log; otherwise, it is directly extracted from the current active log file. In particular, if the file to be processed does not exist, the information such as "Extraction failed: log file has been deleted" can be returned.

[0104] The following describes a log data processing device for a distributed system provided by an embodiment of the present invention. The log data processing device for a distributed system described below and the log data processing method for a distributed system described above can be referred to each other.

[0105] In order to solve the above problems, this specification provides a log data processing device for a distributed system, which aims to reduce resource consumption, simplify the deployment and maintenance process of distributed system log data management, and improve the query efficiency and real-time performance of log data. Figure 4 is a schematic diagram of the structure of a log data processing device of a distributed system according to an embodiment of the present invention. Figure 4 As shown, the device is applied to an application end, and the device may include:

[0106] The tag index module 10 is used to send log index requests to each server in the distributed system and obtain incremental log index data returned by each server. In this embodiment, the server is each application server such as a server, and the server is used to obtain the incremental log index data generated between the current log index request and the previous adjacent log index request after receiving the above-mentioned log index request, and send the incremental log index data to the application.

[0107] The incremental log index data includes: data generation time, server to which it belongs, unique identifier of the log file, unique tracking number, log file name, first offset and second offset, the first offset is the first appearance position of the unique tracking number in the log file, the second offset is the last appearance position of the unique tracking number in the log file, that is, the first offset is the starting position of the corresponding unique tracking number, and the second offset is the ending position of the corresponding unique tracking number. It should be noted that in this embodiment, the unique tracking number adopts the global tracking number.

[0108] The data generation time is used to record the generation time of the incremental log index data; the server (server ID) to which it belongs identifies the application server to which the incremental log index data belongs; the unique identifier of the log file (log file ID) is used to determine whether it is the same log file in subsequent query extraction; the log file name includes the current log file name and the next log file name. The current log file name indicates the name of the currently active log file, and the next log file name records the name of the archived file if the currently active log file is archived.

[0109] Therefore, the method performs an incremental log indexing service of logs, wherein the log indexing request may be generated periodically or based on a specific trigger condition, for example, when a user actively performs a corresponding indexing, a log indexing request is also generated. In this embodiment, the method for generating the log indexing request is not limited.

[0110] Therefore, the log index request is responsible for collecting the index information of incremental logs regularly or based on specific trigger conditions, and storing this index information in the database. In addition, based on the log index request, the application side provides an external log extraction interface for retrieving and extracting log data according to demand.

[0111] Specifically, the application side can be pre-configured by the user with two interfaces to achieve efficient management and retrieval of log data in the distributed system:

[0112] Incremental log indexing interface: It is specifically used to incrementally collect the global tracking number and its corresponding start and end offset information in the log file, which is then reported to the log indexing service for further processing and indexing.

[0113] Log extraction interface: Based on the global tracking number, the start and end offset information of the global tracking number, etc. (which may also include the file name) provided by the user, the receiving server filters and returns the corresponding log fragments, and then integrates the log fragments to obtain a complete log file for the user to view.

[0114] In this embodiment, after obtaining the incremental log index data returned by the server, the incremental log index data will also be converted into a format that can be stored in the database. The converted data still includes fields such as the data generation time, the server to which it belongs, the unique identifier of the log file, the unique tracking number, the log file name, the first offset, and the second offset. These fields indicate that the tracking number has appeared in the specified service, the specified log file (including the file name and file ID), and the specified position segment.

[0115] Through such a setting, the application side can regularly obtain the incremental log data of each application server in the distributed system and store it in the database for subsequent query and analysis. In addition, for log files that have not been completely scanned within the scheduled time period but have been archived, they can also be completely scanned and processed.

[0116] The log query module 20 is used to obtain a log query request from a user, wherein the log query request carries a unique tracking number specified by the user.

[0117] The log extraction module 30 is used to send a log extraction request carrying the same unique tracking number as the log query request to the corresponding server based on the unique tracking number in the log query request, and obtain a log fragment returned by at least one server. In this embodiment, after receiving the log extraction request, the server is used to filter out all the contents of the unique tracking number in the log file based on the unique tracking number specified by the user, the first offset and the second offset corresponding to the unique tracking number, obtain the log fragment, and send the log fragment to the application.

[0118] The entire fragment module 40 is used to merge log fragments by timestamp to obtain integrated log data and send the integrated log data to the user. Since the running order of each application server in the distributed system is not necessarily consistent, the log fragments are first sorted according to the size of the timestamp, and then the corresponding merging and integration operations are performed.

[0119] With this setup, the application can effectively extract and merge the log records of the user-specified global tracking number and return them to the user in chronological order. In addition, it can also effectively handle the situation where the file being scanned has been archived when it is actually extracted.

[0120] In this embodiment, the database is a data storage center that records the mapping relationship between the global tracking number and the service node, file name, start and end offset. These index information plays a key role in extracting log data, allowing users to quickly locate and retrieve the required log information.

[0121] The log data processing device of the distributed system of the present invention sends a log index request to each server in the distributed system to obtain incremental log index data returned by each server, that is, to perform incremental log indexing, wherein the incremental log index data contains information of a unique tracking number, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file, that is, the first offset is the starting position of the corresponding unique tracking number, and the second offset is the ending position of the corresponding unique tracking number, and then sends a log extraction request carrying the same unique tracking number as the user's log query request to the corresponding server, obtains log fragments returned by at least one server, and then merges the log fragments according to timestamps to obtain an integrated log data. The system can collect and integrate log data and send it to users. By only recording the first and last occurrence of each global tracking number in the log file, a lightweight log index database is formed. This not only reduces the demand for storage resources, simplifies the deployment and maintenance process, reduces dependence on professional knowledge, and improves the manageability of distributed systems, but also improves the efficiency and real-time performance of log queries, and realizes rapid positioning and indexing of log data. The incremental log index and log extraction method improve the scalability of distributed systems, allowing the system to more flexibly respond to the increase in log data volume and maintain efficient log processing capabilities. It solves the shortcomings of existing log management technologies in resource consumption, deployment and maintenance complexity, query efficiency, and scalability, and provides a more efficient, flexible, and cost-effective log management solution for distributed systems.

[0122] In order to solve the above problems, this specification provides a log data processing device for a distributed system, which aims to reduce resource consumption, simplify the deployment and maintenance process of distributed system log data management, and improve the query efficiency and real-time performance of log data. Figure 5 is a schematic diagram of the structure of a log data processing device of a distributed system according to an embodiment of the present invention. Figure 5 As shown, the device is applied to the server, and the device may include:

[0123] The log traversal module 50 is used to receive a log index request from the application end and traverse all activity logs.

[0124] The incremental scanning module 60 is used to obtain the incremental log index data generated between the current log index request and the previous adjacent log index request in the activity log, and send the incremental log index data to the application end.

[0125] The incremental log index data includes: data generation time, server to which it belongs, unique identifier of the log file, unique tracking number, log file name, first offset and second offset, the first offset is the first appearance position of the unique tracking number in the log file, the second offset is the last appearance position of the unique tracking number in the log file, that is, the first offset is the starting position of the corresponding unique tracking number, and the second offset is the ending position of the corresponding unique tracking number. It should be noted that in this embodiment, the unique tracking number adopts the global tracking number.

[0126] The data generation time is used to record the generation time of the incremental log index data; the server (server ID) to which it belongs identifies the application server to which the incremental log index data belongs; the unique identifier of the log file (log file ID) is used to determine whether it is the same log file in subsequent query extraction; the log file name includes the current log file name and the next log file name. The current log file name indicates the name of the currently active log file, and the next log file name records the name of the archived file if the currently active log file is archived.

[0127] The fragment sending module 70 is used to receive a log extraction request carrying a unique tracking number from the application end, filter out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain the log fragment, and send the log fragment to the application end. In this embodiment, the application end is used to parse the unique tracking number in the user's log query request, determine the server end corresponding to the incremental log index data containing the unique tracking number, and send a log extraction request carrying the same unique tracking number as the log query request to the corresponding server end.

[0128] The log data processing device of the distributed system of the present invention obtains the incremental log index data generated between the current log index request and the adjacent previous log index request in the activity log, and sends the incremental log index data to the application end, that is, performs incremental log indexing, wherein the incremental log index data contains information of a unique tracking number, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file, that is, the first offset is the starting position of the corresponding unique tracking number, and the second offset is the ending position of the corresponding unique tracking number. Thereafter, when the user performs a query and extraction of the log file, all contents of the unique tracking number in the log file are filtered out based on the unique tracking number, the first offset, and the second offset of the unique tracking number. Get the log fragments and send them to the application end. By only recording the first and last occurrences of each global tracking number in the log file, a lightweight log index database is formed. This not only reduces the demand for storage resources, simplifies the deployment and maintenance process, reduces dependence on professional knowledge, and improves the manageability of distributed systems, but also improves the efficiency and real-time performance of log queries, and achieves rapid positioning and indexing of log data. The incremental log index and log extraction method improve the scalability of the distributed system, allowing the system to more flexibly respond to the increase in the amount of log data, maintain efficient log processing capabilities, and solve the shortcomings of existing log management technologies in resource consumption, deployment and maintenance complexity, query efficiency, and scalability, providing a more efficient, flexible, and cost-effective log management solution for distributed systems.

[0129] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 710 (processor), a communication interface 720 (Communications Interface), a memory 730 (memory) and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic command in the memory 730 to execute the log data processing method of the distributed system, and the method includes:

[0130] Sending a log index request to each server in the distributed system to obtain incremental log index data returned by the server; after receiving the log index request, the server is used to obtain the incremental log index data generated between the current log index request and the previous adjacent log index request, and send the incremental log index data to the application end;

[0131] The incremental log index data includes: data generation time, the server to which it belongs, a unique identifier of the log file, a unique tracking number, a log file name, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file;

[0132] Obtaining a log query request from a user; the log query request carries the unique tracking number;

[0133] Based on the unique tracking number in the log query request, a log extraction request carrying the unique tracking number identical to the log query request is sent to the corresponding server to obtain a log fragment returned by at least one of the servers; after receiving the log extraction request, the server is used to filter out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain the log fragment, and send the log fragment to the application;

[0134] Merging the log fragments according to timestamps to obtain integrated log data, and sending the integrated log data to a user;

[0135] Alternatively, receive a log index request from the application and traverse all activity logs;

[0136] Obtaining incremental log index data generated between the current log index request and the previous adjacent log index request in the activity log, and sending the incremental log index data to the application end;

[0137] The incremental log index data includes: data generation time, the server to which it belongs, a unique identifier of the log file, a unique tracking number, a log file name, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file;

[0138] Receive a log extraction request carrying the unique tracking number from an application end, filter out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain the log fragment, and send the log fragment to the application end; the application end is used to parse the unique tracking number in the user's log query request, determine the server end corresponding to the incremental log index data containing the unique tracking number, and send a log extraction request carrying the unique tracking number that is the same as the log query request to the corresponding server end.

[0139] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memor), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0140] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, the computer can execute the log data processing method of the distributed system provided by the above methods, the method comprising:

[0141] Sending a log index request to each server in the distributed system to obtain incremental log index data returned by the server; after receiving the log index request, the server is used to obtain the incremental log index data generated between the current log index request and the previous adjacent log index request, and send the incremental log index data to the application end;

[0142] The incremental log index data includes: data generation time, the server to which it belongs, a unique identifier of the log file, a unique tracking number, a log file name, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file;

[0143] Obtaining a log query request from a user; the log query request carries the unique tracking number;

[0144] Based on the unique tracking number in the log query request, a log extraction request carrying the unique tracking number identical to the log query request is sent to the corresponding server to obtain a log fragment returned by at least one of the servers; after receiving the log extraction request, the server is used to filter out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain the log fragment, and send the log fragment to the application;

[0145] Merging the log fragments according to timestamps to obtain integrated log data, and sending the integrated log data to a user;

[0146] Alternatively, receive a log index request from the application and traverse all activity logs;

[0147] Obtaining incremental log index data generated between the current log index request and the previous adjacent log index request in the activity log, and sending the incremental log index data to the application end;

[0148] The incremental log index data includes: data generation time, the server to which it belongs, a unique identifier of the log file, a unique tracking number, a log file name, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file;

[0149] Receive a log extraction request carrying the unique tracking number from an application end, filter out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain the log fragment, and send the log fragment to the application end; the application end is used to parse the unique tracking number in the user's log query request, determine the server end corresponding to the incremental log index data containing the unique tracking number, and send a log extraction request carrying the unique tracking number that is the same as the log query request to the corresponding server end.

[0150] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the above-mentioned log data processing method for executing a distributed system, the method comprising:

[0151] Sending a log index request to each server in the distributed system to obtain incremental log index data returned by the server; after receiving the log index request, the server is used to obtain the incremental log index data generated between the current log index request and the previous adjacent log index request, and send the incremental log index data to the application end;

[0152] The incremental log index data includes: data generation time, the server to which it belongs, a unique identifier of the log file, a unique tracking number, a log file name, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file;

[0153] Obtaining a log query request from a user; the log query request carries the unique tracking number;

[0154] Based on the unique tracking number in the log query request, a log extraction request carrying the unique tracking number identical to the log query request is sent to the corresponding server to obtain a log fragment returned by at least one of the servers; after receiving the log extraction request, the server is used to filter out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain the log fragment, and send the log fragment to the application;

[0155] Merging the log fragments according to timestamps to obtain integrated log data, and sending the integrated log data to a user;

[0156] Alternatively, receive a log index request from the application and traverse all activity logs;

[0157] Obtaining incremental log index data generated between the current log index request and the previous adjacent log index request in the activity log, and sending the incremental log index data to the application end;

[0158] The incremental log index data includes: data generation time, the server to which it belongs, a unique identifier of the log file, a unique tracking number, a log file name, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file;

[0159] Receive a log extraction request carrying the unique tracking number from an application end, filter out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain the log fragment, and send the log fragment to the application end; the application end is used to parse the unique tracking number in the user's log query request, determine the server end corresponding to the incremental log index data containing the unique tracking number, and send a log extraction request carrying the unique tracking number that is the same as the log query request to the corresponding server end.

[0160] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0161] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log data processing method for a distributed system, characterized in that: The method is applied to an application end, and the method includes: Sending a log index request to each server in the distributed system to obtain incremental log index data returned by the server; after receiving the log index request, the server is used to obtain the incremental log index data generated between the current log index request and the previous adjacent log index request, and send the incremental log index data to the application end; The incremental log index data includes: data generation time, the server to which it belongs, a unique identifier of the log file, a unique tracking number, a log file name, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file; Obtaining a log query request from a user; the log query request carries the unique tracking number; Based on the unique tracking number in the log query request, a log extraction request carrying the unique tracking number identical to the log query request is sent to the corresponding server to obtain a log fragment returned by at least one of the servers; after receiving the log extraction request, the server is used to filter out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain the log fragment, and send the log fragment to the application; Merging the log fragments according to timestamps to obtain integrated log data, and sending the integrated log data to a user; The merging of the log segments according to timestamps to obtain integrated log data and sending the integrated log data to the user specifically includes: Obtain the timestamp of the log fragment and add an integration tag to each line of the log; All log fragments returned by the server are merged in the order of timestamps to obtain the integrated log data, and the integrated log data is sent to the user.

2. The log data processing method of a distributed system according to claim 1, characterized in that: Before sending a log extraction request carrying the same unique tracking number as the log query request to the corresponding server based on the unique tracking number in the log query request to obtain a log fragment returned by at least one of the servers, the method further includes: Determine whether the unique tracking number in the log query request is recorded and the server corresponding to the incremental log index data containing the unique tracking number.

3. A log data processing method for a distributed system, characterized in that: The method is applied to the server, and the method includes: Receive log indexing requests from the application and traverse all activity logs; Obtaining incremental log index data generated between the current log index request and the previous adjacent log index request in the activity log, and sending the incremental log index data to the application end; The incremental log index data includes: data generation time, the server to which it belongs, a unique identifier of the log file, a unique tracking number, a log file name, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file; Receive a log extraction request carrying the unique tracking number from an application end, filter out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain log fragments, and send the log fragments to the application end; the application end is used to parse the unique tracking number in the user's log query request, determine the server end corresponding to the incremental log index data containing the unique tracking number, and send a log extraction request carrying the unique tracking number that is the same as the log query request to the corresponding server end.

4. The log data processing method of a distributed system according to claim 3, characterized in that: The obtaining of incremental log index data generated between the current log index request and the previous adjacent log index request in the activity log specifically includes: Determine a first log to be processed in the activity log based on the archiving status of the activity log; Determine the end offset position of the last adjacent log index request scanning the log file, and use the first to-be-processed log after the end offset position as incremental log data; Find the first and last occurrence positions of each unique tracking number in the incremental log data, and use the first occurrence position of the unique tracking number as the first offset of the unique tracking number and the last occurrence position of the unique tracking number as the second offset of the unique tracking number; Generate incremental log index data based on each of the unique tracking numbers, the first offset of the unique tracking number, the second offset, and basic information of the incremental log data; The basic information includes: data generation time, the server to which it belongs, a unique identifier of the log file, and the log file name.

5. The log data processing method of a distributed system according to claim 3, characterized in that: The receiving a log extraction request carrying the unique tracking number from the application end, filtering out all contents of the unique tracking number in the log file based on the unique tracking number, the first offset of the unique tracking number, and the second offset to obtain the log fragment, and sending the log fragment to the application end specifically includes: Based on the archiving status of the activity log, extracting a second log to be processed from the activity log; Locate the pointer in the second log to be processed to the position corresponding to the first offset of the unique tracking number, and determine the correctness of the second log to be processed based on the position; When it is determined that the second log to be processed is correct, the pointer in the second log to be processed is located at the positions corresponding to the first offset and the second offset of the unique tracking number, and all contents containing the unique tracking number are filtered out from the second log to be processed between the two positions to obtain the log fragment.

6. A log data processing device for a distributed system, characterized in that: The device is applied to an application end, and the device includes: The log index module is used to send log index requests to each server in the distributed system to obtain incremental log index data returned by the server; after receiving the log index request, the server obtains the incremental log index data generated between the current log index request and the previous adjacent log index request, and sends the incremental log index data to the application; The incremental log index data includes: data generation time, the server to which it belongs, a unique identifier of the log file, a unique tracking number, a log file name, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file; A log query module, used to obtain a log query request from a user; the log query request carries the unique tracking number; A log extraction module, configured to send a log extraction request carrying the same unique tracking number as the log query request to the corresponding server based on the unique tracking number in the log query request, and obtain a log fragment returned by at least one of the servers; after receiving the log extraction request, the server is configured to filter out all contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain the log fragment, and send the log fragment to the application; A fragment integration module, used for merging the log fragments according to timestamps to obtain integrated log data, and sending the integrated log data to the user; The fragment integration module specifically includes: Obtain the timestamp of the log fragment and add an integration tag to each line of the log; All log fragments returned by the server are merged in the order of timestamps to obtain the integrated log data, and the integrated log data is sent to the user.

7. A log data processing device for a distributed system, characterized in that: The device is applied to a server, and includes: The log traversal module is used to receive log index requests from the application side and traverse all activity logs; An incremental scanning module, used to obtain incremental log index data generated between the current log index request and the previous adjacent log index request in the activity log, and send the incremental log index data to the application end; The incremental log index data includes: data generation time, the server to which it belongs, a unique identifier of the log file, a unique tracking number, a log file name, a first offset, and a second offset, wherein the first offset is the position where the unique tracking number first appears in the log file, and the second offset is the position where the unique tracking number last appears in the log file; A fragment sending module is used to receive a log extraction request carrying the unique tracking number from an application end, filter out all the contents of the unique tracking number in the log file based on the unique tracking number, the first offset and the second offset of the unique tracking number, obtain log fragments, and send the log fragments to the application end; the application end is used to parse the unique tracking number in the user's log query request, determine the server end corresponding to the incremental log index data containing the unique tracking number, and send a log extraction request carrying the unique tracking number that is the same as the log query request to the corresponding server end.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the log data processing method of the distributed system as described in any one of claims 1 to 5 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the log data processing method of the distributed system as claimed in any one of claims 1 to 5 are implemented.

Images