A method and device for processing log files

Abstract

The present invention discloses a method and device for processing log files, and relates to the field of computer technology. A specific implementation of the method includes: obtaining a business request; collecting the log file corresponding to the business request according to a preset SDK file, the SDK file including at least one feature point, so that the log file includes at least one feature point node; generating an aggregate root corresponding to the business request according to the business type of the business request; generating a log call link corresponding to the log file according to the aggregate root and the at least one feature point node; and storing the log file according to the log call link. This implementation can accurately locate the log content that needs to be analyzed, improve the efficiency and accuracy of log content query, and then facilitate the analysis and other processing of the log content, so that the post-processing efficiency of the log content is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for processing a log file. Background Art

[0002] Various logs will be continuously recorded during the operation of the system. Efficient management of various logs has an important impact on improving system performance.

[0003] In the prior art, various logs are generally simply stacked and stored. When a system problem needs to be analyzed, these stacked and stored logs need to be fully scanned to query and locate logs related to the problem.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] The query and location of logs based on full file scanning is inefficient and the location accuracy is not high. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a log file processing method and device, which can accurately locate the log content to be analyzed, improve the efficiency and accuracy of log content query, and further facilitate the analysis and other processing of the log content, thereby improving the post-processing efficiency of the log content.

[0007] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for processing a log file is provided.

[0008] A log file processing method according to an embodiment of the present invention includes:

[0009] Get business request;

[0010] According to a preset SDK file, a log file corresponding to the business request is collected, wherein the SDK file includes at least one feature point, so that the log file includes at least one feature point node;

[0011] Generate an aggregate root corresponding to the business request according to the business type of the business request;

[0012] Generate a log call link corresponding to the log file according to the aggregate root and the at least one feature point node;

[0013] The log file is stored according to the log call link.

[0014] Optionally, collecting the log file corresponding to the service request includes:

[0015] Collecting log data corresponding to the at least one feature point node through at least one thread, and generating a session identifier corresponding to each of the threads;

[0016] According to the session identifier, the corresponding log data of the at least one feature point node is written into the log file in sequence.

[0017] Optionally, in the case where there are multiple threads executed asynchronously, generating a session identifier corresponding to each of the threads includes:

[0018] Determine whether the current thread has a parent thread. If so, append the session identifier of the parent thread to the session identifier of the current thread, and use the appended session identifier as the session identifier of the current thread.

[0019] Optionally, the SDK file further includes: a sampling strategy; and storing the log file according to the log call link includes:

[0020] Capture the locally stored log file according to the aggregate root and / or the session identifier through a log capture service;

[0021] In the case that the captured log data meets the sampling strategy, the captured log data is sent to the remote database so that the remote database stores the log data.

[0022] Optionally, the SDK file further includes: an aggregate root instruction; and generating an aggregate root corresponding to the business request according to the business type of the business request includes:

[0023] Determining the service type of the service request according to the type of the acquisition interface of the service request;

[0024] According to the aggregate root instruction, generate the aggregate root corresponding to the business type.

[0025] Optionally, generating an aggregate root corresponding to the business request according to the business type of the business request includes:

[0026] According to the correspondence between multiple pre-configured business types and multiple aggregate roots, the aggregate root corresponding to the business request is generated.

[0027] Optionally, the SDK file is deployed in multiple system instances, and the log files are collected respectively according to the SDK files in the multiple system instances; further comprising:

[0028] The log files collected by the multiple system instances are deduplicated, and the log call link is generated according to the deduplicated log files.

[0029] Optionally, it also includes:

[0030] According to the log call link, the business system that processes the business request is tested.

[0031] Optionally, the feature point includes any one or more of the following: output parameters, input parameters, keywords, pre-configured key point tags and timestamps of a calling function corresponding to the business request.

[0032] To achieve the above objective, according to another aspect of an embodiment of the present invention, a log file processing device is provided.

[0033] A log file processing device according to an embodiment of the present invention includes:

[0034] An acquisition module is used to acquire business requests;

[0035] A collection module, used for collecting a log file corresponding to the business request according to a preset SDK file, wherein the SDK file includes at least one characteristic point, so that the log file includes at least one characteristic point node;

[0036] A first generating module, used to generate an aggregate root corresponding to the business request according to the business type of the business request;

[0037] A second generating module is used to generate a log call link corresponding to the log file according to the aggregate root and the at least one characteristic point node;

[0038] A storage module is used to store the log file according to the log call link.

[0039] To achieve the above objective, according to another aspect of an embodiment of the present invention, a log file processing electronic device is provided.

[0040] An electronic device for processing a log file according to an embodiment of the present invention comprises: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a method for processing a log file according to an embodiment of the present invention.

[0041] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided.

[0042] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, a log file processing method according to an embodiment of the present invention is implemented.

[0043] An embodiment of the above invention has the following advantages or beneficial effects: because the feature points are determined according to the SDK to obtain a log file including feature point nodes; the log call link is determined based on the calibrated or customized aggregate root and feature point node, and then the key log data in the log file is captured according to the log call link to generate a log document and store it in the required remote database technical means, so the technical problems of low efficiency and low positioning accuracy of the existing log query and positioning methods are overcome, so that when collecting log files, the log file including the feature point nodes can be obtained according to the feature points set in the SDK file, and a systematic and orderly log file can be obtained. This makes it easier to process log files later; the log call link is determined based on the aggregate root and feature point nodes, the document structure of the log document is further defined, and the log call link is used to capture and save key log data to generate a log document, which can filter the log data required for subsequent query and analysis processes, eliminate the interference of invalid / duplicate / useless log data, and greatly reduce the storage space required for log documents; in the subsequent process, queries based on keywords of the log call link can accurately locate the log content that needs to be analyzed, improve the efficiency and accuracy of log content queries, and facilitate analysis and other processing of the log content, thereby achieving the technical effect of improving the post-processing efficiency of the log content.

[0044] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0046] Figure 1 is a schematic diagram of main steps of a method for processing a log file according to an embodiment of the present invention;

[0047] Figure 2 is a schematic diagram of the main steps of the log file collection method according to an embodiment of the present invention;

[0048] Figure 3 is a schematic diagram of the main steps of the method for determining an aggregate root according to the first embodiment of the present invention;

[0049] Figure 4 is a schematic diagram of the main steps of a method for determining an aggregate root according to a second embodiment of the present invention;

[0050] Figure 5 is a schematic diagram of the main steps of a method for generating a log document according to an embodiment of the present invention;

[0051] Figure 6is a schematic diagram of main modules of a log file processing device according to an embodiment of the present invention;

[0052] Figure 7 is a schematic diagram of a system for processing log files according to an embodiment of the present invention;

[0053] Figure 8 is a schematic diagram of an application of a log file processing system according to an embodiment of the present invention Figure 1 ;

[0054] Fig. 9 is a schematic diagram of an application of a log file processing system according to an embodiment of the present invention Figure 2 ;

[0055] Fig.10 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0056] Fig.11 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0058] It should be pointed out that the embodiments of the present invention and the technical features therein may be combined with each other without conflict.

[0059] SDK (Software Development Kit): Software Development Kit refers to a collection of development tools used by software engineers to build application software for specific software packages, software frameworks, hardware platforms, operating systems, etc.

[0060] AOP (Aspect Oriented Programming) is a technology that achieves unified maintenance of program functions through pre-compilation and dynamic proxy during runtime. It can dynamically and uniformly add certain specific functions to the program without modifying the source code.

[0061] ES cluster: ES stands for Elasticsearch, which is a search-oriented database.

[0062] Figure 1FIG. 1 is a schematic diagram of the main steps of the method for processing a log file according to an embodiment of the present invention. Figure 1 As shown, the log file processing method of the embodiment of the present invention mainly includes the following steps:

[0063] Step S101: Obtain a service request.

[0064] In the embodiment of the present invention, the business request may be an order query request, an order placement request, a payment request, a virtual resource acquisition request, etc.

[0065] Step S102: According to a preset SDK file, a log file corresponding to the business request is collected, wherein the SDK file includes at least one feature point, so that the log file includes at least one feature point node.

[0066] In an embodiment of the present invention, a business request corresponds to an SDK file, so that the SDK file can process the corresponding business request. The feature points in the SDK file correspond to the feature point nodes in the corresponding log file. The SDK file includes one or more calling functions, and the calling function can be a nested structure.

[0067] In the embodiment of the present invention, the SDK file is deployed in multiple system instances, and log files are collected respectively according to the SDK files in the multiple system instances.

[0068] In the embodiment of the present invention, the SDK file includes multiple types of instructions, such as keyword instructions, key point label instructions, etc. Among them, the SDK file can identify keywords according to the keyword instructions by pre-configuring the keyword instructions in the calling function; the SDK file can identify key point labels according to the key point label instructions by pre-configuring the key point label instructions in the calling function.

[0069] In the embodiment of the present invention, the feature point node of the log file indicates the node in the log file corresponding to the feature point of the SDK file when the SDK file processes the service request by calling the function.

[0070] In the embodiment of the present invention, the SDK file can determine the feature points and the corresponding processing process and results through tools such as AOP, interceptor or standard instructions, so that the collected log files can be standardized.

[0071] In the embodiment of the present invention, Figure 2 As shown, the log file collection method of the present invention includes the following steps:

[0072] Step S201: collect log data corresponding to at least one feature point node through at least one thread, and generate a session identifier corresponding to each thread.

[0073] In the embodiment of the present invention, when a service request is processed through an SDK file, at least one thread is created to collect log data, and a session identifier is generated when the thread is started.

[0074] Step S202, determine whether there are multiple threads executed asynchronously, if yes, go to step S203; if not, go to step S205.

[0075] Step S203, determine whether the current thread has a parent thread, if yes, go to step S204; if no, go to step S205.

[0076] Step S204: append the session identifier of the parent thread to the session identifier of the current thread, and use the appended session identifier as the session identifier of the current thread.

[0077] Step S205: Using the session identifier corresponding to each thread as the session identifier of the current thread.

[0078] Step S206: Write the corresponding log data of at least one feature point node into the log file in sequence according to the session identifier of the current thread.

[0079] In an embodiment of the present invention, through the log file collection method of the present invention, the log file is written in sequence according to the session identifier, which can ensure the integrity of the log file. Compared with the existing scattered log files, the orderliness of the log file can be improved, thereby facilitating the subsequent extraction of log data.

[0080] Step S103: Generate an aggregate root corresponding to the business request according to the business type of the business request.

[0081] In an embodiment of the present invention, the types of aggregate roots include natural aggregate roots and calibrated aggregate roots.

[0082] In an embodiment of the present invention, the SDK file also includes aggregate root instructions. Figure 3 As shown, the method for determining the aggregate root of the first embodiment of the present invention includes the following steps:

[0083] Step S301, determining the service type of the service request according to the type of the acquisition interface of the service request.

[0084] In the embodiment of the present invention, the acquisition interface corresponds to the service type, different acquisition interfaces correspond to different service types, and different service types correspond to different SDK files.

[0085] Step S302: Generate an aggregate root corresponding to the business type according to the aggregate root instruction.

[0086] In an embodiment of the present invention, the aggregate root instruction may be the start and end functions of the SDK file calling function, and accordingly, the aggregate root is a natural aggregate root.

[0087] For example, F1, F2, ..., Fn are natural aggregate roots. Specifically,

[0088] ①F1 – input – [keyword] – time;

[0089] ②Fn – input parameter – [keyword] – time;

[0090] ③Fn – output parameter – [keyword] – time;

[0091] ④F2 – Output parameter – [keyword] – time;

[0092] ⑤F1 – output parameters – [keyword] – time.

[0093] In an embodiment of the present invention, the SDK file also includes a marking instruction. Figure 4 As shown, the method for determining the aggregate root of the second embodiment of the present invention includes the following steps:

[0094] Step S401, determine the correspondence between multiple business types and multiple aggregate roots according to the marking instructions.

[0095] In the embodiment of the present invention, in order to facilitate the subsequent analysis and other processing of the log data, the aggregate root of the log data can be customized according to the log data to be analyzed, so that the log data to be analyzed and other processing can be extracted according to the aggregate root. For example, through the annotation instruction, one or more child nodes in the SDK file call function are marked as aggregate roots, and accordingly, the aggregate root is calibrated as a customized aggregate root.

[0096] Step S402, generating an aggregate root corresponding to the business request according to the correspondence between the pre-configured business type and the aggregate root.

[0097] For example, F2 is a calibration aggregate root. Specifically,

[0098] ⑥F2 – Input parameters – [keywords] – time – [aggregate root label].

[0099] In an embodiment of the present invention, the aggregate roots in the log file are determined through the method for determining the aggregate roots of the present invention, and then the required log data can be extracted according to the aggregate roots for post-processing, which can improve the efficiency and accuracy of locating the log data. In addition, the required log data can be saved according to the aggregate roots to avoid the existing logs being deleted due to time or storage space limitations, and the required log data cannot be obtained, thereby improving the effective utilization of the logs.

[0100] Step S104: Generate a log call link corresponding to the log file based on the aggregate root and the at least one feature point node.

[0101] In the embodiment of the present invention, the log call link integrates the aggregate root and the feature point nodes of the log file based on the call function to determine the document structure of the log data that is convenient for subsequent processing. For example, a document is automatically created based on the aggregate root, and the relevant information of the feature point nodes corresponding to the feature point output parameters, input parameters, keywords, pre-configured key point tags and timestamps in the log file can be combined into the document structure of the log call link.

[0102] Furthermore, after collecting the log files, the log files collected by multiple system instances are deduplicated, and a log call link is generated based on the deduplicated log files.

[0103] Step S105: storing the log file according to the log call link.

[0104] In an embodiment of the present invention, the log data corresponding to the log call link in the locally stored log file is captured through the log capture service according to the log call link. For example, when capturing log data, the output parameters and input parameters of the calling function are captured through the interceptor to obtain the full parameters of the function. Among them, the log capture service can include multiple processes.

[0105] In the embodiment of the present invention, Figure 5 As shown, the log document generation method of the present invention includes the following steps:

[0106] Step S501: Determine the log file and the log data portion corresponding to the aggregate root according to the aggregate root and / or session identifier.

[0107] Step S502: Capture log data corresponding to the log file according to the log call link.

[0108] In an embodiment of the present invention, key log data in the log file can be captured based on the aggregate root in the log file, the feature point output parameters, input parameters, keywords, pre-configured key point tags and feature point nodes corresponding to the timestamp.

[0109] Furthermore, the relevant information corresponding to the keywords in the log call link, as key log data in the log file, can be used as a search field for subsequent processing.

[0110] In the embodiment of the present invention, in the process of capturing log data, the log data can be deduplicated. For example, duplicate exception information can be deduplicated:

[0111] When the SDK file processes a business request, it may not be able to connect to the data source for a certain period of time. The corresponding log file may contain a large amount of duplicate abnormal log data. Deduplication of such a large amount of duplicate abnormal log data can further alleviate the pressure caused by the huge amount of data, facilitate the subsequent processing of log data, and improve the efficiency of subsequent processing.

[0112] Step S503: Combine the captured log data into corresponding log documents.

[0113] Step S504: Send the log document to the remote database so that the remote database stores the log document.

[0114] In an embodiment of the present invention, the log document can be sent to a remote database via network protocol delivery (e.g., TCP / HTTP, etc.), so that the remote database stores the log document. Further, the remote database can be an ES cluster or other databases, and the database can be determined according to the actual application scenario and the amount of data, and the log document can also be sent to different databases.

[0115] In an embodiment of the present invention, through the log document generation method of the present invention, key log data in the log file is captured, and there is no need to save useless or invalid log data. When saving, the log data is preliminarily screened and cleaned in a targeted manner, which greatly saves log storage space while providing the required log data for subsequent processing.

[0116] In the embodiment of the present invention, before sending the log document to the remote database, the log document can be processed by the collection service to obtain a log document set that meets the actual needs, which is convenient for subsequent processing. The collection service can include multiple processes.

[0117] For example, the collection service performs preliminary analysis on the batch received log documents, classifies the log documents into different log document sets and caches them, and sends the log document sets to the remote database at a specified frequency when required for subsequent processing.

[0118] Furthermore, in the process of collecting log documents, the log documents may be deduplicated to exclude duplicate exception information that may exist in the process of different system instances or SDK files processing business requests.

[0119] The collection service can meet actual processing needs, such as different databases storing different types of log documents corresponding to business requests, or log documents need to be stored independently or centrally in different databases, or sample log documents need to be stored independently, etc.

[0120] In the embodiment of the present invention, the crawling service may be implemented through a process of a deployed independent container, or may be implemented through a module embedded in a subsequent processing flow.

[0121] In an embodiment of the present invention, the SDK file also includes a sampling strategy. When capturing log data, the log data that meets the sampling strategy is sent to a remote database so that the remote database stores the log data. Further, the log data that meets the sampling strategy is combined into a corresponding log document and sent to the remote database so that the remote database stores the log document.

[0122] In an embodiment of the present invention, whether the current log document is sent to the remote database is determined according to a sampling strategy. For example, the sampling strategy may be to randomly sample 10% of the log documents of the order service requests on July 10, 2021.

[0123] Through the sampling strategy, you can determine the sampled log documents to meet the needs of the actual environment. For example, you can sample one or more log documents for automatic inspection, analysis, and early warning of the application health status in the production environment.

[0124] In an embodiment of the present invention, after obtaining a log document set through a collection service, the log document set can be sent to a designated remote database through a storage service. Different from the cache in the collection service, the storage service can save all log document sets obtained by the collection service and provide the log document set to the designated remote database as needed. Among them, the storage service can include multiple processes, any process of the collection service can correspond to any process of the storage service, and the storage service can correspond to multiple remote databases.

[0125] Furthermore, the storage service can store the log document set into the remote database in transactional batches according to different index formats and requirements of the remote database.

[0126] Through the storage services of multiple processes and the collection services of multiple processes, a loose coupling relationship between the storage services and the collection services is achieved. Even if one of the processes crashes, it can be adjusted to other processes in time to prevent the problem that the crash of the centralized collection service corresponding to the centralized storage service will cause the system service to crash. When encountering a scale bottleneck, the service can be supported by means of service degradation, business migration, etc. to ensure service capabilities and service quality; and the storage service can correspond to multiple remote databases, or even different log system databases, such as log systems divided by function or log systems of specified business domains, which further improves the applicability of the log file processing method of the present invention and improves the utilization efficiency of the logs.

[0127] In an embodiment of the present invention, the storage service may also send the log document set to a streaming data message system, such as an Apache Kafka message system, so that the message system receives subscribed log data and then analyzes the log data in real time for timely discovery of problems, early warnings, etc. The Kafka message system may analyze the streaming log data through the FLink component to determine the operating status of the business system, thereby timely discovering business system problems, and may also automatically trigger repair services or generate early warning information.

[0128] Through the streaming data message system, log data can be directed through routing, so that problems can be discovered and responded to in a timely manner, further ensuring the stability and security of the system and improving business processing efficiency.

[0129] In an embodiment of the present invention, when the storage service sends a log document set to the ES cluster, the document structure of the log document may be:

[0130] 1.F1 – input – [keyword] – time;

[0131] 2. [keyword OR key point];

[0132] 3.F2 – Input – [keyword] – Time;

[0133] 4. [keyword OR key point];

[0134] 5.F2 – Output parameters – [keyword] – time – [aggregate root annotation];

[0135] 6. [keyword OR key point];

[0136] 7.F1 – Output – [keyword] – Time.

[0137] Among them, F1 and F2 are aggregate roots. An independent MAP (table) is automatically created based on the aggregate root F1. The feature point nodes of the log file can be combined into the document structure of the log call link.

[0138] In an embodiment of the present invention, the correspondence between SDK files, crawling services, collection services, and storage services can be configured through a unified configuration center, and the correspondence between them can be dynamically adjusted through the configuration center. For example, the configuration center can change the collection service corresponding to the crawling service in real time.

[0139] In the embodiment of the present invention, the business system that processes the business request is tested according to the log call link, wherein the test may be query, analysis, and the like.

[0140] In the test environment / grayscale environment, the log data is queried and analyzed according to the specified aggregate root, and / or the specified session identifier, and / or the specified log call link to discover problems, analyze needs, etc., to achieve automatic testing and regression testing in the test environment, or to discover potential risks of the system in the grayscale environment.

[0141] Furthermore, based on the log call link, during the testing phase, it is possible to quickly analyze at different levels whether the functions meet the requirements. Combined with test cases, it is also possible to efficiently and automatically analyze functional expectations.

[0142] In the production environment, you can combine on-site feedback and query the log data according to the log call link of the specified aggregate root to quickly locate the problem point, so as to achieve timely discovery and early warning of problems in the production environment, improve problem solving efficiency and application availability.

[0143] In the embodiment of the present invention, keywords may be used as search fields to search for required log data.

[0144] In the embodiment of the present invention, the user can query the system log document through the function page and customize the log data for analysis, such as the log data of the complete log call link under the specified aggregate root, keyword log data, key point tag log data, etc. The log data under a specific aggregate root can also be pulled through the pull interface to analyze and process it.

[0145] In an embodiment of the present invention, in order to cope with possible sudden demands, an extraction service can be used to build an extraction interface to extract log data, so as to handle sudden demands, automatically test, analyze and verify the extracted log data, so as to detect abnormal situations as early as possible.

[0146] When saving existing log files, they are usually scattered and stacked, and there is no unified saving standard / saving format. On the one hand, the stacked log files lack integrity, and the efficiency of scanning, querying and locating log data is low, and the positioning accuracy is also poor, which makes it difficult to analyze the problems in the log data and get to the core of the problem. Even if the entire program logic is sorted out and the cause of the problem is analyzed, it is usually not accurate, and it takes repeated repairs and tests to solve the problem, or even repeated repairs and tests cannot solve the problem; on the other hand, the stacked log files are difficult to process again, and valuable information cannot be extracted; on the other hand, the stacked log files are disorganized, and monitoring them is out of the question.

[0147] Through the log file processing method of the present invention, the log data in the log file is captured in a targeted manner according to the log call link, and the log data is saved in a decisive manner, which fully guarantees the integrity of the key log data. In addition, the log data is orderly and well-organized, so the log data can be quickly found and located, and then the possible problems can be quickly analyzed and repaired in a targeted manner, which greatly improves the query and positioning efficiency of the log data, accurately locates the possible problems and repairs them in time, and ensures the stability of the system; log data can be accurately extracted according to the aggregate root, session identifier, keyword, key point, etc., and valuable information can be extracted, which further improves the utilization rate of the log data and provides more references for actual production; log data is monitored according to the log call link, so that system problems can be quickly responded to and repaired in time to ensure system security.

[0148] The common way to clean up existing log files is: after the log file reaches the preset file size, the log output middleware will create a new log file and start a new thread to asynchronously delete the original log file; or, deploy a monitoring service on the server to monitor the disk space ratio in real time, and automatically clean up some or all log files when the disk space ratio reaches the preset storage space threshold; or, compress the log files for storage and delete them by time, etc. In the existing log file cleaning, when the system business volume increases suddenly, the number of log files will increase sharply, the system will frequently trigger the log cleaning mechanism, and the storage time of the log files will be shortened. When the storage time is shortened to a shorter time, even if some business scenarios have problems, they cannot be reproduced based on the logs, causing the business scenarios to crash directly.

[0149] Through the log file processing method of the present invention, key log data in the log file is captured and saved, and there is no need to save useless or invalid log data, which greatly improves the utilization rate of storage space. The log data is preliminarily screened and cleaned when saving; the log data in the required log file can be customized and saved according to the required business scenarios, analysis requirements, etc., thereby providing a data basis for subsequent analysis, review, etc., preventing system crashes caused by messy log data cleaning methods, and improving the security and stability of the system.

[0150] In the case of log data loss caused by a sudden increase in business volume, the scale of the remote database and the life cycle of the log document can be determined according to actual needs to ensure the smooth and healthy operation of the log strategy.

[0151] In an embodiment of the present invention, by obtaining a business request; collecting a log file corresponding to the business request according to a preset SDK file, the SDK file includes at least one feature point, so that the log file includes at least one feature point node; generating an aggregate root corresponding to the business request according to the business type of the business request; generating a log call link corresponding to the log file according to the aggregate root and the at least one feature point node; storing the log file according to the log call link, etc., the log data (including the source and storage of the log data) can be saved in a targeted manner, which can greatly facilitate subsequent problem query and analysis, and can provide a reference for the study and analysis of actual business processes, especially sampling monitoring and analysis can be performed according to the log call link, further improving the utilization rate of the log data.

[0152] Figure 6 is a schematic diagram of the main modules of the log file processing device according to an embodiment of the present invention. Figure 6 As shown, the log file processing device 600 of the embodiment of the present invention includes: an acquisition module 601, a collection module 602, a first generation module 603, a second generation module 604 and a storage module 605, wherein:

[0153] The acquisition module 601 is used to acquire a service request.

[0154] In the embodiment of the present invention, the business request acquired by the acquisition module 601 may be an order query request, an order placement request, a payment request, a virtual resource acquisition request, etc.

[0155] The collection module 602 is used to collect the log file corresponding to the business request according to a preset SDK file, wherein the SDK file includes at least one feature point, so that the log file includes at least one feature point node.

[0156] In an embodiment of the present invention, a business request corresponds to an SDK file, so that the SDK file can process the corresponding business request. The feature points in the SDK file correspond to the feature point nodes in the corresponding log file. The SDK file includes one or more calling functions, and the calling function can be a nested structure.

[0157] In the embodiment of the present invention, the characteristic point includes any one or more of the following: the output parameter, input parameter, keyword, pre-configured key point tag and timestamp of the calling function corresponding to the service request, wherein the timestamp is the system time when the SDK file processes the service request.

[0158] In the embodiment of the present invention, the SDK file is deployed in multiple system instances, and the collection module 602 collects log files respectively according to the SDK files in the multiple system instances.

[0159] In the embodiment of the present invention, the SDK file includes multiple types of instructions, such as keyword instructions, key point label instructions, etc. Among them, the SDK file can identify keywords according to the keyword instructions by pre-configuring the keyword instructions in the calling function; the SDK file can identify key point labels according to the key point label instructions by pre-configuring the key point label instructions in the calling function.

[0160] In the embodiment of the present invention, the characteristic point node of the log file indicates the node in the log file corresponding to the characteristic point of the SDK file when the SDK file processes the service request by calling the function.

[0161] In the embodiment of the present invention, the SDK file can determine the feature points and the corresponding processing process and results through tools such as AOP, interceptor or standard instructions, so that the collected log files can be standardized.

[0162] The aggregate root generation module 603 is used to generate the aggregate root corresponding to the business request according to the business type of the business request.

[0163] In the embodiment of the present invention, the aggregate root generation module 603 generates an aggregate root according to the business type of the business request. The types of aggregate roots include natural aggregate roots and calibrated aggregate roots.

[0164] Link generation module 604, used to

[0165] In an embodiment of the present invention, the link generation module 604 generates a log call link corresponding to the log file based on the aggregate root and the feature point node. On the basis of the call function, the log call link integrates the aggregate root and the feature point node of the log file to determine the document structure of the log data that is convenient for subsequent processing. For example, a document is automatically created based on the aggregate root, and the relevant information of the feature point nodes corresponding to the feature point output parameters, input parameters, keywords, pre-configured key point tags and timestamps in the log file can be combined into the document structure of the log call link.

[0166] Furthermore, after the collection module 602 collects the log files, the link generation module 604 deduplicates the log files collected by multiple system instances, and generates a log call link based on the deduplicated log files.

[0167] The storage module 605 is used to store the log file according to the log call link.

[0168] In an embodiment of the present invention, the log data corresponding to the log call link in the locally stored log file is captured through the log capture service according to the log call link. For example, when capturing log data, the output parameters and input parameters of the calling function are captured through the interceptor to obtain the full parameters of the function. Among them, the log capture service can include multiple processes.

[0169] In an embodiment of the present invention, a log file and the portion of log data therein corresponding to the aggregate root are determined based on the aggregate root and / or session identifier; log data corresponding to the log file is captured through a log capture service based on a log call link; the captured log data is combined into a corresponding log document; and the storage module 605 sends the log document to a remote database so that the remote database stores the log document.

[0170] In an embodiment of the present invention, the log file processing device of the present invention can accurately locate the log content to be analyzed, improve the efficiency and accuracy of log content query, and then facilitate the analysis and other processing of the log content, thereby improving the post-processing efficiency of the log content.

[0171] Figure 7 is a schematic diagram of a system for processing log files according to an embodiment of the present invention. Figure 7 As shown, the log file processing system of the embodiment of the present invention includes:

[0172] According to the preset SDK file, the log file corresponding to the business request is collected. The SDK file includes feature points so that the log file includes feature point nodes. The feature points include any one or more of the following: the output parameters, input parameters, keywords, pre-configured key point tags and timestamps of the calling function corresponding to the business request. The feature point nodes of the log file indicate the nodes in the log file corresponding to the feature points of the SDK file when the SDK file processes the business request by calling the function.

[0173] Through the crawling service, according to the log call link, the log data corresponding to the log file is crawled, and the crawled log data is combined into the corresponding log document and sent.

[0174] Through the collection service, log documents are processed to obtain a log document set that meets actual needs.

[0175] The log document set is sent to a specified remote database through the storage service, such as an ES cluster or an Apache Kafka messaging system.

[0176] The ES cluster can provide query services, and in order to cope with possible sudden demands, you can use the extraction service to build an extraction interface to extract the required log data to facilitate handling of sudden demands.

[0177] When sending a set of log documents to the Apache Kafka messaging system, they can be published in the form of messages, and the Kafka messaging system can analyze the streaming log data through the FLink component.

[0178] In the embodiment of the present invention, through the log file processing system of the present invention, the log content to be analyzed can be accurately located, the efficiency and accuracy of log content query can be improved, and the analysis and other processing of the log content can be facilitated, thereby improving the post-processing efficiency of the log content.

[0179] Figure 8 is a schematic diagram of an application of a log file processing system according to an embodiment of the present invention Figure 1 ,like Figure 8 As shown, the application of the log file processing system of the embodiment of the present invention includes: an edge domain and a core domain, wherein:

[0180] The edge domain includes SDK files and crawling services, and the core domain includes collection services, storage services, and ES clusters. The ES cluster can provide a variety of services, such as query and analysis functions, data interfaces, automatic testing, and business analysis services.

[0181] In the embodiment of the present invention, Fig. 9 As shown, the application of the log file processing system of the embodiment of the present invention includes:

[0182] The edge domain corresponds to the target system, and the core domain corresponds to the log system.

[0183] Logs are collected from instances in target systems A, B, and C through SDK files to form log files. Log files are determined based on aggregate roots and / or session identifiers through the crawling service, and log data is captured according to the log call link to form log documents. After deduplication, each crawling service delivers the log document to the collection service in the core domain through the network protocol. After the collection service receives the log document delivered by the edge domain, it deduplications the log document and filters out duplicate log files that may exist between multiple edge domains. The log document is stored in the ES cluster through the storage service. Among them, each instance in the log system can include part or all of the log data of each instance of the target system. For example, instance 1 of the log system includes log data of instance A1 and instance A2 in target system A; instance 2 of the log system includes log data of instance B1, instance B2, and instance B3 in target system B; instance 3 of the log system includes log data of instance C4 in target system C; instance 4 of the log system includes log data of instance C1, instance C2, instance C3, and instance C4 in target system C.

[0184] In the embodiment of the present invention, through the application of the log file processing system of the present invention, the log content to be analyzed can be accurately located, the efficiency and accuracy of log content query can be improved, and the analysis and other processing of the log content can be facilitated, thereby improving the post-processing efficiency of the log content.

[0185] Fig.10 An exemplary system architecture 1000 is shown to which the log file processing method or log file processing device according to the embodiment of the present invention can be applied.

[0186] like Fig.10 As shown, the system architecture 1000 may include terminal devices 1001, 1002, 1003, a network 1004, and a server 1005. The network 1004 is used to provide a medium for communication links between the terminal devices 1001, 1002, 1003 and the server 1005. The network 1004 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0187] Users can use terminal devices 1001, 1002, 1003 to interact with server 1005 through network 1004 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 1001, 1002, 1003, such as business applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0188] The terminal devices 1001 , 1002 , and 1003 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0189] The server 1005 may be a server that provides various services, such as a background management server that provides support for business websites browsed by users using the terminal devices 1001, 1002, and 1003. The background management server may analyze and process the received data such as business processing requests, and feed back the processing results (such as business logs) to the terminal device.

[0190] It should be noted that the log file processing method provided in the embodiment of the present invention is generally executed by the server 1005 , and accordingly, the log file processing device is generally set in the server 1005 .

[0191] It should be understood that Fig.10 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0192] Reference below Fig.11, which shows a schematic diagram of the structure of a computer system 1100 of a terminal device suitable for implementing an embodiment of the present invention. Fig.11 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0193] like Fig.11 As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage part 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the system 1100 are also stored. The CPU 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0194] The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed, so that a computer program read therefrom is installed into the storage section 1108 as needed.

[0195] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the removable medium 1111. When the computer program is executed by the central processing unit (CPU) 1101, the above-mentioned functions defined in the system of the present invention are executed.

[0196] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0197] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0198] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be arranged in a processor. For example, they may be described as: a processor including an acquisition module, a collection module, an aggregate root generation module, a link generation module and a storage module. The names of these modules do not constitute a limitation on the modules themselves in certain cases. For example, the aggregate root generation module may also be described as "a module for generating aggregate roots corresponding to business requests according to the business type of the business requests."

[0199] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining a business request; collecting a log file corresponding to the business request according to a preset SDK file, the SDK file includes at least one feature point, so that the log file includes at least one feature point node; generating an aggregate root corresponding to the business request according to the business type of the business request; generating a log call link corresponding to the log file according to the aggregate root and the at least one feature point node; and storing the log file according to the log call link.

[0200] According to the technical solution of the embodiment of the present invention, the log content to be analyzed can be accurately located, the efficiency and accuracy of log content query can be improved, and the analysis and other processing of the log content can be facilitated, thereby improving the post-processing efficiency of the log content.

[0201] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for processing a log file, characterized in that: include: Get business request; According to a preset SDK file, a log file corresponding to the business request is collected, wherein the SDK file includes at least one feature point and one or more calling functions, so that the log file includes at least one feature point node; the feature point node of the log file indicates a node in the log file corresponding to the feature point of the SDK file when the SDK file processes the business request by calling a function; Generate an aggregate root corresponding to the business request according to the business type of the business request; Generate a log call link corresponding to the log file according to the aggregate root and the at least one feature point node; The log file is stored according to the log call link.

2. The method according to claim 1, characterized in that The collecting of log files corresponding to the business request includes: Collecting log data corresponding to the at least one feature point node through at least one thread, and generating a session identifier corresponding to each of the threads; According to the session identifier, the corresponding log data of the at least one feature point node is written into the log file in sequence.

3. The method according to claim 2, characterized in that In the case where there are multiple threads of asynchronous execution, generating a session identifier corresponding to each of the threads includes: Determine whether the current thread has a parent thread. If so, append the session identifier of the parent thread to the session identifier of the current thread, and use the appended session identifier as the session identifier of the current thread.

4. The method according to claim 2, characterized in that: The SDK file also includes: a sampling strategy; the storing of the log file according to the log call link includes: Capture the locally stored log file according to the aggregate root and / or the session identifier through a log capture service; In the case that the captured log data meets the sampling strategy, the captured log data is sent to a remote database so that the remote database stores the log data.

5. The method according to claim 1, characterized in that The SDK file also includes: an aggregate root instruction; generating an aggregate root corresponding to the business request according to the business type of the business request, including: Determining the service type of the service request according to the type of the acquisition interface of the service request; According to the aggregate root instruction, generate the aggregate root corresponding to the business type.

6. The method according to claim 1, characterized in that The generating, according to the business type of the business request, an aggregate root corresponding to the business request comprises: According to the correspondence between multiple pre-configured business types and multiple aggregate roots, the aggregate root corresponding to the business request is generated.

7. The method according to claim 1, characterized in that The SDK file is deployed in multiple system instances, and the log files are collected respectively according to the SDK files in the multiple system instances; further comprising: The log files collected by the multiple system instances are deduplicated, and the log call link is generated according to the deduplicated log files.

8. The method according to claim 1, characterized in that: Also includes: According to the log call link, the business system that processes the business request is tested.

9. The method according to any one of claims 1 to 8, characterized in that: The feature points include any one or more of the following: output parameters, input parameters, keywords, pre-configured key point tags and timestamps of the calling function corresponding to the business request.

10. A log file processing device, characterized in that: include: An acquisition module is used to obtain business requests; A collection module, used for collecting a log file corresponding to the business request according to a preset SDK file, wherein the SDK file includes at least one feature point and one or more calling functions, so that the log file includes at least one feature point node; the feature point node of the log file indicates a node in the log file corresponding to the feature point of the SDK file when the SDK file processes the business request by calling a function; A first generating module, used to generate an aggregate root corresponding to the business request according to the business type of the business request; A second generating module is used to generate a log call link corresponding to the log file according to the aggregate root and the at least one characteristic point node; A storage module is used to store the log file according to the log call link.

11. An electronic device for processing log files, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images