Distributed log data processing method, device and system

Abstract

The present disclosure relates to the field of data processing technology, and specifically to a distributed log data processing method, a distributed log data processing device, a distributed log data processing system, a storage medium, and a computer system. The method comprises: receiving a compliance check log of a microservice container; wherein the compliance check log comprises a business data identifier; merging the associated compliance check logs according to the business data identifier to obtain a merged log; checking the merged log according to a first inspection rule for verifying the main business logic; when there is an abnormality in the inspection result of the merged log, sending a log extraction instruction to the corresponding microservice container according to the business data identifier, so as to obtain the complete microservice log corresponding to the compliance check log. The present disclosure can save log data in abnormal situations and reduce the difficulty of finding log data.

Description

Background Art

[0002] As cloud computing technology matures, most large-scale application systems or services use distributed microservice clusters to split a single application system or service into several supporting microservices to ensure the stable operation of the application system or service. For example, a microservice cluster based on Docker. A user's business request may contain multiple task links and involve multiple microservice containers. At the same time, the log data of the application system or service is also stored in each microservice container.

[0003] However, when an application system or service has an exception or error, or a user's business request has an error and needs to query the relevant logs, it is necessary to check the logs of multiple microservice containers, resulting in a huge amount of data to be checked, which increases the difficulty of log troubleshooting. In addition, since most microservice container logs have a periodic deletion function, when querying logs, there may be a situation where the logs have been deleted, resulting in the inability to find the logs.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a distributed log data processing method, a distributed log data processing method device, a distributed log data processing system, a storage medium and a computer system, thereby at least to a certain extent overcoming the situation in which log data is difficult to find due to the limitations and defects of related technologies.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, a distributed log data processing method is provided, comprising:

[0008] Receive a compliance check log of a microservice container; wherein the compliance check log includes a business data identifier;

[0009] Merging the associated compliance check logs according to the business data identifier to obtain a merged log;

[0010] Checking the merged log according to a first check rule for checking the main business logic;

[0011] When there is an abnormality in the inspection result of the merged log, a log extraction instruction is sent to the corresponding microservice container according to the business data identifier, so as to obtain the complete microservice log corresponding to the compliance check log.

[0012] In an exemplary embodiment of the present disclosure, the method further includes:

[0013] When it is verified that there is no abnormality in the merged log data, the merged log and / or the compliance check log corresponding to the merged log is deleted.

[0014] In an exemplary embodiment of the present disclosure, the business data identifier includes: a business request identifier for marking a business request, and a microservice call identifier for identifying a call sequence of the microservice container.

[0015] In an exemplary embodiment of the present disclosure, merging the associated compliance check logs according to the business data identifier to obtain a merged log includes:

[0016] Acquire at least one of the compliance check logs according to the service request identifier;

[0017] The compliance check logs are merged in a calling order according to the microservice call identifier.

[0018] In an exemplary embodiment of the present disclosure, when it is determined that the merged log has abnormal data, sending a log extraction instruction to the corresponding microservice container according to the business data identifier to obtain a complete microservice log corresponding to the compliance check log includes:

[0019] When it is determined that the merged log contains abnormal data, generating a log extraction instruction;

[0020] The log extraction instruction is added to the current period log extraction instruction list so as to send the log extraction instruction to the corresponding microservice container at a preset period time.

[0021] In an exemplary embodiment of the present disclosure, the complete microservice log includes:

[0022] The operation log of the microservice container and the compliance check log of the microservice container.

[0023] In an exemplary embodiment of the present disclosure, an interactive interface is provided, and the method further includes:

[0024] Receive log data query request;

[0025] The corresponding complete microservice log is extracted according to the log data query request.

[0026] According to a second aspect of the present disclosure, a distributed log data processing method is provided, comprising:

[0027] Checking the current task according to a second detection rule for checking the business logic to generate a current compliance check log;

[0028] generating a business data identifier in the current compliance check log to obtain the compliance check log;

[0029] The compliance check log is sent to a central log server so that the central log server can check the compliance check log.

[0030] In an exemplary embodiment of the present disclosure, the adding a business data identifier to the current compliance check log to obtain the compliance check log includes:

[0031] Extracting current log data in the current compliance check log according to preset fields;

[0032] The business data identifier is generated and added to the current log data to obtain the compliance check log.

[0033] In an exemplary embodiment of the present disclosure, the method further includes:

[0034] If a log extraction instruction corresponding to the compliance check log is received, extracting the complete microservice log corresponding to the compliance check log according to the log extraction instruction;

[0035] The complete microservice log includes: an operation log of a microservice container and a compliance detection log of the microservice container.

[0036] According to a third aspect of the present disclosure, there is provided a distributed log data processing system, characterized in that it includes: a central log server, and at least one microservice container;

[0037] Wherein, the central log server is used to execute the method in the embodiment of the first aspect above;

[0038] The microservice container is used to execute the method in the above-mentioned second aspect embodiment.

[0039] According to a fourth aspect of the present disclosure, a distributed log data processing device is provided, comprising:

[0040] A log data receiving module, used to receive a compliance check log of a microservice container; wherein the compliance check log includes a business data identifier;

[0041] A log data merging module, used for merging the associated compliance check logs according to the business data identifier to obtain a merged log;

[0042] A log data verification module, used to check the merged log according to a first verification rule for verifying the main logic of the business;

[0043] The complete log data extraction module is used to send a log extraction instruction to the corresponding microservice container according to the business data identifier when there is an abnormality in the inspection result of the merged log, so as to obtain the complete microservice log corresponding to the compliance check log.

[0044] According to a fifth aspect of the present disclosure, a distributed log data processing device is provided, comprising:

[0045] A log data generation module, used to check the current task according to the second detection rule for checking the business logic to generate a current compliance check log;

[0046] An identification loading module, used for adding a business data identification to the current compliance check log to obtain a compliance check log;

[0047] The log data sending module is used to send the compliance check log to the central log server so that the central log server can check the compliance check log.

[0048] According to a sixth aspect of the present disclosure, there is provided a storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned distributed log data processing method is implemented.

[0049] According to a seventh aspect of the present disclosure, there is provided a computer system, comprising:

[0050] Processor; and

[0051] A memory, configured to store executable instructions of the processor;

[0052] Wherein, the processor is configured to perform the above-mentioned distributed log data processing method by executing the executable instructions.

[0053] In a distributed log data processing method provided by an embodiment of the present disclosure, at least one compliance check log from multiple microservice containers is received and merged to obtain a merged log, and the merged log is checked according to preset rules, thereby accurately locating and quickly troubleshooting abnormal data. At the same time, by actively obtaining the complete microservice log of the corresponding microservice container when it is determined that the merged log has an abnormality, the log data is saved, which is convenient for subsequent data processing and reduces the difficulty of finding the log data.

[0054] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0056] Figure 1 A schematic diagram schematically illustrates a distributed log data processing method in an exemplary embodiment of the present disclosure;

[0057] Figure 2 A schematic diagram schematically illustrates a log data query method in an exemplary embodiment of the present disclosure;

[0058] Figure 3 A schematic diagram schematically illustrates another distributed log data processing method in an exemplary embodiment of the present disclosure;

[0059] Figure 4 A schematic diagram schematically shows the composition of a distributed log data processing system in an exemplary embodiment of the present disclosure;

[0060] Figure 5 A schematic diagram schematically illustrates a distributed log data processing device in an exemplary embodiment of the present disclosure;

[0061] Figure 6 A schematic diagram schematically illustrates a distributed log data processing device in an exemplary embodiment of the present disclosure;

[0062] Figure 7 Another schematic diagram schematically illustrates a distributed log data processing device in an exemplary embodiment of the present disclosure;

[0063] Figure 8 Another schematic diagram schematically illustrates a distributed log data processing device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0064] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0065] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0066] This example implementation first provides a distributed log data processing method, which can be applied to the inspection and storage of log data of applications based on microservice clusters. Figure 1 As shown in , the above-mentioned distributed log data processing method may include the following steps:

[0067] S101, receiving a compliance check log of a microservice container; wherein the compliance check log includes a business data identifier;

[0068] S102, merging the associated compliance check logs according to the business data identifier to obtain a merged log;

[0069] S103, checking the merged log according to a first checking rule for checking the main business logic;

[0070] S104, when there is an abnormality in the inspection result of the merged log, a log extraction instruction is sent to the corresponding microservice container according to the business data identifier, so as to obtain the complete microservice log corresponding to the compliance check log.

[0071] The distributed log data processing method provided in this example implementation, on the one hand, receives at least one compliance check log from multiple microservice containers and merges them to obtain a merged log, and checks the merged log according to preset rules, thereby accurately locating and quickly troubleshooting abnormal data. On the other hand, by actively obtaining the complete microservice log of the corresponding microservice container when it is determined that there is an abnormality in the merged log, the log data is saved, which is convenient for subsequent data processing, thereby reducing the difficulty of finding the log data.

[0072] Below, each step of the distributed log data processing method in this example implementation will be described in more detail with reference to the accompanying drawings and embodiments.

[0073] Step S101, receiving a compliance check log of a microservice container; wherein the compliance check log includes a business data identifier.

[0074] In this example implementation, for applications that use distributed and microservice clustering, each microservice needs to be deployed in a container, such as a microservice cluster based on a Docker container, and the application logs are stored in each corresponding Docker container.

[0075] A user's business request may pass through n microservice containers, and each microservice container performs its own tasks. For example, a user generates a hotel order request. In order to identify the user's business request, a business request identifier can be configured for the business request first to mark the user's order request. The business request identifier can be the business request number.

[0076] If the user's hotel order request includes four links, for example: verification link, business 1 link for checking the check-in personnel, MQ message queue link for message transmission, and business 2 link for checking other data, then the four links correspond to four microservices, and each microservice performs its own tasks and corresponding task data. For example, the input parameters, output parameters, execution results (such as an exception and the execution process or task completion), execution time, and the IP address of the corresponding microservice container of the task executed by each microservice container.

[0077] In addition, when executing tasks, each microservice can also configure a microservice call identifier for each microservice according to the calling sequence and calling process of the microservices in order to identify the multiple microservice containers that the user's business request passes through. The microservice container involved in the business request and the specific execution sequence can be determined based on the above business request identifier and microservice call identifier.

[0078] For each microservice container, a second detection rule can be established based on the information of the executed tasks and the task flow, etc., to check the compliance, business logic and specific parameters of the tasks executed by each microservice container, and the inspection results can generate a compliance inspection log and send it to the central log server. In addition, the second detection rule can be adjusted according to the specific content and parameters of the tasks executed by each microservice. For example, different compliance inspection rules can be used in the above-mentioned verification link and business 1 link. The present disclosure does not specifically limit the specific content of the above-mentioned second detection rule.

[0079] Specifically, the compliance check log may include compliance information of normal execution of the task, or non-compliance information due to abnormal exit of the execution process of the task. Specifically, when the microservice container executes the task normally and no abnormality occurs, it can also use the above-mentioned second detection rule to perform compliance detection and generate a corresponding compliance detection log. For example, the compliance detection log may include data such as the input parameters and output parameters corresponding to the microservice and the IP address of the corresponding microservice container.

[0080] If a microservice encounters an exception and ends the task when executing a task, its corresponding compliance check log may include input parameters, output parameters, and execution results, and may also include data such as the reason for the abnormal termination and related parameters, execution time, and the IP address of the corresponding microservice container.

[0081] In addition, in order to facilitate identification of each compliance check log, a business data identifier can also be configured for it, and the business data identifier can include the above-mentioned business request identifier and the microservice call identifier. In addition, the above-mentioned second detection rule can also be executed in the form of a plug-in to monitor each running task of the microservice.

[0082] By setting the central log server to only receive compliance check logs containing important information, the amount of data transmission can be effectively reduced, the data pressure on the central log server can be alleviated, and network congestion can be avoided.

[0083] Step S102: merging the associated compliance check logs according to the business data identifier to obtain a merged log.

[0084] In this example implementation, after receiving multiple compliance check logs from multiple microservice containers, the central log server can merge the compliance check logs according to the business data identifier, and then obtain a complete merged log of each link of the user's business request. Specifically, it includes:

[0085] Step S1021, obtaining at least one compliance check log according to the service request identifier;

[0086] Step S1022: merging the compliance check logs in the order of calling according to the microservice calling identifier.

[0087] Step S103: Check the merged log according to the first check rule for checking the main business logic.

[0088] In this example implementation, a first detection rule can be predefined in the central log server, and the first detection rule can be used to perform a global compliance check on the merged log to check the log information of each microservice process that the user business request passes through. For example, multiple input parameters, output parameters and other data involved in the concatenated merged log can be verified, thereby realizing the investigation and verification of abnormal data in the merged log.

[0089] Step S104, when there is an abnormality in the inspection result of the merged log, a log extraction instruction is sent to the corresponding microservice container according to the business data identifier, so as to obtain the complete microservice log corresponding to the compliance check log.

[0090] In this example implementation, specifically, the above step S104 may include:

[0091] Step S1041, when the inspection result of the merged log is abnormal, generating a log extraction instruction;

[0092] Step S1042: adding the log extraction instruction to the current period log extraction instruction list so as to send the log extraction instruction to the corresponding microservice container at a preset period time.

[0093] In this exemplary embodiment, the abnormality in the above-mentioned inspection result may be an abnormality in the log data, or an abnormality in the logic of the data. For example, the above-mentioned user business request passes through four microservices. For each microservice container, in this embodiment, if the four microservices have passed the compliance check, but when the central log server uses the first inspection rule to perform a global compliance check on the merged log, some global-based anomalies may be detected. For example, the input parameters and output parameters of each microservice container contained in the merged log are globally checked using the first inspection rule. At this time, if abnormal data or logical anomalies are found after inspection, a log extraction instruction can be sent to the corresponding microservice container, so that the microservice container uploads the complete log information to the central log server.

[0094] In another embodiment, among the four microservice containers, if the task of the first microservice container is executed normally, but an exception occurs in the second microservice container and causes the execution process to exit, the central log server can find that the merged log is incomplete and the abnormal data of the second microservice container when checking the merged log. At this time, it can be determined whether the abnormal data of the second microservice container can be integrated and ignored according to the first detection rule mentioned above. If the abnormal data of the second microservice container is ignored based on the global situation, that is, only abnormal data exists but does not affect the operation logic, then the task of the second microservice can be determined to be normal and no other processing is performed on it.

[0095] In another embodiment, among the above four microservice containers, if the first and second microservices are executed normally, but an exception occurs in the third microservice and causes the execution process to exit, when the central log server checks the merged log, it can be found that the merged log is incomplete and an exception is found in the third microservice. When the first detection rule determines that the abnormal data of the third microservice container is not integrable and cannot be ignored, that is, when there is a data anomaly and it may cause a business logic anomaly, a log extraction instruction can be sent to the third microservice container to obtain the complete log information of the third microservice container when executing the task.

[0096] Alternatively, in other exemplary embodiments of the present disclosure, log extraction instructions may be sent to the first, second, and third microservice containers, thereby obtaining complete log information of the three microservice containers when executing the task, so as to facilitate accurate investigation and verification of abnormal data.

[0097] Of course, in other exemplary embodiments of the present disclosure, the above-mentioned complete microservice log may include the operation log of the microservice container when executing the task, and may also include the compliance check log of the microservice container. In addition, the above-mentioned first inspection rule can verify the integrity of the merged log, the validity of each parameter in the merged log, and whether there are any anomalies, and different detection rules can be specifically set according to the content of different business requests and different data types. The present disclosure does not specifically limit the content of the first inspection rule.

[0098] Furthermore, the periodic sending of log extraction instructions can be set, for example, the sending period of the instructions can be set to 5 minutes or 10 minutes. After receiving the instructions, the microservice container can search for local log information and send it to the central log server, so that the central log server can obtain multiple log data from the microservice container at one time. By setting the instructions to be sent to each microservice container at a certain period, which requires it to upload local operation log information, the number of data transmissions can be reduced, the data pressure of the microservice container and the central log server can be reduced, and the management of log data can be facilitated.

[0099] Based on the above content, in other exemplary embodiments of the present disclosure, after receiving the complete microservice logs uploaded by each microservice container, the central log server can merge the operation logs and / or compliance check logs of the microservice container in sequence according to the microservice call identifier, and then save the complete log data corresponding to the user business request with abnormalities in the central log server.

[0100] Based on the above content, in this exemplary embodiment, the above method may further include:

[0101] Step S105: when it is verified that there is no abnormality in the merged log data, the merged log and / or the compliance check log corresponding to the merged log is deleted.

[0102] By setting the central log server to extract the complete microservice log to the microservice container when an abnormality is detected in the merged log, and deleting the compliance check log and the merged log when no abnormality is found in the merged log check, important log data can be saved effectively and timely, and the amount of log data stored can be reduced.

[0103] In addition, in other exemplary embodiments of the present disclosure, an interactive interface may be provided for the central log server, and a user may input log information to be queried or verified in the interactive interface. Figure 2 As shown, the above method may further include:

[0104] Step S201, receiving a log data query request;

[0105] Step S202: extracting the corresponding complete microservice log according to the log data query request.

[0106] In this exemplary embodiment, the user can submit relevant information of the business request in the interactive interface, such as user ID, order number, time, product information, etc. The uploaded logs or merged complete log data are searched locally on the central log server through the relevant information of the business request. This reduces the number of log retrievals and eliminates the need to query log information in each microservice container separately, facilitating unified retrieval of log data and rapid troubleshooting of abnormal data.

[0107] For example, when a technician receives an online troubleshooting task, he or she can search for the corresponding log data in the central log server based on the relevant information submitted by the user, thereby obtaining the merged log corresponding to the user's business request. Of course, the operation log or / and compliance check log of the microservice container corresponding to the user's business request can also be found. This makes it easier to troubleshoot abnormal or erroneous data in the log data.

[0108] Furthermore, this example embodiment also provides another distributed log data processing method, referring to Figure 3 As shown, including:

[0109] Step S301, checking the current task according to the second detection rule for checking the business logic to generate a current compliance check log;

[0110] Step S302, generating a business data identifier in the current compliance check log to obtain the compliance check log;

[0111] Step S303: Send the compliance check log to a central log server so that the central log server can check the compliance check log.

[0112] In this exemplary embodiment, for the microservice container, when executing the task corresponding to the user business request, a local compliance check can be performed on the current task according to the preset second detection rule, and a business data identifier is configured for the generated current compliance check log, thereby generating a compliance check log of the current task by the microservice container. The compliance check log is sent to the central log server.

[0113] For example, the compliance check log may include compliance information when the task is executed normally, such as the input parameters, output parameters of the task, and the IP address of the corresponding microservice container. Alternatively, the compliance check log may also include non-compliance information caused by the abnormal exit of the task from the execution process, such as input parameters, output parameters, execution results, execution time, related abnormal data, and the IP address of the corresponding microservice container. In addition, the second detection rule may also be executed in the microservice container in the form of a plug-in to monitor each running task of the microservice.

[0114] Further, in this exemplary embodiment, the above step S302 of adding a business data identifier to the current compliance check log to obtain the compliance check log also includes:

[0115] Step S3021, extracting current log data from the current compliance check log according to preset fields;

[0116] Step S3022: Generate the business data identifier and add it to the current log data to obtain the compliance check log.

[0117] For compliance check logs, fixed parameters or important information can also be extracted according to preset fields or preset rules to generate summaries, so that the information in the compliance check logs is as small and effective as possible, avoiding submitting all local log information to the central log server, thereby avoiding network congestion.

[0118] Based on the above content, in other exemplary embodiments of the present disclosure, the above method may further include: if a log extraction instruction corresponding to the compliance check log is received, extracting the complete microservice log corresponding to the compliance check log according to the log extraction instruction.

[0119] The complete microservice log includes: an operation log of the microservice container and a compliance detection log of the microservice container.

[0120] When a microservice container receives a log extraction instruction from a central log server, it can search for the corresponding task's operation log and compliance check log locally based on the business data identifier contained in the instruction information, and send all log information to the central log server. The central log server can merge all operation logs based on the business request identifier and microservice call identifier, thereby restoring the microservice internal operations experienced by the user's business request and generating related exception information.

[0121] The distributed log data processing method provided by the present disclosure enables the microservice container to perform compliance checks on the tasks executed by each of them, generates compliance check logs and sends them to the central log server; enables the central log server to merge multiple related compliance check logs according to the calling order of the microservices based on the business data identifier, and then checks the merged logs according to the preset detection rules. Only when the central log server detects that there is an abnormality in the merged log, the corresponding complete data is obtained from the microservice container, so that the central log server does not need to receive all the log information of each microservice container, reduces the data transmission pressure between the microservice container and the central log server and the storage pressure of the central log server, and can upload all logs in the case of abnormalities to the central log server for storage, which is convenient for rapid investigation and acquisition of abnormal data.

[0122] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0123] Furthermore, in this exemplary embodiment, a distributed log data processing system is also provided, referring to Figure 4 As shown, it includes: a central log server, and at least one microservice container;

[0124] Wherein, the microservice container is used to check the current task according to the second detection rule for checking the business logic to generate a compliance detection log, so as to send the compliance detection log to the central log server; wherein the compliance detection log includes a business data identifier;

[0125] The central log server is used to merge the compliance detection logs according to the business data identifier to obtain a merged log, and to check the merged log according to a first check rule for checking the main business logic to obtain abnormal data.

[0126] The specific details of each server in the above-mentioned distributed log data processing system have been described in detail in the corresponding distributed log data processing method, so they are not repeated here.

[0127] For further reference, Figure 5 As shown, in the embodiment of this example, a distributed log data processing device 50 is also provided, comprising: a log data receiving module 501, a log data merging module 502, a log data checking module 503 and a complete log data extracting module 504. Among them:

[0128] The log data receiving module 501 may be used to receive a compliance check log of a microservice container; wherein the compliance check log includes a business data identifier.

[0129] The log data merging module 502 may be configured to merge the associated compliance check logs according to the business data identifier to obtain a merged log.

[0130] The log data verification module 503 may be used to verify the merged log according to a first verification rule for verifying the main logic of the service.

[0131] The complete log data extraction module 504 can be used to send a log extraction instruction to the corresponding microservice container according to the business data identifier when there is an abnormality in the inspection result of the merged log, so as to obtain the complete microservice log corresponding to the compliance check log.

[0132] For further reference, Figure 6 As shown, in the embodiment of this example, a distributed log data processing device 60 is also provided, including: a log data generating module 601, an identification loading module 602 and a log data sending module 603. Among them,

[0133] The log data generation module 601 may be used to check the current task according to the second detection rule for checking the business logic to generate a current compliance check log.

[0134] The identifier loading module 602 may be used to add a business data identifier to the current compliance check log to obtain a compliance check log.

[0135] The log data sending module 603 may be used to send the compliance check log to a central log server so that the central log server can check the compliance check log.

[0136] The specific details of each module in the above-mentioned distributed log data processing device have been described in detail in the corresponding distributed log data processing method, so they will not be repeated here.

[0137] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0138] In an exemplary embodiment of the present disclosure, a computer system capable of implementing the above method is also provided.

[0139] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0140] Refer to the following Figure 6 hereinafter, a computer system 600 according to this embodiment of the present invention is described. Figure 6 The computer system 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0141] like Figure 6 As shown, the computer system 600 is in the form of a general-purpose computing device. The components of the computer system 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, and a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610).

[0142] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Step S101 shown in: receiving the compliance check log of the microservice container; wherein the compliance check log includes a business data identifier; S102: merging the associated compliance check logs according to the business data identifier to obtain a merged log; S103: checking the merged log according to the first inspection rule for checking the main business logic; S104: when there is an abnormality in the inspection result of the merged log, sending a log extraction instruction to the corresponding microservice container according to the business data identifier, so as to obtain the complete microservice log corresponding to the compliance check log; or executing as follows Figure 3 The steps shown in: Step S301: Check the current task according to the second detection rule for checking the business logic to generate a current compliance check log; Step S302: Generate a business data identifier in the current compliance check log to obtain a compliance check log; Step S303: Send the compliance check log to the central log server so that the central log server can check the compliance check log.

[0143] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0144] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0145] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0146] The computer system 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the computer system 600, and / or any device that enables the computer system 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. In addition, the computer system 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the computer system 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer system 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0147] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0148] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0149] refer to Figure 7 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0150] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0151] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0152] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0153] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0154] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0155] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0156] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A distributed log data processing method, characterized in that: include: Receive a compliance check log of a microservice container; wherein the compliance check log includes a business data identifier, wherein a second detection rule is established for each microservice container based on the information of the executed task and the task process, for checking the compliance, business logic and specific parameters of the tasks executed by each microservice container, and the inspection result generates a compliance check log and sends it to a central log server, when the microservice container executes the task normally and no exception occurs, the compliance check log generated by the compliance check using the second detection rule includes the input parameters, output parameters and IP address corresponding to the microservice container; when the microservice container has an exception when executing the task and ends the task, the compliance check log generated by the compliance check using the second detection rule includes the input parameters, output parameters, execution results, reasons for the abnormal termination and related parameters, execution time and IP address corresponding to the microservice container; the business data identifier includes: a business request identifier for marking a business request, and a microservice call identifier for identifying the calling order of the microservice container; Merging the associated compliance check logs according to the business data identifier to obtain a merged log, which includes: obtaining at least one compliance check log according to the business request identifier; merging the compliance check logs according to the calling order according to the microservice call identifier; Perform a global compliance check on the integrity of the merged log, the validity of each parameter, and whether there are any exceptions according to the first inspection rule for verifying the main business logic, check the compliance check log of each microservice container passed by the business request, and verify the abnormal data or abnormal logic of multiple input parameters and output parameters involved in the merged log; When there is an abnormality in the inspection result of the merged log, a log extraction instruction is sent to the corresponding microservice container according to the business data identifier, so as to obtain the complete microservice log corresponding to the compliance check log.

2. The method according to claim 1, characterized in that The method further comprises: When it is verified that there is no abnormality in the merged log data, the merged log and / or the compliance check log corresponding to the merged log is deleted.

3. The method according to claim 1, characterized in that When it is determined that there is abnormal data in the merged log, sending a log extraction instruction to the corresponding microservice container according to the business data identifier to obtain the complete microservice log corresponding to the compliance check log includes: When it is determined that the merged log contains abnormal data, generating a log extraction instruction; The log extraction instruction is added to the current period log extraction instruction list so as to send the log extraction instruction to the corresponding microservice container at a preset period time.

4. The method according to claim 3, characterized in that The complete microservice log includes: The operation log of the microservice container and the compliance check log of the microservice container.

5. The method according to claim 1, characterized in that Providing an interactive interface, the method further includes: Receive log data query request; The corresponding complete microservice log is extracted according to the log data query request.

6. A distributed log data processing method, characterized in that: include: Inspecting the current task according to a second detection rule for inspecting the business logic to generate a current compliance inspection log, wherein the current compliance inspection log is generated by inspecting the compliance, business logic, and specific parameters of the current task executed according to the second detection rule; Generate a business data identifier in the current compliance check log to obtain a compliance check log, wherein, for each microservice container, a second detection rule is established according to the information of the executed task and the task process, which is used to check the compliance, business logic and specific parameters of the tasks executed by each microservice container, and the inspection result generates a compliance check log and sends it to the central log server, when the microservice container executes the task normally and no exception occurs, the compliance check log generated by the compliance check using the second detection rule includes the input parameters, output parameters and IP address corresponding to the microservice container; when the microservice container has an exception when executing the task and ends the task, the compliance check log generated by the compliance check using the second detection rule includes the input parameters, output parameters, execution results, reasons for the abnormal termination and related parameters, execution time and IP address corresponding to the microservice container; the business data identifier includes: a business request identifier for marking a business request, and a microservice call identifier for identifying the calling order of the microservice container; The compliance check log is sent to a central log server so that the central log server can merge the associated compliance check logs according to the business data identifier included in the compliance check log to obtain a merged log, which includes: obtaining at least one of the compliance check logs according to the business request identifier; merging the compliance check logs in the order of calling according to the microservice call identifier; performing a global compliance check on the integrity of the merged log, the validity of each parameter, and whether there is an abnormality according to the first inspection rule for checking the main business logic, checking the compliance check log of each microservice container passed by the business request, and checking the abnormal data or abnormal logic of multiple input parameters and output parameters involved in the merged log; when there is an abnormality in the inspection result of the merged log, sending a log extraction instruction according to the business data identifier; If a log extraction instruction corresponding to the compliance check log is received, the complete microservice log corresponding to the compliance check log is extracted according to the log extraction instruction and sent to the central log server.

7. The method according to claim 6, characterized in that Adding a business data identifier to the current compliance check log to obtain the compliance check log includes: Extracting current log data in the current compliance check log according to preset fields; The business data identifier is generated and added to the current log data to obtain the compliance check log.

8. The method according to claim 6, characterized in that The complete microservice log includes: an operation log of a microservice container and a compliance check log of the microservice container.

9. A distributed log data processing system, characterized in that: include: A central log server, and at least one microservice container; Wherein, the central log server is used to execute the method described in any one of claims 1 to 5; The microservice container is used to execute the method according to any one of claims 6 to 8.

10. A distributed log data processing device, characterized in that: include: A log data receiving module, used to receive a compliance check log of a microservice container; wherein the compliance check log includes a business data identifier, wherein a second detection rule is established for each microservice container according to the information of the executed task and the task process, for checking the compliance, business logic and specific parameters of the tasks executed by each microservice container, and the inspection result generates a compliance check log and sends it to a central log server, when the microservice container executes the task normally and no exception occurs, the compliance check log generated by the compliance check using the second detection rule includes the input parameters, output parameters and IP address corresponding to the microservice container; when the microservice container has an exception when executing the task and ends the task, the compliance check log generated by the compliance check using the second detection rule includes the input parameters, output parameters, execution results, reasons for abnormal termination and related parameters, execution time and IP address corresponding to the microservice container; the business data identifier includes: a business request identifier for marking a business request, and a microservice call identifier for identifying the calling order of the microservice container; A log data merging module, used to merge the associated compliance check logs according to the business data identifier to obtain a merged log, comprising: obtaining at least one compliance check log according to the business request identifier; merging the compliance check logs in a calling order according to the microservice call identifier; A log data verification module is used to perform a global compliance check on the integrity of the merged log, the validity of each parameter, and whether there are any anomalies according to the first verification rule for verifying the main business logic, check the compliance check log of each microservice container passed by the business request, and verify the multiple input parameters and output parameters involved in the merged log for abnormal data or abnormal logic; The complete log data extraction module is used to send a log extraction instruction to the corresponding microservice container according to the business data identifier when there is an abnormality in the inspection result of the merged log, so as to obtain the complete microservice log corresponding to the compliance check log.

11. A distributed log data processing device, characterized in that: include: A log data generation module, used to check the current task according to the second detection rule for checking the business logic to generate a current compliance check log; An identification loading module is used to add a business data identification to the current compliance check log to obtain a compliance check log, wherein a second detection rule is established for each microservice container according to the information of the executed task and the task process, which is used to check the compliance, business logic and specific parameters of the tasks executed by each microservice container, and the compliance check log generated by the inspection result is sent to the central log server, when the microservice container executes the task normally and no exception occurs, the compliance check log generated by the compliance check using the second detection rule includes the input parameters, output parameters and IP address corresponding to the microservice container; when the microservice container has an exception when executing the task and ends the task, the compliance check log generated by the compliance check using the second detection rule includes the input parameters, output parameters, execution results, reasons for abnormal termination and related parameters, execution time and IP address corresponding to the microservice container; the business data identification includes: a business request identification for marking business requests, and a microservice call identification for identifying the calling order of the microservice container; A log data sending module, used for sending the compliance check log to a central log server so that the central log server can merge the associated compliance check logs according to the business data identifier included in the compliance check log to obtain a merged log, which includes: obtaining at least one of the compliance check logs according to the business request identifier; merging the compliance check logs in the order of calling according to the microservice call identifier; performing a global compliance check on the integrity of the merged log, the validity of each parameter, and whether there is an abnormality according to the first inspection rule for checking the main business logic, checking the compliance check log of each microservice container passed by the business request, and checking the abnormal data or abnormal logic of multiple input parameters and output parameters involved in the merged log; when there is an abnormality in the inspection result of the merged log, sending a log extraction instruction according to the business data identifier; The log data sending module is also used to, if a log extraction instruction corresponding to the compliance check log is received, extract the complete microservice log corresponding to the compliance check log according to the log extraction instruction and send it to the central log server.

12. A storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the distributed log data processing method according to any one of claims 1 to 5 or claims 6 to 8 is implemented.

13. A computer system, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the distributed log data processing method according to any one of claims 1 to 5 or claims 6 to 8 by executing the executable instructions.

Images