Log processing method and apparatus, electronic device, and medium

Abstract

This application provides a log processing method, apparatus, electronic device, and medium. The method includes: receiving log information and sending the log information to a message queue; wherein the message queue aggregates multiple log messages for consumption; at least two log messages generated sequentially by the same system operation correspond to the same asynchronous identifier; retrieving target log information to be consumed from the message queue and searching for associated log information with the same asynchronous identifier as the target log information; performing a population operation on the target log information based on the associated log information, and storing the populated target log information. This method effectively alleviates the storage pressure on log data.

Description

Technical Field

[0001] This application relates to cloud computing technology, and more particularly to a log processing method, apparatus, electronic device, and medium. Background Technology

[0002] With the development of cloud computing products, logging access to and modification of data in the cloud has become increasingly important. By performing data mining on these logs and making full use of them, we can provide support for customer resources in areas such as problem localization, resource tracking, and behavior analysis.

[0003] Currently, there is an urgent need for a log collection system that can alleviate the burden of log data storage and be applied to the collection of operational logs. Summary of the Invention

[0004] This application provides a log processing method, apparatus, electronic device, and medium to alleviate the storage pressure on operation log data.

[0005] On the one hand, this application provides a log processing method applied to a log collection system of a cloud platform, the method comprising:

[0006] Receive log information and send the log information to a message queue; wherein, the message queue collects multiple log information for consumption; at least two log information generated successively by the same system operation correspond to the same asynchronous identifier;

[0007] Retrieve the target log information to be consumed from the message queue, and find the associated log information that has the same asynchronous identifier as the target log information;

[0008] Based on the associated log information, the target log information is populated, and the populated target log information is stored.

[0009] In another possible implementation, the log information with an asynchronous identifier also has a sequence identifier to indicate the order of the log information among at least two log information generated in the same system operation; the step of retrieving the target log information to be consumed from the message queue and searching for associated log information with the same asynchronous identifier as the target log information includes:

[0010] Retrieve the target log information to be consumed from the message queue, and determine whether the target log information has an asynchronous identifier;

[0011] If the target log information has an asynchronous identifier, then based on the sequence identifier of the target log information, determine whether the target log information is the first log information generated by the system operation;

[0012] If the target log information is not the first log information generated by the system operation, then the associated log information of the target log information is found according to the asynchronous identifier.

[0013] In another possible implementation, if the target log information is not the first log information generated by the system operation, then searching for related log information of the target log information based on the asynchronous identifier includes:

[0014] Based on the asynchronous identifier, find the first log information generated by the system operation and use it as the associated log information;

[0015] The step of filling the target log information according to the associated log information includes: filling the target log information according to the attribute information contained in the first log information; wherein, the attribute information includes: operation source, operator, and operation time.

[0016] In another possible implementation, receiving log information and sending the log information to a message queue includes:

[0017] Obtain the log information, and determine whether the log information belongs to the first log information of the system operation based on the sequence identifier of the log information;

[0018] If it belongs to the first log information, then check whether the log information contains the attribute information of the system operation; if it does not belong to the first log information, then check whether the log information has the asynchronous identifier.

[0019] After successful verification, the sub-identifier of the log information is set according to the main identifier of the user operation, and the log information is sent to the message queue. Each user operation corresponds to at least one system operation, and each system operation corresponds to at least one log information. The main identifier of the at least one log information corresponding to the same user operation is the same, but the sub-identifiers are different.

[0020] In another possible implementation, the log collection system provides a log collection interface for collecting log information from various cloud computing modules in the cloud platform;

[0021] The step of receiving log information and sending the log information to a message queue includes:

[0022] Obtain log information pushed by each cloud computing module through the log collection interface, and determine whether the log information belongs to the log information corresponding to a preset operation event;

[0023] If it does, the log information will be sent to the message queue.

[0024] In another possible implementation, before sending the log information to the message queue, the method further includes:

[0025] The asynchronous information of the log information is determined based on the system operation, and the asynchronous information is used to indicate whether the asynchronous identifier and the sequence identifier exist in the log information;

[0026] Determine whether the asynchronous identifier exists in the log information, and send the log information to the message queue based on the determination result.

[0027] In another possible implementation, sending the log information to the message queue based on the judgment result includes:

[0028] If the log information contains the asynchronous identifier, then the target partition is determined based on the asynchronous identifier, and the log information is sent to the first target partition of the message queue; the first target partition is used to store log information with the same asynchronous identifier.

[0029] If the asynchronous identifier is not present in the log information, the log information is sent to the message queue using a preset algorithm.

[0030] Secondly, this application provides a log processing apparatus, comprising:

[0031] A receiving module is used to receive log information and send the log information to a message queue; wherein, the message queue collects multiple log information for consumption; at least two log information generated successively by the same system operation correspond to the same asynchronous identifier;

[0032] The lookup module is used to obtain the target log information to be consumed from the message queue and to find the associated log information that has the same asynchronous identifier as the target log information;

[0033] The filling module is used to fill the target log information according to the associated log information and store the filled target log information.

[0034] In another possible implementation, the lookup module is specifically used for:

[0035] The system retrieves target log information to be consumed from the message queue and determines whether the target log information has an asynchronous identifier. Log information with an asynchronous identifier also has a sequence identifier, which is used to indicate the order of the log information among at least two log information generated in the same system operation.

[0036] If the target log information has an asynchronous identifier, then based on the sequence identifier of the target log information, determine whether the target log information is the first log information generated by the system operation;

[0037] If the target log information is not the first log information generated by the system operation, then the associated log information of the target log information is found according to the asynchronous identifier.

[0038] In another possible implementation, the lookup module is specifically used for:

[0039] If the target log information is not the first log information generated by the system operation, the first log information generated by the system operation is found according to the asynchronous identifier and used as the associated log information;

[0040] The step of filling the target log information according to the associated log information includes: filling the target log information according to the attribute information contained in the first log information; wherein, the attribute information includes: operation source, operator, and operation time.

[0041] In another possible implementation, the receiving module is specifically used for:

[0042] Obtain the log information, and determine whether the log information belongs to the first log information of the system operation based on the sequence identifier of the log information;

[0043] If it belongs to the first log information, then check whether the log information contains the attribute information of the system operation; if it does not belong to the first log information, then check whether the log information has the asynchronous identifier.

[0044] After successful verification, the sub-identifier of the log information is set according to the main identifier of the user operation, and the log information is sent to the message queue. Each user operation corresponds to at least one system operation, and each system operation corresponds to at least one log information. The main identifier of the at least one log information corresponding to the same user operation is the same, but the sub-identifiers are different.

[0045] In another possible implementation, the receiving module is specifically used for:

[0046] The system acquires log information pushed by each cloud computing module through the log collection interface and determines whether the log information belongs to the log information corresponding to a preset operation event; the log collection system provides a log collection interface for collecting log information from each cloud computing module in the cloud platform.

[0047] If it does, the log information will be sent to the message queue.

[0048] In another possible implementation, the receiving module is further configured to:

[0049] The asynchronous information of the log information is determined based on the system operation, and the asynchronous information is used to indicate whether the asynchronous identifier and the sequence identifier exist in the log information;

[0050] Determine whether the asynchronous identifier exists in the log information, and send the log information to the message queue based on the determination result.

[0051] In another possible implementation, the receiving module is specifically used for:

[0052] If the log information contains the asynchronous identifier, then the target partition is determined based on the asynchronous identifier, and the log information is sent to the first target partition of the message queue; the first target partition is used to store log information with the same asynchronous identifier.

[0053] If the asynchronous identifier is not present in the log information, the log information is sent to the message queue using a preset algorithm.

[0054] Thirdly, this application provides an electronic device, including a memory and a processor, wherein the memory is used to store computer-executed instructions;

[0055] The processor is used to invoke the computer to execute instructions so that the electronic device performs the method described in any one of the first aspects.

[0056] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any of the first aspects.

[0057] This application provides a log processing method, apparatus, electronic device, and medium. The method of this application, when retrieving target log information to be consumed from a message queue, performs a population operation on the target log information by searching for associated log information with the same asynchronous identifier as the target log information to obtain the corresponding data. This eliminates the need for some target log information in the message queue to contain attribute information corresponding to the operation, thereby effectively reducing the data storage pressure on the message queue. Attached Figure Description

[0058] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0059] Figure 1 A schematic diagram illustrating a log processing method provided in an embodiment of this application;

[0060] Figure 2 A flowchart illustrating a log processing method provided in an embodiment of this application;

[0061] Figure 3 A flowchart illustrating a method for sending log information to a message queue, as provided in this application embodiment;

[0062] Figure 4 A flowchart illustrating a method for retrieving log information from a message queue, provided in an embodiment of this application;

[0063] Figure 5 This is a schematic diagram of the structure of a log processing device provided in an embodiment of this application;

[0064] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0065] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0066] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0067] With the development of cloud computing products, cloud computing operation logs are gradually gaining attention as a potential resource. Data mining of cloud computing product operation logs can provide support for customer resource problem localization and resource tracking, as well as for developer behavior analysis.

[0068] Cloud computing product logs are divided into system logs and operation logs. Currently, system logs are mostly collected using the ELK log analysis platform. Operation logs are typically displayed directly as cloud computing operation logs. ELK is an abbreviation for Elasticsearch + Logstash + Kibana architecture. The ELK log analysis platform is an open-source log analysis platform that collects logs through Logstash, analyzes them using Elasticsearch, and then displays them in Kibana (a web interface).

[0069] The aforementioned methods for collecting cloud computing operation logs are not conducive to the simple integration of operation logs into cloud computing products. Therefore, this application provides a log processing method for use in a log collection system. This method interfaces with various cloud computing products via APIs, eliminating the need to directly inherit message queues or deploy collection tools for operation log collection. Furthermore, when collecting logs, the log collection system of this application, for at least two asynchronous operations belonging to the same system, only ensures that the log information of one of the asynchronous operations contains attribute information when pushing them to the message queue. When the log information of these at least two asynchronous operations is consumed, the log information without attribute information is filled with data using the log information containing attribute information from the at least two asynchronous operations to obtain complete log information. This method reduces the unnecessary duplication of log data to the message queue, thereby alleviating the storage pressure on log data.

[0070] Figure 1 This is a schematic diagram illustrating an application scenario of a log processing method provided in an embodiment of this application, such as... Figure 1 As shown, the application scenario of this application embodiment includes an operation log pusher and a log collection system. The log collection system provides a log collection interface and uses an API interface to collect logs. Specifically, the operation log pusher directly calls the API interface of the log collection system to push the logs, and the log collection system receives the operation logs pushed by the operation log pusher and stores the log data of the operation logs.

[0071] like Figure 1 As shown, the operation log pusher can be any cloud computing product that generates operation logs, such as a cloud host, cloud disk, or cloud desktop. In this application, the log collection system is responsible for receiving the operation logs from these operation log pushers and processing them according to the log processing method of this application. The technical solution of this application and how it solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0072] Figure 2 This is a flowchart illustrating a log processing method provided in an embodiment of this application, as shown below. Figure 2 As shown, the method in this application embodiment includes:

[0073] S201 receives log information and sends the log information to the message queue.

[0074] The message queue aggregates multiple log messages for consumption; at least two log messages generated sequentially by the same system operation correspond to the same asynchronous identifier.

[0075] In this embodiment, the message queue is specifically a Kafka message queue, used to receive operation logs pushed by the operation log pusher and for consumption by the log consumer. In this embodiment, the log collection system provides a log collection interface for collecting log information from various cloud computing modules of the operation log pusher.

[0076] In this embodiment, when the same system operation generates at least two log messages sequentially, an asynchronous flag is set for the at least two log messages to indicate that they are asynchronous operations generated by the same system operation. For example, for a cloud server, after a user clicks "Buy Cloud Server" in the console, the process goes through the order module and the instance system, but the server is not created immediately. At this time, a "Create Cloud Server" operation log is generated. After the cloud server is actually successfully created, another operation log is generated. These two operation logs represent the two asynchronous operations generated by the system operation of purchasing a cloud server.

[0077] In this embodiment, each operation log corresponds to a primary identifier and a secondary identifier. If the operation log is an asynchronous operation within the same system operation, the log information also includes the asynchronous identifier corresponding to the operation log. The primary identifier uniquely corresponds to a user operation, each user operation corresponds to at least one system operation, and each system operation corresponds to at least one log entry. Each log entry uniquely corresponds to a secondary identifier, and log entries belonging to the same user operation share the same primary identifier. If the log entry is generated by an asynchronous operation of a system operation, the log entry also corresponds to an asynchronous identifier.

[0078] For example, if one log message has a main identifier of 1 and a sub-identifier of 11, and does not have an asynchronous identifier; another log message has a main identifier of 1 and a sub-identifier of 12, has an asynchronous identifier, and the asynchronous identifier is 1; and yet another log message has a main identifier of 1 and a sub-identifier of 13, has an asynchronous identifier, and the asynchronous identifier is 1, then it means that the first, second, and third log messages all correspond to the same user operation, and are three log messages generated by different operations. Specifically, the second and third log messages correspond to different asynchronous operations within the same system operation.

[0079] S202: Retrieve the target log information to be consumed from the message queue and find the associated log information with the same asynchronous identifier as the target log information.

[0080] In this embodiment, when a log consumer requests log information from a message queue, the log collection system retrieves the target log information to be consumed from the message queue. Furthermore, if the target log information corresponds to an asynchronous operation of a system operation, the log collection system will first search for associated log information with the same asynchronous identifier as the target log information.

[0081] Understandably, the log collection system first searches for log information with an asynchronous identifier. Then, it searches for log information with the same asynchronous identifier as the target log information to be consumed, and uses that log information as the associated log information.

[0082] S203, based on the associated log information, populate the target log information and store the populated target log information.

[0083] In this embodiment, the log collection system determines that the target log information has associated log information, and after determining the associated log information, it populates the target log information according to the associated log information.

[0084] Understandably, for log messages with asynchronous identifiers, when the log collection system sends log messages with the same asynchronous identifier to the message queue, it only includes the system operation attribute information in the log message when sending one of the log messages. Specifically, the attribute information includes the operation source, operator, operation time, etc.

[0085] Based on this, associated log information refers to log information containing attribute information that corresponds to the same asynchronous identifier. When the log collection system obtains target log information to be consumed, it fills the target log information with the attribute information contained in the associated log information, thus including the attribute information in the target log information, and stores the target log information containing the attribute information.

[0086] It is understandable that in this embodiment, there may be cases where the target log information is the same as the associated log information. Therefore, before searching for associated log information, the log collection system first determines whether the target log information contains attribute information.

[0087] The method provided in this embodiment allows the log collection system to use only one log message as the associated log message when sending it to the message queue, provided that the log message contains the attribute information of the corresponding system operation, for at least two log messages corresponding to the same asynchronous identifier. Based on this, when storing the target log message, the system first retrieves the target log message from the message queue and then searches for the associated log message with the same asynchronous identifier as the target log message—that is, the log message containing attribute information and corresponding to the same asynchronous identifier as the target log message. Finally, the system fills the target log message with the associated log message and stores the filled target log message.

[0088] Using the method of this embodiment, when the log collection system sends received log information to the message queue, for at least two log messages corresponding to the same asynchronous identifier, it can ensure that only one of the log messages contains the attribute information of the corresponding system operation, thereby alleviating the data storage pressure on the message queue. When retrieving target log information to be consumed from the message queue, the log collection system searches for its corresponding associated log information and populates the target log information based on the associated log information, thereby achieving the effect of alleviating the data storage pressure on the message queue.

[0089] Figure 3 This document provides a flowchart illustrating a method for sending log information to a message queue, based on previous embodiments. Specifically, this embodiment focuses on the detailed description of the process by which a log collection system receives log information and sends it to a message queue. Figure 3 As shown, the method in this embodiment includes:

[0090] S301, obtain log information and determine asynchronous information of the log information based on system operations.

[0091] Among them, asynchronous information is used to indicate whether the log information has asynchronous and sequential identifiers.

[0092] In this embodiment, the log collection system determines whether to set an asynchronous identifier for the log information by judging whether the operation log corresponding to the log information was generated by one of the asynchronous operations corresponding to the same system operation. For the log information corresponding to the asynchronous operation, a sequence identifier is further set to indicate which asynchronous operation within the same system operation generated the log information.

[0093] In this embodiment, Q represents the asynchronous identifier and T represents the sequential identifier. Specifically, Q = q, T = t, where q and t are both integers greater than or equal to 0. When at least two log messages are not generated by different asynchronous operations of the same system operation, q and t are set to 0. When at least two log messages are generated by different asynchronous operations of the same system operation, q is set to a non-zero integer to identify which system operation's asynchronous operation generated the log message. The value of t is set according to which asynchronous operation of the same system operation the asynchronous operation corresponding to the log message belongs to.

[0094] Based on this, if the q values ​​of at least two log messages are not zero and are equal, it means that the at least two log messages correspond to different asynchronous operations of the same system operation. In this case, the execution order of the asynchronous operations corresponding to the at least two log messages can be known by the value of t.

[0095] S302, determine whether the log information has an asynchronous identifier based on the asynchronous information; if it does, execute S303; otherwise, execute S307.

[0096] In this embodiment, the log collection system determines whether the log information has an asynchronous flag by checking whether the value of q is zero.

[0097] S303: Based on the sequence identifier, determine whether the log information belongs to the first log information of the system operation; if yes, execute S304; otherwise, execute S305.

[0098] It is understandable that if log information has an asynchronous identifier, it means that it must also have a sequential identifier to indicate which log information it belongs to within a system operation. Specifically, in this embodiment, the log collection system determines which log information belongs to within a system operation by checking the value of t. When t is 1, it means that the corresponding log information is the first log information; if t is not 1, it means that the corresponding log information is not the first log information.

[0099] S304, check whether the log information contains attribute information of system operation, and determine whether the log information belongs to the log information corresponding to the preset operation event. If yes, execute S306; otherwise, return an error.

[0100] In this embodiment, the preset operation event is the log action agreed upon with the operation log pusher.

[0101] S305 checks whether the log information has an asynchronous flag. If so, proceed to S306; otherwise, return an error.

[0102] S306: Set the sub-identifier of the log information according to the main identifier of the user operation, determine the first target partition according to the asynchronous identifier, and send the log information to the first target partition of the message queue.

[0103] Each user operation corresponds to at least one system operation, and each system operation corresponds to at least one log message; the primary identifier of at least one log message corresponding to the same user operation is the same, but the sub-identifiers are different; the first target partition is used to store log messages with the same asynchronous identifier.

[0104] In this embodiment, N represents the primary identifier and M represents the secondary identifier, where N = n and M = m, and n and m are both positive integers greater than 0. Log messages with different values ​​of n correspond to different user operations, and log messages with different values ​​of m correspond to different operations. If at least two log messages have the same value of n, it means that the at least two log messages correspond to the same user operation.

[0105] After acquiring log information, the log collection system first sets the primary identifier of the log information based on the user operation to which it belongs, ensuring that at least one log information corresponding to the same user operation has the same primary identifier. Secondly, it sets the sub-identifier of the log information based on which operation log it corresponds to.

[0106] In this embodiment, for log messages whose sequence identifier indicates they are not the first log message, the log collection system only needs to verify whether the log message contains an asynchronous identifier that must be transmitted. If it does, it will be transmitted; it will not verify whether it contains attribute information. If it does not contain an asynchronous identifier, an error will be returned.

[0107] In this embodiment, the asynchronous identifier is used as the region name of the first target partition so that log information with the same asynchronous identifier can be received.

[0108] S307: Set the sub-identifier of the log information according to the main identifier of the user operation, and send the log information to the second target partition of the message queue according to the preset algorithm.

[0109] In this embodiment, when the log collection system receives log information, it first sets the asynchronous information corresponding to the log information based on the system operation to which it belongs. Specifically, if the log information belongs to an asynchronous operation of the same system operation, its asynchronous information is set to indicate that it includes an asynchronous identifier and a sequence identifier. The asynchronous identifier is set according to the system operation, and the sequence identifier is determined according to the execution order of the asynchronous operation.

[0110] Secondly, when transmitting log information to the message queue, the system first determines whether the log information belongs to the first asynchronous operation within the same system operation based on its sequence identifier. If so, it checks whether the log information contains attribute information and whether it belongs to a preset operation event. If the log information does not contain attribute information or does not belong to a preset operation event, an error is returned. If the log information contains attribute information and belongs to a preset operation event, the system further sets the primary and secondary identifiers based on the user operation to which the log information belongs.

[0111] Finally, the log collection system sends the log information to the first target partition of the message queue according to the asynchronous identifier of the log information.

[0112] If the log information is not generated by an asynchronous operation of the same system and there is no asynchronous identifier, the log information will be sent to the second target partition of the message queue according to the preset algorithm.

[0113] The method in this embodiment allows the log collection system to verify and transmit attribute information only when the received log information with an asynchronous identifier is the first log information of a system operation. For log information that is not the first log information, attribute information transmission is not required, thereby effectively reducing data storage pressure. Simultaneously, for log information that is not the first log information, the corresponding asynchronous identifier must be transmitted, providing effective support for subsequent message queue consumption.

[0114] Figure 4 This document provides a flowchart illustrating a method for retrieving log information from a message queue, based on previous embodiments. Specifically, this embodiment focuses on detailing the process of retrieving log information from a message queue. Figure 4 As shown, the method in this embodiment includes:

[0115] S401: Retrieve target log information to be consumed from the message queue.

[0116] S402, determine whether the target log information has an asynchronous flag; if yes, execute S403; otherwise, execute S405.

[0117] Specifically, in this embodiment, the log collection system determines whether the target log information has an asynchronous identifier by using the region name of the target partition to which the target log information belongs.

[0118] S403, determine whether the target log information is not the first log information generated by the system operation; if yes, execute S404; otherwise, execute S405.

[0119] In this embodiment, the log collection system determines whether the target log information is the first log information generated by the system operation based on the sequence identifier of the target log information. Specifically, the log collection system determines whether the target log information is the first log information generated by the system operation by checking whether the value of t is 1.

[0120] S404: Based on the asynchronous identifier, find the first log information generated by the corresponding system operation and use it as the associated log information; based on the attribute information contained in the first log information, perform a fill operation on the target log information and store the filled target log information.

[0121] The attribute information includes the source of the operation, the operator, and the time of the operation.

[0122] S405 directly stores target log information.

[0123] In this embodiment, the log collection system first searches for log messages with the same asynchronous identifier q value. Then, among these log messages, it searches for log messages with a sequence identifier t value of 1, i.e., the first log message generated by the corresponding system operation, and uses this as the associated log message. As can be seen from the previous embodiment, when the log collection system sends the log message to the message queue, it requires that the log message must contain attribute information. Therefore, for the target log message, the log collection system can perform a fill operation based on the attribute information contained in the first log message.

[0124] Optionally, in this embodiment, for target log information with an asynchronous identifier, target log information with the same asynchronous identifier is stored in the same asynchronous table according to the asynchronous identifier. For target log information without an asynchronous identifier, it is stored in the log table. It can be understood that the asynchronous table is a part of the log table and is used to store target log information with an asynchronous identifier.

[0125] Optionally, after storing the target log information in the log table, the message queue can be notified manually via ACK so that the message queue can delete the corresponding target log information.

[0126] The method provided in this embodiment, for each target log information to be consumed, the log collection system determines whether there is an asynchronous identifier based on the target partition to which it belongs. For target log information with an asynchronous identifier, it further determines whether it needs to be filled based on its sequence identifier, so that the log consumer can obtain complete log data.

[0127] This application also provides a log processing apparatus. Figure 5 A log processing apparatus provided in the embodiments of this application, such as Figure 5As shown, the device in this embodiment includes a receiving module 51, a searching module 52, and a filling module 53.

[0128] The receiving module 51 is used to receive log information and send the log information to a message queue; the message queue collects multiple log information for consumption; at least two log information generated successively by the same system operation correspond to the same asynchronous identifier;

[0129] The lookup module 52 is used to retrieve the target log information to be consumed from the message queue and to find the associated log information that has the same asynchronous identifier as the target log information;

[0130] The filling module 53 is used to fill the target log information according to the associated log information and store the filled target log information.

[0131] In another possible implementation of this application embodiment, the lookup module 52 is specifically used for:

[0132] Retrieve the target log information to be consumed from the message queue and determine whether the target log information has an asynchronous identifier; log information with an asynchronous identifier also has a sequence identifier, which is used to indicate the order of the log information among at least two log information generated in the same system operation;

[0133] If the target log information has an asynchronous identifier, then determine whether the target log information is the first log information generated by the system operation based on the sequence identifier of the target log information;

[0134] If the target log information is not the first log information generated by the system operation, then the associated log information of the target log information is found based on the asynchronous identifier.

[0135] In another possible implementation of this application embodiment, the lookup module 52 is specifically used for:

[0136] If the target log information is not the first log information generated by the system operation, the first log information generated by the system operation is found based on the asynchronous identifier and used as the associated log information.

[0137] The target log information is populated based on the associated log information, including: populating the target log information based on the attribute information contained in the first log information; wherein, the attribute information includes: operation source, operator, and operation time.

[0138] In another possible implementation of this application embodiment, the receiving module 51 is specifically used for:

[0139] Retrieve log information and determine whether the log information is the first log information of the system operation based on the sequence identifier of the log information;

[0140] If it is the first log message, check whether the log message contains attribute information of system operation; if it is not the first log message, check whether the log message has an asynchronous identifier.

[0141] After successful verification, the sub-identifier of the log information is set according to the main identifier of the user operation, and the log information is sent to the message queue. Each user operation corresponds to at least one system operation, and each system operation corresponds to at least one log information. The main identifier of the at least one log information corresponding to the same user operation is the same, but the sub-identifiers are different.

[0142] In another possible implementation of this application embodiment, the receiving module 51 is specifically used for:

[0143] The system acquires log information pushed by various cloud computing modules through the log collection interface and determines whether the log information belongs to the log information corresponding to the preset operation event. The log collection system provides a log collection interface for collecting log information from various cloud computing modules in the cloud platform.

[0144] If it does, the log information will be sent to the message queue.

[0145] In another possible implementation of this application embodiment, the receiving module 51 is further configured to:

[0146] The asynchronous information in the log information is determined based on the system operation. The asynchronous information is used to indicate whether the log information has an asynchronous identifier and a sequence identifier.

[0147] Determine if the log information has an asynchronous flag, and send the log information to the message queue based on the determination result.

[0148] In another possible implementation of this application embodiment, the receiving module 51 is specifically used for:

[0149] If the log information has an asynchronous identifier, the target partition is determined based on the asynchronous identifier, and the log information is sent to the first target partition of the message queue; the first target partition is used to store log information with the same asynchronous identifier.

[0150] If the log information does not have an asynchronous identifier, the log information will be sent to the message queue using a preset algorithm.

[0151] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 6 As shown, Figure 6The illustrated electronic device includes a processor 61 and a memory 62. The processor 61 and the memory 62 are connected, for example, via a bus 63. Optionally, the electronic device may also include a transceiver 64. It should be noted that in practical applications, the transceiver 64 is not limited to one type, and the structure of this electronic device does not constitute a limitation on the embodiments of this application.

[0152] Processor 61 may be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 61 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0153] Bus 63 may include a pathway for transmitting information between the aforementioned components. Bus 63 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 63 may be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus 63 is represented by only one thick line, but this does not mean that there is only one bus 63 or one type of bus 63.

[0154] The memory 62 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0155] The memory 62 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 61. The processor 61 is used to execute the application code stored in the memory 62 to implement the content shown in the foregoing method embodiments.

[0156] This application also provides a computer-readable storage medium, which may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. Specifically, the computer-readable storage medium stores program instructions, which are used to implement the methods in the above embodiments.

[0157] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the technical solution of the above method embodiments. Its implementation principle and technical effects are similar, and will not be repeated here.

[0158] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0159] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A log processing method, characterized in that, A log collection system applied to a cloud platform, the log collection system providing a log collection interface for collecting log information from various cloud computing modules within the cloud platform; the method includes: The system acquires log information pushed by various cloud computing modules through the log collection interface, determines asynchronous information of the log information based on system operations, and uses the asynchronous information to indicate whether the log information has an asynchronous identifier and a sequence identifier; the sequence identifier is used to indicate the order of the log information among at least two log information generated by the same system operation; at least two log information generated sequentially by the same system operation correspond to the same asynchronous identifier. Determine whether the asynchronous identifier exists in the log information; If the log information contains the asynchronous identifier, then based on the sequence identifier of the log information, determine whether the log information belongs to the first log information of the system operation; If it belongs to the first log information, then verify whether the log information contains the attribute information of the system operation; If the log information contains attribute information of the system operation, then the sub-identifier of the log information is set according to the main identifier of the user operation, and the first target partition is determined according to the asynchronous identifier, and the log information is sent to the first target partition of the message queue; the first target partition is used to store log information with the same asynchronous identifier; each user operation corresponds to at least one system operation, each system operation corresponds to at least one log information, and the main identifier of at least one log information corresponding to the same user operation is the same, but the sub-identifiers are different; If it does not belong to the first log information, then check whether the log information has the asynchronous identifier; If the log information has the asynchronous identifier, then the sub-identifier of the log information is set according to the main identifier of the user operation, and the first target partition is determined according to the asynchronous identifier, and the log information is sent to the first target partition of the message queue; If the asynchronous identifier is not present in the log information, the log information is sent to the second target partition of the message queue using a preset algorithm. Obtain the target log information to be consumed from the message queue, and determine whether the target log information has an asynchronous identifier based on the target partition to which the target log information belongs; If the target log information has an asynchronous identifier, then based on the sequence identifier of the target log information, determine whether the target log information is the first log information generated by the system operation; If the target log information is not the first log information generated by the system operation, then the first log information generated by the system operation is found according to the asynchronous identifier and used as the associated log information. Based on the attribute information contained in the first log information, the target log information is populated, and the populated target log information is stored.

2. The method according to claim 1, characterized in that, The attribute information includes: operation source, operator, and operation time.

3. The method according to claim 1, characterized in that, Sending the log information to the message queue includes: Determine whether the log information belongs to the log information corresponding to a preset operation event; if it does, send the log information to the message queue.

4. A log processing apparatus for implementing the log processing method as described in any one of claims 1-3, characterized in that, The device includes: A receiving module is used to receive log information and send the log information to a message queue; wherein, the message queue collects multiple log information for consumption; at least two log information generated successively by the same system operation correspond to the same asynchronous identifier; for log information corresponding to the same asynchronous identifier, when the log information is sent to the message queue, only one log information contains the operating system's attribute information; The lookup module is used to obtain the target log information to be consumed from the message queue and to find the associated log information that has the same asynchronous identifier as the target log information; the associated log information is the log information containing attribute information among the log information corresponding to the same asynchronous identifier. The filling module is used to fill the target log information according to the associated log information, so that the target log information includes the attribute information, and to store the filled target log information.

5. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is used to store computer-executed instructions; The processor is used to invoke the computer to execute instructions so that the electronic device performs the method of any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 3.

7. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 3.

Images