Log processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a log processing method, apparatus, electronic device, and storage medium. The method includes: receiving a target log sent by a client, wherein the target log is obtained by encoding an original log of a preset field type by the client based on a preset log transmission protocol, and the log structure of the target log includes a header area and a content area, the header area includes a routing field area, the routing field area is used to store storage attribute information of the log, and the content area is used to store log content; decoding the routing field area based on the preset log transmission protocol to obtain the storage attribute information of the target log, determining a storage address of the target log based on the storage attribute information of the target log, and storing the target log in the storage address. In this way, there is no need to decode the entire content of the target log to obtain the storage address, which helps reduce the overhead of obtaining the storage address.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a log processing method, device, electronic device, and storage medium. Background Art

[0002] Logs are records of events, activities, or status within an application or system. They are commonly used by developers and system administrators for troubleshooting, monitoring, and performance analysis. Logging and analysis strategies can accelerate troubleshooting and performance optimization.

[0003] In related technologies, the ProtoBuffer (PB) transmission protocol is usually used to encode logs and store the encoded logs on the server. However, during the transmission process, the server needs to decode the entire content of the encoded log to obtain the storage address and store the encoded log to the corresponding storage address. Due to the large amount of log data, the process of obtaining the storage address takes up a lot of overhead. Summary of the Invention

[0004] The embodiments of the present disclosure at least provide a log processing method, device, electronic device, and storage medium, which can reduce the overhead of a server in obtaining the storage address of a target log.

[0005] The present disclosure provides a log processing method, including:

[0006] Receive a target log sent by a client; the target log is obtained by encoding an original log of a preset field type by the client based on a preset log transmission protocol; the log structure of the target log includes a header area and a content area, the header area includes a routing field area, the routing field area is used to store storage attribute information of the log, and the content area is used to store log content;

[0007] Based on the preset log transmission protocol, the routing field area is decoded to obtain storage attribute information of the target log, and the storage address of the target log is determined based on the storage attribute information of the target log, and the target log is stored in the storage address.

[0008] In the disclosed embodiment, the log structure of the target log obtained by encoding includes a routing field area and a content area, wherein the routing field area is used to store the storage location of the log, and the content area is used to store the log content. In this way, after receiving the target log, the server can decode the routing field area to determine the storage address corresponding to the target log. Compared with the method of transmitting through the PB transmission protocol in the related art, the server does not need to decode the entire log to obtain the storage address of the target log. This is conducive to reducing the overhead occupied in the process of obtaining the storage address. In addition, since the space occupied by the routing field area is relatively small, the efficiency of obtaining the storage address can also be improved.

[0009] In a possible implementation, determining the storage address of the target log based on the storage attribute information of the target log includes:

[0010] Based on the storage attribute information of the target log, a target storage address corresponding to the storage attribute information of the target log is searched from a pre-established correspondence between storage attribute information and storage addresses, and the target storage address is determined as the storage address of the target log, wherein the target log is a log generated by the call of the target microservice run by the client, and the storage attribute information includes at least one of the name of the microservice, the host name corresponding to the microservice, and the host network address corresponding to the microservice.

[0011] In an embodiment of the present disclosure, the storage attribute information may be at least one of the name of the target microservice, the host name corresponding to the target microservice, or the host network address corresponding to the target microservice. The storage address of the target log is determined based on the correspondence between the preset storage attribute information and the storage address. In this way, the accuracy of determining the storage address of the target log can be improved.

[0012] In one possible implementation, the header area further includes an index field area, the index field area includes multiple index fields; the target log includes multiple sub-logs, each sub-log corresponds to at least one index field, and the method further includes:

[0013] For each sub-log, log attribute information of the sub-log is determined, and the log attribute information of the sub-log is stored in a corresponding index field; the log attribute information is used to query the corresponding sub-log.

[0014] In the embodiment of the present disclosure, since each sub-log has corresponding log attribute information, by storing the log attribute information of each sub-log in the corresponding index field, when querying, the index fields in the index field area can be traversed without querying the log content. In this way, the query scope can be reduced, thereby improving the query efficiency.

[0015] In a possible implementation, after determining the storage address of the target log based on the storage attribute information of the target log and storing the target log in the storage address, the method further includes:

[0016] Receive a log query request sent by a client, wherein the log query request carries log information to be queried;

[0017] Based on the log information to be queried, target log content corresponding to the log information to be queried is searched, and the target log content is sent to the client.

[0018] In the embodiment of the present disclosure, after receiving the log query request from the client, the server can search for the target log content corresponding to the log information to be queried based on the log query request, and return the target log content to the client, thereby realizing the query of the log content, and the user can intuitively view the target log content through the client.

[0019] In one possible implementation, the log information to be queried includes log attribute information to be queried or log content information to be queried; the header area further includes an index field area, the index field area includes multiple index fields, and the index fields are used to store log attribute information; and querying target log content corresponding to the log information to be queried based on the log information to be queried includes:

[0020] Based on the log attribute information to be queried, searching the index field area for target log attribute information that matches the log attribute information to be queried, and based on the target log attribute information, searching the storage address for target log content associated with the target log attribute information; or

[0021] Based on the log content information to be queried, target log content matching the log content information to be queried is searched from the storage address.

[0022] In the disclosed embodiments, the server can search for target log content using the two methods described above, thereby increasing the diversity of log content search methods and enhancing the user's log search experience. Furthermore, by matching the log attribute information to be queried with the target log attribute information stored in the index field area, index search is implemented, thereby improving search efficiency. Searching for target log content based on the log content information to be queried can also improve the accuracy of log content search.

[0023] In one possible implementation, the target log includes multiple sub-logs; and searching the storage address for target log content that matches the log content information to be queried based on the log content information to be queried includes:

[0024] Decoding the target log to obtain an original sub-log corresponding to each sub-log, where each original sub-log includes multiple log contents;

[0025] For each log content of each original sub-log, determine a pointer of each log content in the memory;

[0026] Based on the log content corresponding to each pointer, a target pointer matching the log content information to be queried is determined, and the target log content is searched from the storage address indicated by the target pointer.

[0027] In the embodiment of the present disclosure, since the decoding process of the target log is completed in the memory, after the log content of each original sub-log is obtained by decoding, the log content will be stored in the memory, and a pointer to the log content in the memory will be generated, so that the corresponding target log content can be found based on the pointer. Compared with the PB transmission protocol in the related art that needs to copy each log content (in byte form) into a string object for parsing, it is beneficial to save memory space and can also reduce decoding overhead.

[0028] In one possible implementation, the log structure of the target log also includes a protocol header, which is used to store protocol information, including the version number of the log transmission protocol, the log compression algorithm type, the log length, and protocol verification information; the protocol information includes at least one of the version number of the log transmission protocol, the log compression algorithm type, the log length, and the protocol verification information, wherein the version number of the log transmission protocol is used to characterize the version of the log transmission protocol used to generate the target log; the log compression algorithm type refers to the compression algorithm used to compress the original log; the log length is used to indicate the maximum data amount of the encoded log; and the protocol verification information is used to verify whether there are errors in the target log relative to the original log.

[0029] In the embodiment of the present disclosure, since the protocol header of the log transmission protocol includes metadata information such as the version number of the log transmission protocol, the log compression algorithm type, the log length, and the protocol verification information, the client and the server can correctly parse and process the log. Among them, the protocol verification information can verify whether the target log is damaged or tampered with during the transmission process between the client and the server, thereby ensuring the integrity and correctness of the log.

[0030] The present disclosure provides a log processing method, including:

[0031] Obtaining an original log and encoding the original log based on a preset log transmission protocol to obtain a target log; the log structure of the target log includes a header area and a content area, the header area includes a routing field area, the routing field area is used to store storage attribute information of the log, and the content area is used to store log content;

[0032] The target log is sent to a server, so that the server determines a storage address of the target log based on the storage attribute information in the routing field area, and stores the target log in the storage address.

[0033] In the embodiment of the present disclosure, since the log structure of the target log includes a routing field area and a content area, wherein the routing field area is used to store the storage attribute information of the log, and the content area is used to store the log content, in this way, after the client encodes the original log based on the log transmission protocol, it can obtain the target log and send the target log to the server. In this way, after receiving the target log, the server can decode the routing field area to determine the storage address corresponding to the target log. Compared with the method of transmitting through the PB transmission protocol in the related art, the server does not need to decode the entire log to obtain the storage address of the target log. This is conducive to reducing the overhead occupied in the process of obtaining the storage address. In addition, since the space occupied by the routing field area is small, the efficiency of obtaining the storage address can also be improved.

[0034] In one possible implementation, the original log includes multiple original sub-logs, each of which includes multiple key-value pairs, each of which includes a keyword and a key value. The encoding of the original log based on a preset log transmission protocol to obtain a target log includes:

[0035] For the plurality of original sub-logs, determining at least one identical original keyword existing in the plurality of original sub-logs;

[0036] For each identical original keyword, the original keyword is replaced with a target keyword to obtain multiple new original sub-logs, and the multiple new original sub-logs are encoded, wherein the byte length of the target keyword is smaller than the byte length of the original keyword.

[0037] In the disclosed embodiment, by replacing each identical original keyword in each original sub-log with a target keyword of smaller byte length, the storage space of the original log can be reduced, achieving the purpose of log compression, thereby improving encoding efficiency. In addition, when the target log is transmitted in the subsequent step, transmission bandwidth can be saved.

[0038] In one possible implementation, the method further includes:

[0039] In response to the query operation, a log query request is generated; the log query request carries the log information to be queried;

[0040] Sending the log query request to the server; the log query request is used to find the target log content corresponding to the log information to be queried;

[0041] Receive target log content corresponding to the log information to be queried, sent by the server.

[0042] In the embodiment of the present disclosure, the client can send a log query request to the server, so that the target log content corresponding to the log information to be queried carried by the log query request can be obtained from the server, so that the user can view the target log content, which facilitates problem locking and processing.

[0043] The present disclosure provides a log processing device, including:

[0044] A log receiving module is configured to receive a target log sent by a client; the target log is obtained by encoding an original log of a preset field type by the client based on a preset log transmission protocol; the log structure of the target log includes a header area and a content area, the header area includes a routing field area, the routing field area is used to store storage attribute information of the log, and the content area is used to store log content;

[0045] The log storage module is used to decode the routing field area based on the preset log transmission protocol to obtain storage attribute information of the target log, determine the storage address of the target log based on the storage attribute information of the target log, and store the target log in the storage address.

[0046] In a possible implementation, the log storage module is specifically configured to:

[0047] Based on the storage attribute information of the target log, a target storage address corresponding to the storage attribute information of the target log is searched from a pre-established correspondence between storage attribute information and storage addresses, and the target storage address is determined as the storage address of the target log, wherein the target log is a log generated by the call of the target microservice run by the client, and the storage attribute information includes at least one of the name of the microservice, the host name corresponding to the microservice, and the host network address corresponding to the microservice.

[0048] In one possible implementation, the header area further includes an index field area, the index field area includes multiple index fields; the target log includes multiple sub-logs, each sub-log corresponds to at least one index field, and the log storage module is further configured to:

[0049] For each sub-log, log attribute information of the sub-log is determined, and the log attribute information of the sub-log is stored in a corresponding index field; the log attribute information is used to query the corresponding sub-log.

[0050] In a possible implementation, the apparatus further includes a request receiving module, wherein the request receiving module is configured to:

[0051] Receive a log query request sent by a client, wherein the log query request carries log information to be queried;

[0052] Based on the log information to be queried, target log content corresponding to the log information to be queried is searched, and the target log content is sent to the client.

[0053] In one possible implementation, the log information to be queried includes log attribute information to be queried or log content information to be queried; the header area further includes an index field area, the index field area includes multiple index fields, and the index fields are used to store log attribute information; the request receiving module is specifically configured to:

[0054] Based on the log attribute information to be queried, searching the index field area for target log attribute information that matches the log attribute information to be queried, and based on the target log attribute information, searching the storage address for target log content associated with the target log attribute information; or

[0055] Based on the log content information to be queried, target log content matching the log content information to be queried is searched from the storage address.

[0056] In a possible implementation, the request receiving module is specifically configured to:

[0057] Decoding the target log to obtain an original sub-log corresponding to each sub-log, where each original sub-log includes multiple log contents;

[0058] For each log content of each original sub-log, the pointer of the log content in memory;

[0059] Based on the log content corresponding to each pointer, a target pointer matching the log content information to be queried is determined, and the target log content is searched from the storage address indicated by the target pointer.

[0060] In one possible implementation, the log structure of the target log also includes a protocol header, which is used to store protocol information. The protocol information includes at least one of the version number of the log transmission protocol, the log compression algorithm type, the log length, and protocol verification information. The version number of the log transmission protocol is used to characterize the version of the log transmission protocol used to generate the target log; the log compression algorithm type refers to the compression algorithm used to compress the original log; the log length is used to indicate the maximum amount of data in the encoded log; and the protocol verification information is used to verify whether there are errors in the target log relative to the original log.

[0061] The present disclosure also provides another log processing device, including:

[0062] A log acquisition module is configured to acquire an original log and, based on a preset log transmission protocol, encode the original log to obtain a target log; the log structure of the target log includes a header area and a content area, the header area includes a routing field area, the routing field area is used to store storage attribute information of the log, and the content area is used to store the log content;

[0063] The log sending module is used to send the target log to the server, so that the server determines the storage address of the target log based on the storage attribute information in the routing field area, and stores the target log in the storage address.

[0064] In a possible implementation, the original log includes multiple original sub-logs, each of which includes multiple key-value pairs, each of which includes a keyword and a key value; and the log acquisition module is specifically configured to:

[0065] For the plurality of original sub-logs, determining at least one identical original keyword existing in the plurality of original sub-logs;

[0066] For each identical original keyword, the original keyword is replaced with a target keyword to obtain multiple new original sub-logs, and the multiple new original sub-logs are encoded, wherein the byte length of the target keyword is smaller than the byte length of the original keyword.

[0067] In a possible implementation, the apparatus further includes a request sending module, wherein the request sending module is specifically configured to:

[0068] In response to the query operation, a log query request is generated; the log query request carries the log information to be queried;

[0069] Sending the log query request to the server; the log query request is used to find the target log content corresponding to the log information to be queried;

[0070] Receive target log content corresponding to the log information to be queried, sent by the server.

[0071] An embodiment of the present disclosure provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the log processing method described in any of the above embodiments is performed.

[0072] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the log processing method described in any of the above embodiments is executed.

[0073] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0075] Figure 1 A flow chart of a log processing method provided by an embodiment of the present disclosure is shown;

[0076] Figure 2 A schematic diagram showing the protocol structure of a log transmission protocol provided by an embodiment of the present disclosure;

[0077] Figure 3 A flow chart showing another log processing method provided by an embodiment of the present disclosure is shown;

[0078] Figure 4 A schematic diagram of the structure of a log processing device provided by an embodiment of the present disclosure is shown;

[0079] Figure 5 A schematic diagram showing the structure of another log processing device provided by an embodiment of the present disclosure is shown;

[0080] Figure 6 A schematic diagram showing the structure of another log processing device provided by an embodiment of the present disclosure is shown;

[0081] Figure 7 A schematic structural diagram of another log processing device provided by an embodiment of the present disclosure is shown;

[0082] Figure 8 A schematic diagram of an electronic device provided by some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0084] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0085] The term "and / or" herein simply describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, and the existence of B alone. In addition, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0086] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0087] Logs are records of events, activities, or status within an application or system. They are commonly used by developers and system administrators for troubleshooting, monitoring, and performance analysis. Logging and analysis strategies can expedite troubleshooting and performance optimization. To facilitate log management, logs are typically transferred to a server for storage. Due to the lengthy log transmission chain, after logs are generated on the client, they are sequentially transmitted through the server's gateway layer, consumption layer, and storage layer before being stored.

[0088] In related technologies, the ProtoBuffer (PB) transmission protocol is usually used to encode logs and store the encoded logs on the server. However, during the transmission process, this method requires decoding the entire content of the encoded log to obtain the storage address and store the encoded log to the corresponding storage address. Due to the large amount of log data, the decoding overhead consumed in the process of obtaining the storage address is relatively high.

[0089] Based on the above research, an embodiment of the present disclosure provides a log processing method, which first receives a target log sent by a client; the target log is obtained by the client encoding the original log of a preset field type based on a preset log transmission protocol; the log structure of the target log includes a header area and a content area, the header area includes a routing field area, the routing field area is used to store the storage attribute information of the log, and the content area is used to store the log content; based on the preset log transmission protocol, the routing field area is decoded to obtain the storage attribute information of the target log, and the storage address of the target log is determined based on the storage attribute information of the target log, and the target log is stored in the storage address. That is, in the embodiment of the present disclosure, when determining the storage address, it is only necessary to decode the routing field area, and there is no need to decode the entire content of the target log, which is beneficial to reducing the overhead occupied by the storage address acquisition process.

[0090] To facilitate understanding of this embodiment, the execution subject of the log processing method provided by the embodiment of the present disclosure is first introduced in detail. The execution subject of the log processing method provided by the embodiment of the present disclosure is a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data and artificial intelligence platforms. In other implementations, the log processing method can also be implemented by a processor calling computer-readable instructions stored in a memory.

[0091] The log processing method provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings. Figure 1FIG. 1 is a flow chart of a log processing method provided by an embodiment of the present disclosure, and the log processing method includes the following S101 to S102:

[0092] S101, receiving a target log sent by a client; the target log is obtained by encoding an original log of a preset field type based on a preset log transmission protocol; the log structure of the target log includes a header area and a content area, the header area includes a routing field area, the routing field area is used to store storage attribute information of the log, and the content area is used to store log content.

[0093] Here, the client generates the original log, encodes and compresses the original log to obtain the target log, and sends the target log to the server.

[0094] The target log may be a log obtained by encoding and compressing an original log of a preset field type based on a preset log transmission protocol. The original log may be a microservice log. Here, the microservice log refers to a log generated when the microservice is called. The microservice is a software development technology - a variant of the service-oriented architecture (SOA) architectural style, which divides a single application into many loosely coupled and independently deployable smaller components or services, and each microservice runs in its independent process.

[0095] For example, the client runs ten microservices, each microservice corresponds to one thousand logs, and the one thousand logs corresponding to each microservice are encoded to obtain ten target logs. The client will send these ten target logs to the server in the form of a log stream.

[0096] It should be understood that, since the original logs may be generated continuously, the client will encode and compress a preset number of original logs to obtain a target log. Therefore, the target log may include multiple sub-logs.

[0097] The log structure of the target log is described in detail below. Here, it should be noted that since the target log is obtained by encoding the original log based on the log transfer protocol, the log structure of the target log is similar to the protocol structure of the log transfer protocol. Therefore, in this embodiment, the preset log transfer protocol is first introduced in detail.

[0098] See Figure 2 , is a protocol structure diagram of a log transmission protocol provided by an embodiment of the present disclosure. Figure 2As shown in , the protocol structure of the log transmission protocol includes a protocol header (ProtoHeaders), a header area (HeaderRegion), a content area (LogContents) and a temporary area (PaddingRegion).

[0099] Among them, the protocol header (ProtoHeaders) is used to store protocol information, which includes the version number of the log transfer protocol, the log compression algorithm type, the log length, and protocol verification information. Here, it should be understood that the log transfer protocol can have multiple versions, and the version numbers of different versions are different from each other, such as version V1, version V2, etc., among which the versions of the log transfer protocol used by different clients may be different. In order to enable the server to decode the target log sent by the client, it is necessary to agree on the version number in the log transfer protocol.

[0100] The log compression algorithm type refers to the algorithm used to compress the original log. In this embodiment, the log compression algorithm can be a ZSTD (Zstandard) compression algorithm, wherein the ZSTD compression algorithm has the advantages of high compression ratio, high performance, adjustable compression rate, and support for streaming compression and decompression. In other embodiments, it can also be a lossless compression algorithm (such as LZ77 or LZ78 compression algorithm), a fast compression algorithm (such as Snappy compression algorithm or LZ4 compression algorithm), etc., which is not limited here.

[0101] The log length refers to the maximum amount of data that can be encoded by the transmission protocol. It should be understood that since logs are usually streaming, that is, logs are generated continuously, setting the log length can avoid the compressed original log having a large amount of data, occupying a large bandwidth and compression overhead.

[0102] The protocol verification information is used to verify the entire content of the protocol. Specifically, in the log transmission protocol, the verification information is used to verify the integrity of the data during the transmission process to ensure that the server can detect errors or tampering that may occur during the transmission process, that is, it can detect whether there are errors in the target log relative to the original log. In this embodiment, a cyclic redundancy check CRC32 can be used for verification. In other embodiments, other verification methods can also be used, such as verification through a checksum or verification based on a hash function, which is not limited here.

[0103] The header region (HeaderRegion) includes a dictionary field Pattern, a routing field area CommonHeaders, and an index field area LogHeaders. In this embodiment, to simplify the header region, the field length of the header region is no more than 256 bytes. In other embodiments, the field length of the header region can also be other lengths, which are not limited here.

[0104] The Pattern dictionary field is used to convert repeated strings in the protocol into dictionaries. Specifically, some strings have many bytes and appear frequently in logs. Therefore, they can be converted into dictionaries to replace the original strings with strings with fewer bytes. For example, if each original log has a "message" field, "message" can be converted into the number "1" in the dictionary. For another example, if each original log has a "Level" field, "level" can be converted into the number "2" in the dictionary. This improves the log compression rate.

[0105] The routing field area is used to store storage attribute information of the log, and the storage attribute information may include at least one of the name of the microservice (service-name), the host name (hostname) corresponding to the microservice, and the host network address (ip) corresponding to the microservice.

[0106] It should be understood that after the client sends the target log to the server, the server needs to store the target log. Therefore, different target logs can be stored based on the storage attribute information stored in the routing field area. In this way, when storing the target log in subsequent steps, there is no need to decode the entire content of the target log, only the routing field area needs to be decoded, which can improve decoding efficiency.

[0107] The routing field area may include multiple routing fields (also known as common fields), and the routing fields are used to store the storage attribute information of the log. For example, the content of routing field 1 is "service-name: 1". The routing fields in the routing field area may be pre-configured by the client. For example, client 1 may set the name of the microservice as the routing field, client 2 may set the host name as the routing field, and client 3 may also set the name of the microservice and the host name as routing fields.

[0108] Here, it should be noted that since microservices run on different hosts in a distributed structure, for example, microservice 1 runs on hosts A, B, and C, and microservice 2 runs on hosts A and B, then service 1 and service 2 have the same host names of A and B. If the host name is used as the routing field, the logs corresponding to different microservices can be stored in the same storage address.

[0109] The index field area LogHeaders includes multiple index fields, which are used to store attribute information of the log, wherein the attribute information may include a timestamp, a log identifier LogID, etc. In other implementations, the attribute information may also include a log level Log-level, etc., which is not limited here.

[0110] Here, the preset field type includes at least one of a string type (String), a numeric type (e.g., long, int, double), and a composite type (e.g., UUID, IPv4, IPv6), wherein a UUID (Universally Unique Identifier) ​​is a standardized 128-bit identifier used to track and identify log records, which has the advantages of high uniqueness and low duplication rate; IPv4 (Internet Protocol version 4) is the fourth version of the Internet Protocol and is the currently widely used IP address format. An IPv4 address consists of 32 bits of binary and is usually represented in dotted decimal notation. In logs, IPv4 addresses are often used to record information such as network connections, client requests, and server access; IPv6 (Internet Protocol version 6) is the sixth version of the Internet Protocol and is an upgraded version of IPv4. An IPv6 address consists of 128 bits of binary and is usually represented in colon-separated hexadecimal notation. With its larger address space and better network support, IPv6 addresses are gradually replacing IPv4 and becoming the foundation of the future Internet. In logs, IPv6 addresses can record information about network connections, IP traffic, and network devices. The above-mentioned preset field types can also be filled or reduced according to the actual needs of the client, which is not limited here. Since the log transmission protocol also supports at least one of the above-mentioned field types, this can improve the flexibility of log processing, provide log richness and scalability, adapt to various log processing needs, and provide a more reliable, scalable and easy-to-understand data transmission and processing mechanism.

[0111] The content area (LogContents) is used to store log content, which may include message or stack content, etc. Message refers to the actual text message or description in the log entry, and the stack information stack records the hierarchical relationship and location information of the function call, as well as the stack trace of the error when an error or exception occurs. Here, the content area includes multiple content fields. Since the log content usually exists in the form of key-value pairs, the key-value pairs of the log content can be stored in the content field. Since the data volume of the log content is large, there is no restriction on the field length of the content area.

[0112] The temporary area (PaddingRegion) includes at least one temporary field. In some implementations, the temporary field can be used for zero padding alignment when performing log storage.

[0113] Therefore, based on the above analysis, it can be seen that after the original log is encoded based on the above log transmission protocol, the target log can be obtained. Therefore, the target log will also include a protocol header, a header area, a content area and a temporary area. Therefore, the target log obtained by encoding based on the above log transmission protocol can include a header area and a content area, wherein the routing field area is used to store the storage attribute information of the target log, and the content area is used to store the log content of the target log.

[0114] S102: Based on the preset log transmission protocol, decode the routing field area to obtain storage attribute information of the target log, determine the storage address of the target log based on the storage attribute information of the target log, and store the target log in the storage address.

[0115] It should be understood that, as mentioned above, the routing field area is used to store the storage attribute information of the log. Therefore, the routing field area of ​​the target log will store the storage attribute information of the target log. After receiving the target log, since the target log is obtained by encoding the original log based on the preset log transmission protocol, the server needs to decode the routing field area based on the preset log transmission protocol. In this way, the storage attribute information of the target log can be obtained, and the storage address of the target log can be determined based on the storage attribute information, thereby storing the target log in the storage address. This is conducive to reducing the overhead occupied in the process of obtaining the storage address.

[0116] In addition, since the routing field area occupies a smaller space, the efficiency of obtaining the storage address can also be improved.

[0117] It should be noted that when the server stores the target log, it usually receives the target log sent by the client based on the server's gateway, and then the gateway performs fixed-point parsing on the routing field in the routing field area to obtain the storage attribute information of the target log.

[0118] In some embodiments, when determining the storage address of the target log based on the storage attribute information, the storage address of the target log can be determined based on the preset correspondence between the storage attribute information and the storage address. Specifically, based on the storage attribute information of the target log, the target storage address corresponding to the storage attribute information of the target log is searched from the pre-constructed correspondence between the storage attribute information and the storage address, and the target storage address is determined as the storage address of the target log.

[0119] That is, the client can pre-construct a correspondence between storage attribute information and storage addresses. For example, if the storage attribute information is the name of a microservice, the corresponding storage address can be pre-assigned for the microservice name. For example, the corresponding storage address for microservice 1 is A, and the corresponding storage address for microservice 2 is B. It should be noted that the correspondence between storage attribute information and storage addresses created by different clients can be the same or different. For example, client S1 creates a correspondence based on the microservice name, and client S2 creates a correspondence based on the host name of the host where the microservice is located. Client S2 can also create a correspondence based on both the microservice name and the host name of the host where the microservice is located.

[0120] It should be understood that since the routing field area can only be used to determine the storage address where the target log is stored, and since the user may need to query the target log after the target log is stored on the server, in some embodiments, the log attribute information of each sub-log in the target log can be determined, and the log attribute information of the sub-log can be stored in the corresponding index field. In this way, the client can query the log content of the corresponding sub-log based on the log attribute information.

[0121] Here, since each sub-log is independent of each other, each sub-log has corresponding attribute information, where the attribute information may include timestamp, log identifier, and other information. The attribute information may be in the form of a key-value pair. For example, the attribute information of sub-log 1 is Timestamp: 2023-08-14 14:00:00 or LogID: 1234; the attribute information of sub-log 2 is Timestamp: 2023-08-14 14:01:00 or LogID: 1237.

[0122] Specifically, the server can receive a log query request sent by any client, wherein the log query request carries the log information to be queried, and then based on the log information to be queried, determine the target log content corresponding to the log information to be queried, and send the target log content to the client.

[0123] The log information to be queried carried in the log query request may refer to the log information that the client needs to query, for example, may be a timestamp or a log identifier.

[0124] It can be understood that the client side can respond to the user's query operation, input a log query request through the client, and send the log query request to the server. In this way, after receiving the log query request from the client, the server can determine the target log content corresponding to the log information to be queried based on the log information to be queried in the log query request. Here, since the log information to be queried input by different clients is different, the content of the determined target log content is also different. In this way, after determining the target log content, the target log content can be sent to the client.

[0125] In some embodiments, since the log information to be queried may include log attribute information to be queried, when querying the target log content corresponding to the log information to be queried based on the log information to be queried, the target log attribute information matching the log attribute information to be queried can be searched from the index field area based on the log attribute information to be queried, and the target log content associated with the target log attribute information can be searched from the storage address based on the target log attribute information.

[0126] Here, the log attribute information of the target log and the storage address are pre-associated.

[0127] It is understandable that after the log attribute information of each sub-log is stored in the index field in the index field area, the user can query the log content of any sub-log based on the log attribute information to be queried, wherein the log attribute information to be queried can be one of the log attribute information, or it can be not included in the log attribute information. Here, if the log attribute information to be queried is not included in the log attribute information, the server will return a null value. For example, the user can enter a query statement (query expression) or query condition for the timestamp Timestamp, so that the server returns the corresponding log content according to the entered query statement or query condition. That is, the speed and efficiency of log queries can be improved by using the log information to be queried, and the query time can be reduced.

[0128] In other embodiments, the log information to be queried may also include log content information to be queried. Therefore, when querying the target log content corresponding to the log information to be queried based on the log information to be queried, it is also possible to search for the target log content matching the log content information to be queried from the storage address based on the log content information to be queried, and send the target log content to the client.

[0129] Similar to the aforementioned implementation, users can also directly query log content. Specifically, users can input the log content information to be queried on the client to generate a log query request and send the log query request to the server. At this time, the server will determine the target log to be searched based on the log content information to be queried carried in the log query request, and then decode and traverse the entire target log to obtain the target log content that matches the log content information to be queried, and return the target log content to the client.

[0130] Here, when searching for target log content that matches the log content information to be queried from the storage address based on the log content information to be queried, the target log can be first decoded in the memory to obtain the original sub-log corresponding to each sub-log, wherein each original sub-log includes multiple log contents, and for each log content of each original sub-log, the pointer of each log content in the memory is determined, and then, based on the log content corresponding to each pointer, the target pointer that matches the log content information to be queried is determined, and the target log content is searched from the storage address indicated by the target pointer.

[0131] Here, the process of decoding the target log is implemented in memory. Therefore, after obtaining the log content of the original sub-log corresponding to each sub-log, the log content is stored in the memory space, and for each log content, a pointer to the log content in the memory is determined. In this way, based on the log content corresponding to each pointer, a target pointer matching the log content information to be queried can be determined, and the corresponding target log content can be searched from the storage address indicated by the target pointer. In the encoding rules of the PB transmission protocol, string type fields are encoded in byte form. When parsing byte data and converting it to a string type, the PB transmission protocol parses by copying the byte data into a new String object. When frequently parsing a large number of strings, a new String object must be created each time, which may cause gaps in the memory and gradually increase the degree of memory fragmentation. On the other hand, this copy operation may affect parsing performance. For a large number of string copy operations, especially when frequently parsing a large number of strings, the operation of copying the entire string involves memory allocation, data duplication, and related memory management overhead. Therefore, compared with the PB transmission protocol, this embodiment can improve decoding efficiency.

[0132] See Figure 3 , which is a flowchart of another log processing method provided by an embodiment of the present disclosure. This log processing method is applied to a client, wherein the client runs at least one microservice, wherein the client can run on a terminal device, and the terminal device can also be a mobile device, user terminal, terminal, handheld device, computing device, vehicle-mounted device, wearable device, etc., without limitation herein.

[0133] like Figure 3 As shown in , the log processing method includes the following steps S301-S302:

[0134] S301, obtaining an original log, and encoding the original log based on a preset log transmission protocol to obtain a target log; the log structure of the target log includes a header area and a content area, the header area includes a routing field area, the routing field area is used to store storage attribute information of the log, and the content area is used to store log content.

[0135] Here, the original log may refer to a log generated when the microservice running on the client is called by the application program, and the original log includes multiple original sub-logs.

[0136] It should be understood that after the client generates the original log, the original log can be encoded based on the preset log transmission protocol to obtain the target log. Here, the relevant description of the log transmission protocol and the target log has been explained in the above embodiment and will not be repeated here.

[0137] Optionally, when encoding the original log, encoding can be performed in different modes, which may include High-Level mode and Low-level mode. The High-Level mode adds each key-value pair through a preset interface and calls the Serialize interface to serialize each key-value pair into continuous memory; the Low-level mode completes the log encoding by appending the key-value pair to the buffer and finally calling the Finish interface to fill in the protocol header information.

[0138] S302: Send the target log to a server, so that the server determines a storage address of the target log based on the storage attribute information in the routing field area, and stores the target log in the storage address.

[0139] It can be understood that after obtaining the encoded target log, the target log can be sent to the server for storage, so that the target log can be managed uniformly. Here, the storage process of the server storing the target log can be referred to the description in the above embodiment and will not be repeated here.

[0140] In some embodiments, since the original log includes multiple original sub-logs, each original sub-log includes multiple key-value pairs, and the multiple original sub-logs usually contain a large number of repeated keywords, such as the keyword message, the keyword timestamp, etc., during encoding, these identical keywords will be encoded multiple times, which requires a large overhead. Based on this, in this embodiment, when the original log is encoded based on a preset log transmission protocol to obtain a target log, for the multiple original sub-logs, at least one identical original keyword existing in the multiple original sub-logs is determined, and for each identical original keyword, the original keyword is replaced with the target keyword to obtain multiple new original sub-logs, and the new multiple original sub-logs are encoded, wherein the byte length of the target keyword is less than the byte length of the identical keyword.

[0141] For each identical original keyword, the corresponding target keyword is unique, and the target keyword can be in English or in numbers, which is not limited here.

[0142] For example, similar to the keyword "message", its byte length is 7, it can be replaced by the target keyword "a". It can be seen that the byte length of "a" is 1. For another example, the keyword "LogID" can be replaced by the target keyword "b". In this way, the byte length is changed from 5 to 1. In this way, the storage space occupied by the original log can be reduced, and the overhead and bandwidth occupied during the encoding process can also be reduced.

[0143] Here, the header area of ​​the transmission protocol includes a Pattern field, which can store the original keyword and the target keyword in the dictionary, so that the keyword can be converted into a dictionary.

[0144] Optionally, after storing the target log on the server, the user can query the target log stored on the server based on the client. Specifically, a log query request can be generated in response to the query operation, wherein the log query request carries the log information to be queried. Then the client can send the log query request to the server. Since the log query request carries the log information to be queried, the server can search for log content corresponding to the log information to be queried based on the log information to be queried.

[0145] Here, since log attribute information is stored in the index field in the protocol structure of the log transmission protocol in this application, and log content information is stored in the content area, the log information to be queried may include the log attribute information to be queried or the log content information to be queried. In this way, the user can perform log query based on the log attribute information to be queried or the log content information to be queried, thereby improving the diversity of query methods.

[0146] For example, the query is performed using the log attribute information LogID to be queried, and the corresponding query statement can be in the following form:

[0147] select * from table_name

[0148] where LogID = '123456'

[0149] limit 1000;

[0150] Based on the above query statement, you can find the log content with LogID "123456" in the first 1000 logs.

[0151] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0152] Based on the same technical concept, the embodiment of the present disclosure also provides a log processing device corresponding to the log processing method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned log processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0153] Reference Figure 4 FIG. 4 is a schematic diagram of a log processing device provided by an embodiment of the present disclosure, wherein the device 400 includes:

[0154] The log receiving module 401 is configured to receive a target log sent by a client; the target log is obtained by encoding an original log of a preset field type by the client based on a preset log transmission protocol; the log structure of the target log includes a header area and a content area, the header area includes a routing field area, the routing field area is used to store storage attribute information of the log, and the content area is used to store the log content;

[0155] The log storage module 402 is used to decode the routing field area based on the preset log transmission protocol to obtain the storage attribute information of the target log, determine the storage address of the target log based on the storage attribute information of the target log, and store the target log in the storage address.

[0156] In a possible implementation, the log storage module 402 is specifically configured to:

[0157] Based on the storage attribute information of the target log, a target storage address corresponding to the storage attribute information of the target log is searched from a pre-established correspondence between storage attribute information and storage addresses, and the target storage address is determined as the storage address of the target log, wherein the target log is a log generated by the call of the target microservice run by the client, and the storage attribute information includes at least one of the name of the microservice, the host name corresponding to the microservice, and the host network address corresponding to the microservice.

[0158] In one possible implementation, the header area of ​​the log transmission protocol further includes an index field area, the index field area includes multiple index fields; the target log includes multiple sub-logs, each sub-log corresponds to at least one index field, and the log storage module 402 is further configured to:

[0159] For each sub-log, log attribute information of the sub-log is determined, and the log attribute information of the sub-log is stored in a corresponding index field; the log attribute information is used to query the corresponding sub-log.

[0160] Reference Figure 5 FIG. 4 is a schematic diagram of another log processing device provided by an embodiment of the present disclosure. The device 400 further includes a request receiving module 403. The request receiving module 403 is configured to:

[0161] Receive a log query request sent by a client, wherein the log query request carries log information to be queried;

[0162] Based on the log information to be queried, target log content corresponding to the log information to be queried is searched, and the target log content is sent to the client.

[0163] In one possible implementation, the log information to be queried includes log attribute information to be queried or log content information to be queried, the header area further includes an index field area, the index field area includes multiple index fields, and the index fields are used to store log attribute information; the request receiving module 403 is specifically used to:

[0164] Based on the log attribute information to be queried, searching the index field area for target log attribute information that matches the log attribute information to be queried, and based on the target log attribute information, searching the storage address for target log content associated with the target log attribute information; or

[0165] Based on the log content information to be queried, target log content matching the log content information to be queried is searched from the storage address.

[0166] In a possible implementation, the target log includes multiple sub-logs; the request receiving module 403 is specifically configured to:

[0167] Decoding the target log to obtain an original sub-log corresponding to each sub-log, where each original sub-log includes multiple log contents;

[0168] For each log content of each original sub-log, determine a pointer of each log content in the memory;

[0169] Based on the log content corresponding to each pointer, a target pointer matching the log content information to be queried is determined, and the target log content is searched from the storage address indicated by the target pointer.

[0170] In one possible implementation, the log structure of the target log also includes a protocol header, which is used to store protocol information. The protocol information includes at least one of the version number of the log transmission protocol, the log compression algorithm type, the log length, and protocol verification information. The version number of the log transmission protocol is used to characterize the version of the log transmission protocol used to generate the target log; the log compression algorithm type refers to the compression algorithm used to compress the original log; the log length is used to indicate the maximum amount of data in the encoded log; and the protocol verification information is used to verify whether there are errors in the target log relative to the original log.

[0171] See also Figure 6FIG. 6 is a schematic diagram of another log processing device provided by an embodiment of the present disclosure, wherein the device 600 includes:

[0172] The log acquisition module 601 is configured to acquire an original log and encode the original log based on a preset log transmission protocol to obtain a target log. The log structure of the target log includes a header area and a content area. The header area includes a routing field area for storing log storage attribute information, and the content area is for storing log content.

[0173] The log sending module 602 is configured to send the target log to a server, so that the server determines a storage address of the target log based on the storage attribute information in the routing field area, and stores the target log in the storage address.

[0174] In a possible implementation, the original log includes multiple original sub-logs, each of which includes multiple key-value pairs, each of which includes a keyword and a key value. The log acquisition module 601 is specifically configured to:

[0175] For the plurality of original sub-logs, determining at least one identical original keyword existing in the plurality of original sub-logs;

[0176] For each identical original keyword, the original keyword is replaced with a target keyword to obtain multiple new original sub-logs, and the multiple new original sub-logs are encoded, wherein the byte length of the target keyword is smaller than the byte length of the original keyword.

[0177] See also Figure 7 FIG. 6 is a schematic diagram of another log processing device provided by an embodiment of the present disclosure. The device 600 further includes a request sending module 603. The request sending module 603 is specifically configured to:

[0178] In response to the query operation, a log query request is generated; the log query request carries the log information to be queried;

[0179] Sending the log query request to the server; the log query request is used to find the target log content corresponding to the log information to be queried;

[0180] Receive target log content corresponding to the log information to be queried, sent by the server.

[0181] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0182] Based on the same technical concept, the embodiment of the present disclosure also provides an electronic device. Figure 8 8 is a schematic diagram of the structure of an electronic device 800 provided in an embodiment of the present disclosure, including a processor 801, a memory 802, and a bus 803. The memory 802 is used to store execution instructions and includes a memory 8021 and an external memory 8022. The memory 8021 is also referred to as internal memory and is used to temporarily store operation data in the processor 801 and data exchanged with an external memory 8022 such as a hard disk. The processor 801 exchanges data with the external memory 8022 through the memory 8021.

[0183] In the embodiment of the present application, the memory 802 is specifically used to store application code for executing the solution of the present application, and the execution is controlled by the processor 801. That is, when the electronic device 800 is running, the processor 801 communicates with the memory 802 via the bus 803, so that the processor 801 executes the application code stored in the memory 802, thereby performing the method described in any of the aforementioned embodiments.

[0184] Among them, the memory 802 can be, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0185] The processor 801 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0186] It should be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 800. In other embodiments of the present application, the electronic device 800 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0187] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program executes the steps of the log processing method in the above method embodiment. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0188] The present disclosure also provides a computer program product that carries program code. The program code includes instructions that can be used to execute the steps of the log processing method in the above method embodiment. For details, please refer to the above method embodiment and will not be repeated here.

[0189] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0190] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0191] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0192] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0193] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0194] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. A log processing method, characterized in that: include: Receive a target log sent by a client; the target log is obtained by encoding an original log of a preset field type by the client based on a preset log transmission protocol; the log structure of the target log includes a header area and a content area, the header area includes a routing field area, the routing field area is used to store storage attribute information of the log, and the content area is used to store log content; Based on the preset log transmission protocol, the routing field area is decoded to obtain the storage attribute information of the target log, and the storage address of the target log is determined based on the storage attribute information of the target log, and the target log is stored in the storage address. The step of determining the storage address of the target log based on the storage attribute information of the target log includes: Based on the storage attribute information of the target log, a target storage address corresponding to the storage attribute information of the target log is searched from a pre-established correspondence between storage attribute information and storage addresses, and the target storage address is determined as the storage address of the target log, wherein the target log is a log generated by the call of the target microservice run by the client, and the storage attribute information includes at least one of the name of the microservice, the host name corresponding to the microservice, and the host network address corresponding to the microservice.

2. The method according to claim 1, characterized in that The header area further includes an index field area, the index field area includes a plurality of index fields; the target log includes a plurality of sub-logs, each sub-log corresponds to at least one index field, and the method further includes: For each sub-log, log attribute information of the sub-log is determined, and the log attribute information of the sub-log is stored in a corresponding index field; the log attribute information is used to query the corresponding sub-log.

3. The method according to claim 1, characterized in that After determining the storage address of the target log based on the storage attribute information of the target log and storing the target log in the storage address, the method further includes: Receive a log query request sent by the client, wherein the log query request carries log information to be queried; Based on the log information to be queried, target log content corresponding to the log information to be queried is searched, and the target log content is sent to the client.

4. The method according to claim 3, characterized in that The log information to be queried includes log attribute information to be queried or log content information to be queried; the header area also includes an index field area, the index field area includes a plurality of index fields, and the index fields are used to store log attribute information; The searching, based on the log information to be queried, for target log content corresponding to the log information to be queried includes: Based on the log attribute information to be queried, searching the index field area for target log attribute information that matches the log attribute information to be queried, and based on the target log attribute information, searching the storage address for target log content associated with the target log attribute information; or Based on the log content information to be queried, target log content matching the log content information to be queried is searched from the storage address.

5. The method according to claim 4, characterized in that The target log includes multiple sub-logs; and searching the storage address for target log content that matches the log content information to be queried based on the log content information to be queried includes: Decoding the target log to obtain an original sub-log corresponding to each sub-log, where each original sub-log includes multiple log contents; For each log content of each original sub-log, determine a pointer of each log content in the memory; Based on the log content corresponding to each pointer, a target pointer matching the log content information to be queried is determined, and the target log content is searched from the storage address indicated by the target pointer.

6. The method according to any one of claims 1 to 5, characterized in that: The log structure of the target log also includes a protocol header, which is used to store protocol information. The protocol information includes at least one of the version number of the log transmission protocol, the log compression algorithm type, the log length, and protocol verification information. The version number of the log transmission protocol is used to characterize the version of the log transmission protocol used to generate the target log; the log compression algorithm type refers to the compression algorithm used to compress the original log; the log length is used to indicate the maximum data volume of the encoded log; and the protocol verification information is used to verify whether there are errors in the target log relative to the original log.

7. A log processing method, characterized in that: include: Obtaining an original log and encoding the original log based on a preset log transmission protocol to obtain a target log; the log structure of the target log includes a header area and a content area, the header area includes a routing field area, the routing field area is used to store storage attribute information of the log, and the content area is used to store log content; Sending the target log to a server so that the server determines a storage address of the target log based on the storage attribute information in the routing field area, and stores the target log in the storage address. The step of determining the storage address of the target log based on the storage attribute information of the routing field area includes: Based on the storage attribute information of the routing field area, a target storage address corresponding to the storage attribute information of the routing field area is searched from a pre-constructed correspondence between storage attribute information and storage addresses, and the target storage address is determined as the storage address of the target log, wherein the target log is a log generated by the call of the target microservice running on the client, and the storage attribute information includes at least one of the name of the microservice, the host name corresponding to the microservice, and the host network address corresponding to the microservice.

8. The method according to claim 7, characterized in that The original log includes multiple original sub-logs, each of which includes multiple key-value pairs, each of which includes a keyword and a key value. The encoding of the original log based on a preset log transmission protocol to obtain a target log includes: For the plurality of original sub-logs, determining at least one identical original keyword existing in the plurality of original sub-logs; For each identical original keyword, the original keyword is replaced with a target keyword to obtain multiple new original sub-logs, and the multiple new original sub-logs are encoded, wherein the byte length of the target keyword is smaller than the byte length of the original keyword.

9. The method according to claim 7 or 8, characterized in that The method further comprises: In response to the query operation, a log query request is generated; the log query request carries the log information to be queried; Sending the log query request to the server; the log query request is used to find the target log content corresponding to the log information to be queried; Receive target log content corresponding to the log information to be queried, sent by the server.

10. A log processing device, characterized in that: include: A log receiving module is configured to receive a target log sent by a client; the target log is obtained by encoding an original log of a preset field type by the client based on a preset log transmission protocol; the log structure of the target log includes a header area and a content area, the header area includes a routing field area, the routing field area is used to store storage attribute information of the log, and the content area is used to store log content; A log storage module is configured to decode the routing field area based on the preset log transmission protocol to obtain storage attribute information of the target log, determine the storage address of the target log based on the target log storage attribute information, and store the target log in the storage address. The step of determining the storage address of the target log based on the storage attribute information of the target log includes: Based on the storage attribute information of the target log, a target storage address corresponding to the storage attribute information of the target log is searched from a pre-established correspondence between storage attribute information and storage addresses, and the target storage address is determined as the storage address of the target log, wherein the target log is a log generated by the call of the target microservice run by the client, and the storage attribute information includes at least one of the name of the microservice, the host name corresponding to the microservice, and the host network address corresponding to the microservice.

11. A log processing device, characterized in that: include: A log acquisition module is used to obtain original logs and encode the original logs based on a preset log transmission protocol to obtain target logs; The log structure of the target log includes a header area and a content area, the header area includes a routing field area, the routing field area is used to store storage attribute information of the log, and the content area is used to store log content; a log sending module, configured to send the target log to a server, so that the server determines a storage address of the target log based on the storage attribute information stored in the routing field area, and stores the target log in the storage address; The step of determining the storage address of the target log based on the storage attribute information of the routing field area includes: Based on the storage attribute information of the routing field area, a target storage address corresponding to the storage attribute information of the routing field area is searched from a pre-constructed correspondence between storage attribute information and storage addresses, and the target storage address is determined as the storage address of the target log, wherein the target log is a log generated by the call of the target microservice running on the client, and the storage attribute information includes at least one of the name of the microservice, the host name corresponding to the microservice, and the host network address corresponding to the microservice.

12. An electronic device, characterized in that: include: A processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the steps of the log processing method described in any one of claims 1-6 or 7-9 are performed.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the log processing method according to any one of claims 1 to 6 or 7 to 9.

Images