Log sending method, log receiving method, log sending end and log receiving end

Abstract

The invention relates to the technical field of communication, and discloses a log sending method, a log receiving method, a log sending end and a log receiving end. The log sending method comprises the steps that an original log is converted into a compressed log based on a target log template, the target log template is predefined by a log sending end and a log receiving end and comprises structured data keys in the original log, the original log comprises structured data key value pairs, and the compressed log comprises a target template ID of the target log template; the structured data values are in one-to-one correspondence with the structured data keys in the target log template; the at least one compressed log is aggregated and packaged, a packaged compressed log is generated, and the packaged compressed log comprises a unique identifier of the packaged compressed log; and sending the packaged and compressed log to a log receiving end through a connectionless transmission protocol. According to the technical scheme, the network bandwidth in the log transmission process is saved, the waste of storage resources is avoided, and the system performance overhead is reduced.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a log sending and receiving method, a log sending and receiving terminal. Background Technology

[0002] In cloud computing, industrial networks, the Internet of Things (IoT), and microservice architectures, log data volume is growing exponentially. However, for current log information based on the RFC (Request for Comments) 5424 standard, the structured data portion requires recording complete key-value pairs, resulting in severe structural redundancy. For example, a typical network firewall log might have the structured data formatted as [traffic src_ip="1.2.3.4" dst_ip="5.6.7.8" src_port="12345" dst_port="443" proto="tcp" action="allow"]. In this example, the total string length of the descriptive keys (src_ip, dst_ip, etc.) is approximately 50 bytes, while the actual values ​​(IP address, port, etc.) are approximately 35 bytes long. This means that over 58% of the load consists of duplicate keys. In a large network that generates billions of such logs daily, this redundancy leads to a staggering waste of network bandwidth and storage resources. Furthermore, to ensure reliable data transmission, TCP / TLS (Transmission Control Protocol / Transport Layer Security) is often used. However, the establishment (three-way handshake) and teardown (four-way handshake) of TCP connections impose significant performance overhead on massive, short-connection log sources (such as containers and serverless functions). In addition, maintaining the TCP connection state consumes a large amount of memory and file handle resources on the receiving end, limiting the receiving capacity of a single log collector.

[0003] Therefore, to address the above shortcomings, it is necessary to provide a log sending and receiving method, as well as a log sending and receiving terminal. Summary of the Invention

[0004] The technical problem to be solved by this invention is how to save network bandwidth, avoid waste of storage resources and reduce system performance overhead when log transmission is required. In view of the defects in the prior art, this invention provides a log sending and receiving method, and a log sending and receiving terminal.

[0005] To address the aforementioned technical problems, this invention provides a log sending method, executed by a log sending end, comprising: Based on the target log template, the original log is converted into a compressed log. The target log template is predefined by the log sending end and the log receiving end, and includes the structured data keys in the original log. The original log includes structured data key-value pairs, and the compressed log includes the target template ID of the target log template, as well as structured data values ​​that correspond one-to-one with the structured data keys in the target log template. At least one compressed log is aggregated and encapsulated to generate an encapsulated compressed log, wherein the encapsulated compressed log includes a unique identifier for the encapsulated compressed log. The encapsulated and compressed log is sent to the log receiving end via a connectionless transmission protocol.

[0006] The present invention also provides a log receiving method, executed by a log receiving end, comprising: The receiver receives encapsulated and compressed logs sent by the log sender via a connectionless transmission protocol. The encapsulated and compressed logs are generated by the log sender by aggregating and encapsulating at least one compressed log. The encapsulated and compressed logs include a unique identifier. The compressed logs are generated by the log sender by converting the original logs based on a target log template. The target log template is predefined by the log receiver and the log sender and includes structured data keys from the original logs. The original logs include structured data key-value pairs. The compressed logs include the target template ID of the target log template and structured data values ​​that correspond one-to-one with the structured data keys in the target log template. The encapsulated and compressed log is parsed to obtain each compressed log entry in the encapsulated and compressed log; For each compressed log entry in the encapsulated compressed log, the target log template is determined using the target template ID. The structured data key corresponding to the original log is determined from the target log template, and the structured data values ​​in the compressed log are matched one-to-one with the structured data key to obtain the corresponding original log.

[0007] The present invention also provides a log sending terminal, comprising: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the log sending method as described in any of the preceding claims.

[0008] The present invention also provides a log receiver, comprising: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the log receiving method as described in any of the preceding claims.

[0009] The log processing method and log interaction processing method of the present invention have the following beneficial effects: By introducing a target log template that includes structured data keys from the original log, the original log is compressed into a compressed log containing the target log template ID and structured data values. This significantly reduces structural redundancy in the log information, thereby saving network bandwidth during log transmission and avoiding waste of storage resources. By aggregating and encapsulating the compressed log and using batch processing, the number of network packets and system calls is reduced, further saving network bandwidth. Furthermore, using a connectionless transmission protocol for the encapsulated compressed log offers high throughput and low latency, while also reducing system performance overhead. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating a log sending method provided in an embodiment of the present invention; Figure 2 This is a message structure diagram of a raw log provided in an embodiment of the present invention; Figure 3 This is a message structure diagram of a compressed log provided in an embodiment of the present invention; Figure 4 This is a message structure diagram of a packaged and compressed log provided in an embodiment of the present invention; Figure 5 This is an interactive diagram illustrating a log successful transmission and log retransmission mechanism provided in an embodiment of the present invention; Figure 6 This is a flowchart illustrating a log receiving method provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a log sending device provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of a log receiving device provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of a log sending end provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the structure of a log receiving end provided in an embodiment of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] Figure 1 This is a flowchart illustrating a log sending method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the log sending method is executed by the log sending end and includes: S110. Based on the target log template, convert the original log into a compressed log, wherein the target log template is predefined by the log sending end and the log receiving end, and includes the structured data keys in the original log. The original log includes structured data key-value pairs, and the compressed log includes the target template ID of the target log template, and structured data values ​​that correspond one-to-one with the structured data keys in the target log template.

[0013] Preferably, the log in this embodiment is a system log using the existing standard system log protocol RFC5424. The RFC5424 standard introduces a STRUCTURED-DATA field in the system log, meaning that each system log (including the original log in this embodiment) includes structured data key-value pairs. Figure 2 This is a message structure diagram of a raw log provided in an embodiment of the present invention, such as... Figure 2 As shown, the raw log includes a message header, structured data keys (including key1, key2, ... and keyN) and corresponding structured data values ​​(including value1, value2, ... and valueN).

[0014] Log templates are pre-negotiated and defined by the log sender and receiver. The negotiation process supports out-of-band configuration or dynamic in-band negotiation using specific network information. Each log template includes a target template ID (Identitydocument) and template content. The template ID is the identity identifier of the corresponding log template, used to uniquely identify the corresponding log template. Each log template corresponds to a type of system log, and its template content includes the structured data keys of the corresponding type of system log. For example, the template content can be an ordered list of structured data keys.

[0015] Understandably, in this embodiment, the target log template is the log template corresponding to the structured data keys in the original log. Since the target log template maps the structured data keys in the original log to the target template ID, the compressed log can include the target template ID and structured data values ​​that correspond one-to-one with the structured data keys in the target log template. Preferably, the compressed log statement is expressed as [sd-id@template-id value1 value2 ...valueN], where sd-id is the ID corresponding to the predefined structured data key, @template-id is the target template ID, and value1, value2, ..., valueN are N structured data values ​​that correspond one-to-one with the structured data keys in the target log template. Through the [sd-id@template-id......] syntax design, the parsing of other elements by the traditional log parser is not affected, maintaining good forward compatibility. Figure 3 This is a message structure diagram of a compressed log provided in an embodiment of the present invention, such as... Figure 3 As shown, the compressed log includes a message header (including the target template ID) and structured data values ​​(including value 1, value 2, ... and value N) that correspond one-to-one with the structured data keys in the target log template.

[0016] The following example, using a high-traffic Web Application Firewall (WAF) log reporting scenario, illustrates the above-mentioned log compression and transformation process. The log sender is the WAF, and the log receiver is the central SIEM (Security Information Event Management) platform.

[0017] When the firewall (WAF) and the central SIEM platform start, they both load the shared templates.json configuration file. This templates.json configuration file is the target log template. The following is the specific content of the templates.json configuration file.

[0018] JSON { "templates": [ { "id": 1001, "name": "WAF_BLOCK_LOG", "keys": ["client_ip", "host", "rule_id", "attack_type", "user_agent"] } ]} When the firewall (WAF) blocked an SQL injection attack, the following information was captured: Client IP: 203.0.113.10, Host: api.example.com, Rule ID: 942100, Attack Type: SQLInjection, User-Agent: sqlmap / 1.5. The structured data portion of the original log was [waf@... client_ip="203.0.113.10" host="api.example.com" rule_id="942100" attack_type="SQLInjection" user_agent="sqlmap / 1.5"]. Using the technical solution of this invention, the firewall (WAF) searches the template library and matches the target log template templates.json with target template ID 1001. A compressed log is generated, whose structured data portion is: [w@1001 "203.0.113.10" "api.example.com" "942100" "SQL Injection" "sqlmap / 1.5"], where w is the ID (i.e., sd-id) corresponding to a predefined structured data key. Compared with the original log, the compressed log's data volume is significantly reduced.

[0019] S120. At least one compressed log is aggregated and encapsulated to generate an encapsulated compressed log, wherein the encapsulated compressed log includes a unique identifier for the encapsulated compressed log.

[0020] For example, the encapsulated compressed logs can be in the form of UDP (User Datagram Protocol) packets, and the number of compressed logs encapsulated can be a preset number, for example, 20 logs. A unique identifier for each encapsulated compressed log is used to uniquely identify it; this unique identifier can be a sequence number, for example, 15987.

[0021] S130. The encapsulated and compressed log is sent to the log receiving end via a connectionless transmission protocol.

[0022] Preferably, the connectionless transmission protocol in this embodiment is the UDP transmission protocol.

[0023] This invention proposes a log sending method, executed by a log sending end, comprising: converting raw logs into compressed logs based on a target log template, wherein the target log template is predefined by the log sending end and the log receiving end, and includes structured data keys from the raw log, the raw log including structured data key-value pairs, and the compressed log including a target template ID of the target log template, and structured data values ​​corresponding one-to-one with the structured data keys in the target log template; aggregating and encapsulating at least one compressed log to generate an encapsulated compressed log, the encapsulated compressed log including a unique identifier; and sending the encapsulated compressed log to the log receiving end via a connectionless transmission protocol. By introducing a target log template including structured data keys from the raw log, the raw log is compressed into a compressed log including the target log template ID and structured data values, significantly reducing the structural redundancy of log information, thereby saving network bandwidth during log transmission and avoiding waste of storage resources. By aggregating and encapsulating the compressed logs and using batch processing of compressed logs, the number of network data packets and system calls is reduced, further saving network bandwidth. Furthermore, using a connectionless transmission protocol to transmit encapsulated and compressed logs offers high throughput and low latency, while also reducing system performance overhead.

[0024] Based on the above embodiments, further, at least one compressed log entry is aggregated and encapsulated to generate an encapsulated compressed log entry, including: A compressed log aggregation buffer is pre-set in memory, which is used to store at least one compressed log; For the compressed logs in the current compressed log aggregation buffer, if it is determined that a send operation will be performed, then encapsulated compressed log header information is generated. The encapsulated compressed log header information includes the encapsulated compressed log unique identifier, which is used to uniquely identify the compressed logs in the current compressed log aggregation buffer. The encapsulated compressed log header information and the compressed log in the current compressed log aggregation buffer are concatenated into a UDP data packet, which is the encapsulated compressed log.

[0025] Optionally, the number of compressed log aggregation buffers in this embodiment can be multiple. Preferably, the encapsulated compressed log header information may further include the total number of compressed logs in the current compressed log aggregation buffer, for example, the total number of compressed logs is 20, the data type is uint16_t, and the unique identifier of the encapsulated compressed log can be the encapsulated compressed log sequence number, for example, the encapsulated compressed log sequence number is 15987, the data type is uint64_t. Figure 4 This is a message structure diagram of an encapsulated and compressed log provided in an embodiment of the present invention, such as... Figure 4 As shown, the encapsulated and compressed logs are UDP packets, including UDP header information (including the unique identifier of the encapsulated and compressed logs and the total number of compressed logs), message header of each compressed log (including the target template ID), and structured data values ​​that correspond one-to-one with the structured data keys in the target log template.

[0026] The following section will continue to use the log reporting scenario of a high-traffic Web Application Firewall (WAF) as an example to explain the above-mentioned process of generating encapsulated and compressed logs.

[0027] The firewall's Web Application Firewall (WAF) allocates a 1400-byte compressed log aggregation buffer in memory for aggregating compressed logs. Within 50 milliseconds, the WAF generates 20 compressed logs of the same type, totaling approximately 1200 bytes. The WAF decides to send these 20 compressed logs from the current aggregation buffer, generating an encapsulated compressed log header. If the unique identifier of the encapsulated compressed log is sequence number 15987, then the header includes sequence number 15987 and the total number of compressed logs, 20. The WAF concatenates the encapsulated compressed log header and the 20 compressed logs into a single UDP packet and sends it to the central SIEM platform.

[0028] In this embodiment, the encapsulated and compressed logs are in the form of UDP data packets, which inherit the high throughput and low latency characteristics of UDP, and realize lightweight and reliable transmission of logs.

[0029] Based on the above embodiments, further comprising, after sending the encapsulated and compressed log to the log receiving end via a connectionless transmission protocol, the method further includes: The unique identifier of the encapsulated and compressed log is stored in an identifier hash table, and a timer is set for the unique identifier of the encapsulated and compressed log. Determine whether a notification of successful reception of encapsulated and compressed logs is received from the log receiving end within the time period set by the timer; If received, the unique identifier of the encapsulated compressed log in the identifier hash table is deleted, and the current compressed log aggregation buffer is released; If not received, the encapsulated and compressed log is resent to the log receiver, and the operation of determining whether the encapsulated and compressed log reception success notification sent by the log receiver is received within the time period set by the timer continues until the current compressed log aggregation buffer is released.

[0030] For example, the timer can be set to a period of 500ms. Preferably, the successful reception notification of the encapsulated and compressed log is an ACK (Acknowledge character) data packet, which includes the unique identifier of the encapsulated and compressed log. For example, if the unique identifier of the encapsulated and compressed log is sequence number 15987, then the ACK data packet includes sequence number 15987. The above-mentioned lightweight ACK data packet and log retransmission mechanism provide a "at least once" delivery guarantee, meeting critical business requirements.

[0031] The following section will use the log reporting scenario of a high-traffic Web Application Firewall (WAF) as an example to explain the above log acknowledgment and retransmission mechanism in detail.

[0032] The firewall WAF receives the ACK packet sent by the central SIEM platform, parses out the sequence number 15987, confirms that the log receiver has received the encapsulated and compressed log, deletes the unique identifier 15987 of the encapsulated and compressed log from the identifier hash table, cancels the timer and releases the current compressed log aggregation buffer.

[0033] If a UDP packet sent by the firewall WAF is lost in the network, the central SIEM platform will not receive any information or send ACK data. Correspondingly, the firewall WAF will not receive a notification of successful encapsulation and compression log reception. If the 500ms timer expires, the firewall WAF will extract sequence number 15987 from the identifier hash table and resend the UDP packet with sequence number 15987 to the log receiver.

[0034] Figure 5 This is an interactive diagram illustrating a log successful transmission and log retransmission mechanism provided in an embodiment of the present invention, such as... Figure 5 As shown, the log sender sends UDP packets 1, 2, and 3 to the log receiver. UDP packets 1 and 3 are successfully received by the log receiver, while UDP packet 2 is lost during transmission. The log receiver then sends an ACK packet 1 (corresponding to UDP packet 1) and an ACK packet 3 (corresponding to UDP packet 3) back to the log sender. At the log sender, the timer corresponding to UDP packet 2 times out, and the log sender resends UDP packet 2 to the log receiver. The log receiver successfully receives UDP packet 2 and sends its corresponding ACK packet 2 back to the log sender, completing the transmission of UDP packets 1, 2, and 3.

[0035] Figure 6 This is a flowchart illustrating a log receiving method provided in an embodiment of the present invention, as shown below. Figure 6As shown, the log receiving method is executed by the log receiving end and includes: S610. Receive encapsulated and compressed logs sent by a log sender via a connectionless transmission protocol. The encapsulated and compressed logs are generated by the log sender aggregating and encapsulating at least one compressed log. The encapsulated and compressed logs include a unique identifier. The compressed logs are generated by the log sender by converting the original logs based on a target log template. The target log template is predefined by the log receiver and the log sender and includes structured data keys from the original log. The original log includes structured data key-value pairs. The compressed log includes the target template ID of the target log template and structured data values ​​that correspond one-to-one with the structured data keys in the target log template. S620. Parse the encapsulated and compressed log to obtain each compressed log entry in the encapsulated and compressed log; S630. For each compressed log in the encapsulated compressed log, the target log template is determined using the target template ID, the structured data key corresponding to the original log is determined from the target log template, and the structured data values ​​in the compressed log are matched one-to-one with the structured data key to obtain the corresponding original log.

[0036] Preferably, parsing the encapsulated and compressed log further includes: parsing the encapsulated and compressed log to obtain a unique identifier for the encapsulated and compressed log; correspondingly, before determining the target log template using the target template ID and determining the structured data key corresponding to the original log from the target log template, the method further includes: determining whether the encapsulated and compressed log has been processed based on the unique identifier of the encapsulated and compressed log; if it is determined that it has not been processed, then determining the target log template using the target template ID and determining the structured data key corresponding to the original log from the target log template.

[0037] The following section will continue to use the log reporting scenario of a high-traffic Web Application Firewall (WAF) as an example to explain in detail the process by which the central SIEM platform parses the encapsulated and compressed logs to obtain the original logs.

[0038] The central SIEM platform receives a UDP packet, parses it to find that the encapsulated and compressed log has a unique identifier of sequence number 15987 and contains 20 messages. The central SIEM platform confirms whether the encapsulated and compressed log with sequence number 15987 has been processed to achieve deduplication. If it is confirmed that it has not been processed, the central SIEM platform uses the target template ID to determine the target log template and reconstructs the original log based on the target log template. Specifically, taking the compressed log listed in the example above as an example, the process of reconstructing the original log is explained. The central SIEM platform parses the target template ID 1001, then obtains the structured data keys ["client_ip", "host", "rule_id", "attack_type", "user_agent"] from the target log template in the local template library, and maps the subsequent structured data values ​​["203.0.113.10", "api.example.com", "942100", "SQLInjection", "sqlmap / 1.5"] to the prestructured data keys one by one to reconstruct the original log corresponding to the compressed log. Repeat the above steps to reconstruct all 20 compressed log entries into their corresponding original log entries.

[0039] This invention proposes a log receiving method, executed by a log receiving end, comprising: receiving encapsulated and compressed logs sent by a log sending end via a connectionless transmission protocol, wherein the encapsulated and compressed logs are generated by the log sending end by aggregating and encapsulating at least one compressed log, and the encapsulated and compressed logs include a unique identifier for the encapsulated and compressed logs; wherein the compressed logs are generated by the log sending end by converting original logs based on a target log template, and the target log template is predefined by the log receiving end and the log sending end, including structured data keys in the original logs, the original logs including structured data key-value pairs, and the compressed logs including a target template ID of the target log template, and structured data values ​​corresponding one-to-one with the structured data keys in the target log template; parsing the encapsulated and compressed logs to obtain each compressed log in the encapsulated and compressed logs; for each compressed log in the encapsulated and compressed logs, determining the target log template using the target template ID, determining the structured data key corresponding to the original log from the target log template, and mapping the structured data values ​​in the compressed logs one-to-one with the structured data keys to obtain the corresponding original log. The above-mentioned connectionless transmission protocol for transmitting encapsulated and compressed logs features high throughput and low latency, reducing the state maintenance overhead of the log receiver. Encapsulating and compressing logs significantly reduces the structural redundancy of log information, thereby saving network bandwidth during log transmission and avoiding waste of storage resources. The batch processing method for compressed logs reduces the number of network data packets and system calls, further saving network bandwidth and reducing system performance overhead.

[0040] Based on the above embodiments, further, after obtaining the corresponding original logs, the method also includes: A successful notification of receiving the generated encapsulated and compressed log is sent to the log sending end.

[0041] After obtaining the corresponding raw logs, the central SIEM platform sends an ACK packet back to the source port of the firewall (WAF). For example, this ACK packet only contains sequence number 15987. This ACK packet is used to confirm that the central SIEM platform has successfully received the encapsulated and compressed logs.

[0042] Figure 7 This is a schematic diagram of the structure of a log receiving device provided in an embodiment of the present invention, as shown below. Figure 7 As shown, the log sending device includes: The compressed log conversion module 710 is used to convert the original log into a compressed log based on a target log template. The target log template is predefined by the log sending end and the log receiving end and includes the structured data keys in the original log. The original log includes structured data key-value pairs. The compressed log includes the target template ID of the target log template and structured data values ​​that correspond one-to-one with the structured data keys in the target log template. The encapsulated and compressed log generation module 720 is used to aggregate and encapsulate at least one compressed log to generate an encapsulated and compressed log, wherein the encapsulated and compressed log includes a unique identifier for the encapsulated and compressed log. The encapsulated and compressed log sending module 730 is used to send the encapsulated and compressed log to the log receiving end via a connectionless transmission protocol.

[0043] This invention proposes a log sending device that utilizes a compressed log conversion module to convert raw logs into compressed logs based on a target log template. The target log template is predefined by the log sending and receiving ends and includes structured data keys from the raw log. The raw log includes structured data key-value pairs, and the compressed log includes a target template ID of the target log template and structured data values ​​corresponding one-to-one with the structured data keys in the target log template. An encapsulated compressed log generation module aggregates and encapsulates at least one compressed log entry to generate an encapsulated compressed log, which includes a unique identifier. The encapsulated compressed log sending module then sends the encapsulated compressed log to the log receiving end via a connectionless transmission protocol. By introducing a target log template including structured data keys from the raw log, the device compresses the raw log into a compressed log including the target log template ID and structured data values, significantly reducing structural redundancy in the log information. This saves network bandwidth during log transmission and avoids wasting storage resources. Furthermore, by aggregating and encapsulating the compressed logs and using batch processing, the device reduces the number of network data packets and system calls, further saving network bandwidth. Furthermore, using a connectionless transmission protocol to transmit encapsulated and compressed logs offers high throughput and low latency, while also reducing system performance overhead.

[0044] Based on the above technical solutions, the encapsulated compressed log generation module 720 can be further configured to: pre-set a compressed log aggregation buffer in memory, the compressed log aggregation buffer being used to store at least one compressed log; for the compressed log in the current compressed log aggregation buffer, if it is determined to perform a sending operation, generate encapsulated compressed log header information, the encapsulated compressed log header information including the encapsulated compressed log unique identifier, the encapsulated compressed log unique identifier being used to uniquely identify the compressed log in the current compressed log aggregation buffer; and concatenate the encapsulated compressed log header information and the compressed log in the current compressed log aggregation buffer into a UDP data packet, the UDP data packet being the encapsulated compressed log.

[0045] Based on the above technical solutions, the log sending device may further include a log monitoring and retransmission module. Specifically, after sending the encapsulated and compressed log to the log receiving end via a connectionless transmission protocol, the module stores the unique identifier of the encapsulated and compressed log in an identifier hash table and sets a timer for the unique identifier. It then determines whether a successful reception notification of the encapsulated and compressed log is received from the log receiving end within the time period set by the timer. If received, the unique identifier of the encapsulated and compressed log is deleted from the identifier hash table, and the current compressed log aggregation buffer is released. If not received, the encapsulated and compressed log is retransmitted to the log receiving end, and the operation of determining whether a successful reception notification of the encapsulated and compressed log is received from the log receiving end within the time period set by the timer continues until the current compressed log aggregation buffer is released.

[0046] Based on the above technical solutions, the successful reception notification of the encapsulated and compressed log is further defined as an ACK data packet, and the ACK data packet includes the unique identifier of the encapsulated and compressed log.

[0047] Based on the above technical solutions, the encapsulated compressed log header information further includes the total number of compressed logs in the current compressed log aggregation buffer, and the encapsulated compressed log is uniquely identified by its sequence number.

[0048] Based on the above technical solutions, the compressed log statement is further expressed as [sd-id@template-id value1 value2 ...valueN], where sd-id is the ID corresponding to the predefined structured data key, @template-id is the target template ID, and value1, value2, ..., valueN are N structured data values ​​that correspond one-to-one with the structured data keys in the target log template.

[0049] The log sending device provided in this embodiment of the invention can execute the steps in the log sending method provided in the method embodiment of this disclosure, and has the execution steps and beneficial effects, which will not be repeated here.

[0050] Figure 8 This is a schematic diagram of the structure of a log receiving device provided in an embodiment of the present invention, as shown below. Figure 8 As shown, the log receiving device includes: The encapsulated and compressed log receiving module 810 is used to receive encapsulated and compressed logs sent by a log sending end via a connectionless transmission protocol. The encapsulated and compressed logs are generated by the log sending end by aggregating and encapsulating at least one compressed log. The encapsulated and compressed logs include a unique identifier for the encapsulated and compressed logs. The compressed logs are generated by the log sending end by converting the original logs based on a target log template. The target log template is predefined by the log receiving end and the log sending end and includes structured data keys from the original logs. The original logs include structured data key-value pairs. The compressed logs include the target template ID of the target log template and structured data values ​​that correspond one-to-one with the structured data keys in the target log template. The parsing module 820 is used to parse the encapsulated and compressed log to obtain each compressed log entry in the encapsulated and compressed log. The raw log conversion module 830 is used to determine the target log template using the target template ID for each compressed log in the encapsulated compressed log, determine the structured data key corresponding to the raw log from the target log template, and match the structured data values ​​in the compressed log with the structured data key one by one to obtain the corresponding raw log.

[0051] Based on the above technical solutions, the log receiving device may further include a module for sending a successful encapsulated and compressed log reception notification, which is used to generate a successful encapsulated and compressed log reception notification and send it to the log sending end after obtaining the corresponding original log.

[0052] Based on the above technical solutions, the parsing module 820 can further be used to: parse the encapsulated and compressed log to obtain a unique identifier for the encapsulated and compressed log; correspondingly, the log receiving device can also include an encapsulated and compressed log processing confirmation module, which can be used to: before determining the target log template using the target template ID and determining the structured data key corresponding to the original log from the target log template, determine whether the encapsulated and compressed log has been processed based on the unique identifier of the encapsulated and compressed log; if it is determined that it has not been processed, then determine the target log template using the target template ID and determine the structured data key corresponding to the original log from the target log template.

[0053] The log receiving device provided in this embodiment of the invention can execute the steps in the log receiving method provided in the method embodiment of this disclosure, and has the execution steps and beneficial effects, which will not be repeated here.

[0054] Figure 9 This is a schematic diagram of the structure of a log sending end provided in an embodiment of the present invention. Figure 9 As shown, the log sending end 900 includes one or more processors 901 and a storage device 902, wherein the storage device 902 is used to store one or more programs; when the one or more programs are executed by the one or more processors 901, the one or more processors 901 implement the log sending method as described above.

[0055] The processor 901 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the log sender 900 to perform the desired functions.

[0056] Storage device 902 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and processor 901 may execute the program instructions to implement the log sending method of any embodiment of the present invention described above and / or other desired functions. Various contents such as initial external parameters and thresholds may also be stored in the computer-readable storage medium.

[0057] In one example, the log sender 900 may further include an input device 903 and an output device 904, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 903 may include, for example, a keyboard, a mouse, etc. The output device 904 may output various information to the outside, including warning messages, braking force, etc. The output device 904 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0058] Of course, for the sake of simplicity, Figure 9 Only some of the components in the log sender 900 relevant to this invention are shown, omitting components such as buses, input / output interfaces, etc. In addition, the log sender 900 may include any other suitable components depending on the specific application.

[0059] In addition to the methods and devices described above, embodiments of the present invention may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the log sending method provided in any embodiment of the present invention.

[0060] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of the present invention. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0061] Figure 10 This is a schematic diagram of a log receiving end provided in an embodiment of the present invention. Figure 10 As shown, the log sending end 1000 includes one or more processors 1001 and a storage device 1002, wherein the storage device 1002 is used to store one or more programs; when the one or more programs are executed by the one or more processors 1001, the one or more processors 1001 implement the log receiving method as described above.

[0062] The processor 1001 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the log receiver 1000 to perform desired functions.

[0063] Storage device 1002 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and processor 1001 may execute the program instructions to implement the log receiving method of any embodiment of the present invention described above and / or other desired functions. Various contents such as initial external parameters and thresholds may also be stored in the computer-readable storage medium.

[0064] In one example, the log receiver 1000 may further include an input device 1003 and an output device 1004, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 1003 may include, for example, a keyboard, a mouse, etc. The output device 1004 may output various information to the outside, including warning messages, braking force, etc. The output device 1004 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0065] Of course, for the sake of simplicity, Figure 10 Only some of the components in the log receiver 1000 relevant to the present invention are shown, omitting components such as buses, input / output interfaces, etc. In addition, the log receiver 1000 may include any other suitable components depending on the specific application.

[0066] In addition to the methods and devices described above, embodiments of the present invention may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the log receiving method provided in any embodiment of the present invention.

[0067] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of the present invention. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0068] Furthermore, embodiments of the present invention may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the log sending method or log receiving method provided in any embodiment of the present invention.

[0069] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log sending method, characterized in that, Executed by the log sender, including: Based on the target log template, the original log is converted into a compressed log. The target log template is predefined by the log sending end and the log receiving end, and includes the structured data keys in the original log. The original log includes structured data key-value pairs, and the compressed log includes the target template ID of the target log template, as well as structured data values ​​that correspond one-to-one with the structured data keys in the target log template. At least one compressed log is aggregated and encapsulated to generate an encapsulated compressed log, wherein the encapsulated compressed log includes a unique identifier for the encapsulated compressed log. The encapsulated and compressed log is sent to the log receiving end via a connectionless transmission protocol.

2. The method according to claim 1, characterized in that, Aggregate and encapsulate at least one compressed log entry to generate an encapsulated compressed log entry, including: A compressed log aggregation buffer is pre-set in memory, which is used to store at least one compressed log; For the compressed logs in the current compressed log aggregation buffer, if it is determined that a send operation will be performed, then encapsulated compressed log header information is generated. The encapsulated compressed log header information includes the encapsulated compressed log unique identifier, which is used to uniquely identify the compressed logs in the current compressed log aggregation buffer. The encapsulated compressed log header information and the compressed log in the current compressed log aggregation buffer are concatenated into a UDP data packet, which is the encapsulated compressed log.

3. The method according to claim 1 or 2, characterized in that, After sending the encapsulated and compressed log to the log receiver via a connectionless transmission protocol, the process further includes: The unique identifier of the encapsulated and compressed log is stored in an identifier hash table, and a timer is set for the unique identifier of the encapsulated and compressed log. Determine whether a notification of successful reception of encapsulated and compressed logs is received from the log receiving end within the time period set by the timer; If received, the unique identifier of the encapsulated compressed log in the identifier hash table is deleted, and the current compressed log aggregation buffer is released; If not received, the encapsulated and compressed log is resent to the log receiver, and the operation of determining whether the encapsulated and compressed log reception success notification sent by the log receiver is received within the time period set by the timer continues until the current compressed log aggregation buffer is released.

4. The method according to claim 3, characterized in that, The successful reception notification of the encapsulated and compressed log is an ACK data packet, and the ACK data packet includes the unique identifier of the encapsulated and compressed log.

5. The method according to claim 2, characterized in that, The header information of the encapsulated compressed log also includes the total number of compressed logs in the current compressed log aggregation buffer, and the unique identifier of the encapsulated compressed log is the encapsulated compressed log sequence number.

6. The method according to claim 1 or 2, characterized in that, The compressed log statement is expressed as [sd-id@template-id value1 value2 ...valueN], where sd-id is the ID corresponding to the predefined structured data key, @template-id is the target template ID, and value1, value2, ..., valueN are N structured data values ​​that correspond one-to-one with the structured data keys in the target log template.

7. A log receiving method, characterized in that, Executed by the log receiver, including: The receiver receives encapsulated and compressed logs sent by the log sender via a connectionless transmission protocol. The encapsulated and compressed logs are generated by the log sender by aggregating and encapsulating at least one compressed log. The encapsulated and compressed logs include a unique identifier. The compressed logs are generated by the log sender by converting the original logs based on a target log template. The target log template is predefined by the log receiver and the log sender and includes structured data keys from the original logs. The original logs include structured data key-value pairs. The compressed logs include the target template ID of the target log template and structured data values ​​that correspond one-to-one with the structured data keys in the target log template. The encapsulated and compressed log is parsed to obtain each compressed log entry in the encapsulated and compressed log; For each compressed log entry in the encapsulated compressed log, the target log template is determined using the target template ID. The structured data key corresponding to the original log is determined from the target log template, and the structured data values ​​in the compressed log are matched one-to-one with the structured data key to obtain the corresponding original log.

8. The method according to claim 7, characterized in that, After obtaining the corresponding raw logs, the following are also included: A successful notification of receiving the generated encapsulated and compressed log is sent to the log sending end.

9. A log sending terminal, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the log sending method as described in any one of claims 1-6.

10. A log receiver, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the log receiving method as described in claim 7 or 8.

Images