Log transmission method and system of an acknowledgement-free log transmission protocol

Abstract

The application relates to a log transmission method and system of an unacknowledged log transmission protocol, wherein the method is suitable for a log collection scene in which a log source does not have the capability of confirmation and retransmission, and is especially suitable for a syslog log transmission environment based on UDP. Without modifying log source equipment and without changing a log transmission protocol, by constructing a sending state awareness mechanism, a non-interruption receiving mechanism, a local persistence retention mechanism and a unified processing link historical log re-consumption mechanism at a collection end, reliable retention of log data is continuously received under limited downstream processing capability, and log sending is automatically recovered after an exception is removed. In the case of not changing the log transmission protocol and not depending on the retransmission capability of the log source, the responsibility for guaranteeing log reliability is moved to the collection end receiving link, log data continuity, integrity and recoverable consumption capability are realized under the condition of no confirmation and non-retransmission, log loss is avoided and continuous operation of the collection process is guaranteed.

Description

Technical Field

[0001] This application relates to the field of log acquisition technology, and in particular to a log transmission method and system without acknowledgment log transmission protocol. Background Technology

[0002] With the development of applications such as security auditing, situational awareness, and operation and maintenance monitoring, log data from various network and security devices has become an important foundation for upper-level analysis and decision-making. In actual deployment environments, many log source devices (such as switches and firewalls) are limited by system isolation or security policies and cannot deploy third-party collection agents. They typically adopt a remote collection mode based on the Syslog protocol and widely use the UDP protocol for log transmission to reduce performance overhead.

[0003] Existing log collection solutions mainly include local collection mode and remote collection mode. Local collection mode requires installing an agent program on the log source device to read files or listen for events, and in case of transmission failure, it achieves limited retransmission recovery by pausing collection or rolling back the read position. However, this mode is not suitable for closed devices. In remote collection mode, the log source sends logs unidirectionally to the collection end via protocols such as Syslog, and the collection end then forwards them to the backend analysis service. When the downstream processing capacity of the log collection end is insufficient, the service is abnormal, or the collection end itself triggers a rate-limiting mechanism, existing technologies typically address this by stopping reception, simple caching and retrying, or relying on the fault tolerance capabilities of the backend distributed system. Some existing technologies (such as patent CN200710103164.7) improve processing performance through asynchronous caching and concurrent storage.

[0004] However, in Syslog log collection scenarios based on the UDP protocol (scenarios without acknowledgment log transmission protocols), the aforementioned existing technologies have significant limitations. The UDP protocol itself lacks acknowledgment and retransmission mechanisms. Log sources typically do not retransmit data when transmission fails or the receiver becomes unavailable, leading to irrecoverable log loss should the collection end stop receiving data or discard data due to an anomaly. Especially under conditions of continuous high-concurrency log input and prolonged downstream anomalies, existing solutions struggle to simultaneously ensure real-time log reception, sequential retention, and unified recovery and consumption after anomaly resolution, easily resulting in data omissions or inconsistent processing.

[0005] Therefore, for log transmission methods without acknowledgment log transmission protocols, how to achieve reliable log retention and continuous recovery through autonomous acquisition at the collection end without modifying the log source device or relying on the protocol acknowledgment mechanism has not yet been proposed. Summary of the Invention

[0006] This application provides a log transmission method and system without acknowledgment log transmission protocol, which at least solves the problem in related technologies of how to achieve reliable log retention and continuous recovery through autonomous acquisition end without modifying the log source device or relying on protocol acknowledgment mechanism.

[0007] In a first aspect, embodiments of this application provide a log transmission method without an acknowledgment log transmission protocol, the method comprising: It continuously receives log data sent from the log source, preprocesses the received log data, and generates logs to be sent. The log collection terminal sends the logs to be sent to the log analysis server and obtains the sending status information; When the sending status information is in an abnormal state, the log collection terminal continues to receive the log data and writes the log to be sent in the abnormal state into a local storage file; When the sending status information recovers from an abnormal state to a normal state, the log collection terminal reads the log to be sent from the local storage file and sends the log to be sent to the log analysis server through the same processing link as the real-time log.

[0008] In one embodiment, the step of continuously receiving log data sent by a log source and preprocessing the received log data to generate logs to be sent includes: The log collection terminal receives raw log data through either remote collection mode or local collection mode; Based on the protocol type and log source identifier of the original log data, the corresponding log parsing rules are matched, the core log fields are extracted, and semi-structured log data is generated. Based on the local system clock of the log collection terminal, add a unified receiving timestamp and unique log source identifier metadata to the semi-structured log data; The semi-structured log data is filtered to remove those that do not conform to the filtering rules; the retained semi-structured log data is then formatted according to standard formats to generate logs to be sent in a unified format.

[0009] In one embodiment, the log collection terminal sends the log to be sent to the log analysis server and obtains sending status information, including: The log collection terminal sends a request to the log analysis server to send the log to be sent based on a preset sending strategy. The log collection terminal determines whether the log to be sent has been successfully sent based on the response result of the sending request; If the number of logs to be sent that fail to be sent consecutively reaches a preset threshold, or the number of timeouts of the sending request reaches a preset timeout threshold, then the log collection terminal determines that the sending status information is the abnormal status. Otherwise, the log collection terminal determines that the sending status information is in the normal state.

[0010] In one embodiment, when the log collection terminal is in remote collection mode and the sending status information is in an abnormal state, the log collection terminal continues to receive the log data and writes the log to be sent in the abnormal state to a local storage file, including: After confirming that it is in the abnormal state based on the sending status information, the log collection terminal continuously writes the logs to be sent into the memory queue. When the used capacity of the memory queue reaches a preset capacity threshold, the log collection terminal will transfer the logs to be sent temporarily stored in the memory queue to the local storage file in the order of writing, and release the used space of the memory queue. The log data is continuously received during the transfer process.

[0011] In one embodiment, when the log collection terminal is in local collection mode and the sending status information is in an abnormal state, the method further includes: The log collection terminal writes the logs to be sent into a memory queue; When the used capacity of the memory queue reaches a preset capacity threshold, the log collection terminal stops reading log data from the local log file and records the current consumed position of the local log file.

[0012] In one embodiment, the log collection terminal transfers the logs to be sent temporarily stored in the memory queue to the local storage file in the order they were written, including: The log collection terminal performs rolling management of the local storage files according to a preset rolling strategy; the rolling strategy includes: Scroll by file size: When the size of the current local storage file reaches a preset file size threshold, close the current local storage file and create a new local storage file for writing; Rolling by time interval: When the continuous writing duration of the current local storage file reaches a preset time threshold, the current local storage file is closed and a new local storage file is created for writing.

[0013] In one embodiment, when the log collection terminal is in remote collection mode and the sending status information recovers from an abnormal state to a normal state, the log collection terminal reads the log to be sent from the local storage file and sends the log to be sent to the log analysis server through the same processing link as the real-time log, including: After the log collection terminal confirms that the current state has recovered from the abnormal state to the normal state based on the sending status information, it triggers the historical log re-consumption process. In the historical log re-consumption process, the log collection terminal reads the logs to be sent from the local storage file in the order of writing. The log collection terminal reads the date to be sent and sends it to the log analysis server through a simulated real-time log collection and processing link.

[0014] In one embodiment, when the log collection terminal is in local collection mode and the sending status information recovers from an abnormal state to a normal state, the method further includes: The log collection terminal sends the logs to be sent that are temporarily stored in the memory queue to the log analysis server; After all the logs to be sent in the memory queue have been sent, the log collection terminal restarts reading the logs from the consumed position of the local log file, based on the recorded consumed position of the local log file, and continues log collection.

[0015] In one embodiment, the log collection terminal reads the logs to be sent stored in the local storage file in the order they were written, including: The log collection terminal determines the list of local storage files to be read based on the creation time order of the local storage files; The log collection terminal opens the local storage files sequentially according to their creation time, from earliest to latest, based on the local storage file list. The log collection terminal reads the logs to be sent line by line from the opened local storage file. Each line of the logs to be sent corresponds to a log data record retained during the writing process.

[0016] Secondly, embodiments of this application provide a log transmission system without an acknowledgment log transmission protocol. The system is used to execute the method described above, and includes a preprocessing module, a status information module, a local storage writing module, and a sending module; wherein: The preprocessing module is used to continuously receive log data sent by the log source, and preprocess the received log data to generate logs to be sent. The status information module is used by the log collection terminal to send the log to be sent to the log analysis server and to obtain the sending status information. The local storage writing module is used to continuously receive the log data and write the log to be sent in the abnormal state to a local storage file when the sending status information is in an abnormal state. The sending module is used to read the log to be sent from the local storage file and send the read log to be sent to the log analysis server when the sending status information recovers from an abnormal state to a normal state.

[0017] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a log transmission method with an unacknowledged log transmission protocol as described in the first aspect above.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a log transmission method with an unacknowledged log transmission protocol as described in the first aspect above.

[0019] The log transmission and system based on the unacknowledged log transmission protocol provided in this application embodiment has at least the following technical effects.

[0020] By continuously receiving log data from the log source and preprocessing the received log data, logs to be sent are generated. Without relying on the log source's retransmission capability or changing the log transmission protocol, complete reception and standardized processing of log data are achieved, providing a unified data format foundation for subsequent sending and retention. The log collection end sends the logs to be sent to the log analysis server and obtains the sending status information; this enables real-time awareness of downstream processing capabilities, providing a reliable basis for determining whether to switch data processing paths. When the sending status information is in an abnormal state, the log collection end continues to receive log data and writes the logs to be sent in the abnormal state to a local storage file. During periods of downstream service anomalies or limited processing capabilities, the log receiving process is not interrupted to avoid log loss, and log data that failed to be sent is reliably retained in chronological order, providing a data source for recovery consumption after the anomaly is resolved. When the sending status information returns from an abnormal state to a normal state, the log collection end reads the logs to be sent from the local storage file and sends the read logs to the log analysis server. This system enables automatic recovery and transmission of historical logs after anomalies are resolved, allowing historical logs to be consumed using the same processing chain as real-time logs, thus ensuring the continuity and consistency of log data processing. It addresses the challenge in related technologies of achieving reliable log retention and continuous recovery through autonomous acquisition of logs without modifying the log source device or relying on protocol confirmation mechanisms.

[0021] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a log transmission method using an unacknowledged log transmission protocol; Figure 2 This is a flowchart illustrating step S103 according to an exemplary embodiment; Figure 3 This is a system architecture block diagram illustrating a log transmission system with an unacknowledged log transmission protocol according to an exemplary embodiment; Figure 4 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0024] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0025] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0026] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0027] In this document, it should be understood that the terms used may be technical means used to implement part of the present invention or other summary technical terms. For example, the terms may include: Log analysis server: Deployed downstream of the log collection terminal, it is a backend computer system used to receive logs to be sent forward by the collection terminal, and to centrally store, normalize and analyze, correlate security events, conduct compliance audits and visualize the log data.

[0028] Local system clock: refers to the hardware or software system clock of the log acquisition terminal itself, which is the unified time reference for all log data in this application. Unlike the system clock of the log source device, its time accuracy and consistency are maintained independently by the acquisition terminal and are not affected by the clock deviation, asynchrony or error of the log source device.

[0029] Semi-structured log data refers to the intermediate data format generated after extracting fields from the original log data. This format is organized in key-value pairs, retaining the core business information of the original logs while possessing structured characteristics that can be directly processed by computers. It is a key intermediate product in the log preprocessing process.

[0030] Receive timestamp: This refers to the time stamp generated by the log collection end based on its local system clock at the precise moment it receives the raw log data. This timestamp is appended to the semi-structured log data as metadata and is the core technical means of this invention to solve the problem of clock asynchrony among multiple log sources. It is used to ensure that all log data has a unified and accurate time dimension, and to ensure the time sequence consistency between historical logs and real-time logs after anomaly recovery.

[0031] Log source unique identifier metadata: This refers to the additional information added by the log collection end to each piece of semi-structured log data to uniquely identify the source of the log. It consists of the IP address, port number, device serial number, or pre-configured device ID of the log source device. It is used to achieve accurate traceability of the log source, solve the problem of source differentiation when different log sources send logs with the same format, and provide a basis for subsequent log filtering, classification analysis, and accountability auditing.

[0032] Firstly, embodiments of this application provide a log transmission method without an acknowledgment log transmission protocol. Figure 1 This is a flowchart of a log transmission method using an unacknowledged log transmission protocol, such as... Figure 1 As shown, the method includes: Step S101: Continuously receive log data sent by the log source, and preprocess the received log data to generate logs to be sent.

[0033] Step S102: The log collection terminal sends the logs to be sent to the log analysis server and obtains the sending status information.

[0034] Step S103: When the sending status information is in an abnormal state, the log collection terminal continues to receive log data and writes the logs to be sent in the abnormal state to a local storage file.

[0035] Step S104: When the sending status information recovers from the abnormal state to the normal state, the log collection terminal reads the log to be sent from the local storage file and sends the log to be sent to the log analysis server through the same processing link as the real-time log.

[0036] In summary, this application provides a log transmission method without an acknowledgment log transmission protocol. It continuously receives log data from a log source and preprocesses the received log data to generate logs to be sent. Without relying on the log source's retransmission capability or changing the log transmission protocol, it achieves complete reception and standardized processing of log data, providing a unified data format foundation for subsequent transmission and retention. The log collection end sends the logs to be sent to the log analysis server and obtains transmission status information, enabling real-time awareness of downstream processing capabilities and providing a reliable basis for determining whether to switch data processing paths. When the transmission status information is in an abnormal state, the log collection end continues to receive log data and writes the logs to be sent in the abnormal state to a local storage file. During periods of downstream service anomalies or limited processing capabilities, it neither interrupts the log reception process to avoid log loss nor fails to reliably retains log data that was not successfully sent in chronological order, providing a data source for recovery consumption after the anomaly is resolved. When the transmission status information returns from an abnormal state to a normal state, the log collection end reads the logs to be sent from the local storage file and sends the read logs to the log analysis server. This mechanism enables automatic resumption and transmission of historical logs after anomalies are resolved, allowing historical logs to be consumed using the same processing chain as real-time logs, thus ensuring the continuity and consistency of log data processing. Based on a unified processing chain, the automatic re-consumption mechanism for historical logs achieves a high degree of consistency in the processing flow: historical and real-time logs use a unified processing chain, avoiding data discrepancies; it also boasts strong deployment adaptability: no modification to the log source device or transmission protocol is required, making it easy to promote and apply.

[0037] This addresses the challenge of achieving reliable log retention and continuous recovery through autonomous acquisition without modifying the log source device or relying on protocol confirmation mechanisms.

[0038] In one example, step S101 involves continuously receiving log data from a log source and preprocessing the received log data to generate logs to be sent. Specifically, this includes the following steps: Step S1011: The log collection terminal receives raw log data through remote collection mode or local collection mode; Step S1012: Based on the protocol type and log source identifier of the original log data, match the corresponding log parsing rules, extract the core log fields, and generate semi-structured log data; Step S1013: Based on the local system clock of the log collection terminal, add a unified receiving timestamp and unique log source identifier metadata to the semi-structured log data; Step S1014: Filter the semi-structured log data and remove semi-structured log data that does not meet the filtering rules; perform standard formatting on the retained semi-structured log data to generate logs to be sent in a unified format.

[0039] Optionally, after the log collection terminal starts, it first continuously listens to the preset log transmission port (the default port of the syslog protocol) or periodically reads the local log files to be collected, continuously receiving raw log data from various log sources such as firewalls, switches, and servers; then, based on the transmission protocol type (UDP syslog) and the IP address of the log source, it matches the pre-configured log parsing rules of the corresponding vendor's device, such as matching the parsing rules from the Huawei firewall at 192.168.1.10, extracting core fields such as event time, source IP, destination IP, port, and event type from the plain text log, and generating semi-structured log data in key-value pair format; next, based on the local system clock synchronized by NTP on the collection terminal itself, it adds a receiving timestamp accurate to milliseconds and unique identifier metadata composed of the log source IP + port to each semi-structured log data; finally, according to the preset filtering rules (such as filtering out logs containing the keyword "not connected"), invalid semi-structured log data is removed, and the retained semi-structured log data is converted into JSON format logs with unified fields to be sent.

[0040] Step S101 ensures the validity and consistency of log data from the source, solving the problems of chaotic log formats and inconsistent data quality among different manufacturers and types of devices. Specifically, by uniformly adding a receiving timestamp and a unique log source identifier at the collection end, the industry pain point of clock asynchrony among multiple log source devices and inability to accurately trace the log source is completely avoided. This provides a reliable time benchmark and source basis for the correct sorting and correlation analysis of historical logs after subsequent anomaly recovery. At the same time, the standardized log format to be sent ensures that real-time logs and historical logs can reuse the same set of sending and processing links, avoiding inconsistencies in processing results caused by differences in data formats.

[0041] In one example, step S102 involves the log collection terminal sending the logs to be sent to the log analysis server and obtaining the sending status information. This specifically includes the following steps: Step S1021: The log collection terminal sends a request to the log analysis server to send the logs to be sent based on the preset sending strategy. Step S1022: The log collection terminal determines whether the log to be sent has been successfully sent based on the response result of the sending request; Step S1023: If the number of logs to be sent that fail to be sent consecutively reaches a preset threshold, or the number of timeouts for sending requests reaches a preset timeout threshold, then the log collection terminal determines that the sending status information is in an abnormal state. Step S1024: Otherwise, the log collection terminal determines that the sending status information is in a normal state.

[0042] Optionally, when the log collection terminal initiates a sending request to the log analysis server based on a preset sending strategy, the preset sending strategy includes, but is not limited to, asynchronous non-blocking sending, connection timeout settings, and retry limit. The log collection terminal records the corresponding response result for each sending request, which may include: successful sending confirmation, connection rejection, connection timeout, and error code returned by the receiver. The log collection terminal maintains a counter related to the sending status to accumulate the number of consecutively failed sending logs or the number of consecutive timeouts. For example, if the log collection terminal receives a "connection timeout" response when sending the first log to the log analysis server, and receives another "connection timeout" response when sending the second log, the consecutive sending failure count is accumulated to 2. If the preset threshold for consecutive sending failures is 5, the log collection terminal determines that the current sending status information is an abnormal state when the counter reaches 5. Similarly, if a sending request times out due to a lack of response from the downstream service, and the number of consecutive timeouts reaches a preset timeout threshold (e.g., 3 times), the abnormal state determination is also triggered. When the number of consecutive failed transmissions does not reach a preset threshold, and the number of timeouts does not reach a preset timeout threshold, the log collection terminal maintains the transmission status information as normal and continues to process subsequent logs to be sent along the real-time transmission path. This judgment process is continuously executed after each transmission operation at the log collection terminal to ensure real-time status awareness.

[0043] Step S102 determines the transmission status by accumulating the number of consecutive failures or consecutive timeouts, avoiding erroneous triggering of abnormal status switching due to occasional network jitter or instantaneous load fluctuations. This improves the accuracy and stability of status judgment and provides a reliable decision-making basis for subsequent local persistent retention operations under uninterrupted reception.

[0044] In one instance, Figure 2 This is a flowchart illustrating step S103 according to an exemplary embodiment, as follows: Figure 2 As shown, in step S103, when the log collection terminal is in remote collection mode and the sending status information is in an abnormal state, the log collection terminal continues to receive log data and writes the logs to be sent in the abnormal state to a local storage file. Specifically, this includes the following steps: Step S1031: After confirming that the current state is abnormal based on the sending status information, the log collection terminal continuously writes the logs to be sent into the memory queue. Step S1032: When the used capacity of the memory queue reaches a preset capacity threshold, the log collection terminal transfers the logs temporarily stored in the memory queue to the local storage file in the order they were written, and releases the used space of the memory queue; specifically including: The log collection terminal performs rolling management of local storage files according to a preset rolling strategy; the rolling strategy includes: Scroll by file size: When the size of the current local storage file reaches the preset file size threshold, close the current local storage file and create a new local storage file for writing; Rolling by time interval: When the continuous writing duration of the current local storage file reaches a preset time threshold, the current local storage file is closed and a new local storage file is created for writing.

[0045] Step S1033: Continuously receive log data during the transfer process.

[0046] Optionally, when the log collection end is in remote collection mode, after confirming an abnormal state based on the sending status information, the log collection end first continuously writes the logs that failed to be sent into a memory queue for temporary storage. This memory queue adopts a first-in-first-out ordered structure to ensure that the logs to be sent are arranged in the order of their generation time. When the used capacity of the memory queue reaches a preset capacity threshold (e.g., 80% of the total capacity), a transfer operation is triggered: the log collection end writes the logs temporarily stored in the memory queue into a local storage file in batches according to the writing order. After the writing is completed, the corresponding part of the used space in the memory queue is immediately released to free up capacity for subsequent continuously received logs to be sent to continue to be enqueued.

[0047] During the data transfer process, the log acquisition terminal implements rolling management of local storage files: if a file size-based rolling strategy is adopted, for example, with a preset file size threshold of 100MB, then when the log acquisition terminal continuously writes until the current local storage file reaches 100MB, the file is automatically closed, and a new local storage file is created to continue receiving transferred data; if a time interval-based rolling strategy is adopted, for example, with a preset time threshold of 1 hour, then after 1 hour from the creation of the first local storage file, the current file is closed, and a new file is created for writing. Both the transfer and rolling management processes are executed asynchronously, without affecting the log acquisition terminal's continuous listening and receiving of log data sent by the log source.

[0048] Step S103 effectively smooths the rate difference between downstream anomalies and continuous log reception by setting up a memory queue as a buffer layer for logs to be sent in abnormal states, avoiding reception blockage that may be caused by direct disk writing. The entire retention process runs in parallel with the log reception operation, truly achieving reliable persistence of abnormal data without interrupting log collection, thereby fundamentally solving the log loss problem caused by downstream anomalies in scenarios without acknowledgment log transmission protocols.

[0049] In one instance, when the log collection terminal is in local collection mode and the sent status information is in an abnormal state, the method also includes: The log collection end writes the logs to be sent into the memory queue; When the used capacity of the memory queue reaches the preset capacity threshold, the log collection terminal stops reading log data from the local log file and records the current consumed position of the local log file.

[0050] Optionally, in local acquisition mode, when the log acquisition end confirms an abnormal state based on the sending status information, it continuously writes the pre-processed logs to be sent into a preset memory queue for temporary storage. This memory queue uses a first-in-first-out ordered structure to ensure that the logs to be sent are arranged in chronological order of their generation. When the log acquisition end detects that the used capacity of the memory queue has reached a preset capacity threshold, it determines that the current memory resources are insufficient to cache more logs to be sent. At this point, the log acquisition end stops reading from the local log file and stops processing the current consumed logs.

[0051] In one example, step S104, when the log collection terminal is in remote collection mode and the sending status information recovers from an abnormal state to a normal state, the log collection terminal reads the logs to be sent from the local storage file and sends the logs to be sent to the log analysis server through the same processing link as the real-time logs. Specifically, this includes the following steps: Step S1041: After the log collection terminal confirms that the current state has recovered from the abnormal state to the normal state based on the sent status information, it triggers the historical log re-consumption process. Step S1042: In the historical log re-consumption process, the log collection end reads the logs to be sent from the local storage file in the order they were written. Specifically, this includes: The log collection terminal determines the list of local storage files to be read based on the creation time order of the local storage files; The log collection terminal opens local storage files sequentially based on the list of local storage files, in order of their creation time from earliest to latest. The log collection terminal reads the logs to be sent line by line from the opened local storage file. Each line of logs to be sent corresponds to a log data record that was saved when it was written.

[0052] Step S1043: The log collection terminal reads the date to be sent and sends it to the log analysis server through a simulated real-time log collection and processing link.

[0053] Optionally, after confirming that the sending status has recovered from an abnormal state to a normal state based on the sending status information, the log collection terminal automatically triggers the historical log re-consumption process. In this process, the log collection terminal first scans the local storage file directory and determines the list of local storage files to be read based on the creation timestamp of each local storage file. For example, if there are three retained files in the local storage directory: log_20240101_001.log (created at 08:00), log_20240101_002.log (created at 09:00), and log_20240101_003.log (created at 10:00), the log collection terminal generates a read list by sorting them from earliest to latest creation time. Subsequently, the log collection terminal opens each local storage file sequentially and reads the logs to be sent line by line within each file. Each line of data corresponds to a complete log record to be sent written during the abnormal period. The read logs to be sent are sent to a unified processing link with the same processing logic as the real-time sending path, and finally sent to the log analysis server.

[0054] Step S104, through an automatically triggered historical log re-consumption process, enables unattended recovery and transmission of retained data after anomaly resolution, significantly reducing operational costs. Simultaneously, the mechanism of parallel execution of historical log consumption and real-time log transmission ensures that the system does not need to pause or degrade real-time acquisition capabilities during recovery, guaranteeing the continuity and real-time nature of log collection services.

[0055] In one embodiment, when the log collection terminal is in local collection mode and the sending status information recovers from an abnormal state to a normal state, the method further includes: The log collection terminal sends the logs temporarily stored in the memory queue to the log analysis server; After all the logs to be sent in the memory queue have been sent, the log collection terminal restarts reading the local log file from the consumed position recorded in the local log file and continues log collection.

[0056] Optionally, in local acquisition mode, after the log acquisition end confirms that the sending status has recovered from an abnormal state to a normal state based on the sending status information, the log acquisition end first checks whether there are still logs to be sent in the memory queue. The log acquisition end prioritizes retrieving the logs to be sent from the memory queue one by one in the enqueue order and sends them to the log analysis server through the same unified processing link as the real-time logs. After all the logs to be sent in the memory queue have been successfully sent and cleared, the log acquisition end reads the consumed position of the local log file recorded during the abnormal period, reopens the local log file from that position, continues to read the newly added log data line by line, and resumes the normal log acquisition and sending process.

[0057] In summary, this application provides a log transmission method without an acknowledgment log transmission protocol, applicable to log collection scenarios where the log source lacks acknowledgment and retransmission capabilities, particularly suitable for UDP-based syslog log transmission environments. With the core design principle of ensuring log continuity through end-to-end autonomy, this method, without modifying the log source device or changing the log transmission protocol, establishes a sending status awareness mechanism, an uninterrupted receiving mechanism, a local persistent storage mechanism, and a unified historical log re-consumption mechanism at the collection end. This ensures continuous reception and reliable storage of log data even when downstream processing capabilities are limited or services are abnormal, and automatically resumes log transmission after the anomaly is resolved. Unlike existing solutions that rely on log source retransmission or downstream system fault tolerance, this application, without changing the log transmission protocol or relying on log source retransmission capabilities, shifts the responsibility for log reliability to the receiving stage at the collection end. This achieves log data continuity, integrity, and recoverable consumption capability under conditions of no acknowledgment and no retransmission, preventing log loss and ensuring continuous operation of the collection process. This addresses the challenge of achieving reliable log retention and continuous recovery through autonomous acquisition without modifying the log source device or relying on protocol confirmation mechanisms.

[0058] Secondly, embodiments of this application provide a log transmission system without an acknowledgment log transmission protocol. Figure 3 This is a system architecture block diagram illustrating a log transmission system with an unacknowledged log transmission protocol according to an exemplary embodiment. For example... Figure 3 As shown, the system includes a preprocessing module 310, a status information module 320, a local storage writing module 330, and a sending module 340; wherein: The preprocessing module 310 is used to continuously receive log data sent by the log source and preprocess the received log data to generate logs to be sent. The status information module 320 is used by the log collection terminal to send logs to be sent to the log analysis server and to obtain the sending status information. The local storage module 330 is used to continuously receive log data and write the logs to be sent in the abnormal state to the local storage file when the sending status information is in an abnormal state. The sending module 340 is used to read the logs to be sent from the local storage file when the sending status information recovers from the abnormal state to the normal state, and then send the read logs to the log analysis server.

[0059] In summary, the log transmission system without acknowledgment log transmission protocol provided in this application solves the problem in related technologies of how to achieve reliable log retention and continuous recovery through autonomous acquisition without modifying the log source device or relying on the protocol acknowledgment mechanism, through the synergistic effect of the preprocessing module, status information module, local storage writing module and sending module.

[0060] It should be noted that the log transmission system without acknowledgment log transmission protocol provided in this embodiment is used to implement the above-described embodiments, and details already described will not be repeated. As used above, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the above embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0061] Thirdly, embodiments of this application provide an electronic device, Figure 4 This is a block diagram illustrating an electronic device according to an exemplary embodiment. (e.g.) Figure 4 As shown, the electronic device may include a processor 81 and a memory 82 storing computer program instructions.

[0062] Specifically, the processor 81 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0063] The memory 82 may include a mass storage device for data or instructions. For example, and not limitingly, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include removable or non-removable (or fixed) media. Where appropriate, the memory 82 may be internal or external to a data processing device. In a particular embodiment, the memory 82 is non-volatile memory. In a particular embodiment, the memory 82 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0064] The memory 82 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 81.

[0065] The processor 81 implements any of the log transmission methods of the unacknowledged log transmission protocol in the above embodiments by reading and executing computer program instructions stored in the memory 82.

[0066] In one embodiment, a log transmission device without an acknowledgment log transmission protocol may further include a communication interface 83 and a bus 80. Wherein, as Figure 4 As shown, the processor 81, memory 82, and communication interface 83 are connected through bus 80 and complete communication with each other.

[0067] The communication interface 83 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 83 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0068] Bus 80 includes hardware, software, or both, that couples together components of a log transmission device with an unacknowledged log transmission protocol. Bus 80 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0069] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a log transmission method with an unacknowledged log transmission protocol provided in the first aspect.

[0070] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0071] In a possible implementation, the present invention can also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform steps of a log transmission method implementing an unacknowledged log transmission protocol provided in the first aspect.

[0072] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0073] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0074] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A log transmission method without acknowledgment log transmission protocol, characterized in that, The method includes: It continuously receives log data sent from the log source, preprocesses the received log data, and generates logs to be sent. The log collection terminal sends the logs to be sent to the log analysis server and obtains the sending status information; When the sending status information is in an abnormal state, the log collection terminal continues to receive the log data and writes the log to be sent in the abnormal state into a local storage file; When the sending status information recovers from an abnormal state to a normal state, the log collection terminal reads the log to be sent from the local storage file and sends the log to be sent to the log analysis server through the same processing link as the real-time log.

2. The log transmission method according to claim 1, characterized in that, When the log collection terminal is in remote collection mode and the sending status information is in an abnormal state, the log collection terminal continues to receive the log data and writes the log to be sent in the abnormal state to a local storage file, including: After confirming that it is in the abnormal state based on the sending status information, the log collection terminal continuously writes the logs to be sent into the memory queue. When the used capacity of the memory queue reaches a preset capacity threshold, the log collection terminal will transfer the logs to be sent temporarily stored in the memory queue to the local storage file in the order of writing, and release the used space of the memory queue. The log data is continuously received during the transfer process.

3. The log transmission method according to claim 2, characterized in that, When the log collection terminal is in local collection mode and the sending status information is in an abnormal state, the method further includes: The log collection terminal writes the logs to be sent into a memory queue; When the used capacity of the memory queue reaches a preset capacity threshold, the log collection terminal stops reading log data from the local log file and records the current consumed position of the local log file.

4. The log transmission method according to claim 1, characterized in that, When the log collection terminal is in remote collection mode and the sending status information recovers from an abnormal state to a normal state, the log collection terminal reads the log to be sent from the local storage file and sends the log to be sent to the log analysis server through the same processing link as the real-time log, including: After the log collection terminal confirms that the current state has recovered from the abnormal state to the normal state based on the sending status information, it triggers the historical log re-consumption process. In the historical log re-consumption process, the log collection terminal reads the logs to be sent from the local storage file in the order of writing. The log collection terminal reads the date to be sent and sends it to the log analysis server through the same processing link as the real-time log.

5. A log transmission method with an unacknowledged log transmission protocol according to claim 4, characterized in that, When the log collection terminal is in local collection mode and the sending status information recovers from an abnormal state to a normal state, the method further includes: The log collection terminal sends the logs to be sent that are temporarily stored in the memory queue to the log analysis server; After all the logs to be sent in the memory queue have been sent, the log collection terminal restarts reading the logs from the consumed position of the local log file, based on the recorded consumed position of the local log file, and continues log collection.

6. A log transmission method with an unacknowledged log transmission protocol according to claim 2, characterized in that, The log collection terminal transfers the logs to be sent temporarily stored in the memory queue to the local storage file in the order they were written, including: The log collection terminal performs rolling management of the local storage files according to a preset rolling strategy; the rolling strategy includes: Scroll by file size: When the size of the current local storage file reaches a preset file size threshold, close the current local storage file and create a new local storage file for writing; Rolling by time interval: When the continuous writing duration of the current local storage file reaches a preset time threshold, the current local storage file is closed and a new local storage file is created for writing.

7. A log transmission method using a non-acknowledgmented log transmission protocol according to claim 4, characterized in that, The log collection terminal reads the logs to be sent from the local storage file in the order they were written, including: The log collection terminal determines the list of local storage files to be read based on the creation time order of the local storage files; The log collection terminal opens the local storage files sequentially according to their creation time, from earliest to latest, based on the local storage file list. The log collection terminal reads the logs to be sent line by line from the opened local storage file. Each line of the logs to be sent corresponds to a log data record retained during the writing process.

8. A log transmission method with an unacknowledged log transmission protocol according to claim 1, characterized in that, The process of continuously receiving log data from a log source and preprocessing the received log data to generate logs to be sent includes: The log collection terminal receives raw log data through either remote collection mode or local collection mode; Based on the protocol type and log source identifier of the original log data, the corresponding log parsing rules are matched, the core log fields are extracted, and semi-structured log data is generated. Based on the local system clock of the log collection terminal, add a unified receiving timestamp and unique log source identifier metadata to the semi-structured log data; The semi-structured log data is filtered to remove those that do not conform to the filtering rules; the retained semi-structured log data is then formatted according to standard formats to generate logs to be sent in a unified format.

9. A log transmission method using a non-acknowledgmented log transmission protocol according to claim 1, characterized in that, The log collection terminal sends the logs to be sent to the log analysis server and obtains sending status information, including: The log collection terminal sends a request to the log analysis server to send the log to be sent based on a preset sending strategy. The log collection terminal determines whether the log to be sent has been successfully sent based on the response result of the sending request; If the number of logs to be sent that fail to be sent consecutively reaches a preset threshold, or the number of timeouts of the sending request reaches a preset timeout threshold, then the log collection terminal determines that the sending status information is the abnormal status. Otherwise, the log collection terminal determines that the sending status information is in the normal state.

10. A log transmission system without an acknowledgment log transmission protocol, characterized in that, The system is used to execute the method according to any one of claims 1 to 9, and the system includes a preprocessing module, a status information module, a local storage writing module, and a sending module; wherein: The preprocessing module is used to continuously receive log data sent by the log source, and preprocess the received log data to generate logs to be sent. The status information module is used by the log collection terminal to send the log to be sent to the log analysis server and to obtain the sending status information. The local storage writing module is used to continuously receive the log data and write the log to be sent in the abnormal state to a local storage file when the sending status information is in an abnormal state. The sending module is used to read the log to be sent from the local storage file and send the read log to be sent to the log analysis server when the sending status information recovers from an abnormal state to a normal state.

Images