Dual-mode distributed log acquisition and processing method and system

Abstract

The invention relates to a dual-mode distributed log collecting and processing method and system, and belongs to the technical field of computers.The method comprises the steps that parameters input by each distributed node are obtained, the parameters input by each distributed node are analyzed, and an analysis result of each distributed node is obtained; under the condition of determining that a target log file exists based on the analysis result, determining an acquisition mode of the target log file based on a mode log parameter, performing elastic acquisition on the target log file based on the acquisition mode, preprocessing the acquired target log file, and writing a preprocessing result into an intermediate result file; converting the intermediate result file into a request script and sending the request script to a dual-mode receiving interface; and the dual-mode receiving interface determines an acquisition mode based on the request script and executes a data receiving operation corresponding to the acquisition mode. According to the method, a lightweight acquisition mechanism is deployed on each distributed node, and two working modes of a real-time mode and a re-running mode are supported.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to a dual-mode distributed log collection and processing method and system. Background Technology

[0002] With the widespread adoption of microservices and distributed architectures, business applications are typically deployed across numerous server nodes, resulting in fragmented log data storage. Existing log collection solutions (such as Filebeat and Fluentd in the ELK architecture) primarily focus on centralized log collection and forwarding. However, when dealing with scenarios involving state statistics and data consistency requirements, they lack flexible data processing strategies and the process of correcting erroneous data is cumbersome. Therefore, how to better implement log collection has become an urgent problem to be solved. Summary of the Invention

[0003] In view of the shortcomings of the prior art, the purpose of the invention is to provide a dual-mode distributed log collection and processing method and system.

[0004] In a first aspect, the present invention proposes a dual-mode distributed log collection and processing method, comprising: S1, acquiring parameters input by each distributed node, and parsing the parameters input by each distributed node to obtain the parsing result of each distributed node, wherein each distributed node has a preset log directory, intermediate file directory, line number record directory, and log file name, wherein the line number record directory is used to store line number offset files named by date; S2, if a target log file is determined to exist based on the parsing result, determining the collection mode of the target log file based on the mode log parameters, and performing elastic collection of the target log file and preprocessing of the collected target log file based on the collection mode, and writing the preprocessing result into an intermediate result file, wherein the collection mode is a real-time mode or a rerun mode; S3, converting the intermediate result file into a request script and sending it to a dual-mode receiving interface; S4, the dual-mode receiving interface determining the collection mode based on the request script and executing the data receiving operation corresponding to the collection mode.

[0005] Further, determining the collection mode of the target log file based on the mode log parameter includes: determining whether the mode log parameter is empty; if the mode log parameter is empty, determining the collection mode of the target log file as the real-time mode; if the mode log parameter is not empty, determining the collection mode of the target log file as the rerun mode.

[0006] Further, collecting the target log file based on the collection mode includes: when the collection mode of the target log file is the real-time mode, determining whether a line record file for a preset date exists; if it exists, obtaining the previous line number of the line record file for the preset date, and using the next line after the previous line number as the starting line and the last line of the log file for the preset date as the ending line to extract incremental data, and recording the ending line in the line record file for the preset date as the basis for the next starting line; if it does not exist, creating the line record file for the preset date, and if it is determined that the line record file for the day before the preset date exists, processing the remaining logs of the day before the preset date; wherein, processing the remaining logs of the day before the preset date includes: obtaining the previous line number of the line record file for the day before the preset date, and using the next line after the previous line number as the starting line and the last line of the log file for the day before the preset date as the ending line to extract incremental data, and recording the ending line in the line record file for the day before the preset date.

[0007] Furthermore, collecting the target log file based on the collection mode includes: when the collection mode of the target log file is the rerun mode, reading the target log file in its entirety.

[0008] Furthermore, the collected target log file is preprocessed, including: filtering log lines that conform to preset business rules from the collected target log file, wherein the preset business rules include filtering conditions based on log fields; and grouping and counting the log lines that conform to the preset business rules according to different defined dimensions.

[0009] Furthermore, the collection frequency of the target log file based on the collection mode is configured by an external scheduled task, wherein the collection frequency is dynamically adjusted by modifying the expression of the external scheduled task.

[0010] Furthermore, the request script includes the URL of the dual-mode receiving interface, the data of the intermediate result file, key data types, and mode log parameters. Executing the data receiving operation corresponding to the acquisition mode includes: when the acquisition mode is the real-time mode, the dual-mode receiving interface performs an accumulation operation on the received data of the intermediate result file according to the business key; when the acquisition mode is the rerun mode, the dual-mode receiving interface performs an overwrite operation on data with the same business key to replace historical data.

[0011] A second aspect of the present invention proposes a dual-mode distributed log acquisition and processing system, characterized in that it includes: an acquisition module, configured to acquire parameters input by each distributed node and parse the parameters input by each distributed node to obtain the parsing result of each distributed node, wherein each distributed node has a preset log directory, intermediate file directory, line number record directory, and log file name, the line number record directory being used to store line number offset files named by date; and an acquisition and processing module, configured to determine the acquisition mode of the target log file based on mode log parameters when the parsing result indicates the existence of a target log file, and based on the parsing result... The system performs elastic data collection on the target log file in the specified collection mode and preprocesses the collected target log file, writing the preprocessing result to an intermediate result file. The collection mode can be either real-time or rerun mode. The collection frequency of the target log file based on the collection mode is configured by an external scheduled task, and the collection frequency is dynamically adjusted by modifying the expression of the external scheduled task. A sending module converts the intermediate result file into a request script and sends it to the dual-mode receiving interface. An execution module is used by the dual-mode receiving interface to determine the collection mode based on the request script and execute the data receiving operation corresponding to the collection mode.

[0012] A third aspect of the present invention provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method described in any one aspect of the present invention.

[0013] A fourth aspect of the present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described in any one of the first aspects of the present invention.

[0014] The beneficial effects of this invention are as follows:

[0015] The dual-mode distributed log collection and processing method and system of this invention acquires parameters input from each distributed node and parses these parameters to obtain the parsing results for each distributed node. Each distributed node has a preset log directory, intermediate file directory, line number record directory, and log file name. The line number record directory stores line number offset files named by date. If the parsing results indicate the existence of a target log file, the collection mode for the target log file is determined based on the mode log parameters. Elastic collection and preprocessing of the collected target log file are then performed based on the collection mode, and the preprocessing results are written to an intermediate result file. The collection mode can be either real-time mode or rerun mode. The intermediate result file is converted into a request script and sent to the dual-mode receiving interface. The dual-mode receiving interface determines the collection mode based on the request script and executes the data receiving operation corresponding to the collection mode. This method deploys a lightweight collection mechanism on each distributed node, supports both real-time and rerun modes, and achieves efficient and reliable processing of log data based on the dual-mode intelligent data receiving mechanism. Simultaneously, the dual-mode mechanism ensures data consistency and recoverability. Attached Figure Description

[0016] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0017] Figure 1 This is a flowchart of a dual-mode distributed log collection and processing method according to an embodiment of the present invention;

[0018] Figure 2 This is a flowchart of a dual-mode distributed log collection and processing method according to a specific embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram of a dual-mode distributed log collection and processing system according to an embodiment of the present invention;

[0020] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0023] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0025] With the widespread adoption of microservices and distributed architectures, business applications are typically deployed across a large number of server nodes, resulting in fragmented log data storage. Existing log collection solutions (such as Filebeat and Fluentd in the ELK architecture) primarily focus on centralized log collection and forwarding, but they suffer from the following drawbacks when dealing with scenarios involving state statistics and data consistency requirements:

[0026] Lack of flexible data processing strategies: Existing tools typically forward log lines simply, while complex filtering and statistical (such as counting by dimension) business logic needs to be completed at the central end, increasing the processing pressure on the central node and the degree of data coupling.

[0027] The process of fixing erroneous data is cumbersome: When it is found that the historical log processing logic is wrong or needs to be re-statistically analyzed, the existing solution often requires stopping real-time collection, writing a one-time script to reprocess the entire log file, which is cumbersome, error-prone, and affects the real-time data stream.

[0028] Rigid acquisition frequency: The acquisition frequency is usually fixed at startup, making it difficult to dynamically adjust the acquisition cycle according to the peaks and troughs of data volume, and it is impossible to achieve a flexible balance between real-time performance and system overhead.

[0029] To address this, the present invention proposes a dual-mode distributed log collection and processing method, system, and related equipment. Specifically, the dual-mode distributed log collection and processing method, system, and related equipment of the present invention are described below with reference to the accompanying drawings.

[0030] Figure 1 This is a flowchart of a dual-mode distributed log collection and processing method according to an embodiment of the present invention. It should be noted that the dual-mode distributed log collection and processing method of this invention can be applied to the dual-mode distributed log collection and processing system of this invention. This dual-mode distributed log collection and processing system can be configured on an electronic device or in a server. This application does not limit the scope of the application.

[0031] like Figure 1 As shown, the dual-mode distributed log collection and processing method includes:

[0032] S110: Obtain the parameters input by each distributed node, parse the parameters input by each distributed node, and obtain the parsing result of each distributed node. Each distributed node has a preset log directory, intermediate file directory, line number record directory, and log file name. The line number record directory is used to store line number offset files named by date.

[0033] In embodiments of the present invention, preset log directories, intermediate file directories, line number record directories, and log file names can be set on each distributed node to obtain the parameters input by each distributed node. In other words, each distributed node (e.g., a server) can independently run a collection agent to process local log files.

[0034] The input parameters include, but are not limited to, IP address, date, and mode log parameters. The IP address is used to distinguish the data collection node, the date is used to distinguish which day's log file is being collected, and the mode log parameters are used to distinguish the data collection mode.

[0035] S120, if the existence of the target log file is determined based on the parsing results, the collection mode of the target log file is determined based on the mode log parameters, and the target log file is elastically collected based on the collection mode and preprocessed after collection. The preprocessing results are written to an intermediate result file. The collection mode is either real-time mode or rerun mode.

[0036] In an embodiment of the present invention, after obtaining the parsing results of each distributed node, if the existence of a target log file is determined based on the results, the collection mode of the target log file is determined based on the pattern log parameters. Then, an elastic collection operation is performed based on the collection mode. Afterwards, the collected target log file is preprocessed, and the preprocessing result is written to an intermediate result file. Specific implementation details can be found in subsequent embodiments.

[0037] S130 converts the intermediate result file into a request script and sends it to the dual-mode receiving interface.

[0038] In an embodiment of the present invention, the request script includes the URL of the dual-mode receiving interface, the data of the intermediate result file, key data types, and mode log parameters.

[0039] The request script can be an HTTP request script.

[0040] S140, the dual-mode receiving interface determines the acquisition mode based on the request script and executes the data receiving operation corresponding to the acquisition mode.

[0041] In embodiments of the present invention, when the acquisition mode is real-time, the dual-mode receiving interface performs an accumulation operation on the data of the received intermediate result file according to the service key; when the acquisition mode is rerun mode, the dual-mode receiving interface performs an overwrite operation on the data with the same service key to replace historical data. Specific implementation details can be found in subsequent embodiments.

[0042] According to an embodiment of the present invention, a dual-mode distributed log collection and processing method is used to acquire parameters input from each distributed node and parse these parameters to obtain the parsing results for each distributed node. Each distributed node has a preset log directory, intermediate file directory, line number record directory, and log file name. The line number record directory stores line number offset files named by date. If the existence of a target log file is determined based on the parsing results, the collection mode of the target log file is determined based on the mode log parameters. Based on the collection mode, elastic collection of the target log file is performed, and the collected target log file is preprocessed. The preprocessing results are written to an intermediate result file. The collection mode can be either real-time mode or rerun mode. The intermediate result file is converted into a request script and sent to a dual-mode receiving interface. The dual-mode receiving interface determines the collection mode based on the request script and executes the data receiving operation corresponding to the collection mode. This method deploys a lightweight collection mechanism on each distributed node, supports both real-time and rerun modes, and achieves efficient and reliable processing of log data based on a dual-mode intelligent data receiving mechanism. Simultaneously, the dual-mode mechanism ensures data consistency and recoverability.

[0043] To enable those skilled in the art to more readily understand the present invention, Figure 2 This is a dual-mode distributed log collection and processing method according to a specific embodiment of the present invention, such as... Figure 2 As shown, this dual-mode distributed log collection and processing method includes:

[0044] S210, obtain the parameters input by each distributed node, parse the parameters input by each distributed node, and obtain the parsing result of each distributed node. Each distributed node has a preset log directory, intermediate file directory, line number record directory and log file name. The line number record directory is used to store line number offset files named by date.

[0045] S220, Based on the parsing results, it is determined that the target log file exists.

[0046] In embodiments of the present invention, the parsing result carries parameters of the distributed nodes, and each distributed node has a preset log directory, intermediate file directory, line number record directory, and log file name. Therefore, the existence of the target log file can be determined by the preset log directory and log file name. For example, the existence can be determined by the log file name.

[0047] S230, if the existence of the target log file is determined based on the parsing results, the collection mode of the target log file is determined based on the mode log parameters.

[0048] In an embodiment of the present invention, if the existence of a target log file is determined based on the parsing result, it is determined whether the mode log parameter (i.e., the rerun parameter) is empty; if the mode log parameter is empty, the collection mode of the target log file is determined to be real-time mode; if the mode log parameter is not empty, the collection mode of the target log file is determined to be rerun mode.

[0049] S240 collects target log files when the collection mode is set to real-time.

[0050] In an embodiment of the present invention, when the target log file is collected in real-time mode, it is determined whether a line record file for a preset date exists. If it exists, the previous line number of the line record file for the preset date is obtained, and the next line after the previous line number is used as the starting line, and the last line of the log file for the preset date is used as the ending line to extract incremental data. The ending line is recorded in the line record file for the preset date and used as the basis for the next starting line. If it does not exist, a line record file for the preset date is created, and if a line record file for the day before the preset date exists, the remaining logs of the day before the preset date are processed.

[0051] The process of processing the remaining logs of the day before the preset date includes: obtaining the previous line number of the previous day's line record file, using the next line after the previous line number as the starting line, extracting incremental data using the last line of the previous day's log file as the ending line, and recording the ending line in the previous day's line record file.

[0052] For example, if the file does not exist, create a line record file for the preset date and initialize the offset to 0. This also means that a portion of the log file from the day before the preset date has not been fully transmitted. Subtract one day from the preset date, i.e., the day before the preset date, obtain the last line number of the previous day's line record file, and use the line following the last line number as the starting line and the last line of the previous day's log file as the ending line to extract incremental data.

[0053] The S250 collects target log files when the collection mode is set to rerun mode.

[0054] In an embodiment of the present invention, when the target log file is collected in rerun mode, the target log file is read in its entirety.

[0055] In other words, when the collection mode is in rerun mode, line number records are ignored, and the entire target log file is read from the beginning.

[0056] In embodiments of the present invention, the collection frequency of target log file collection based on the collection mode is configured by an external scheduled task. The collection frequency is dynamically adjusted by modifying the expression of the external scheduled task. This achieves seamless collection agent operation and eliminates the need for restarts throughout the entire process, decoupling the collection rhythm from the collection logic and enabling dynamic frequency configuration. This allows the system to better adapt to changes in business traffic, striking a balance between real-time requirements and system overhead.

[0057] In embodiments of this invention, two working modes—real-time incremental and full rerun—are integrated within the same acquisition mechanism and can be switched via input parameters. Based on real-time acquisition logs, a one-click data rerun capability is integrated, reducing the complexity and manual operation risks associated with backtracking and correction due to historical data errors. Furthermore, each distributed node independently maintains a lightweight line number record file locally, enabling persistence of the acquisition status and fault recovery.

[0058] S260 preprocesses the collected target log file and writes the preprocessing results into an intermediate result file.

[0059] In an embodiment of the present invention, after collecting the target log file based on the collection mode, log lines that conform to preset business rules can be filtered out from the collected target log file. The preset business rules include filtering conditions based on log fields. The log lines that conform to the preset business rules are grouped and counted according to different defined dimensions.

[0060] For example, after collecting data from the target log file, where each line represents a payment record, based on pre-defined business rules, the system "filters out all orders with successful payments and virtual goods" from the target log file. For instance, the first line might be: ...0 1...-> Meets the criteria (successful payment and is a virtual product); the second line might be: ...1 1...-> Does not meet the criteria (payment failed); the third line might be: ...0 1...-> Meets the criteria; the fourth line might be: ...0 0...-> Does not meet the criteria (is not a virtual product); and the fifth line might be: ...0 1...-> Meets the criteria, resulting in three valid log lines. This filtering process obtains log lines that meet the pre-defined business rules, reducing the amount of data processed subsequently and allowing the extraction of data relevant to specific business scenarios (such as "virtual product revenue statistics").

[0061] Next, the filtered data is grouped according to multiple dimensions, such as username, product name, and server IP. Assuming the server IP is 192.168.1.xx, the filtered three records are grouped as follows:

[0062] UserA,ProductX,192.168.1.xx;

[0063] UserA, ProductX, 192.168.1.xx (same as the first dimension);

[0064] UserA,ProductY,192.168.1.xx.

[0065] Among them, group UserA|||ProductX|||192.168.1.xx appeared twice, and group UserA|||ProductY|||192.168.1.xx appeared once. This reduces the amount of data that needs to be transmitted and stored. That is, what used to require transmitting 3 complete log lines now only requires transmitting 2 statistical results.

[0066] Therefore, by performing log filtering, dimensional aggregation, and statistics on distributed nodes, the central processing pressure is reduced, network transmission volume is decreased, and the system architecture becomes more scalable.

[0067] S270 converts the intermediate result file into a request script and sends it to the dual-mode receiving interface.

[0068] In an embodiment of the present invention, the request script includes the URL of the dual-mode receiving interface, the data of the intermediate result file, key data types, and mode log parameters.

[0069] The S280 dual-mode receiving interface determines the acquisition mode based on the request script and executes the data receiving operation corresponding to the acquisition mode.

[0070] In an embodiment of the present invention, when the acquisition mode is real-time mode, the dual-mode receiving interface performs an accumulation operation on the data of the received intermediate result file according to the service key; when the acquisition mode is rerun mode, the dual-mode receiving interface performs an overwrite operation on the data with the same service key to replace the historical data.

[0071] In real-time acquisition mode, this means the uploaded data is a periodic, incremental statistical result. In other words, the dual-mode receiving interface extracts the business key and the current incremental statistical value from the received intermediate result file, and performs an accumulation operation on records with the same business key and date in the database. For example;

[0072] Existing records in the database: (Key='UserA|Pay|192.168.1.1', Day='2024-01-01', Count=10);

[0073] Real-time uploaded data: (Key='UserA|Pay|192.168.1.1', Day='2024-01-01', Count=2);

[0074] Operation result: The records in the database were updated to Count = 12 (i.e., 10 + 2). This is because real-time mode uploads in batches. The first upload might have counted 5 records, the second 3, and the third 4. Only by accumulating these counts can the correct total of 12 be obtained.

[0075] In the case of rerun mode, the currently uploaded data is a complete recalculation of data from a specific historical time period. In other words, the dual-mode receiving interface extracts the business key (or dimension combination, such as user ID|operation type|node IP) and the corresponding new statistical value based on the request script, and directly uses this new value to overwrite existing records in the database with the same business key and date. For example:

[0076] Existing records in the database: (Key='UserA|Pay|192.168.1.1', Day='2024-01-01', Count=10);

[0077] Re-uploaded data: (Key='UserA|Pay|192.168.1.1', Day='2024-01-01', Count=12);

[0078] Operation result: The record in the database was updated to Count=12. The old 10 was completely discarded. Because the rerun mode has performed a complete recount of the entire log file, it yielded the correct new result. This new result should completely replace the potentially erroneous old result.

[0079] Therefore, the dual-mode receiving interface can automatically distinguish between "incremental data" and "repair data" and take the correct actions. It handles real-time streaming data processing through "accumulation" and achieves data repair through "overwriting," ensuring the final correctness of data in a distributed environment. The complex data repair process is simplified into a single command (triggering rerun mode), with the backend automatically completing all overwriting operations without manual database intervention. In other words, the interface design, which performs different operations (accumulation or overwriting) on ​​data with the same business key based on the mode log parameter `rerun`, ensures that even when data is uploaded in batches and arrives out of order, the final result remains accurate.

[0080] Based on the above scheme, regardless of the number of data collection nodes or the number of uploads, the results in the database can be guaranteed to be correct. Furthermore, by distributing the data processing logic to various distributed nodes for independent execution, the computational burden on the central node is effectively reduced, improving the overall processing performance and log collection efficiency of the system.

[0081] According to an embodiment of the present invention, a dual-mode distributed log collection and processing method deploys a lightweight collection mechanism on each distributed node. This mechanism achieves incremental breakpoint resumption by reading the line number offset of local records. It operates in either real-time or rerun mode at a configurable frequency based on external instructions (such as timing signals or rerun commands). After the collected log lines are filtered locally and statistically analyzed according to business rules, the resulting data is sent to a backend receiving interface with dual-mode processing capabilities. This receiving interface determines whether to accumulate or overwrite the data based on the mode log parameters, thereby achieving eventual consistency of distributed data.

[0082] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0083] According to one aspect of the present invention, a dual-mode distributed log collection and processing system is also proposed. Figure 3 This is a schematic diagram of a dual-mode distributed log collection and processing system according to an embodiment of the present invention; as shown below. Figure 3 As shown, it includes:

[0084] The acquisition module 310 is used to acquire the parameters input by each distributed node and parse the parameters input by each distributed node to obtain the parsing result of each distributed node. Each distributed node has a preset log directory, intermediate file directory, line number record directory and log file name. The line number record directory is used to store line number offset files named by date.

[0085] The acquisition and processing module 320 is used to determine the acquisition mode of the target log file based on the mode log parameters when the existence of the target log file is determined based on the parsing result, and to perform elastic acquisition of the target log file and preprocess the acquired target log file based on the acquisition mode, and write the preprocessing result to an intermediate result file. The acquisition mode is either real-time mode or rerun mode. The acquisition frequency of the target log file acquisition based on the acquisition mode is configured by an external timed task, and the acquisition frequency is dynamically adjusted by modifying the expression of the external timed task.

[0086] Sending module 330 is used to convert the intermediate result file into a request script and send it to the dual-mode receiving interface;

[0087] The execution module 340 is used for the dual-mode receiving interface to determine the acquisition mode based on the request script and to execute the data receiving operation corresponding to the acquisition mode.

[0088] According to an embodiment of the present invention, a dual-mode distributed log acquisition and processing system acquires parameters input from each distributed node and parses these parameters to obtain the parsing results for each distributed node. Each distributed node has a preset log directory, intermediate file directory, line number record directory, and log file name. The line number record directory stores line number offset files named by date. If the existence of a target log file is determined based on the parsing results, the acquisition mode of the target log file is determined based on the mode log parameters. Based on the acquisition mode, elastic acquisition of the target log file is performed, and the acquired target log file is preprocessed. The preprocessing results are written to an intermediate result file. The acquisition mode can be either real-time mode or rerun mode. The intermediate result file is converted into a request script and sent to the dual-mode receiving interface. The dual-mode receiving interface determines the acquisition mode based on the request script and executes the data receiving operation corresponding to the acquisition mode. Thus, a lightweight acquisition mechanism is deployed on each distributed node, supporting both real-time and rerun modes. Based on the dual-mode intelligent data receiving mechanism, efficient and reliable processing of log data is achieved, while ensuring data consistency and recoverability through the dual-mode mechanism.

[0089] Optionally, the acquisition and processing module 320 is specifically used to determine whether the mode log parameter is empty; if the mode log parameter is empty, determine that the acquisition mode of the target log file is the real-time mode; if the mode log parameter is not empty, determine that the acquisition mode of the target log file is the rerun mode.

[0090] Optionally, the acquisition and processing module 320 is specifically used to determine whether a line record file for a preset date exists when the acquisition mode of the target log file is the real-time mode; if it exists, it obtains the previous line number of the line record file for the preset date, and extracts incremental data with the next line of the previous line number as the starting line and the last line of the log file for the preset date as the ending line, and records the ending line in the line record file for the preset date as the basis for the next starting line; if it does not exist, it creates a line record file for the preset date, and if it is determined that a line record file for the day before the preset date exists, it processes the remaining logs of the day before the preset date; wherein, processing the remaining logs of the day before the preset date includes: obtaining the previous line number of the line record file for the day before the preset date, and extracting incremental data with the next line of the previous line number as the starting line and the last line of the log file for the day before the preset date as the ending line, and recording the ending line in the line record file for the day before the preset date.

[0091] Optionally, the acquisition and processing module 320 is specifically used to read the target log file in its entirety when the acquisition mode of the target log file is the rerun mode.

[0092] Optionally, the acquisition and processing module 320 is used to filter log lines that conform to preset business rules from the acquired target log file, wherein the preset business rules include filtering conditions based on log fields; and to group and count the log lines that conform to the preset business rules according to different defined dimensions.

[0093] Optionally, the request script includes the URL of the dual-mode receiving interface, the data of the intermediate result file, key data types, and mode log parameters. Specifically, the execution module 340 is used to perform an accumulation operation on the data of the received intermediate result file according to the business key when the acquisition mode is the real-time mode; and to perform an overwrite operation on the data with the same business key to replace historical data when the acquisition mode is the rerun mode.

[0094] According to one aspect of the present invention, an electronic device is provided.

[0095] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Figure 4 As shown, an electronic device may include one or more ( Figure 4Only one is shown in the image. A processor 102 (which may include, but is not limited to, a microprocessor unit (MPU) or a programmable logic device (PLD)) and a memory 104 for storing data are also shown. In one exemplary embodiment, the electronic device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 4 The structure shown is for illustrative purposes only and does not limit the structure of the terminal device described above. For example, the terminal device may also include components that are more... Figure 4 The more or fewer components shown, or having the same Figure 4 Equivalent functions or ratios shown Figure 4 The functions shown have more different configurations.

[0096] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the dual-mode distributed log collection and processing method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to terminal devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0097] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the switching device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0098] This invention proposes a non-transitory computer-readable storage medium storing computer instructions, which are used to enable the computer to execute a dual-mode distributed log collection and processing method.

[0099] The applicant of this invention has provided a detailed description of the embodiments of the invention in conjunction with the accompanying drawings. However, those skilled in the art should understand that the above embodiments are merely preferred embodiments of the invention. The detailed description is only intended to help readers better understand the spirit of the invention and is not intended to limit the scope of protection of the invention. On the contrary, any improvements or modifications made based on the inventive spirit of the invention should fall within the scope of protection of the invention.

[0100] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0101] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A dual-mode distributed log collection and processing method, characterized in that, include: S1, obtain the parameters input by each distributed node, and parse the parameters input by each distributed node to obtain the parsing result of each distributed node. Each distributed node has a preset log directory, intermediate file directory, line number record directory and log file name. The line number record directory is used to store line number offset files named by date. S2, if the existence of a target log file is determined based on the parsing result, the collection mode of the target log file is determined based on the mode log parameters, and the target log file is elastically collected based on the collection mode and preprocessed after collection. The preprocessing result is written into an intermediate result file, wherein the collection mode is real-time mode or rerun mode. S3, convert the intermediate result file into a request script and send it to the dual-mode receiving interface; S4, the dual-mode receiving interface determines the acquisition mode based on the request script and executes the data receiving operation corresponding to the acquisition mode.

2. The dual-mode distributed log collection and processing method according to claim 1, characterized in that, Determining the collection mode of the target log file based on the pattern log parameters includes: Determine whether the mode log parameter is empty; If the mode log parameter is empty, the collection mode of the target log file is determined to be the real-time mode; If the mode log parameter is not empty, the collection mode of the target log file is determined to be the rerun mode.

3. The dual-mode distributed log collection and processing method according to claim 2, characterized in that, The target log file is collected based on the aforementioned collection mode, including: If the target log file is collected in real-time mode, determine whether a line record file for a preset date exists. If it exists, obtain the last line number of the line record file of the preset date, and take the next line of the last line number as the starting line and the last line of the log file of the preset date as the ending line to extract incremental data, and record the ending line in the line record file of the preset date as the basis for the next starting line; If it does not exist, create a line log file for the preset date, and if a line log file for the day before the preset date exists, process the remaining logs for the day before the preset date; The process of processing the remaining logs of the day before the preset date includes: obtaining the previous line number of the previous day's line record file, taking the next line after the previous line number as the starting line, taking the last line of the previous day's log file as the ending line to extract incremental data, and recording the ending line in the previous day's line record file.

4. The dual-mode distributed log collection and processing method according to claim 2, characterized in that, The target log file is collected based on the aforementioned collection mode, including: When the target log file is collected in the rerun mode, the target log file is read in its entirety.

5. The dual-mode distributed log collection and processing method according to claim 2, characterized in that, The collected target log files are preprocessed, including: Log lines that conform to preset business rules are selected from the collected target log files, wherein the preset business rules include filtering conditions based on log fields; Log lines that conform to the preset business rules are grouped and counted according to different defined dimensions.

6. The dual-mode distributed log collection and processing method according to claim 1, characterized in that, The collection frequency of the target log file based on the collection mode is configured by an external scheduled task, wherein the collection frequency is dynamically adjusted by modifying the expression of the external scheduled task.

7. The dual-mode distributed log collection and processing method according to claim 1, characterized in that, The request script includes the URL of the dual-mode receiving interface, the data of the intermediate result file, key data types, and mode log parameters. Executing the data receiving operation corresponding to the acquisition mode includes: When the acquisition mode is the real-time mode, the dual-mode receiving interface performs an accumulation operation on the data of the received intermediate result file according to the service key; When the acquisition mode is the rerun mode, the dual-mode receiving interface performs an overwrite operation on data with the same service key to replace historical data.

8. A dual-mode distributed log collection and processing system, characterized in that, include: The acquisition module is used to acquire the parameters input by each distributed node and parse the parameters input by each distributed node to obtain the parsing result of each distributed node. Each distributed node has a preset log directory, intermediate file directory, line number record directory and log file name. The line number record directory is used to store line number offset files named by date. The acquisition and processing module is used to determine the acquisition mode of the target log file based on the mode log parameters when the existence of the target log file is determined based on the parsing result, and to perform elastic acquisition of the target log file based on the acquisition mode and preprocess the acquired target log file, and write the preprocessing result to an intermediate result file. The acquisition mode is either real-time mode or rerun mode. The acquisition frequency of the target log file acquisition based on the acquisition mode is configured by an external scheduled task, and the acquisition frequency is dynamically adjusted by modifying the expression of the external scheduled task. The sending module is used to convert the intermediate result file into a request script and send it to the dual-mode receiving interface; An execution module is used for the dual-mode receiving interface to determine the acquisition mode based on the request script and to execute the data receiving operation corresponding to the acquisition mode.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 7.

Images