Log processing method and device, electronic equipment and computer program product

Abstract

The invention relates to a log processing method and device, electronic equipment and a computer program product, and belongs to the technical field of computers. The method comprises the following steps: reading a whole line of log data from each process in sequence, and putting the read whole line of log data into a write cache multi-grid queue in sequence; according to the storage sequence of the whole line of log data in the write cache multi-grid queue, sequentially obtaining the whole line of log data from the write cache multi-grid queue; and performing exception processing on the whole line of log data, and writing the processed whole line of log data into a memory for storage. The log processing efficiency can be improved, the log integrity is ensured, and the problems of data loss and blocking in the processing process are avoided.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a log processing method, a log processing apparatus, an electronic device, and a computer program product. Background Technology

[0002] In server operation and maintenance environments such as Linux, process management tools, such as supervisor, are mainly used to perform various operations such as starting, stopping, and monitoring processes.

[0003] However, current process management tools mainly use single-threaded pipes to read and write program log messages, which exposes obvious performance bottlenecks in high-traffic business scenarios. When data traffic increases significantly, log processing efficiency drops sharply, causing frequent application stuttering and log data loss.

[0004] Therefore, there is an urgent need in this field for a log processing method that can improve log processing efficiency, ensure log integrity, and avoid data loss and blocking problems during the processing.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a log processing method, log processing device, electronic device, and computer program product, which can at least to some extent improve log processing efficiency, ensure log integrity, and avoid data loss and blocking problems during processing.

[0007] According to a first aspect of this disclosure, a log processing method is provided, comprising: Read the entire line of log data from each process in sequence, and put the read entire line of log data into the write cache multi-cell queue in sequence; According to the order in which the entire log data is stored in the write cache multi-cell queue, the entire log data is retrieved sequentially from the write cache multi-cell queue; The entire log data is subjected to anomaly processing, and the processed log data is written into the memory for storage.

[0008] According to a second aspect of this disclosure, a log processing apparatus is provided, comprising: The whole line log reading module is used to read whole line log data from each process in sequence, and put the read whole line log data into the write cache multi-cell queue in sequence; The multi-grid queue processing module is used to sequentially retrieve the entire log data from the write cache multi-grid queue according to the storage order of the entire log data in the write cache multi-grid queue; The log data writing module is used to perform anomaly processing on the entire line of log data and write the processed entire line of log data into the memory for storage.

[0009] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the log processing method described in any one of the preceding claims by executing the executable instructions.

[0010] According to a fourth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the log processing method described in any one of the preceding claims.

[0011] The exemplary embodiments disclosed herein can have the following beneficial effects: In the log processing method of this exemplary embodiment, entire lines of log data are read sequentially from each process and placed into a write cache multi-cell queue. Log data is then retrieved and processed sequentially from the write cache multi-cell queue. On the one hand, log integrity is ensured, preventing data loss. On the other hand, the multi-threaded synchronous processing mechanism of the write cache multi-cell queue ensures that the log processing logic of each process is independent, thereby preventing multiple log processes from affecting subsequent processes. This improves log processing efficiency, effectively reduces program lag, avoids blocking issues during processing, and delivers a better user experience.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0014] Figure 1 A flowchart illustrating a log processing method according to a related embodiment of the present disclosure is shown; Figure 2 A flowchart illustrating a log processing method according to an exemplary embodiment of this disclosure is shown. Figure 3A schematic diagram of the process for reading an entire line of log data from various processes is shown in an exemplary embodiment of this disclosure; Figure 4 This illustration shows a flowchart of reading an entire line of log data from the current process according to an example implementation of this disclosure; Figure 5 A schematic diagram of a read cache splicing module according to a specific embodiment of the present disclosure is shown; Figure 6 A flowchart illustrating a log processing method according to a specific embodiment of the present disclosure is shown. Figure 7 A block diagram of a log processing apparatus according to an exemplary embodiment of the present disclosure is shown; Figure 8 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown. Detailed Implementation

[0015] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0016] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0017] In some related implementations, log processing and sending can be performed using the community version of supervisor, a process management tool for Linux systems that can start and stop processes. The supervisor tool can manage multiple processes (with the number n representing the managed process number).

[0018] Figure 1 A flowchart illustrating a log processing method according to a related embodiment of the present disclosure is shown, which may specifically include the following steps: Step S110. Supervisor reads a line of log from process_1 through “anonymous pipe_1” and puts it into a single-cell queue.

[0019] Each queue can only hold data from one process at a time. Furthermore, if a log line is too long, the supervisor will randomly split the log into multiple lines before reading it.

[0020] Step S120. Supervisor retrieves all logs (which may contain multiple lines) of process_1 from the single-cell queue and writes them to third-party storage.

[0021] This step takes a long time. In addition, if there are abnormal or illegal characters in the log, the supervisor will discard all log data and not send it.

[0022] Step S130. Supervisor reads the log of process 2 through “anonymous pipe 2” and puts it into a single queue.

[0023] Step S140. Supervisor retrieves the logs of process_2 from the single-cell queue and sends them to third-party storage.

[0024] Step S150. Process sequentially until the data of process _n has been processed by the Supervisor.

[0025] The above process demonstrates that the community version of Supervisor processes the log data of each process sequentially before forwarding it. If one process has a large log volume, the Supervisor's time spent sending that process's logs increases, affecting the log data of subsequent processes and causing delays in transmission. In summary, the community version of Supervisor has the following drawbacks: 1. When reading logs, if a log line is too long, it will be randomly split into multiple lines, causing abnormal line breaks or loss of logs, which will cause business operation abnormalities.

[0026] 2. When writing logs, if a log line contains illegal characters, the entire log line will be discarded without any notification to the user that the data has been lost.

[0027] 3. If a process has a large log volume, the time taken for sending steps will increase, which will cause the logs of subsequent queued processes to be delayed in processing.

[0028] To address the aforementioned issues, this example implementation first provides a log processing method. (Reference) Figure 2 As shown, the above log processing method may include the following steps: Step S210. Read the entire line of log data from each process in sequence, and put the read entire line of log data into the write cache multi-cell queue in sequence.

[0029] Step S220. Based on the order in which the entire log data is stored in the write cache multi-cell queue, retrieve the entire log data sequentially from the write cache multi-cell queue.

[0030] Step S230. Perform exception handling on the entire line of log data, and write the processed entire line of log data into the memory for storage.

[0031] In the log processing method of this exemplary embodiment, entire lines of log data are read sequentially from each process and placed into a write cache multi-cell queue. Log data is then retrieved and processed sequentially from the write cache multi-cell queue. On the one hand, log integrity is ensured, preventing data loss. On the other hand, the multi-threaded synchronous processing mechanism of the write cache multi-cell queue ensures that the log processing logic of each process is independent, thereby preventing multiple log processes from affecting subsequent processes. This improves log processing efficiency, effectively reduces program lag, avoids blocking issues during processing, and delivers a better user experience.

[0032] Below, in conjunction with Figures 3 to 6 The steps described above in this example implementation will be explained in more detail.

[0033] In step S210, the entire line of log data is read from each process in sequence, and the read entire line of log data is placed into the write cache multi-cell queue in sequence.

[0034] In this example implementation, a process refers to a program's execution activity on a certain set of data in a computer system. A complete log line refers to a single, unbroken log record identified by a newline character or other specific terminator.

[0035] In this example implementation, the write cache multi-cell queue is a cache structure used for temporary storage of log data. Each cell in the write cache multi-cell queue is used to store the log data corresponding to a process, and the write cache multi-cell queue can store log data from multiple processes at the same time.

[0036] In a write-cached multi-cell queue, each cell stores the log data of one process. If a process has multiple lines, they are placed in the same cell. The community version's single-cell queue can only hold the data of one process at a time. However, the write-cached multi-cell queue in this example implementation can store the data of multiple processes at a time. This process data is not sent out immediately but is cached in the multi-cell queue.

[0037] In this example implementation, as Figure 3 As shown, the entire line of log data is read from each process sequentially, and the read log data is placed into the write cache multi-cell queue in turn. Specifically, this may include the following steps: Step S310. Add the read cache splicing module in the process management tool.

[0038] In this example implementation, the process management tool can be, for example, Supervisor. The read cache splicing module is a software component whose main function is to temporarily store partial data read from the data source and splice these partial data into a complete data unit when certain conditions are met. This module may include an internal buffer for temporarily storing data fragments, and a set of logic to determine when the collection of a complete data unit is complete and to splice it.

[0039] Step S320. Read the entire line of log data from the current process through the read cache splicing module, and put the read entire line of log data into the write cache multi-cell queue.

[0040] Reading a complete line of log data from the current process aims to retrieve the entire log line from the currently running process. Its purpose is to ensure the integrity of the log data and avoid information loss or incorrect parsing due to incomplete reading.

[0041] In this example implementation, as Figure 4 As shown, reading an entire line of log data from the current process using the read cache concatenation module can specifically include the following steps: Step S410. Read a log string from the current process using the read cache concatenation module, and determine whether the last character of the log string is a newline character.

[0042] During the process of reading log data from the current process, a log string is first read from the current process through the read cache concatenation module. After obtaining the log string, it is necessary to determine whether the last character of the log string is a newline character. This determination is crucial for identifying the integrity of the log line; in many systems, the newline character is used as the end marker for a log line.

[0043] For example, in a Linux system, a log line starts with... The end of the log string. When the module receives the log, it will first check if the end of the log string is a string ending with a specific character. End of character.

[0044] Step S420. If the last character of the log string is a newline character, then treat the log string as the entire line of log data.

[0045] If the determination result is that the last character of the log string is a newline character, it indicates that the currently read log string is a complete log line and can be directly processed. In this case, the log string is used as the entire log line data.

[0046] Step S430. If the last character of the log string is not a newline character, continue reading the next log string and concatenate the log strings sequentially until a log string with a newline character at the end is read.

[0047] If not, the data is first cached in the read cache concatenation module, and the module continues to receive subsequent log data and concatenate them sequentially until log data ending with a newline character is read. Once these log data are concatenated, the log becomes a complete line of data, and the read cache concatenation module sends out the log line.

[0048] Figure 5 A schematic diagram of a read cache concatenation module according to a specific embodiment of the present disclosure is shown. The read cache concatenation module reads log strings that are not in the format specified in the first and second reads. Since the string ends with a certain character, it is first cached in the read cache concatenation module and concatenated sequentially. When the log string read for the third time ends with a certain character, it is then concatenated sequentially. When the character ends, the concatenated complete line of log data will be sent out.

[0049] Through the above mechanism, the read cache splicing module can dynamically identify and splice log fragments when reading log data from the current process, ensuring that each output is a complete log line. This effectively solves the problem of log data fragmentation caused by incomplete reading, guaranteeing the atomicity and integrity of log data. This provides reliable input for subsequent log processing and avoids errors or anomalies caused by incomplete data.

[0050] Step S330. After the log data in the current process has been read, continue to read the log data in the next process through the read cache splicing module.

[0051] After reading the log data of the current process, the log data of the next process can be read. Similar to the log data processing method of the current process, the data read by the next process is cached in multiple queues, and the data of other processes are processed in sequence.

[0052] In this example implementation, by introducing a read cache concatenation module into the process management tool and having it responsible for reading and concatenating log data from various processes, it ensures that complete log lines are always retrieved. This avoids incomplete or lost log data due to excessively long logs or interrupted reading, thus guaranteeing the originality and accuracy of the log data. Therefore, subsequent anomaly handling and storage operations on log data will be based on complete and accurate log lines, significantly improving the reliability and data quality of the entire log processing flow and avoiding potential errors or analytical biases caused by incomplete data.

[0053] In step S220, the entire log data is retrieved sequentially from the write cache multi-cell queue according to the order in which the log data is stored.

[0054] In this example implementation, after an entire line of log data is stored in the write cache multi-cell queue, it can be retrieved for subsequent processing according to the order in which it was stored. The first line of log data to enter the queue is retrieved from the multi-cell queue and then sent to the exception handling module for processing.

[0055] In this example implementation, a write operation thread can be added to the process management tool. The write operation thread can then retrieve the entire line of log data sequentially from the write cache multi-cell queue and perform exception handling on the entire line of log data.

[0056] A write operation thread is an independent execution unit. While the program is executing the operation logic of the log data reading phase, a new thread will be added to synchronously execute the log data writing operation in order to improve the responsiveness and throughput of the system.

[0057] By introducing a dedicated write operation thread, the log data acquisition and exception handling processes can be optimized. Specifically, after log data is read and stored in the write cache multi-cell queue, there is no need to wait for the read phase to complete. Instead, a newly added write operation thread independently and sequentially retrieves the entire line of log data from the write cache multi-cell queue. Once the data is retrieved, the write operation thread immediately performs exception handling on this data. This design achieves parallel processing, ensuring that the system maintains high responsiveness and processing efficiency even under high log traffic or frequent exceptions, effectively avoiding the risks of program lag and log loss.

[0058] In step S230, the entire log data is subjected to anomaly processing, and the processed log data is written into the memory for storage.

[0059] In this example implementation, after retrieving the entire line of log data from the write cache multi-cell queue, it can be subjected to anomaly handling, i.e., quality checks and necessary corrections. The processed log data is then written to storage for persistent storage, such as to the server's log file system or log database.

[0060] In this example implementation, an exception handling module can be added to the process management tool, and the exception handling module can be used to determine whether there are illegal characters in the entire log data. If there are illegal characters in the entire log data, the illegal characters are replaced with preset characters to obtain the processed entire log data.

[0061] An exception handling module is a software component specifically designed to detect and handle abnormal content in log data. It can be a submodule or plugin within a process management tool, directly integrated into the log processing flow. Its main function is to identify and correct characters in log data that do not conform to specifications or may cause system problems.

[0062] The exception handling module can inspect every character in the entire log line. If an illegal character is found, it will be removed and replaced with a preset character, such as a black square or other preset characters. The criteria for judging illegal characters can be, for example, checking if the character exists in the ASCII encoding table; if so, it conforms to the standard. When the program outputs log data containing illegal characters, the exception handling module can check and replace the illegal characters in the program output, thereby ensuring normal program operation and avoiding log loss issues.

[0063] By introducing an exception handling module into the process management tool, the entire line of log data retrieved from the write cache multi-cell queue undergoes fine-grained processing. Upon detecting illegal characters, the exception handling module does not simply discard the entire log line; instead, it employs a replacement strategy, replacing these illegal characters with preset safe characters. In this way, valid information in the log data is preserved while eliminating illegal content that could lead to storage or parsing problems, thus generating a processed log line. This processed log data is then written to memory for storage, ensuring the integrity and availability of the log. This crucial quality control step effectively avoids the risk of overall data loss due to localized anomalies, significantly improving the robustness of log processing.

[0064] In this example implementation, if there are illegal characters in the entire log data, while writing the processed entire log data into the memory for storage, the corresponding illegal character prompt information is output through the exception handling module.

[0065] If an illegal character is found in the entire log line, the exception handling module will output a corresponding prompt message to the user while sending the processed log data to the storage, informing the user of the presence of an illegal character. The prompt message may include phrases like "Illegal character encountered," or it may display the specific illegal character. The prompt message can be sent to administrators or relevant personnel via email, SMS, or notifications to monitoring dashboards or instant messaging platforms. This synchronous mechanism ensures that log data cleaning and transparent reporting of anomalies occur simultaneously, enabling the system to not only correct problems in the logs but also promptly inform users of the occurrence of the problem, thus effectively solving the problem of opaque log information. By integrating the notification function into the existing exception handling module, additional processing steps and resource consumption are avoided, improving overall processing efficiency and reliability.

[0066] like Figure 6 The diagram shown is a flowchart illustrating a log processing method in a specific embodiment of this disclosure. It mainly consists of two stages: reading and writing. This flowchart serves as an example of the steps described above in this exemplary embodiment. The specific steps of the flowchart are as follows: The steps for reading log data include: Step S610. Read the log data in process 1 through anonymous pipe 1 and write it to the read cache splicing module.

[0067] Step S620. After the read cache concatenation module receives the log, it will first determine whether the log string ends with "...". The end of the string. If not, it is cached in the module first, and subsequent log data is received and concatenated sequentially; until the end of the string is read. Log entries ending with a specific character. After concatenating these log entries, the log now consists of a complete line of data, which the module then sends out.

[0068] Step S630. The read cache concatenation module sends the logs to the write cache multi-cell queue. The write cache multi-cell queue can store data from multiple processes at a single point in time. This process data is not sent out immediately, but is first cached in the multi-cell queue.

[0069] Step S640. After reading the log data of process_1, start reading the log data of process_2 through anonymous pipe_2.

[0070] Step S650. The read cache splicing module splices the log data in process_2 to obtain the complete line of log data and sends it out.

[0071] Step S660. Cache the entire line of log data read into the write cache multi-cell queue.

[0072] Similar to the data processing steps of the aforementioned processes, the subsequently read data is cached in a multi-cell queue and the data of other processes is processed sequentially.

[0073] While the program is executing the read phase operations, a new thread will be added to synchronously execute the write phase operations. The steps for writing log data include: Step S670. Retrieve the process log data that first entered the queue from the multi-cell queue, and then send that log data to the exception handling module.

[0074] Step S680. The exception handling module checks each character in the log line. If there is an illegal character, it removes the illegal character, replaces it with a black square, and outputs the relevant information to the standard output of the process itself, informing the user of the situation and the specific illegal character.

[0075] Step S690. Send the processed log data to the storage module for storage.

[0076] Therefore, by sequentially reading entire lines of log data from each process and placing them into a multi-cell write buffer queue, the blocking and log loss problems that may occur in traditional single-threaded pipes when processing high-concurrency logs from multiple processes are solved. Simultaneously, the multi-cell queue achieves initial isolation and caching of log data. Next, log data is retrieved sequentially according to the order of storage, ensuring the timeliness and integrity of log processing. Finally, exception handling is performed on the log data before writing it to storage, effectively preventing logs from being discarded due to illegal characters, thus improving the quality and reliability of log data. The entire process forms an efficient and stable log processing flow, improving the performance and robustness of log processing.

[0077] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0078] Furthermore, this disclosure also provides a log processing apparatus. (See reference...) Figure 7 As shown, the log processing device may include a whole-line log reading module 710, a multi-cell queue processing module 720, and a log data writing module 730. Wherein: The whole line log reading module 710 can be used to read whole line log data from each process in sequence and put the read whole line log data into the write cache multi-cell queue in sequence; The multi-grid queue processing module 720 can be used to retrieve the entire log data from the write cache multi-grid queue sequentially according to the order in which the entire log data is stored in the write cache multi-grid queue; The log data writing module 730 can be used to perform exception handling on the entire line of log data and write the processed entire line of log data into the memory for storage.

[0079] In some exemplary embodiments of this disclosure, the whole-line log reading module 710 may include a splicing function adding unit, a current process reading unit, and a next process reading unit. Wherein: The splicing function addition unit can be used to add a read cache splicing module in the process management tool; The current process reading unit can be used to read the entire line of log data from the current process through the read cache splicing module, and put the read entire line of log data into the write cache multi-cell queue; The next process reading unit can be used to continue reading log data from the next process through the read cache splicing module after the log data in the current process has been read.

[0080] In some exemplary embodiments of this disclosure, the current process reading unit may include a tail character determination unit, a whole line log determination unit, and a string concatenation unit. Wherein: The tail character judgment unit can be used to read a log string from the current process through the read cache concatenation module and determine whether the tail character of the log string is a newline character; The whole line log determination unit can be used to treat the log string as the whole line log data if the last character of the log string is a newline character; The string concatenation unit can be used to continue reading the next log string if the last character of the log string is not a newline character, and concatenate the log strings one by one until a log string with a newline character at the end is read.

[0081] In some exemplary embodiments of this disclosure, the log data writing module 730 may include an exception handling function adding unit, an illegal character judgment unit, and an illegal character replacement unit. Wherein: The exception handling functionality addition unit can be used to add exception handling modules to process management tools; The illegal character detection unit can be used by the exception handling module to determine whether there are illegal characters in the entire log data; The illegal character replacement unit can be used to replace illegal characters with preset characters if there are illegal characters in the entire log data, so as to obtain the processed entire log data.

[0082] In some exemplary embodiments of this disclosure, the log data writing module 730 may further include a prompt information output unit, which can be used to output the corresponding illegal character prompt information through the exception handling module while writing the processed log data into the memory for storage if there are illegal characters in the entire line of log data.

[0083] In some exemplary embodiments of this disclosure, the log processing apparatus provided in this disclosure may further include a thread addition module, which may include a write operation thread addition unit and a write operation thread processing unit. Wherein: The write operation thread addition unit can be used to add write operation threads in the process management tool; The write operation thread processing unit can be used to retrieve the entire line of log data sequentially from the write cache multi-cell queue through the write operation thread, and to perform exception handling on the entire line of log data.

[0084] The specific details of each module / unit in the above-mentioned log processing device have been described in detail in the corresponding method embodiment section, and will not be repeated here.

[0085] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0086] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0087] It should be noted that, Figure 8 The computer system 800 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0088] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for system operation. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0089] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0090] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs various functions defined in the system of this disclosure.

[0091] Exemplary embodiments of this disclosure also provide a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the above-described log processing method.

[0092] In one implementation, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing a computer program, such as read-only memory, NAND flash memory, etc.

[0093] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.

[0094] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, and C++. Program code can execute entirely on the user's computing device, partially on the user's computing device, or as a standalone software package. It can also execute partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0095] Computer programs can be carried or transmitted via signals such as electrical, magnetic, optical, electromagnetic, and infrared rays. Electronic devices can convert the signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, to execute) the method steps of various exemplary embodiments of this disclosure, such as the log processing method described above.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0097] It should be noted that although several modules for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0098] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0099] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A log processing method, characterized in that, include: Read the entire line of log data from each process in sequence, and put the read entire line of log data into the write cache multi-cell queue in sequence; According to the order in which the entire log data is stored in the write cache multi-cell queue, the entire log data is retrieved sequentially from the write cache multi-cell queue; The entire log data is subjected to anomaly processing, and the processed log data is written into the memory for storage.

2. The log processing method according to claim 1, characterized in that, The step of sequentially reading entire lines of log data from each process and placing the read lines of log data sequentially into a write cache multi-cell queue includes: Add a read cache splicing module to the process management tool; The read cache splicing module reads the entire line of log data from the current process and puts the read entire line of log data into the write cache multi-cell queue; After the log data in the current process is read, the log data in the next process is read through the read cache splicing module.

3. The log processing method according to claim 2, characterized in that, The step of reading the entire line of log data from the current process through the read cache splicing module includes: The read cache concatenation module reads a log string from the current process and determines whether the last character of the log string is a newline character. If the last character of the log string is a newline character, then the log string is taken as the entire line of log data; If the last character of the log string is not a newline character, continue reading the next log string and concatenate the log strings sequentially until a log string with a newline character at the end is read.

4. The log processing method according to claim 1, characterized in that, The anomaly handling of the entire log data includes: Add an exception handling module to the process management tool; The exception handling module determines whether there are illegal characters in the entire log data. If there are illegal characters in the entire log data, the illegal characters are replaced with preset characters to obtain the processed entire log data.

5. The log processing method according to claim 4, characterized in that, The method further includes: If there are illegal characters in the entire log data, while writing the processed entire log data into the memory for storage, the exception handling module outputs the corresponding illegal character prompt information.

6. The log processing method according to claim 1, characterized in that, Each cell in the write cache multi-cell queue is used to store log data corresponding to a process, and the write cache multi-cell queue can store log data from multiple processes simultaneously.

7. The log processing method according to claim 1, characterized in that, The method further includes: Add a write operation thread in the process management tool; The write operation thread sequentially retrieves the entire line of log data from the write cache multi-cell queue and performs exception handling on the entire line of log data.

8. A log processing apparatus, characterized in that, include: The whole line log reading module is used to read whole line log data from each process in sequence, and put the read whole line log data into the write cache multi-cell queue in sequence; The multi-grid queue processing module is used to sequentially retrieve the entire log data from the write cache multi-grid queue according to the storage order of the entire log data in the write cache multi-grid queue; The log data writing module is used to perform anomaly processing on the entire line of log data and write the processed entire line of log data into the memory for storage.

9. An electronic device, characterized in that, include: processor; as well as A memory for storing one or more programs that, when executed by the processor, cause the processor to implement the log processing method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the log processing method as described in any one of claims 1 to 7.

Images