A multi-threaded log processing method

Abstract

The present invention discloses a multithreaded log processing method, comprising the following steps: S1. defining a structure S for log preservation, defining a function for creating a structure variable of the structure; S2. defining a mutex M for multithreaded shared resource preemption, assigning a value to a mutex pointer inside the variable when initializing the structure variable, and calling a mutex initialization function pointer inside the variable to initialize the mutex M, and at the same time, assigning a log printing level, a preservation level, and a maximum size of a log file of the structure variable; S3. when the log is saved, if multiple threads need to occupy the access rights of the log file under the same path from each other, each thread seizes the permission to write the log file under the structure variable by locking the mutex M. The present invention can effectively handle the mutual exclusion problem of accessing the same log file when logging between different threads in a multithreaded environment.

Description

Technical Field

[0001] The present invention relates to the field of log data processing, and in particular to a multi-threaded log processing method. Background Art

[0002] In system-level projects, the functions and data processing involved are complex. Therefore, in order to debug or analyze the system's functional points and processes afterwards, it is often necessary to log or print logs for each key point. The content of the logs will become more and more, and the files for saving the logs will become larger and larger.

[0003] For system projects that need to be powered on and off multiple times, the log may not only need to record the current log, but also need to record the log about the current power-on each time the device is powered on again. If the information about power on and off can be reflected in the log file name, it will be convenient to analyze and locate the problem.

[0004] The logs usually recorded are: operation process log, error information log, prompt information log, fatal log, etc. The logs can be processed by online terminal printing or recording in log files.

[0005] The version of the program is also divided into the debugging stage and the release stage. The log processing methods in different stages are also different. For example, the log in the debugging stage is mainly to analyze the process to see whether the process is proceeding according to the expected results, whether there are any errors in the middle, etc., so the printed log information will be much more than the log content in the release stage. When the project enters the release stage, some debugging information no longer needs to be printed frequently, but only key information can be printed. Then, it is not advisable to delete these previous logs directly in the program. Later, for version updates or function iterations, these blocked logs may need to be re-enabled. If the blocking is simply completed through comments, it will be time-consuming and laborious.

[0006] In multi-threading, log recording is more difficult, and there are generally two situations for processing log files. 1. In order to achieve unified log recording, it is usually necessary to save all system logs in the same file to facilitate unified recording of processes and to facilitate analysis and problem discovery. However, in multi-threading, each thread has logs to record and needs to access the same file, so who comes first and who comes later is a problem, and mutual exclusion between threads is required. When one thread is accessing a log file, another thread needs to wait. 2. Logs of different threads exist in different files, or threads are specified to be planned into the same log file.

[0007] As a file, it must have a file size. It is impossible to store logs in a file endlessly. If the file is too large, it will be very slow to open the file, not to mention that it will be very slow to view key information and search in such a large amount of data. Therefore, when the file size reaches a certain level, it is necessary to divide the log files to continue to save the system logs. If the caller needs to re-plan the file name of the new log, it will not only increase the workload, but also increase the difficulty of controlling the timing. Summary of the invention

[0008] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a multi-threaded log processing method that can effectively handle the mutual exclusion problem of accessing the same log file when logging between different threads in a multi-threaded environment.

[0009] The object of the present invention is to achieve the following technical solution: a multi-threaded log processing method, comprising the following steps:

[0010] S1. Define a structure S for log storage, and define a function to create a structure variable of the structure;

[0011] S2. Define a mutex M for multi-threaded shared resource preemption. When initializing the structure variable, assign a value to the mutex pointer inside the variable, and call the mutex initialization function pointer inside the variable to initialize the mutex M. At the same time, assign the log printing level, storage level, and maximum size of the log file of the structure variable;

[0012] S3. When saving logs, if multiple threads need to occupy the access rights of the log files under the same path, each thread seizes the permission to write the log files under the structure variable by locking the mutex M.

[0013] The structure used for log storage in step S1 includes:

[0014] The file name of the log to be saved, the log save path passed in by the user, the power-on number passed in by the user, the log printing level, the log save level, the maximum number of bytes of the log file, the amount of data already stored in the current log file, the power-on number, the mutex, the mutex initialization function pointer, the mutex destruction function pointer and the log processing function pointer.

[0015] Preferably, in step S1, a function is defined to create the structure variable;

[0016] This function mainly completes the initialization of some members in the structure and returns a variable of the current structure type; the function input parameters are: the basic path of the log file to be saved by the user, the number of power-on times;

[0017] The function integrates the basic path and power-on times into the log name through input parameters, initializes a level for printing and saving logs to prevent users from missing settings after creating structure variables, and initializes the maximum size of a log file;

[0018] And this function saves the amount of data stored in the current log file, the number of power-on times, the mutex, the mutex initialization function pointer, the mutex destruction function pointer and the log processing function pointer.

[0019] Preferably, in step S2, the mutex, also called a mutex lock, is used to ensure the integrity of the operation of the shared data object, so that the object can only be operated by one thread at any time among multiple threads;

[0020] Preferably, in step S2, the initialization function pointer of the internal member mutex is pointed to a defined initialization function, and when the function pointer is called externally, the function is called directly to initialize the mutex M.

[0021] Preferably, the purpose of defining a function pointer in a structure is to associate the defined function with the structure. For example, a structure named STR can be defined, which contains a function pointer opt, which is used to point to a function that performs a specific operation. Then, a function, such as Add, can be defined, which will implement the specific operation. When creating the function implementation of the structure variable, let the function pointer in the structure variable point to the implemented Add function.

[0022] The mutex initialization function pointer points to a mutex initialization function, which completes the initialization of the mutex in the input parameter;

[0023] The mutex destroy function pointer points to a mutex destroy function, which completes the destruction of the input parameter mutex;

[0024] The log processing function pointer points to a function for log data processing. The input parameters of the function are: variables of the structure S, the level of the log, keywords of different levels, and log data information. In the log processing function, the mutex in the input parameter structure variable is taken out to call the mutex locking function in the third-party Pthread library to complete the locking. After the locking is successful, according to a series of operations such as the set maximum size of the log file, the judgment of the log saving and printing level, and the change of the log file name, the log file needs to be written to process whether it needs to be written, and finally the mutex unlocking function in the Pthread library is also used to unlock it.

[0025] Preferably, in step S2, the maximum size of the log file is used in the log processing flow, and the value is judged. When the maximum file byte number set by the structure variable initialization is exceeded, a new file needs to be created to continue saving the log data; the new file name is still composed of the power-on times input by the user and the accumulated number of existing files, so that the user can view the log files according to the size, and the value of the number of recorded files increases with each additional file;

[0026] The log printing level and saving level are used to limit the log levels that can be printed and saved to files.

[0027] Preferably, step S3 comprises:

[0028] Before each thread seizes resources, it locks the mutex M, and then unlocks it after enjoying the resources. The mechanism in the thread library determines who gets the lock first, and the first one to get the lock will use it first. After use, the lock is released, and other threads in the queue wait for allocation by this thread mechanism, and so on.

[0029] Preferably, the function pointed to by the log processing function pointer in the structure requires the user to pass in the log level currently used when calling the method. When the function is implemented, it can only be printed or saved when the level at the time of calling is greater than or equal to the set printing or saving level, including:

[0030] The log level is passed in the input parameter. The user defines the required log level by himself. At the same time, the user customizes different levels of logs according to his own preferences and adds different keywords as input parameters to the log content, so as to facilitate viewing the log content of different levels by keywords later, or dynamically passing in the log content.

[0031] The beneficial effects of the present invention are as follows: 1. The present invention solves the problem that the log files of each power-on and power-off cannot be distinguished in a system project with multiple power-ons and power-offs by adding the number of power-on and power-off times to the name of the log file.

[0032] 2. The present invention solves the problem of needing to comment out a large number of unnecessary logs from the debugging stage to the release stage by defining different usage levels of logs. Some logs used to record processes and key data in the debugging stage need to be completely blocked when switching to release.

[0033] 3. In the multi-threaded environment of the present invention, when different threads record logs, there is a mutual exclusion problem of accessing the same log file.

[0034] 4. The present invention overcomes the problem that users need to regularly divide files and record logs during system operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Use flow chart for log processing module;

[0036] Figure 2 This is a schematic diagram of the initialization content of the structure variable;

[0037] Figure 3 This is the overall flow chart for calling the logging module;

[0038] Figure 4 Implement a flow for the log processing function. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0040] The present invention designs such a structure, which contains member variables and methods involved in managing log processing. When using the log processing method, the user needs to define such a structure variable. When initializing the structure, a power-on number and a path for saving logs are given. After that, all logs generated by calling the method are under the path. If it is necessary to save log data of different threads under different paths, multiple such structure variables can be defined.

[0041] When in a multi-threaded environment, multiple threads use mutex locking to preempt log processing methods. Lock the mutex before all threads process the log file, and unlock it after the processing is completed. When other threads need to call this method to process logs, they need to lock it first. If other threads occupy the lock, wait until other threads release the lock. Before writing log data to the specified file each time, it is necessary to determine whether the size of the current log file exceeds the maximum value of the log file saved set by the user. If it exceeds, it is necessary to continue to save the log data in separate files. The new file name is named according to the power-on number passed by the user and the cumulative number of files generated under the current path. This ensures that each file name is related to the power-on number and is an incremental file name under the power-on number, which is convenient for users to obtain.

[0042] When processing logs, you can set different log levels according to the needs of users, such as debug level, prompt level, warning level, error level, fatal level, etc. Then, in the design of log processing, add level judgment for printing and saving. Some levels need to be printed to the terminal in real time, some levels only need to be printed and do not need to be saved, and some levels must be saved to the log file, etc.

[0043] The general process of the user calling the log processing method is as follows: Figure 1 As shown;

[0044] The present invention needs to define such a structure, which includes: the file name of the log to be saved, the log saving path passed in by the user, the power-on number passed in by the user, the log printing level, the log saving level, the maximum number of bytes of the log file, the amount of data stored in the current log file, the power-on number, the mutex, the mutex initialization function pointer, the mutex destruction function pointer, the log processing function pointer and other information.

[0045] This log processing method needs to complete three major steps: 1) initialization of structure variables, 2) calling of specific log processing methods, and 3) destruction of structure variables.

[0046] Create and initialize structure variables, which involves initializing the internal variables of the structure and assigning function pointers. For details, see Figure 2 ;

[0047] When you need to save the log, create a structure variable and call the initialization interface, then call the mutex function pointer inside the variable to initialize the mutex, and assign the log printing level, saving level, and maximum size of the log file to the structure variable. Among them, the mutex is used to lock the mutex multi-threaded call in the log processing interface function; the maximum size of the log file is used in the log processing process to judge this value. When it exceeds the maximum file byte number set by the structure variable initialization, a new file is needed to continue to save the log data; the new file name is still composed of the number of power-ons passed in by the user and the accumulated number of existing files, so that users can view log files by size. For each additional file, the value of the number of recorded files will increase. The print level and save level of the log are used to limit which levels of logs can be printed and which levels of logs can be saved in files.

[0048] When using, threads need to occupy the access rights of log files under the same path, so they need to grab the mutex resources under the structure variable. The overall process of calling is shown in Figure 3 ;

[0049] Among them, the function pointed to by the log processing function pointer in the structure is as follows Figure 4 As shown, this interface function requires the user to pass in the log level (customizable) used when the method is currently called. When the function is implemented, when the level at the time of the call is greater than or equal to the set print or save level, it can be printed or saved.

[0050] The present invention greatly simplifies the complexity of log recording. All issues related to log recording in a multi-threaded system are simplified. Since the software method can be made into a static library, it can also be reused in different projects that meet the environment, bringing convenient usability of such log recording issues. Let the user focus on the functional implementation of the system, fill in the logs that need to be recorded and printed, and then call them, without worrying about whether there will be problems with the log recording process. When you need to view the saved logs, just go to the filled path to view them.

[0051] The above is a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art do not deviate from the spirit and scope of the present invention, and should be within the scope of protection of the claims attached to the present invention.

Claims

1. A multi-threaded log processing method, characterized in that: The following steps are involved: S1. Define a structure S for log storage, and define a function to create a structure variable of the structure; The structure S used for log storage includes: The file name of the log to be saved, the log saving path passed in by the user, the power-on number passed in by the user, the log printing level, the log saving level, the maximum number of bytes of the log file, the amount of data already stored in the current log file, the power-on number, the mutex, the mutex initialization function pointer, the mutex destruction function pointer and the log processing function pointer; S2. Define a mutex M for multi-threaded shared resource preemption. When initializing the structure variable, assign a value to the mutex pointer inside the variable, and call the mutex initialization function pointer inside the variable to initialize the mutex M. At the same time, assign the log printing level, storage level, and maximum size of the log file of the structure variable; The maximum size of the log file is used in the log processing flow. When the value exceeds the maximum file byte number set by the structure variable initialization, a new file needs to be created to continue saving the log data. The new file name is still composed of the power-on times passed in by the user and the accumulated number of existing files, which is convenient for users to view log files by size. With each additional file, the value of the number of recorded files increases. The log printing level and saving level are used to limit the log levels that can be printed and saved to files; S3. When saving logs, if multiple threads need to occupy the access rights of the log files under the same path, each thread seizes the permission to write the log files under the structure variable by locking the mutex M.

2. A multi-threaded log processing method according to claim 1, characterized in that: In the step S1, a function is defined to create the structure variable; This function mainly completes the initialization of some members in the structure and returns a variable of the current structure type; The function integrates the basic path and power-on times into the log name through input parameters, initializes a level for printing and saving logs to prevent users from missing settings after creating structure variables, and initializes the maximum size of a log file; And this function saves the amount of data stored in the current log file, the number of power-on times, the mutex, the mutex initialization function pointer, the mutex destruction function pointer and the log processing function pointer.

3. A multi-threaded log processing method according to claim 1, characterized in that: In step S2, the mutex, also called a mutual exclusion lock, is used to ensure the integrity of the operation of the shared data object between multiple threads, so that the object can only be operated by one thread at any time.

4. A multi-threaded log processing method according to claim 3, characterized in that: In the step S2, the initialization function pointer of the internal member mutex is pointed to a defined initialization function. When the function pointer is called externally, the function is called directly to initialize the mutex M.

5. A multi-threaded log processing method according to claim 1, characterized in that: The mutex destroy function pointer points to a mutex destroy function, which completes the destruction of the input parameter mutex; The log processing function pointer points to a function for log data processing. The input parameters of the function are: variables of the structure S, the level of the log, keywords of different levels, and log data information. In the log processing function, the mutex in the input parameter structure variable is taken out to call the mutex locking function in the third-party Pthread library to complete the locking. After the locking is successful, according to a series of operations such as the set maximum size of the log file, the judgment of the log saving and printing level, and the change of the log file name, the log file needs to be written to process whether it needs to be written, and finally the mutex unlocking function in the Pthread library is also used to unlock it.

6. A multi-threaded log processing method according to claim 1, characterized in that: The step S3 comprises: Before each thread seizes resources, it locks the mutex M, and then unlocks it after enjoying the resources. The mechanism in the thread library determines who gets the lock first, and the first one to get the lock will use it first. After use, the lock is released, and other threads in the queue wait for allocation by this thread mechanism, and so on.

7. A multi-threaded log processing method according to claim 1, characterized in that: The function pointed to by the log processing function pointer in the structure requires the user to pass in the log level currently used when calling the method. When the function is implemented, it can only be printed or saved when the level at the time of calling is greater than or equal to the set printing or saving level, including: The log level is passed in the input parameter. The user defines the required log level by himself. At the same time, the user customizes different levels of logs according to his own preferences and adds different keywords as input parameters to the log content, so as to facilitate viewing the log content of different levels by keywords later, or dynamically passing in the log content.

Images