Log management method and device, equipment and storage medium

Abstract

The invention discloses a log management method and device, equipment and a storage medium, and relates to the field of log management.The log management method comprises the steps that original logs are read from annular queues corresponding to application programs respectively, and the original logs are sorted to obtain sorted logs; each log is composed of a plurality of fields; counting the occurrence frequency of different fields in the sorted logs, determining target fields which do not exist in the log dictionary from the fields of which the occurrence frequency is greater than a preset frequency, configuring target characters for the target fields, and adding a corresponding relationship between the target fields and the target characters into the log dictionary; different fields correspond to different characters; according to the technical scheme, the to-be-replaced fields located in the log dictionary are determined from the sorted logs, the to-be-replaced fields are replaced with the corresponding characters based on the log dictionary to obtain the replaced logs, and then the replaced logs are stored in the preset area, so that the real-time performance of log storage is improved, and the utilization rate of the storage space is increased.

Description

Technical Field

[0001] The present invention relates to the field of log management, and in particular to a log management method, device, equipment and storage medium. Background Art

[0002] As the integration of chips becomes higher and higher, smart devices integrate more and more applications, such as browsers, game applications, music applications, video applications, cameras, etc., and each application will generate corresponding logs when running. Therefore, more and more applications have also put forward higher requirements for log management. First of all, the logs generated by multiple applications are usually stored in the same queue, but when each application adds logs to the queue, it first needs to check whether the queue is occupied by other applications. If other applications are also adding logs to the queue at this time, it is necessary to wait for other applications to add logs before operating the queue. However, this behavior will cause a lag in the actual storage time of the log, which will eventually manifest as an inconsistency between the log generation time and the log storage time. In addition, as more and more applications generate more and more logs, how to save as many logs as possible in the case of limited storage space is also a problem that needs to be solved. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to provide a log management method, device, equipment and storage medium, which can improve the real-time performance of log storage by configuring a circular queue for each application, and replace the corresponding fields in the log with corresponding characters through the log dictionary, thereby realizing compressed storage of the log, so as to store as many logs as possible in a limited storage space, and improve the utilization rate of the storage space. The specific scheme is as follows:

[0004] In a first aspect, the present invention provides a log management method, comprising:

[0005] Reading original logs from the circular queues corresponding to the respective applications, and sorting the original logs based on preset rules to obtain sorted logs; wherein different applications correspond to different circular queues, and each log is a log composed of several fields;

[0006] Counting the number of occurrences of different fields in the sorted log, and determining a target field that does not exist in the log dictionary from the fields whose number of occurrences is greater than a preset number, and configuring a corresponding target character for the target field, and adding the correspondence between the target field and the target character to the log dictionary; wherein the log dictionary is a dictionary stored at the starting position of a preset log storage area, and different fields correspond to different characters;

[0007] A field to be replaced in the log dictionary is determined from the sorted log, and based on the correspondence between the field and the character stored in the log dictionary, the field to be replaced is replaced with a corresponding character to obtain a replaced log, and then the replaced log is stored in the preset log storage area.

[0008] Optionally, before reading the original log from the circular queues corresponding to the respective applications, the method further includes:

[0009] When any application determines that the corresponding target circular queue is full or the free capacity of the target circular queue is less than the capacity required for the log to be written, obtain the log reading instruction sent by the any application, and trigger the step of reading the original log from the circular queue corresponding to each application based on the log reading instruction, so that when the free capacity of the target circular queue is not less than the capacity required for the log to be written, the any application writes the log to be written to the end of the target circular queue.

[0010] Optionally, reading the original log from the circular queue corresponding to each application program includes:

[0011] Traversing the circular queues corresponding to the respective applications to determine the first circular queue in the circular queues corresponding to the respective applications as the current circular queue, and determining whether the current circular queue is an empty queue;

[0012] If the current circular queue is not an empty queue, read the original log from the head of the current circular queue, and after the original log is read, determine whether the current circular queue is the last circular queue in the circular queues corresponding to the respective applications;

[0013] If the current circular queue is an empty queue, directly determining whether the current circular queue is the last circular queue in the circular queues corresponding to the respective applications;

[0014] If the current circular queue is not the last circular queue in the circular queues corresponding to the respective applications, the next circular queue in the circular queues corresponding to the respective applications is determined as the new current circular queue, and the process jumps again to the step of determining whether the current circular queue is an empty queue;

[0015] If the current circular queue is the last circular queue in the circular queues corresponding to the respective applications, the traversal of the circular queues corresponding to the respective applications is terminated to complete the original log reading of the circular queues corresponding to the respective applications.

[0016] Optionally, the sorting the original logs based on a preset rule to obtain sorted logs includes:

[0017] The header information of each of the original logs is parsed respectively to obtain a timestamp corresponding to each of the original logs, and the original logs are sorted based on the order of the timestamps from front to back to obtain sorted logs.

[0018] Optionally, the sorting the original logs based on a preset rule to obtain sorted logs includes:

[0019] The header information of each of the original logs is parsed respectively to obtain the log level corresponding to each of the original logs, and the original logs are sorted in descending order based on the log level to obtain sorted logs.

[0020] Optionally, after sorting the original logs based on a preset rule to obtain sorted logs, the method further includes:

[0021] When a full log viewing instruction sent by the host side is obtained, the sorted log is sent to the host side via a universal asynchronous receiver and transmitter for visual display based on the full log viewing instruction, so that the host side uses the application identifier corresponding to each of the applications and determines the log capacity corresponding to each of the applications based on the sorted log, and then optimizes the capacity of the circular queue corresponding to each of the applications based on the log capacity.

[0022] Optionally, storing the replaced log in the preset log storage area includes:

[0023] When there is no stored log in the preset log storage area, determining a first target free area adjacent to the starting position from the preset log storage area, and storing the replaced log in the first target free area; wherein the capacity of the first target free area is not less than the capacity required for the replaced log;

[0024] Or, when the stored log exists in the preset log storage area, a second target free area adjacent to the stored log is determined from the preset log storage area, and the replaced log is stored in the second target free area; wherein the capacity of the second target free area is not less than the capacity required for the replaced log.

[0025] In a second aspect, the present invention provides a log management device, comprising:

[0026] A log sorting module is used to read original logs from the circular queues corresponding to each application, and sort the original logs based on preset rules to obtain sorted logs; wherein different applications correspond to different circular queues, and each log is a log composed of several fields;

[0027] a corresponding relationship adding module, used for counting the number of occurrences of different fields in the sorted log, and determining a target field that does not exist in the log dictionary from the fields whose number of occurrences is greater than a preset number, and configuring a corresponding target character for the target field, and adding the corresponding relationship between the target field and the target character to the log dictionary; wherein the log dictionary is a dictionary stored at the starting position of a preset log storage area, and different fields correspond to different characters;

[0028] A field replacement module is used to determine the field to be replaced in the log dictionary from the sorted log, and replace the field to be replaced with corresponding characters based on the correspondence between the field and the characters stored in the log dictionary to obtain a replaced log, and then store the replaced log in the preset log storage area.

[0029] In a third aspect, the present invention provides an electronic device, comprising:

[0030] Memory for storing computer programs;

[0031] A processor is used to execute the computer program to implement the steps of the aforementioned log management method.

[0032] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the aforementioned log management method when executed by a processor.

[0033] In the present invention, original logs are read from circular queues corresponding to respective applications, and the original logs are sorted based on preset rules to obtain sorted logs; wherein different applications correspond to different circular queues, and each log is a log composed of several fields; the number of occurrences of different fields in the sorted logs is counted, and a target field that does not exist in a log dictionary is determined from fields whose number of occurrences is greater than a preset number, and a corresponding target character is configured for the target field, and the correspondence between the target field and the target character is added to the log dictionary; wherein the log dictionary is a dictionary stored at the starting position of a preset log storage area, and different fields correspond to different characters; a field to be replaced in the log dictionary is determined from the sorted log, and based on the correspondence between the field and the character stored in the log dictionary, the field to be replaced is replaced with a corresponding character to obtain a replaced log, and then the replaced log is stored in the preset log storage area.

[0034] Beneficial effect: The present invention configures corresponding circular queues for each application respectively, thereby avoiding the problem of inconsistent log generation time and log storage time caused by multiple applications sharing a circular queue, and improving the real-time performance of log storage. In addition, the present invention sets a log dictionary to store the correspondence between fields and characters, so as to replace the fields to be replaced in the log dictionary with corresponding characters based on the correspondence. Since the character is the smallest data access unit in the data structure, and the field is composed of several characters, that is, the storage capacity required by the field is greater than the storage capacity required by the character; therefore, the corresponding field in the log is replaced with the corresponding character through the log dictionary, thereby reducing the storage capacity required for the log, realizing compressed storage of the log, so as to store as many logs as possible in a limited storage space, and improving the utilization rate of the storage space. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flow chart of a log management method provided by an embodiment of the present invention;

[0037] Figure 2 A log structure diagram provided by an embodiment of the present invention;

[0038] Figure 3A schematic diagram of a ring queue provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of an original log provided by an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of a sorted log provided by an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of a log after replacement provided by an embodiment of the present invention;

[0042] Figure 7 A log management flow chart provided by an embodiment of the present invention;

[0043] Figure 8 A log management timing diagram provided by an embodiment of the present invention;

[0044] Fig. 9 A schematic diagram of the structure of a log management device provided by an embodiment of the present invention;

[0045] Fig.10 A structural diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Since the logs generated by multiple applications are usually stored in the same queue, it is easy to cause a lag problem in the actual storage time of the logs, which is ultimately manifested as inconsistency between the log generation time and the log storage time; in addition, how to save as many logs as possible under limited storage space is also a problem to be solved at present. To this end, the present invention provides a log management method, which improves the real-time performance of log storage by configuring a circular queue for each application, and replaces the corresponding fields in the log with corresponding characters through a log dictionary, thereby realizing compressed storage of the log, so as to store as many logs as possible in a limited storage space and improve the utilization rate of the storage space.

[0048] See also Figure 1 As shown, an embodiment of the present invention provides a log management method, including:

[0049] Step S11, reading original logs from the circular queues corresponding to each application, and sorting the original logs based on preset rules to obtain sorted logs; wherein different applications correspond to different circular queues, and each log is a log composed of several fields.

[0050] In the embodiment of the present invention, in the preparation stage, the host side creates multiple applications according to the business scenario, creates a corresponding circular queue for each application, and binds the application to the circular queue one by one, so that different applications correspond to different circular queues; wherein the capacity of each circular queue can be flexibly configured according to the corresponding application. In addition, the host side also needs to create a log management task in the preparation stage and initialize the universal asynchronous receiver / transmitter (UART), such as initializing the baud rate, stop bit and check mode of the universal asynchronous receiver / transmitter. It should be noted that the log management task running on the host side is the application subject of the embodiment of the present invention, that is, the log management task can be used to read logs from the circular queue, sort the logs, and replace the fields in the logs with corresponding characters.

[0051] Before reading the original log from the circular queue corresponding to each application, for any application among the applications, if any application generates a log to be written during operation, it is necessary to first determine whether the target circular queue corresponding to any application is full. If the target circular queue is not full and the free capacity of the target circular queue is greater than or equal to the capacity required for the log to be written, the log to be written can be directly written to the end of the target circular queue; in this way, the embodiment of the present invention improves the consistency of the log generation time and the log storage time by storing the log generated by any application in the corresponding circular queue, so as to improve the real-time performance of the log storage. If the target circular queue is full or the free capacity of the target circular queue is less than the capacity required for the log to be written, any application sends a log reading instruction to the log management task, so that when the log management task obtains the log reading instruction sent by any application, it triggers the step of reading the original log from the circular queues corresponding to each application based on the log reading instruction; it should be noted that the reading here means taking out, that is, taking out the original log from the circular queues corresponding to each application, thereby increasing the free capacity of the circular queue, so that any application can write the log to be written to the end of the target circular queue when the free capacity of the target circular queue is greater than or equal to the capacity required for the log to be written.

[0052] Furthermore, the log management task running on the host side can also periodically determine whether the circular queues corresponding to each application are empty queues, so as to trigger the step of reading the original log from the circular queues corresponding to each application when any of the circular queues corresponding to each application is not empty.

[0053] Specifically, for reading the original log from the circular queues corresponding to each application, it can include: traversing the circular queues corresponding to each application to determine the first circular queue in the circular queues corresponding to each application as the current circular queue, and judging whether the current circular queue is an empty queue. If the current circular queue is not an empty queue, reading the original log from the head of the current circular queue, and after the original log reading is completed, judging whether the current circular queue is the last circular queue in the circular queues corresponding to each application; if the current circular queue is an empty queue, directly judging whether the current circular queue is the last circular queue in the circular queues corresponding to each application. If the current circular queue is not the last circular queue in the circular queues corresponding to each application, determining the next circular queue in the circular queues corresponding to each application as the new current circular queue, and jumping back to the step of judging whether the current circular queue is an empty queue; if the current circular queue is the last circular queue in the circular queues corresponding to each application, then ending the traversal of the circular queues corresponding to each application to complete the reading of the original logs of the circular queues corresponding to each application. In this way, the embodiment of the present invention sequentially takes out original logs from the circular queues corresponding to the respective applications in a traversal polling manner, so that each circular queue can be equally selected and the risk of missing logs can be reduced.

[0054] After reading the original logs from the circular queues corresponding to each application, the original logs need to be sorted based on preset rules to obtain sorted logs. Figure 2 As shown, each log is composed of several fields, including timestamp, log level, module name, file name, function name, log content, etc., and the fields are composed of several characters. It should be noted that the timestamp indicates the time when the log is generated; the log levels include INFO, DEBUG, ERROR, WARN, etc. INFO indicates normal operation information, which may be the result of an action or event, DEBUG is usually used to provide detailed information when debugging an application, ERROR indicates a real error, which prevents part of the application or the entire application from working properly, and WARN indicates the output of potential harmful situations, which serves as a reminder, but will not affect the operation of the application for the time being.

[0055] According to one embodiment, the header information of each original log is parsed to obtain the timestamps corresponding to each original log, and the original logs are sorted in order from the front to the back of the timestamps to obtain sorted logs. In this way, the business conditions of different applications in continuous time can be better analyzed by sorting the logs according to the timestamps.

[0056] by Figure 3 Taking the three circular queues corresponding to the three applications shown in the figure as an example, the logs of the three circular queues are read in sequence to obtain the following Figure 4 The original logs shown in FIG. 1 are obtained by parsing the header information of each original log to obtain the timestamps corresponding to each original log, and sorting the original logs in order from the front to the back based on the timestamps to obtain the following: Figure 5 The sorted log is shown.

[0057] According to another embodiment, the header information of each original log is parsed to obtain the log level corresponding to each original log, and the original logs are sorted in descending order based on the log level to obtain the sorted logs. It should be noted that sorting logs based on timestamps and sorting logs based on log levels are only two specific feasible situations. Of course, there are other feasible situations, which are not listed here one by one.

[0058] Step S12, counting the number of occurrences of different fields in the sorted log, and determining a target field that does not exist in the log dictionary from the fields whose number of occurrences is greater than a preset number, and configuring a corresponding target character for the target field, and adding the correspondence between the target field and the target character to the log dictionary; wherein the log dictionary is a dictionary stored at the starting position of a preset log storage area, and different fields correspond to different characters.

[0059] In an embodiment of the present invention, the number of occurrences of different fields in the sorted log is counted, and the target field that does not exist in the log dictionary is determined from the fields whose number of occurrences is greater than the preset number; wherein the preset number can be set according to the actual compression requirements, for example, it can be set to 1; the log dictionary is a dictionary for storing the correspondence between fields and characters, that is, different fields in the log dictionary correspond to different characters, and the log dictionary is stored at the starting position of the preset log storage area. After determining the target field that does not exist in the log dictionary in the sorted log, the corresponding target character is configured for each target field, thereby establishing the correspondence between the target field and the target character, and adding the correspondence to the log dictionary, thereby realizing the dynamic update of the log dictionary.

[0060] Step S13, determining the to-be-replaced field in the log dictionary from the sorted log, and based on the correspondence between the field and the character stored in the log dictionary, replacing the to-be-replaced field with the corresponding character to obtain a replaced log, and then storing the replaced log in the preset log storage area.

[0061] In the embodiment of the present invention, after the log dictionary is dynamically updated, the field to be replaced in the log dictionary can be determined from the sorted log, and the correspondence between the field and the character stored in the log dictionary can be used to replace the field to be replaced in the log dictionary with the corresponding character, thereby obtaining the replaced log, and then storing the replaced log in the preset log storage area. In this way, by replacing the field in the log with the corresponding character through the log dictionary, the compressed storage of the log can be realized, and the capacity required for log storage can be reduced, so as to store as many logs as possible in a limited storage space.

[0062] by Figure 6 For example, the left side shows five sorted logs, 0.0, 0.1, 0.2, 0.3 and 0.4 are timestamps, INFO, DEBUG and ERROR are log levels, Module1 and Module2 are module names, File1, File3 and File4 are file names, Function1 and Function2 are function names, and 1111, 2222, 3333, 4444 and 5555 are log contents. Count the number of occurrences of different fields in the five sorted logs, and determine the target fields that do not exist in the log dictionary from the fields with a number of occurrences greater than 1, and configure the corresponding target characters for the target fields, and then add the correspondence between the target fields and the target characters to the log dictionary; for example: the field INFO corresponds to character 1, the field DEBUG corresponds to character 2, the field Module1 corresponds to character 3, the field Module2 corresponds to character 4, the field File1 corresponds to character 5, the field File3 corresponds to character 6, the field Function1 corresponds to character 7, and the field Function2 corresponds to character 8. Furthermore, after the corresponding relationship is added to the log dictionary, the field to be replaced in the log dictionary is determined from the five sorted logs, and the field to be replaced is replaced with the corresponding character based on the log dictionary to obtain the replaced log, and then the replaced log is stored in the preset log storage area.

[0063] It should be noted that, considering that the probability of duplication of the two types of fields, log content and timestamp, is generally low, if the two types of fields, log content and timestamp, are configured with corresponding characters and the corresponding relationship is added to the log dictionary, the storage capacity required for the log dictionary will increase. Therefore, the embodiment of the present invention can pre-set the target field type whose occurrence count is required, such as log level, module name, file name, function name, etc., and then count the occurrence count of different fields corresponding to the target field type in the sorted log, and determine the target field that does not exist in the log dictionary from the fields whose occurrence count is greater than the preset number, and then configure the corresponding characters for the target field, and add the corresponding relationship to the log dictionary, thereby further reducing the storage capacity required for the log dictionary.

[0064] When you want to restore the stored logs in the preset log storage area, you can determine the fields corresponding to the characters in the stored logs from the log dictionary stored at the starting position of the preset log storage area, and replace the characters in the stored logs with the corresponding fields, thereby realizing the restoration of the compressed stored logs.

[0065] For storing the replaced log in the preset log storage area, a target free area adjacent to the stored data and having a capacity not less than the capacity required by the replaced log can be determined from the preset log storage area, and then the replaced log is stored in the target free area; wherein the stored data includes the log dictionary and the stored log. In this way, the embodiment of the present invention can realize the continuous storage of the log dictionary and the log in the preset log storage area, thereby improving the utilization rate of resources.

[0066] According to one of the embodiments, when there is no stored log in the preset log storage area, it indicates that only the log dictionary stored at the starting position is stored in the preset log storage area at this time. Therefore, a first target free area adjacent to the starting position and with a capacity not less than the capacity required for the replaced log can be determined from the preset log storage area, and then the replaced log is stored in the first target free area.

[0067] According to another embodiment, when there are stored logs in the preset log storage area, a second target free area adjacent to the stored logs and with a capacity not less than the capacity required for the replaced logs can be determined from the preset log storage area, and then the replaced logs can be stored in the second target free area.

[0068] It should be noted that after sorting the original log to obtain the sorted log, if the full log viewing instruction sent by the host is obtained, the sorted log can be sent to the host via the universal asynchronous receiver and transmitter for visual display based on the full log viewing instruction, so that the user can view the full log of the application online through the host, and the host can use the application identifier corresponding to each application and based on the sorted log to filter out the log capacity corresponding to each application, so as to help the user understand the usage of each application more accurately, and thus optimize the capacity of the circular queue corresponding to each application based on the log capacity.

[0069] It should also be noted that the embodiment of the present invention sets a corresponding cumulative number for each corresponding relationship in the log dictionary, so that when the number of occurrences of different fields in the log after statistical sorting is calculated, for a field whose number of occurrences is greater than a preset number and exists in the log dictionary, the cumulative number of the corresponding relationship corresponding to the field in the log dictionary can be incremented based on the number of occurrences to obtain a new cumulative number; for example, if the number of occurrences of a certain field is 3, and the cumulative number of the corresponding relationship corresponding to the field in the log dictionary is 2, then the new cumulative number after the increment operation is 5. For a field whose number of occurrences is greater than a preset number and does not exist in the log dictionary, a corresponding character can be configured for the field to establish a corresponding relationship between the field and the character, and the corresponding relationship can be added to the log dictionary, and a corresponding cumulative number can be set for the corresponding relationship in the log dictionary based on the number of occurrences; for example, if the number of occurrences of a certain field is 3, then after the corresponding relationship corresponding to the field is added to the log dictionary, the cumulative number of the corresponding relationship is also 3. Accordingly, when taking out a log from a preset log storage area, the corresponding field can be determined from the log dictionary based on the characters in the log, and the characters in the log can be replaced with the corresponding field, thereby realizing the restoration of the log. At this time, the accumulated number of corresponding relationships in the log dictionary corresponding to the characters in the log needs to be decremented to obtain a new accumulated number; for example, if the accumulated number of corresponding relationships in the log dictionary corresponding to the characters in the log is 2, the accumulated number after the decrement operation is 1. Furthermore, when the accumulated number of a corresponding relationship in the log dictionary is reduced to 0, the corresponding relationship can be deleted from the log dictionary, thereby realizing the dynamic update of the log dictionary, avoiding the resource consumption problem caused by storing many useless corresponding relationships in the log dictionary, and further improving the utilization rate of storage space.

[0070] Beneficial effect: The present invention configures corresponding circular queues for each application respectively, thereby avoiding the problem of inconsistent log generation time and log storage time caused by multiple applications sharing a circular queue, and improving the real-time performance of log storage. In addition, the present invention sets a log dictionary to store the correspondence between fields and characters, so as to replace the fields to be replaced in the log dictionary with corresponding characters based on the correspondence. Since the character is the smallest data access unit in the data structure, and the field is composed of several characters, that is, the storage capacity required by the field is greater than the storage capacity required by the character; therefore, the corresponding field in the log is replaced with the corresponding character through the log dictionary, thereby reducing the storage capacity required for the log, realizing compressed storage of the log, so as to store as many logs as possible in a limited storage space, and improving the utilization rate of the storage space.

[0071] See also Figure 7 and Figure 8 As shown, an embodiment of the present invention provides a log management method, including:

[0072] During the system initialization phase, the host creates multiple applications based on business scenarios, creates log management tasks, and initializes the universal asynchronous receiver and transmitter. It also creates and initializes the corresponding ring queue for each application, thereby binding the application to the ring queue one by one.

[0073] When writing logs to the ring queue, if the application generates logs to be written, it is necessary to first determine whether the corresponding ring queue is full; if the ring queue is not full and the free capacity of the ring queue is greater than or equal to the capacity required for the logs to be written, the logs to be written are directly written to the end of the ring queue.

[0074] During the log retrieval and sorting stage, the log management task will continuously determine whether the circular queue corresponding to each application is an empty queue. When any of the circular queues corresponding to each application is not an empty queue, the original log will be taken out from the head of the circular queue corresponding to each application in turn, and the original log will be sorted in the order of timestamps to obtain the sorted log. The sorted log is then sent to the host side via the universal asynchronous receiver and transmitter for visual display, so that users can view the full log of the application online through the host side.

[0075] In the log compression and storage stage, the log management task first counts the number of occurrences of different fields in the sorted log, and determines the target field that does not exist in the log dictionary from the fields with a number of occurrences greater than the preset number, and configures the corresponding target character for the target field, and then establishes the correspondence between the target field and the target character, and adds the correspondence between the target field and the target character to the log dictionary located at the starting position of the preset log storage area. Further, the log management task determines the field to be replaced in the log dictionary from the sorted log, and based on the correspondence between the field and the character stored in the log dictionary, replaces the field to be replaced with the corresponding character to obtain the replaced log, and then stores the replaced log in the preset log storage area on the host side, thereby realizing the compressed storage of the log.

[0076] Beneficial effect: The present invention configures corresponding circular queues for each application respectively, thereby avoiding the problem of inconsistent log generation time and log storage time caused by multiple applications sharing a circular queue, and improving the real-time performance of log storage. In addition, the present invention sets a log dictionary to store the correspondence between fields and characters, so as to replace the fields to be replaced in the log dictionary with corresponding characters based on the correspondence. Since the character is the smallest data access unit in the data structure, and the field is composed of several characters, that is, the storage capacity required by the field is greater than the storage capacity required by the character; therefore, the corresponding field in the log is replaced with the corresponding character through the log dictionary, thereby reducing the storage capacity required for the log, realizing compressed storage of the log, so as to store as many logs as possible in a limited storage space, and improving the utilization rate of the storage space.

[0077] See also Fig. 9 As shown, an embodiment of the present invention provides a log management device, including:

[0078] The log sorting module 11 is used to read the original logs from the circular queues corresponding to each application, and sort the original logs based on preset rules to obtain sorted logs; wherein different applications correspond to different circular queues, and each log is a log composed of several fields;

[0079] The corresponding relationship adding module 12 is used to count the number of occurrences of different fields in the sorted log, and determine the target field that does not exist in the log dictionary from the fields whose number of occurrences is greater than a preset number, and configure a corresponding target character for the target field, and add the corresponding relationship between the target field and the target character to the log dictionary; wherein the log dictionary is a dictionary stored at the starting position of a preset log storage area, and different fields correspond to different characters;

[0080] The field replacement module 13 is used to determine the field to be replaced in the log dictionary from the sorted log, and replace the field to be replaced with the corresponding character based on the correspondence between the field and the character stored in the log dictionary to obtain the replaced log, and then store the replaced log in the preset log storage area.

[0081] Since the embodiments of the device part correspond to the above embodiments, please refer to the description of the embodiments of the method part for the embodiments of the device part, and will not be repeated here.

[0082] Beneficial effect: The present invention configures corresponding circular queues for each application respectively, thereby avoiding the problem of inconsistent log generation time and log storage time caused by multiple applications sharing a circular queue, and improving the real-time performance of log storage. In addition, the present invention sets a log dictionary to store the correspondence between fields and characters, so as to replace the fields to be replaced in the log dictionary with corresponding characters based on the correspondence. Since the character is the smallest data access unit in the data structure, and the field is composed of several characters, that is, the storage capacity required by the field is greater than the storage capacity required by the character; therefore, the corresponding field in the log is replaced with the corresponding character through the log dictionary, thereby reducing the storage capacity required for the log, realizing compressed storage of the log, so as to store as many logs as possible in a limited storage space, and improving the utilization rate of the storage space.

[0083] Furthermore, the present application also discloses an electronic device. Fig.10 It is a structural diagram of an electronic device according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of this application. The electronic device may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input and output interface 25 and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the log management method disclosed in any of the aforementioned embodiments. In addition, the electronic device in this embodiment may specifically be an electronic computer.

[0084] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0085] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0086] The operating system 221 is used to manage and control various hardware devices on the electronic device and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the log management method performed by the electronic device disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0087] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed log management method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0088] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0089] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0090] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0091] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0092] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A log management method, characterized in that: include: Reading original logs from the circular queues corresponding to the respective applications, and sorting the original logs based on preset rules to obtain sorted logs; wherein different applications correspond to different circular queues, and each log is a log composed of several fields; Counting the number of occurrences of different fields in the sorted log, and determining a target field that does not exist in the log dictionary from the fields whose number of occurrences is greater than a preset number, and configuring a corresponding target character for the target field, and adding the correspondence between the target field and the target character to the log dictionary; wherein the log dictionary is a dictionary stored at the starting position of a preset log storage area, and different fields correspond to different characters; A field to be replaced in the log dictionary is determined from the sorted log, and based on the correspondence between the field and the character stored in the log dictionary, the field to be replaced is replaced with a corresponding character to obtain a replaced log, and then the replaced log is stored in the preset log storage area.

2. The log management method according to claim 1, characterized in that: Before reading the original log from the circular queues corresponding to the respective applications, the method further includes: When any application determines that the corresponding target circular queue is full or the free capacity of the target circular queue is less than the capacity required for the log to be written, obtain the log reading instruction sent by the any application, and trigger the step of reading the original log from the circular queue corresponding to each application based on the log reading instruction, so that when the free capacity of the target circular queue is not less than the capacity required for the log to be written, the any application writes the log to be written to the end of the target circular queue.

3. The log management method according to claim 1, characterized in that: The reading of the original log from the circular queues corresponding to the respective applications includes: Traversing the circular queues corresponding to the respective applications to determine the first circular queue in the circular queues corresponding to the respective applications as the current circular queue, and determining whether the current circular queue is an empty queue; If the current circular queue is not an empty queue, read the original log from the head of the current circular queue, and after the original log is read, determine whether the current circular queue is the last circular queue in the circular queues corresponding to the respective applications; If the current circular queue is an empty queue, directly determining whether the current circular queue is the last circular queue in the circular queues corresponding to the respective applications; If the current circular queue is not the last circular queue in the circular queues corresponding to the respective applications, the next circular queue in the circular queues corresponding to the respective applications is determined as the new current circular queue, and the process jumps again to the step of determining whether the current circular queue is an empty queue; If the current circular queue is the last circular queue in the circular queues corresponding to the respective applications, the traversal of the circular queues corresponding to the respective applications is terminated to complete the original log reading of the circular queues corresponding to the respective applications.

4. The log management method according to claim 1, characterized in that: The step of sorting the original logs based on a preset rule to obtain sorted logs includes: The header information of each of the original logs is parsed respectively to obtain a timestamp corresponding to each of the original logs, and the original logs are sorted based on the order of the timestamps from front to back to obtain sorted logs.

5. The log management method according to claim 1, characterized in that: The step of sorting the original logs based on a preset rule to obtain sorted logs includes: The header information of each of the original logs is parsed respectively to obtain the log level corresponding to each of the original logs, and the original logs are sorted in descending order based on the log level to obtain sorted logs.

6. The log management method according to claim 1, characterized in that: After the original logs are sorted based on the preset rules to obtain sorted logs, the method further includes: When a full log viewing instruction sent by the host side is obtained, the sorted log is sent to the host side via a universal asynchronous receiver and transmitter for visual display based on the full log viewing instruction, so that the host side uses the application identifier corresponding to each of the applications and determines the log capacity corresponding to each of the applications based on the sorted log, and then optimizes the capacity of the circular queue corresponding to each of the applications based on the log capacity.

7. The log management method according to any one of claims 1 to 6, characterized in that: The storing the replaced log in the preset log storage area includes: When there is no stored log in the preset log storage area, determining a first target free area adjacent to the starting position from the preset log storage area, and storing the replaced log in the first target free area; wherein the capacity of the first target free area is not less than the capacity required for the replaced log; Or, when the stored log exists in the preset log storage area, a second target free area adjacent to the stored log is determined from the preset log storage area, and the replaced log is stored in the second target free area; wherein the capacity of the second target free area is not less than the capacity required for the replaced log.

8. A log management device, characterized in that: include: A log sorting module is used to read original logs from the circular queues corresponding to each application, and sort the original logs based on preset rules to obtain sorted logs; wherein different applications correspond to different circular queues, and each log is a log composed of several fields; a corresponding relationship adding module, used for counting the number of occurrences of different fields in the sorted log, and determining a target field that does not exist in the log dictionary from the fields whose number of occurrences is greater than a preset number, and configuring a corresponding target character for the target field, and adding the corresponding relationship between the target field and the target character to the log dictionary; wherein the log dictionary is a dictionary stored at the starting position of a preset log storage area, and different fields correspond to different characters; A field replacement module is used to determine the field to be replaced in the log dictionary from the sorted log, and replace the field to be replaced with corresponding characters based on the correspondence between the field and the characters stored in the log dictionary to obtain a replaced log, and then store the replaced log in the preset log storage area.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the log management method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the log management method according to any one of claims 1 to 7 are implemented.

Images