Log processing method and device, and storage medium

Abstract

The application relates to a log processing method and device and a storage medium, wherein the method comprises the following steps: receiving a target instruction for collecting a log; in the case that the target instruction is a first instruction for collecting a non-system log, collecting a non-system log of an application indicated by transmission information carried in the first instruction based on the transmission information, and reporting the collected non-system log; in the case that the target instruction is a second instruction for collecting a system log, collecting the system log based on the second instruction. Through the application, the problem that only conventional logs can be collected based on the active calling of a developer in the related art and system-level logs cannot be obtained is solved.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a method, apparatus and storage medium for processing logs. Background Technology

[0002] The system enables the collection, reporting, and retrieval of regular logs through developer-initiated calls. However, it currently relies solely on developer-initiated calls to capture regular logs and lacks the ability to obtain system-level logs. Furthermore, it lacks real-time monitoring of system resource consumption. Frequent log writing consumes significant system resources, which can easily lead to issues such as lag and overheating. Additionally, the lack of log file fragmentation results in inefficient use of memory space. Summary of the Invention

[0003] This application provides a log processing method, apparatus, and storage medium to solve the problem in related technologies that can only retrieve regular logs based on developers' active invocation and cannot obtain system-level logs.

[0004] In a first aspect, this application provides a log processing method, including: receiving a target instruction for collecting logs;

[0005] When the target instruction is a first instruction for collecting non-system logs, non-system logs of the application indicated by the transparent information carried in the first instruction are collected, and the collected non-system logs are reported; when the target instruction is a second instruction for collecting system logs, system logs are collected based on the second instruction.

[0006] Secondly, this application provides a log processing apparatus, comprising: a receiving module for receiving a target instruction for collecting logs; a first processing module for collecting non-system logs of the application indicated by the transparent information carried in the first instruction, and reporting the collected non-system logs, when the target instruction is a first instruction for collecting non-system logs; and a second processing module for collecting system logs based on the second instruction, when the target instruction is a second instruction for collecting system logs.

[0007] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0008] Memory, used to store computer programs;

[0009] When a processor executes a program stored in memory, it implements the steps of the log processing method described in any embodiment of the first aspect.

[0010] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the log processing method as described in any embodiment of the first aspect.

[0011] The technical solutions provided in this application have the following advantages compared with the prior art:

[0012] The method provided in this application embodiment can automatically collect system logs and non-system logs according to different received instructions. That is, the relevant non-system logs can be automatically collected without making any changes to the business data content through the transparent information carried in the first instruction, so as to preserve the system environment at that time. System logs can also be collected based on the second instruction, thereby solving the problem in related technologies that can only capture regular logs based on the developer's active call and cannot obtain system-level logs, thus improving the efficiency of log collection. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart illustrating a log processing method provided in an embodiment of this application;

[0016] Figure 2 This is a schematic diagram of the system structure for automated log collection according to an embodiment of this application;

[0017] Figure 3 A schematic flowchart of a log processing apparatus provided in an embodiment of this application;

[0018] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] Figure 1 A flowchart illustrating a log processing method provided in an embodiment of this application is shown below. Figure 1 As shown, the method steps of this application embodiment include:

[0021] Step 102: Receive the target instruction for collecting logs;

[0022] In this application, the target instruction can be sent across processes by an automated data collection client (client A, client B, client C, etc.). In a specific example, the client can reside in the JAVA layer, while the receiving target instruction can be an independent automated log collection process located in the system's JAVA layer. Specifically, it can be a log-related process such as an initialization strategy process, a memory control strategy process, a CPU load strategy process, a temperature strategy process, or a memory strategy process. Figure 2 As shown.

[0023] Before log collection is started, the system script can push push instructions (instructions to push data onto the stack) to the corresponding client for automated log collection in a one-to-one or one-to-many manner based on the client identifier. The push instruction carries transparent information, and the client then determines whether to start the log collection operation.

[0024] Step 104: When the target instruction is a first instruction for collecting non-system logs, collect the non-system logs of the application indicated by the transparent information carried in the first instruction and report the collected non-system logs.

[0025] In a specific example, the first instruction can be passed through to the corresponding application (process) based on this pass-through information. For instance, the pass-through information carried in the first instruction is used to instruct the collection of non-system logs from application A, such as application logs and security logs related to application security. Therefore, relevant non-system logs can be automatically collected based on the pass-through information without altering the business data content. It should be noted that in this specific example, the first instruction can be a log retrieval instruction, and the collected non-system logs are packaged and reported to the system backend.

[0026] Step 106: If the target instruction is a second instruction for collecting system logs, collect system logs based on the second instruction.

[0027] Through steps 102 to 106 above, automatic collection of system logs and non-system logs can be achieved based on different received instructions. That is, the relevant non-system logs can be automatically collected without making any changes to the business data content through the transparent information carried in the first instruction, so as to preserve the system environment at that time; and system logs can be collected based on the second instruction, thereby solving the problem in related technologies that can only capture regular logs (non-system logs) based on the developer's active call and cannot obtain system-level logs, thus improving the efficiency of log collection.

[0028] It should be noted that the execution entity of the method steps in this application embodiment can be a log module built into the system. The log module is built into the system as an independent process service, and provides services to all applications that have integrated the log module client code through cross-process communication, so as to unify the operation of log collection, writing, uploading and other operations.

[0029] The method of collecting system logs based on the second instruction mentioned in step 106 above may further include:

[0030] Step 11, start the timer;

[0031] Step 12: When the timer expires, end system log collection, release the relevant script processes to release system resources, and stop submitting system logs to the log queue to be written.

[0032] Step 13: If the timer duration has not been reached, continue collecting system logs and submit the system logs to the log queue to be written.

[0033] As can be seen, in this embodiment of the application, a timer can be used to avoid excessive log collection, thereby avoiding the need to submit system logs to the log queue even when the log queue is full.

[0034] In addition, in an optional embodiment of this application, the log service can be initialized before writing the logs to the system memory. Specifically, the application process can be started, general configuration parameters can be obtained, and then the log collection client program can be initialized and the server process can be started according to the configuration parameters, and the log service strategy can be obtained by configuring the list interface.

[0035] In optional embodiments of this application, the method may further include:

[0036] Step 108: After collecting system logs based on the second instruction, determine whether the current system memory has reached the first preset threshold.

[0037] It should be noted that the first preset threshold is a safety threshold that can be set according to actual needs. That is, if the current system memory exceeds the first preset threshold, it indicates that the system memory is running low and cannot store any more data. Expired data in the system memory needs to be deleted.

[0038] Step 110: If the current system memory reaches the first preset threshold, delete logs stored in the system for a duration exceeding the second preset threshold, and determine whether the log queue to be written has reached the third preset threshold.

[0039] It is evident that the storage duration exceeding the second preset threshold refers to logs that have expired in the current system memory. The second preset threshold can be set with different expiration times for specific logs, or the expiration time for all logs can be set uniformly, meaning it can be set according to requirements.

[0040] Step 112: If the current system memory has not reached the first preset threshold, process the collected system logs to be written based on whether the log queue to be written has reached the third preset threshold.

[0041] As can be seen from steps 108 to 112 above, even if the current system memory has not reached the safety threshold, it is still necessary to further determine whether the log queue is full in order to process the collected system logs to be written. In this specific example, step 112 may further include:

[0042] Step 21: If it is determined that the system logs in the log queue have not reached the third preset threshold, add the collected system logs to the log queue.

[0043] Step 22: If the system logs in the log queue reach the third preset threshold, delete the currently collected system logs to be written.

[0044] Therefore, if the current log queue is full, the system logs that are currently being collected and are waiting to be written should be deleted instead of being forcibly added to the log queue, in order to avoid system failure. The collected system logs will only be added to the log queue if the log queue is not full.

[0045] In another optional embodiment of this application, the method of this application may further include:

[0046] Step 114: After a preset situation occurs, control the thread used to maintain the log queue to sleep for a preset duration;

[0047] The preset conditions include one of the following: the system's CPU load reaches the fourth preset threshold, the system's temperature reaches the fifth preset threshold, and the system's memory reaches the sixth preset threshold.

[0048] As can be seen, in this embodiment, the log queue is maintained by a separate sub-thread. Therefore, when the system CPU load reaches the fourth preset threshold, the system temperature reaches the fifth preset threshold, or the system memory reaches the sixth preset threshold, the thread needs to be controlled to sleep for a preset duration to alleviate system pressure. This preset duration can be set according to actual needs, such as 100ms, 150ms, 80ms, etc. Similarly, the aforementioned fourth, fifth, and sixth preset thresholds can also be set according to actual conditions. Furthermore, in a specific example, the aforementioned preset conditions can be determined sequentially. For example, if the CPU load exceeds the fourth preset threshold, the thread is controlled to sleep for 100ms. If the CPU load does not exceed the fourth preset threshold, it is necessary to further determine whether the system temperature exceeds the fifth preset threshold. If the system temperature exceeds the fifth preset threshold, the thread is controlled to sleep for 100ms. If the system temperature does not exceed the fifth preset threshold, it is necessary to further determine whether the system memory reaches the sixth preset threshold. If the sixth preset threshold is reached, the thread is controlled to sleep for 100ms. If the sixth preset threshold is not reached, subsequent operations on the system log can be performed based on the thread.

[0049] As can be seen, the system resources, such as CPU, memory, and temperature, can be intelligently monitored in real time in this embodiment. When a threshold is reached, a control strategy will be triggered to avoid problems such as lag and overheating, thus finding a reasonable balance between functionality and user experience.

[0050] In another optional embodiment of this application, the method of this application may further include:

[0051] Step 116: After adding the collected system logs to the log queue, retrieve the system logs from the log queue one by one and write the retrieved system logs to the corresponding local files.

[0052] In other words, in this embodiment, system logs are first collected and then written to the corresponding log queue. This ensures that the corresponding system logs can be retrieved from the log queue in an orderly manner when writing to the local file later. Before writing to the local file, it is necessary to further determine which system application the system log to be written corresponds to, and then write it to the local file corresponding to that system application.

[0053] Of course, if there is no local file corresponding to the system application in the system beforehand, a local file can be created based on the package name corresponding to the system log and the date and time of the collected system log as the file name.

[0054] The method of writing the retrieved system logs to the corresponding local file, as mentioned in step 116 above, can further include:

[0055] Step 31: Segment the system log and write the segmented system log into multiple file fragments in a local file in sequence.

[0056] As can be seen, in this embodiment, the system log is stored in multiple file fragments. Specifically, step 31 may further include:

[0057] Step 41: Segment the system log into N file fragments;

[0058] Step 42: If the M file segments in the local file meet the size of the N file segments, write the N file segments into the M file segments in the local file in sequence.

[0059] Step 43: If the M file segments in the local file do not meet the size requirement of the N file segments, clear them sequentially from the earliest stored file segment of the split system log according to the chronological order of the M file segments being stored, until the last file segment of the N file segments is written.

[0060] As can be seen, in this embodiment, a log file segment 1 can be created first to store the retrieved system logs, and the file handle corresponding to log file segment 1 can be opened to store it. If the current system log exceeds the file segment, log file segment 2 needs to be created. If this still cannot satisfy the storage of the retrieved system logs, log file segments need to be created continuously until a preset log file segment N is reached. If this still cannot satisfy the storage, logs in log file segment 1 need to be deleted, log file segment 2 needs to be deleted, and so on, until the system log is written completely. Therefore, in this embodiment, by segmenting the log file, when memory space is insufficient, the latest log record can be retained by overwriting the previous logs, thus making reasonable use of memory space.

[0061] Corresponding to the above Figure 1 This application also provides a log processing apparatus, such as... Figure 3 As shown, it includes:

[0062] Receiver module 32 is used to receive target instructions for collecting logs;

[0063] The first processing module 34 is used to collect non-system logs of the application indicated by the transparent information carried in the first instruction when the target instruction is a first instruction for collecting non-system logs, and to report the collected non-system logs.

[0064] The second processing module 36 is used to collect system logs based on the second instruction when the target instruction is a second instruction for collecting system logs.

[0065] The apparatus of this application embodiment can automatically collect system logs and non-system logs according to different received instructions. That is, the relevant non-system logs can be automatically collected without making any changes to the business data content through the transparent information carried in the first instruction, so as to preserve the system environment at that time. System logs can also be collected based on the second instruction, thereby solving the problem in related technologies that can only capture regular logs (non-system logs) based on the developer's active call and cannot obtain system-level logs, thus improving the efficiency of log collection.

[0066] Optionally, the apparatus in this application embodiment may further include: a determining module, configured to determine whether the current system memory has reached a first preset threshold after collecting system logs based on a second instruction; a deleting module, configured to delete logs stored in the system for a duration exceeding a second preset threshold when the current system memory has reached the first preset threshold, and determine whether the log queue to be written has reached a third preset threshold; and a third processing module, configured to process the collected system logs to be written based on the result of whether the log queue to be written has reached the third preset threshold when the current system memory has not reached the first preset threshold.

[0067] Optionally, the third processing module in this application embodiment may further include: an adding unit, configured to add the collected system logs to be written to the log queue when it is determined that the system logs in the log queue have not reached a third preset threshold; and a deleting unit, configured to delete the currently collected system logs to be written when it is determined that the system logs in the log queue have reached a third preset threshold.

[0068] Optionally, the apparatus in this application embodiment may further include: a control module, configured to control the thread used to maintain the log queue to sleep for a preset duration after a preset condition occurs; wherein the preset condition includes one of the following: the CPU load of the system reaches a fourth preset threshold, the temperature of the system reaches a fifth preset threshold, and the memory of the system reaches a sixth preset threshold.

[0069] Optionally, the apparatus in this application embodiment may further include: a fourth processing module, used to, after adding the collected system logs to the log queue, sequentially retrieve system logs from the log queue and write the retrieved system logs to the corresponding local file.

[0070] Optionally, the fourth processing module in this application embodiment may further include: a processing unit, used to segment the system log and sequentially write the segmented system log into multiple file segments in a local file.

[0071] Optionally, the processing unit in this embodiment may further include: a segmentation subunit, used to segment the system log into N file fragments; a writing subunit, used to sequentially write the N file fragments into the M file fragments in the local file if the M file fragments in the local file meet the size of the N file fragments; and a processing subunit, used to sequentially clear the system log fragments in the order they were stored, starting from the earliest stored fragment, until the last file fragment among the N file fragments is written, if the M file fragments in the local file do not meet the size of the N file fragments.

[0072] like Figure 4 As shown in the figure, this application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0073] Memory 113 is used to store computer programs;

[0074] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the log processing method provided in any of the foregoing method embodiments, and its function is similar, so it will not be described again here.

[0075] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the log processing method provided in any of the foregoing method embodiments.

[0076] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0077] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A log processing method, characterized in that, include: Receive the target instruction for collecting logs; When the target instruction is a first instruction for collecting non-system logs, non-system logs of the application indicated by the transparent information carried in the first instruction are collected, and the collected non-system logs are reported. When the target instruction is a second instruction for collecting system logs, system logs are collected based on the second instruction. The system log collection based on the second instruction includes: Start the timer; When the timer expires, the system log collection ends, the relevant script processes are released to free up system resources, and the submission of system logs to the log queue to be written is stopped. If the timer's set duration has not been reached, continue collecting system logs and submit the system logs to the log queue to be written.

2. The method according to claim 1, characterized in that, The method further includes: After collecting system logs based on the second instruction, determine whether the current system memory has reached the first preset threshold; If the current system memory reaches the first preset threshold, delete the logs stored in the system for a duration exceeding the second preset threshold, and determine whether the log queue to be written has reached the third preset threshold. If the current system memory has not reached the first preset threshold, the collected system logs to be written are processed based on whether the log queue to be written has reached the third preset threshold.

3. The method according to claim 2, characterized in that, The process of processing the collected system logs to be written based on whether the log queue to be written has reached a third preset threshold includes: If it is determined that the system logs in the log queue have not reached the third preset threshold, the collected system logs to be written will be added to the log queue. If the system logs in the log queue reach the third preset threshold, delete the currently collected system logs to be written.

4. The method according to claim 3, characterized in that, The method further includes: After a preset condition occurs, the thread used to maintain the log queue is controlled to sleep for a preset duration; The preset conditions include one of the following: the CPU load of the system reaches a fourth preset threshold, the temperature of the system reaches a fifth preset threshold, and the memory of the system reaches a sixth preset threshold.

5. The method according to claim 3, characterized in that, The method further includes: After the collected system logs are added to the log queue, the system logs are retrieved sequentially from the log queue and written to the corresponding local files.

6. The method according to claim 5, characterized in that, The step of writing the retrieved system logs to the corresponding local file includes: The system log is segmented, and the segmented system log is sequentially written into multiple file segments of the local file.

7. The method according to claim 6, characterized in that, The process of segmenting the system log and sequentially writing the segmented system log into multiple file fragments in the local file includes: The system log is divided into N file segments; If the M file segments in the local file meet the size of the N file segments, then the N file segments are sequentially written into the M file segments in the local file; If the M file segments in the local file do not meet the size of the N file segments, the system logs after being split into segments are cleared sequentially, starting from the earliest stored system log segment, according to the chronological order of the M file segments being stored, until the last file segment of the N file segments is written.

8. A log processing apparatus, characterized in that, include: The receiving module is used to receive target instructions for collecting logs; The first processing module is configured to, when the target instruction is a first instruction for collecting non-system logs, collect non-system logs of the application indicated by the transparent information carried in the first instruction, and report the collected non-system logs; The second processing module is configured to collect system logs based on the second instruction when the target instruction is a second instruction for collecting system logs; the collection of system logs based on the second instruction includes: starting a timer; When the timer expires, system log collection ends, related script processes are released to free up system resources, and submission of system logs to the log queue to be written ceases; if the timer expires, system log collection continues and system logs are submitted to the log queue to be written.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images