Low-resource android terminal autonomous log collection and rolling cache method

Abstract

This invention primarily relates to the field of terminal log operation and maintenance technology. To address the problem of limited memory and storage resources in low-resource Android terminals, which leads to the easy loss of logs before and after anomalies, this invention provides an autonomous log collection and rolling caching method for low-resource Android terminals. After the terminal starts up, a rolling pre-cache is created independently of the Android native log buffer and its capacity can be dynamically adjusted. Under normal conditions, only high-value logs are selected and written to the cache. At the same time, an anomaly risk value reflecting the probability of system anomalies is calculated in real time through local analysis. When the anomaly risk value increases, the capacity of the rolling pre-cache is expanded in advance to accommodate more logs related to the causes of anomalies, so that the system has sufficient log storage space before an anomaly occurs. Once a critical anomaly event is detected, all logs in the cache are immediately exported and locked to prevent them from being overwritten or reclaimed by subsequent cycles, effectively improving the efficiency of log-based anomaly tracing.

Description

Technical Field

[0001] This invention mainly relates to the field of terminal log operation and maintenance technology, and in particular to a method for autonomous log collection and rolling caching of low-resource Android terminals. Background Technology

[0002] Android terminals typically record their runtime status through logcat, system service logs, kernel logs, and business module logs. For servers or high-performance terminals, logging systems can employ large memory buffers, high write frequencies, and complex log analysis components; however, in smart set-top boxes, carrier-customized terminals, smart TVs, and low-end IoT Android devices, CPU computing power, memory capacity, data partition space, and Flash write endurance are all strictly limited, and log collection itself may become a factor affecting system stability. Existing log management for low-resource Android terminals has the following main shortcomings:

[0003] 1. Logs are captured at a fixed frequency or at a fixed level, and the sampling strategy cannot be dynamically adjusted according to device load, anomaly risk, and storage pressure;

[0004] 2. Continuous reading of logcat for extended periods can increase CPU wake-ups and I / O writes, leading to lag on low-resource terminals, increased power consumption, or accelerated Flash wear.

[0005] 3. Log collection is usually triggered manually, remotely, or by after-sales tools after an anomaly occurs, which can easily lead to the loss of key causal logs before the anomaly occurs;

[0006] 4. Circular overwrite or fixed-capacity deletion strategies cannot distinguish log value, and normal duplicate logs may crowd out critical exception logs such as Crash, ANR, and watchdog logs;

[0007] 5. Cloud-based AI log analysis relies on network and upload links, making it unsuitable for offline, weak network, or privacy-sensitive terminal scenarios;

[0008] 6. While general high-performance log storage solutions can improve the efficiency of concurrent writing or file management, they are usually not designed in a way that is compatible with Android logcat’s multi-buffer, low resource constraints, window retention before and after exceptions, Flash write pressure, and terminal autonomous operation and maintenance. Summary of the Invention

[0009] This invention provides an autonomous log collection and rolling caching method for low-resource Android terminals, aiming to solve the problem that logs are easily lost before and after an anomaly due to the limited memory and storage resources of low-resource Android terminals.

[0010] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0011] A method for autonomous log collection and rolling caching on low-resource Android terminals, including:

[0012] Step 1: After the terminal starts up, set the initial capacity of the rolling pre-cache according to the available memory size of the terminal. The rolling pre-cache is independent of the log buffer.

[0013] Step 2: After filtering the collected logs according to the initial sampling frequency, continuously write them to the rolling pre-cache;

[0014] Step 3: Perform local analysis on the logs written to the rolling pre-cache, extract log features, and calculate the anomaly risk value that represents the probability of the current anomaly occurring in the terminal.

[0015] Step 4: Adjust the subsequent log collection strategy and rolling pre-cache capacity based on the abnormal risk value: When the abnormal risk value is below the first threshold, maintain the current sampling frequency and current cache capacity; when the abnormal risk value reaches the first threshold, increase the sampling frequency and expand the rolling pre-cache capacity; when the abnormal risk value recovers to below the first threshold, decrease the sampling frequency and reduce the rolling pre-cache capacity.

[0016] Step 5: After detecting a critical abnormal event, export the currently stored logs in the rolling pre-cache to the abnormal log file, and attach an abnormal window flag and a forced retention flag to the exported logs to prevent the abnormal logs from being overwritten.

[0017] Furthermore, the rolling pre-cache is a circular memory buffer.

[0018] Furthermore, the abnormal risk value mentioned in step 3 is determined based on log abnormality level characteristics, abnormal keyword characteristics, service restart characteristics, memory pressure characteristics, storage pressure characteristics, and log surge characteristics. The log abnormality level characteristics are the weighted statistical value of the number of ERROR logs and WARNING logs per unit time. The abnormal keyword hit characteristics are the number of times preset abnormal keywords are hit in the logs per unit time. The service restart characteristics are the number of times critical system services are restarted per unit time. The memory pressure characteristics are the ratio of available terminal memory to total memory. The storage pressure characteristics are the ratio of remaining space in the data partition to total capacity. The log surge characteristics are the ratio of the total log volume in the current time window to the total historical baseline log volume.

[0019] Furthermore, the key abnormal events mentioned in step 5 include: Crash, ANR, watchdog timer, kernel panic, frequent system service restarts, insufficient memory, and abnormal exit of business processes.

[0020] Furthermore, the method also includes step 6: setting an active buffer and an archive buffer, continuously writing the collected logs into the current active buffer, and when the active buffer reaches the switching condition, switching the current active buffer to the archive buffer and compressing and archiving it through a background thread, while switching the other buffer to the new active buffer to continue receiving logs.

[0021] Furthermore, the switching conditions include: the active buffer reaches the single file capacity limit, the write duration reaches the rotation cycle, and a critical abnormal event is detected.

[0022] Furthermore, the method also includes step 7: classifying logs according to preset value rules, and performing differentiated compression and retention of logs according to the classification results; the value rules include: whether the log has an abnormal window marker and a forced retention marker, and whether the log level, the number of log repetitions, and the log source are within the whitelist.

[0023] Furthermore, the differentiated retention includes: logs with abnormal window flags and forced retention flags have a higher retention priority than ordinary logs, and deletion is not allowed during the forced retention period.

[0024] Furthermore, step 8 is included: when the storage space or Flash write pressure reaches a preset threshold, logs are reclaimed in ascending order of log value.

[0025] Furthermore, the process of recycling logs in step 8 according to their value from low to high includes: prioritizing the recycling of low-level duplicate logs without anomaly window markers, followed by recycling compressed archive logs without anomaly windows, and finally recycling expired medium-value logs; logs with anomaly window markers and mandatory retention markers are not recycled during the mandatory retention period.

[0026] The beneficial effects of this invention are:

[0027] (1) The initial capacity of the rolling pre-cache is set according to the available memory of the terminal, and the logs are filtered and written, so that the log system consumes less CPU and memory resources under normal operation, effectively reducing power consumption and Flash erase and write wear, extending the service life of low-resource terminals, and improving long-term operation stability.

[0028] (2) Perform local real-time analysis on the logs in the rolling pre-cache and calculate the abnormal risk value. When the risk value increases, expand the cache capacity so that the system can completely retain the abnormal logs before the abnormality occurs, which effectively improves the reliability of log-based abnormal tracing.

[0029] (3) After a critical abnormal event is detected, all logs in the rolling pre-cache are immediately exported and an abnormal window flag and a forced retention flag are attached, which effectively prevents the abnormal logs from being overwritten or recycled in subsequent cycles and improves the efficiency of abnormal location.

[0030] (4) Setting up an active buffer and an archive buffer enables parallel log collection and compression, avoids compression and archiving from blocking the collection thread, and ensures the continuity of log collection;

[0031] (5) The logs are value-based and differentially retained, so that logs with abnormal window markers and forced retention markers are given higher retention priority than ordinary logs. At the same time, by recycling logs from low to high value when the storage space or Flash write pressure reaches the threshold, it can be ensured that the terminal will not affect the main business operation due to the log system occupying too many resources. Attached Figure Description

[0032] Figure 1 This is a flowchart of the low-resource Android terminal autonomous log collection and rolling caching method described in this invention. Detailed Implementation

[0033] This invention addresses the problem of severely limited memory and storage resources in low-resource Android terminals, leading to the easy loss of logs related to the causes of anomalies. It provides an autonomous log collection and rolling caching method for low-resource Android terminals. The core of this method is: after the terminal starts, a rolling pre-cache is created, independent of the native Android log buffer and with dynamically adjustable capacity. Under normal conditions, only high-value logs are selected and written to the cache to maximize memory conservation. Simultaneously, an anomaly risk value reflecting the probability of system anomalies is calculated in real time through local analysis. When the anomaly risk value increases, the rolling pre-cache capacity is pre-expanded to accommodate more logs related to the causes of anomalies, ensuring sufficient log storage space before an anomaly occurs. Once a critical anomaly event is detected, all logs in the cache are immediately exported and locked to prevent subsequent cyclic overwriting or reclamation. This effectively solves the problem of existing Android log solutions in low-resource scenarios where fixed-capacity cyclic overwriting makes it impossible to trace the root cause of anomalies.

[0034] The technical solution of the invention will be further described clearly and completely below with reference to the embodiments and accompanying drawings.

[0035] like Figure 1 As shown, the low-resource Android terminal autonomous log collection and rolling caching method of the present invention includes the following steps.

[0036] Step 1: Create a scrolling pre-cache.

[0037] The low-resource Android terminal described in this embodiment is a carrier-customized set-top box running the Android system. After the set-top box is powered on, the log management service starts as a system daemon. The log management service first reads the product configuration file to obtain configuration parameters such as the log working directory, the maximum total log capacity, the maximum capacity of a single log file, the basic sampling period, the enhanced sampling period, the duration of the window before an anomaly, the duration of the window after an anomaly, the compression algorithm, the compression level, and the retention period. The log management service collects current system running status information, including CPU load, available memory, remaining space in the data partition, Flash write pressure, system temperature, and the status of the logcat buffer. The log management service sets the initial capacity of the rolling pre-cache based on the collected available memory size. If the available memory is lower than the memory safety threshold, the rolling pre-cache uses a smaller capacity; if the available memory is sufficient, the rolling pre-cache uses a larger capacity. At the same time, the log management service initializes the anomaly signature database, which at least contains keywords or event patterns such as "FATAL EXCEPTION", "ANR in", "Watchdog", "kernel panic", "lowmemorykiller", and "system_server died".

[0038] The rolling pre-caching is independent of the Android system's logcat log buffer and is created and maintained by the log management service in its own process space. The rolling pre-caching adopts a circular memory buffer organization, and each log record includes a timestamp, source buffer, process name, process ID, thread ID, log level, module tag, message digest, and original log content.

[0039] Step 2: Collect logs and write them to the rolling pre-cache.

[0040] Under normal system operation, the log service enters a low-resource mode. In low-resource mode, the log service periodically reads logcat and kernel logs at an initial sampling frequency and filters the read logs. The log service restricts the collection of DEBUG level logs, only collecting ERROR level, WARNING level, system critical service, and kernel exception logs. For ordinary logs that appear repeatedly with the same content, only the summary and repetition count are retained. For ERROR level, WARNING level, system critical service, and kernel exception logs, the log service retains the complete content. The log service continuously writes the filtered logs to the rolling pre-cache. When the rolling pre-cache reaches its capacity limit, low-value ordinary logs are removed first, while high-level logs and whitelisted module logs are retained.

[0041] Step 3: Local edge intelligence calculates the abnormal risk value.

[0042] The log service uses a set interval to statistically analyze the characteristics of logs written to the rolling pre-cache at a time. These log characteristics include log anomaly level characteristics, anomaly keyword hit characteristics, service restart characteristics, memory pressure characteristics, storage pressure characteristics, and log surge characteristics. The log anomaly level characteristic is a weighted statistical value of the number of ERROR logs and WARNING logs per unit time, with ERROR logs having a greater weight than WARNING logs. The anomaly keyword hit characteristic is the number of times a preset anomaly keyword is hit in the logs per unit time, with different preset anomaly keywords corresponding to different weights. The service restart characteristic is the number of times critical system services are restarted per unit time, with different critical system services corresponding to different weights. The memory pressure characteristic is the ratio of available terminal memory to total memory. The storage pressure characteristic is the ratio of remaining space in the data partition to the total capacity. The log surge characteristic is the ratio of the total log volume in the current time window to the total historical baseline log volume.

[0043] After normalizing each feature, the log service calculates an anomaly risk value that represents the probability of an anomaly occurring in the terminal. The anomaly risk value is calculated as follows:

[0044] ;

[0045] In the formula, Normalized values ​​of log anomaly level features Normalized values ​​of abnormal keyword hit features For service restart feature normalized value, Normalized values ​​of memory pressure features Normalized values ​​for storage pressure characteristics Normalized values ​​for log burst features, , , , , and The weights are assigned to the normalized values ​​of the corresponding features, and each weight can be adjusted according to the product configuration and terminal model.

[0046] Step 4: Adjust the log collection strategy and the capacity of the rolling pre-cached buffer.

[0047] When the anomaly risk value is below the set first threshold, the log service maintains the current sampling frequency and cache capacity, and the system operates in low-resource mode. When the anomaly risk value reaches the first threshold, the log service increases the sampling frequency, expands the log collection scope, begins recording more WARNING level logs, and simultaneously expands the memory allocation of the rolling pre-cache to accommodate context logs from a longer period before the anomaly occurred. When the anomaly risk value recovers to below the first threshold, the log service reduces the sampling frequency and shrinks the rolling pre-cache capacity. During the recovery process, after the anomaly risk value recovers to below the first threshold and remains below it for a preset duration, the log service then performs the operations of reducing the sampling frequency and shrinking the rolling pre-cache capacity to avoid frequent switching of the system near the threshold, which could cause resource jitter or log loss.

[0048] Step 5: Abnormal event detection and rolling pre-cached export protection.

[0049] The log service continuously monitors for critical anomalies. Critical anomalies include Crash, ANR, watchdog timer, kernel panic, frequent system service restarts, insufficient memory, and abnormal process exits. When the log service detects a critical anomaly, it generates an anomaly event identifier. The log service exports all currently stored logs from the rolling pre-cache to the anomaly log file, attaching an anomaly window flag and a forced retention flag to the exported logs to prevent them from being overwritten or autonomously reclaimed in subsequent cycles. The log service also continues to collect logs after the anomaly occurs, attaching the same anomaly window flag and forced retention flag to these logs. Anomaly log packets are written to the anomaly log file, along with the event identifier, anomaly type, start time, end time, involved processes, and keyword hit information. The anomaly log file is set to a high-value level and is not deleted during the forced retention period. If storage space is limited, the log service prioritizes deleting normal logs, but does not delete anomaly window logs.

[0050] Step 6: Asynchronous compression archiving with dual buffers.

[0051] The log service sets up an active buffer and a sealed buffer. The log service continuously writes collected logs to the current active buffer. The log service monitors the status of the active buffer; when the active buffer reaches its single-file capacity limit, the writing duration reaches its rotation cycle, or a critical abnormal event is detected, the log service switches the current active buffer to the sealed buffer, and simultaneously switches another buffer to become the new active buffer to continue receiving logs. The collection thread only needs to switch the write target to continue working; it does not need to wait for compression to complete. The log service's background thread performs log sorting, duplicate log merging, log value scoring, compression and archiving, index generation, and integrity verification on the sealed buffer. The background thread executes these operations independently, enabling log collection and compression / archiving to be performed asynchronously and in parallel, avoiding the risk of log collection blocking and log loss due to file I / O or compression calculations. The index file records the log file sequence number, time range, abnormal event identifier, log level distribution, compression status, and verification information, facilitating subsequent remote export or local location.

[0052] Step 7: Log value classification and differentiated retention.

[0053] The log service classifies logs according to preset value rules, including: whether the log has an anomaly window marker and a mandatory retention marker, the log level, the number of log repetitions, and whether the log source is on a whitelist. Logs with anomaly window markers and mandatory retention markers receive the highest value score. The log service performs differentiated compression and retention based on the classification results. High-value logs remain immediately readable or are compressed at a lower level with extended retention periods. Medium-value logs are archived using normal compression. Low-value logs are only saved as summaries or are deleted preferentially when they meet the eviction criteria. The specific rules for differentiated retention are: logs with anomaly window markers and mandatory retention markers have higher retention priority than ordinary logs and cannot be deleted during the mandatory retention period. Through tiered retention, the log service prioritizes the retention of high-value anomaly logs within limited storage space, improving the efficiency of remote problem localization.

[0054] Step 8: Perform log reclamation based on storage pressure.

[0055] The log service continuously monitors storage space and Flash write pressure. When the total log volume reaches its upper limit, the remaining storage space falls below the space safety threshold, or the Flash write pressure is too high, the log service initiates autonomous reclamation, reclaiming logs in ascending order of value. The reclamation order is as follows: first, low-level duplicate logs without anomaly window markers are reclaimed; second, compressed archived logs without anomaly windows are reclaimed; and finally, expired medium-value logs are reclaimed. Logs with anomaly window markers and mandatory retention markers are not reclaimed during the mandatory retention period. If storage space still does not recover to the safety threshold after reclamation, the log service further reduces the basic sampling frequency and log retention period, while retaining critical anomaly collection capabilities to ensure that low-resource terminals are not affected by excessive resource consumption by the log system, thus impacting main business operations. Once storage space recovers to above the safety threshold, the log service gradually restores the sampling frequency and log retention period to their previous levels.

Claims

1. A method for autonomous log collection and rolling caching on low-resource Android terminals, characterized in that, The methods include: Step 1: After the terminal starts up, set the initial capacity of the rolling pre-cache according to the available memory size of the terminal. The rolling pre-cache is independent of the log buffer. Step 2: After filtering the collected logs according to the initial sampling frequency, continuously write them to the rolling pre-cache; Step 3: Perform local analysis on the logs written to the rolling pre-cache, extract log features, and calculate the anomaly risk value that represents the probability of the current anomaly occurring in the terminal. Step 4: Adjust the subsequent log collection strategy and rolling pre-cache capacity based on the abnormal risk value: maintain the current sampling frequency and current cache capacity when the abnormal risk value is lower than the first threshold; When the abnormal risk value reaches the first threshold, the sampling frequency is increased and the rolling pre-cache capacity is expanded; when the abnormal risk value recovers to below the first threshold, the sampling frequency is reduced and the rolling pre-cache capacity is decreased. Step 5: After detecting a critical abnormal event, export the currently stored logs in the rolling pre-cache to the abnormal log file, and attach an abnormal window flag and a forced retention flag to the exported logs to prevent the abnormal logs from being overwritten.

2. The method for autonomous log collection and rolling caching on low-resource Android terminals according to claim 1, characterized in that, The rolling pre-cache is a circular memory buffer.

3. The method for autonomous log collection and rolling caching on low-resource Android terminals according to claim 1, characterized in that, The abnormal risk value mentioned in step 3 is determined based on log abnormality level characteristics, abnormal keyword characteristics, service restart characteristics, memory pressure characteristics, storage pressure characteristics, and log surge characteristics. The log abnormality level characteristics are the weighted statistical value of the number of ERROR logs and WARNING logs per unit time. The abnormal keyword hit characteristics are the number of times preset abnormal keywords are hit in the logs per unit time. The service restart characteristics are the number of times critical system services are restarted per unit time. The memory pressure characteristics are the ratio of available terminal memory to total memory. The storage pressure characteristics are the ratio of remaining space in the data partition to total capacity. The log surge characteristics are the ratio of the total log volume in the current time window to the total historical baseline log volume.

4. The method for autonomous log collection and rolling caching on low-resource Android terminals according to claim 1, characterized in that, The key abnormal events mentioned in step 5 include: Crash, ANR, watchdog timer, kernel panic, frequent system service restarts, insufficient memory, and abnormal exit of business processes.

5. The method for autonomous log collection and rolling caching on low-resource Android terminals according to any one of claims 1-4, characterized in that, The method further includes step 6: setting an active buffer and an archive buffer, continuously writing the collected logs into the current active buffer, and when the active buffer reaches the switching condition, switching the current active buffer to the archive buffer and compressing and archiving it through a background thread, while switching another buffer to the new active buffer to continue receiving logs.

6. The method for autonomous log collection and rolling caching on low-resource Android terminals according to claim 5, characterized in that, The switching conditions include: the active buffer reaches the single file capacity limit, the write duration reaches the rotation cycle, and a critical abnormal event is detected.

7. The method for autonomous log collection and rolling caching on low-resource Android terminals according to any one of claims 1-4, characterized in that, The method further includes step 7: classifying logs according to preset value rules, and performing differentiated compression and retention of logs according to the classification results; The value rules include: whether the log has an abnormal window flag and a forced retention flag, and whether the log level, log duplication count, and log source are in the whitelist.

8. The method for autonomous log collection and rolling caching on low-resource Android terminals according to claim 7, characterized in that, The differentiated retention includes: logs with abnormal window flags and forced retention flags have a higher retention priority than ordinary logs, and they are not allowed to be deleted during the forced retention period.

9. The method for autonomous log collection and rolling caching on a low-resource Android terminal according to any one of claims 1-4, characterized in that, It also includes step 8: when the storage space or Flash write pressure reaches a preset threshold, logs are reclaimed in order of log value from low to high.

10. The method for autonomous log collection and rolling caching on a low-resource Android terminal according to claim 9, characterized in that, The process of reclaiming logs in step 8, in order of increasing log value, includes: prioritizing the reclamation of low-level duplicate logs without anomaly window markers, followed by the reclamation of compressed archive logs without anomaly windows, and finally the reclamation of expired medium-value logs; logs with anomaly window markers and mandatory retention markers are not reclaimed during the mandatory retention period.

Images