Embedded device log management method and system

Abstract

The invention relates to the technical field of embedded system log management, and relates to an embedded device log management method and system, and the method comprises the steps: S1, scanning a log stream line by line, analyzing each line of logs, and filtering the logs according to a routing rule; s2a, splitting the log routed to the local storage into a plurality of independent files, and monitoring and writing the independent files in real time; and S2b, for the log routed to the temporary buffer area, if the number of times of occurrence of the same predefined prefix label in a predetermined time exceeds a threshold value, generating a temporary routing rule and adding the temporary routing rule to a routing table. By implementing the scheme, (1) the log management efficiency and the system reliability are improved; (2) efficient storage, resource optimization and fault self-diagnosis of embedded system logs are realized, and the service life of hardware is prolonged; and (3) the log time continuity is ensured, the efficiency of troubleshooting and system analysis is greatly improved, the data security is ensured, convenient access is provided for authorized users, and a complete operation auditing capability is provided.

Description

Technical Field

[0001] This invention relates to the field of embedded system log management technology, and specifically to an embedded device log management method and system. Background Technology

[0002] In embedded devices such as routers and IoT gateways that typically run on customized Linux systems like OpenWrt for extended periods, logging systems are crucial for device monitoring, troubleshooting, and performance analysis. However, existing log processing methods suffer from one or more of the following serious drawbacks: 1. Log information overload: Devices generate massive amounts of system logs when busy. For example, OpenWrt can generate more than 10MB of logs in 5 minutes. If application service customized logs are stored together with system logs, critical debugging information can easily be overwhelmed, making it difficult to trace problems.

[0003] 2. Excessive Flash Memory Wear: Flash memory, as the primary storage medium, has a limited write / erase lifespan. Some simple circular logging methods, such as periodically removing the header of a log file using the `tail` command, functionally achieve circular writing, but their essence is "reading the entire file – writing to a temporary file – renaming and overwriting the original file." This operation produces a huge "write amplification" effect, meaning that rewriting the entire file to delete a small amount of old data leads to frequent erasure and rewriting of flash memory blocks, drastically shortening the lifespan of the device hardware.

[0004] 3. High-concurrency write bottleneck: When faced with a surge of high-concurrency logs, the above-mentioned tail-class log processing methods will cause a serious disk I / O bottleneck due to frequent file read, write and rename operations, which may lead to log recording delays or even loss.

[0005] 4. Log export is insecure: In many scenarios, logs may contain sensitive information. Directly providing download links for log files poses security risks and lacks effective access control and encryption mechanisms.

[0006] Therefore, there is an urgent need for a log management method designed specifically for resource-constrained embedded devices that can effectively distinguish and manage logs from different sources, maximize flash memory lifespan, and provide secure access mechanisms. Summary of the Invention

[0007] Based on the above situation, the present invention aims to overcome the shortcomings of the prior art and provide a method for efficient log diversion, filtering and cyclic writing in resource-constrained embedded systems. This method aims to solve the problems of critical information being overwhelmed by massive log output, flash memory "write amplification" and excessive wear caused by using a simple cyclic writing method, and I / O performance bottlenecks in high-concurrency log writing scenarios, and provides a controllable and secure log viewing and export mechanism.

[0008] The technical solution adopted by this invention to solve its technical problem is as follows: In a first aspect, the present invention provides an embedded device log management method, comprising the following steps: S1 scans the real-time log stream output by the embedded system line by line, parses the predefined prefix label of each log line and filters the logs according to the routing rules. If the predefined prefix label of the log matches the routing rule, the log is routed to the cloud or local storage according to the routing rule; if the predefined prefix label of the log does not match the routing rule, the log is routed to a temporary buffer; if the log has no predefined prefix label, the log is discarded. For logs routed to local storage, the following steps are performed: S2a splits the logs routed to local storage into multiple independent files, determines the initial active shard file and writes it to all independent files; during the writing process, it monitors the size of the active shard file in real time to determine whether to trigger dynamic switching of the active shard and circular overlay. For logs routed to a temporary buffer, the following steps are performed: S2b: For logs routed to the temporary buffer, if the number of times the same predefined prefix label appears exceeds the threshold within a predetermined time, a temporary routing rule is generated and added to the routing table, and the logs containing the predefined prefix label are returned to the execution step S1; logs routed to the temporary buffer whose number of times the predefined prefix label appears within the predetermined time does not exceed the threshold are marked as abnormal logs.

[0009] Furthermore, the predefined prefix tag includes a dynamic anti-tampering verification field. When constructing the tag, a lightweight hash value is calculated based on the original values ​​of all fixed fields within the tag, and this hash value is embedded as an anti-tampering field at the end of the tag.

[0010] Furthermore, the routing rules adopt a dynamic strategy, allowing routing rules to be dynamically loaded, modified, or deleted during system runtime through configuration files or management interfaces based on the real-time status of the device or external commands. The routing rules are dynamically configurable and multi-dimensional. They support pattern matching of prefix labels using two types of wildcards. The first wildcard matches a single label field, while the second wildcard can match zero or more consecutive label fields.

[0011] Furthermore, the method for determining the initial active fragment file in step S2a above specifically includes: Extract key fields from the log to be written and generate hash input values; A lightweight hash function is used to calculate the log hash value from the hash input value; the target shard index is obtained by taking the log hash value modulo the preset total number of shards N. Based on the log writing order, the shard file corresponding to the target shard index of the first log is determined as the initial active shard file; or based on the remaining space size of each shard file, the shard file with the largest remaining space is selected as the initial active shard file. Record the index, starting address, remaining space threshold, and current write pointer position of the initial active fragment file.

[0012] Furthermore, the method for triggering dynamic switching of active fragments and ring coverage in step S2a above specifically includes: During the writing process, the remaining space of the active shard file is read in real time, and the remaining space threshold of the current active shard is obtained through the shard mapping table; When the remaining space of the active shard file is less than or equal to the remaining space threshold, the active shard is dynamically switched. The target shard index for switching is determined by hash order: based on the current active shard index, the next shard index is calculated, and the shard corresponding to the next shard index with a remaining space greater than the remaining space threshold is selected as the new active shard. When the remaining space of all shards is less than or equal to the remaining space threshold, a circular overwrite is triggered; based on the write pointers of each shard in the shard mapping table, the earliest write shard with the smallest write pointer is located as the overwrite target shard; new logs are written to the write pointer position of the overwrite target shard, overwriting the earliest written log data, and the write pointer of the shard is updated. After a switch or overwrite operation is completed, the active shard index, remaining space of each shard, and write pointer information in the shard mapping table are updated in real time to ensure consistency of state.

[0013] Furthermore, when it is necessary to view the complete log, the following steps are also included: S3: Traverse the log directory to extract all files that conform to the segment naming rules, parse and extract the segment sequence number of each file, sort the segment sequence numbers according to the size of the segment file, and output the contents of each segment file in turn to obtain a complete log file with continuous time.

[0014] Furthermore, when it is necessary to safely export log files, the following steps are also included: S4, obtain the call request to export the log file. If the call request does not contain a key parameter, encrypt and compress the log file using a preset internal key, generate and return a first download URL. If the call request contains a key parameter, verify whether the key parameter is correct. When the key parameter is correct, compress the log file without encryption, generate and return a second download URL. When the key parameter is incorrect, perform the same operation as when no key parameter is included, and return the first download URL.

[0015] Secondly, this invention provides an embedded device log management system, comprising: a log filtering module, a routing decision module, a local fragmented writing and dynamic management module, a temporary buffer management module, a log reassembly module, a log export module, and a non-volatile storage module; each module is connected via an on-chip bus or communication interface to collaboratively implement log filtering routing, fragmented writing, rule generation, complete log viewing, and secure export, wherein: The log filtering module receives the log stream output in real time from the embedded system, parses each log tag line by line, extracts the tag content and log metadata, and outputs the parsed log information to the routing decision module. The routing decision module makes decisions on the parsed logs based on preset routing rules: if the log tag matches the routing rule, the log is routed to the local sharding and dynamic management module or the cloud interface module; if the log tag does not match the routing rule, the log is written to the temporary buffer of the temporary buffer management module; if the log has no predefined prefix tag, it is directly discarded and the discard event is recorded to the non-volatile storage module. The local shard writing and dynamic management module receives logs routed to local storage, extracts key log fields to generate hash input, calculates hash values ​​using a lightweight hash function, takes the modulo of the preset total number of shards N to obtain the target shard index, determines the initial active shard file, writes the logs to that file, and records the active shard index, shard physical address, remaining space threshold, and write pointer in the shard mapping table of the non-volatile storage module. It also monitors the remaining space of the active shard in real time. When the remaining space is less than or equal to the threshold, it switches to the next shard with sufficient remaining space as the new active shard according to hash order. If all shards have insufficient remaining space, it locates the earliest write shard with the smallest write pointer, overwrites its earliest log data, and updates the write pointer. The shard mapping table is updated synchronously in real time. The temporary buffer management module is used to store logs routed to the temporary buffer and count the number of times the same predefined prefix label appears within a preset time. If the number exceeds the threshold, a temporary routing rule is generated and updated to the routing table of the routing decision module. At the same time, the log of the label is re-output to the routing decision module for secondary routing. For logs whose number of occurrences of the label within the preset time does not exceed the threshold, they are marked as abnormal logs and stored in the abnormal log partition of the non-volatile storage module for subsequent analysis. The non-volatile storage module is used to store metadata and log files required for system operation, including: routing table, sharding mapping table, temporary buffer log, exception log partition and exported log index.

[0016] Furthermore, the log reassembly module is used to traverse the log directory and extract files that conform to the segmentation naming rules when a request to view the complete log is received; parse the sequence number within each segment file and sort them according to the size of the segment file; read the contents of each segment file in sequence, and concatenate them according to the sequence number to generate a complete log file with continuous time, and output it to the user interface or debugging interface.

[0017] Furthermore, the log export module, upon receiving a secure log export request, retrieves the call parameters of the export request and determines whether a key parameter is included. If no key parameter is included, it encrypts and compresses the complete log file using a preset internal key, generates an encrypted compressed package, and returns the first download URL. If a key parameter is included, it verifies the key's correctness: if correct, it only compresses the file and generates a second download URL; if incorrect, it performs the same encryption and compression operation as when no key is included and returns the first download URL. After the export is complete, it updates the export log index of the non-volatile storage module, recording the export time, file hash, and access permissions.

[0018] The beneficial effects of this invention include: Firstly, this invention significantly reduces embedded system resource consumption through intelligent routing and filtering of logs, solving the problem of critical information being overwhelmed by massive log output, and improving log management efficiency and system reliability. Secondly, this invention solves the problems of flash memory "write amplification" and excessive wear caused by using simple loop-like writing methods through intelligent sharding and adaptive learning, achieving efficient storage, resource optimization, and fault self-diagnosis of embedded system logs. Thirdly, this invention solves the I / O performance bottleneck problem in high-concurrency log writing scenarios through intelligent reassembly of sharded logs and key verification to dynamically control log encryption, providing a controllable and secure log viewing and export mechanism, ensuring log time continuity, greatly improving the efficiency of fault diagnosis and system analysis, providing convenient access for authorized users while ensuring data security, and possessing complete operation auditing capabilities. Attached Figure Description

[0019] Figure 1 This is the embedded device log management method flow provided in the embodiments of the present invention. Figure 1 .

[0020] Figure 2 This is the embedded device log management method flow provided in the embodiments of the present invention. Figure 2 .

[0021] Figure 3 This is a structural block diagram of the embedded device log management system provided in an embodiment of the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0023] Please refer to Figure 1 To achieve efficient filtering, distribution, and storage of logs, this application provides an embedded device log management method, including the following steps: S1 scans the real-time log stream output by the embedded system line by line, parses the predefined prefix label of each log line and filters the logs according to the routing rules. If the predefined prefix label of the log matches the routing rule, the log is routed to the cloud or local storage according to the routing rule; if the predefined prefix label of the log does not match the routing rule, the log is routed to a temporary buffer; if the log has no predefined prefix label, the log is discarded. For logs routed to local storage, the following steps are performed: S2a splits the logs routed to local storage into multiple independent files, determines the initial active shard file and writes it to all independent files; during the writing process, it monitors the size of the active shard file in real time to determine whether to trigger dynamic switching of the active shard and circular overlay. For logs routed to a temporary buffer, the following steps are performed: S2b: For logs routed to the temporary buffer, if the number of times the same predefined prefix label appears exceeds the threshold within a predetermined time, a temporary routing rule is generated and added to the routing table, and the logs containing the predefined prefix label are returned to the execution step S1; logs routed to the temporary buffer whose number of times the predefined prefix label appears within the predetermined time does not exceed the threshold are marked as abnormal logs.

[0024] Specifically, during the operation of the embedded device, a log stream containing device status, fault alarms, and sensor data is generated in real time. First, a routing decision is made for the real-time output log stream. Log tags are matched with the routing rules stored locally, and the logs are routed to the local sharding write module or the cloud. Log tags that do not match the rules are routed to a temporary buffer, and unlabeled logs are discarded directly. Second, initial sharding is performed, extracting key fields to calculate hash values, total number of shards, target shard index, and initializing the sharding mapping table to record active shards, remaining space threshold, and write pointers. After continuous log writing, remaining space triggers a switch, checking the next shard index and remaining space in hash order, switching to the next shard, and updating the active shard in the mapping table. If the number of occurrences of log tags received in the temporary buffer is found to be greater than the threshold, a temporary routing rule is generated, and the routing table is updated. Subsequent tagged logs are re-routed to the local sharding write module, hashed, and then written. If the number of occurrences of another log tag in the buffer is less than the threshold, it is marked as an abnormal log and stored in an abnormal partition. In summary, by employing a closed-loop mechanism of dynamic routing, sharding management, and rule learning, problems such as rigid tag matching, uneven storage, and loss of unmatched logs in embedded log management are solved, achieving efficient log storage, value mining, and improved system reliability.

[0025] Due to the resource constraints, diverse deployment environments, and high real-time requirements of embedded devices, high-value, low-time-sensitivity logs that need long-term retention or cross-device analysis are often stored in the cloud to avoid redundant data consuming local resources and affecting Flash lifespan and bandwidth. For example, when a device detects hardware failures such as Flash bad block rate exceeding the threshold or abnormal temperature, or software failures such as process crashes or memory overflows, it automatically captures the full logs, including system calls, sensor data, and communication packets, from 5 minutes before the failure to 10 minutes after the failure, and attaches the failure event ID to form an independent log package. Furthermore, for intermittent failures that cannot be located locally, such as occasional communication packet loss or random restarts, the device retains a fault snapshot after restarting, including register states, stack traces, and the last 100 operation logs, which is then uploaded to the cloud after connecting to the network. This allows engineers to reproduce and analyze the failures using historical data. Other logs stored in the cloud include those related to unauthorized access attempts, firmware / configuration file tampering, network attack detection, log configuration changes, key parameter modifications, as well as periodic summary logs, version iteration comparison logs, and related logs requiring cross-device collaborative analysis.

[0026] For logs that can be stored in the cloud, a lightweight filtering agent is deployed on embedded devices to determine whether to upload them in real time based on preset rules such as log level, keywords, and event type, reducing invalid data transmission. Each uploaded log is accompanied by metadata such as device ID, timestamp, log type, and associated event ID, supporting categorized storage in the cloud, such as database partitioning and retrieval by device model and fault type. Incremental upload, breakpoint resume, and compression algorithms can be used to reduce bandwidth consumption. In summary, by focusing on high-value logs such as fault diagnosis, security auditing, long-term analysis, and cross-device collaboration, and combining edge-end intelligent filtering with cloud metadata management, we can extend the lifespan of local storage, reduce transmission costs, and centrally utilize cloud computing power to achieve in-depth operation and maintenance and compliance traceability, ultimately building a collaborative log management system that enables efficient local processing and cloud-based value mining.

[0027] The method of storing embedded device logs in the cloud includes: (1) real-time filtering of locally generated logs based on preset filtering rules, and sending logs that need to be uploaded to the cloud, such as fault-related logs, security event logs, and periodic statistical logs, to the cloud message queue through an encrypted channel; (2) storing high-value logs such as fault-related logs and real-time alarm logs within 7 days after uploading in a time-series database, and sharding them by device ID-date, supplemented by memory caching to support millisecond-level queries; storing historical logs such as version iteration comparison logs and periodic statistical summaries from 3 months to 1 year in a distributed NoSQL database or columnar storage file, partitioning them by log type + time window and establishing a secondary index; storing logs such as original debugging logs, etc. from 3 months to 1 year in a distributed NoSQL database or columnar storage file, partitioning them by log type + time window and establishing a secondary index; storing logs such as original debugging logs, etc. from 1 year to 1 year in a distributed NoSQL database or columnar storage file. The low-frequency access logs such as the full package of faults have been migrated to cloud object storage and are automatically managed through lifecycle policies. Expired data is deleted anonymously. (3) The raw logs received from the cloud are processed in sequence: sensitive data is filtered, timestamps are corrected, and metadata is supplemented through the stream processing engine; unstructured logs are converted into a unified JSON format, core fields are extracted, and the integrity and time sequence of logs are verified through the rule engine. Abnormal logs are marked and transferred to the manual review queue. (4) All logs are encrypted with AES-256 when stored statically, and the key is managed by KMS. The permissions for querying, downloading, and deleting operations are restricted based on the RBAC model, and only authorized users are authorized to access the logs through two-way digital certificate authentication. All log management operations are recorded and uploaded to the consortium chain through the hash chain to ensure that they are tamper-proof.

[0028] In embedded system log management, the discarding of logs without predefined prefix tags is typically a proactive filtering strategy based on resource constraints, storage costs, processing efficiency, and log value assessment, rather than random discarding. Predefined prefix tags serve as log identifiers, carrying core metadata such as log type, importance, and source. Therefore, unlabeled logs, lacking this crucial information, are difficult to utilize efficiently and may become a burden on the system. Embedded devices reduce resource consumption through tag filtering, ensuring the operation of core business processes; they also intercept invalid data through tags, reducing storage / processing costs and improving analysis efficiency. The core reason for discarding unlabeled logs is that the system retains only logs whose value can be quickly determined and who can be used efficiently, avoiding being dragged down by data noise, and ultimately maximizing log value with limited resources.

[0029] To ensure the integrity and tamper-proof nature of log tags, the aforementioned predefined prefix tags include a dynamic tamper-proof verification field. When constructing a tag, a lightweight hash value is calculated based on the original values ​​of all fixed fields within the tag, and this hash value is embedded as an tamper-proof field at the end of the tag.

[0030] Specifically, when generating logs, a dynamic anti-tampering verification field is added to the predefined prefix label of each log entry. The log label contains three fixed fields: device ID, log level, and timestamp. These fixed fields are concatenated in order, and a hash value is calculated using a lightweight hash function such as CRC32 hardware acceleration. This hash value is then embedded as the anti-tampering field at the end of the label. Upon receiving the log, the fixed fields and the anti-tampering field are separated from the label, and the hash value of the fixed fields is recalculated using the same hash function. The recalculated hash value is compared with the anti-tampering field in the label: if they match, the label has not been tampered with, and the log is parsed normally; if they do not match, the label is deemed abnormal, the log is discarded, and the tampering event is recorded. The above process, which generates hashes based on fixed fields during construction and recalculates and compares them during verification, ensures label integrity and provides a secure foundation for reliable management of embedded logs.

[0031] To ensure that routing rules are dynamically configurable and can be modified at runtime, a dynamic strategy is adopted, allowing routing rules to be dynamically loaded, modified, or deleted during system runtime through configuration files or management interfaces based on the real-time status of the device or external commands. The routing rules are dynamically configurable multi-dimensional routing rules that support pattern matching of prefix labels using two types of wildcards. The first wildcard matches one label field, while the second wildcard can match zero or more consecutive label fields.

[0032] Specifically, during the operation of embedded devices, log routing rules need to be dynamically adjusted based on real-time status, supporting two types of wildcard matching for tag fields. Basic routing rules are pre-configured at the factory, supporting two types of wildcards: the first matches a single field, and the second matches zero or more consecutive fields. For example, during device operation, if a temperature sensor detects that the temperature has risen above a threshold, the following dynamic adjustment is triggered: the maintenance platform sends a JSON configuration, the system parses the configuration, adds a new rule, routes it to the local alarm module, and removes outdated rules to avoid conflicts. After the device completes temperature adjustment, the maintenance platform sends a command, and the system deletes the original rule, ensuring a streamlined rule set. In this embodiment, when the device temperature is abnormal, rules are automatically added / deleted without requiring a system restart, enabling flexible runtime adjustment of routing rules to adapt to changes in device status and maintenance needs, thus improving the intelligence and adaptability of log management.

[0033] To achieve intelligent determination of the initial active fragment file, the method for determining the initial active fragment file in step S2a specifically includes: Extract key fields from the log to be written and generate hash input values; A lightweight hash function is used to calculate the log hash value from the hash input value; the target shard index is obtained by taking the log hash value modulo the preset total number of shards N. Based on the log writing order, the shard file corresponding to the target shard index of the first log is determined as the initial active shard file; or based on the remaining space size of each shard file, the shard file with the largest remaining space is selected as the initial active shard file. Record the index, starting address, remaining space threshold, and current write pointer position of the initial active fragment file.

[0034] To achieve intelligent determination of the initial active fragment file, the method for triggering dynamic switching and circular overlay of the active fragment in step S2a specifically includes: During the writing process, the remaining space of the active shard file is read in real time, and the remaining space threshold of the current active shard is obtained through the shard mapping table; When the remaining space of the active shard file is less than or equal to the remaining space threshold, the active shard is dynamically switched. The target shard index for switching is determined by hash order: based on the current active shard index, the next shard index is calculated, and the shard corresponding to the next shard index with a remaining space greater than the remaining space threshold is selected as the new active shard. When the remaining space of all shards is less than or equal to the remaining space threshold, a circular overwrite is triggered; based on the write pointers of each shard in the shard mapping table, the earliest write shard with the smallest write pointer is located as the overwrite target shard; new logs are written to the write pointer position of the overwrite target shard, overwriting the earliest written log data, and the write pointer of the shard is updated. After a switch or overwrite operation is completed, the active shard index, remaining space of each shard, and write pointer information in the shard mapping table are updated in real time to ensure consistency of state.

[0035] Specifically, during embedded device operation, filtered logs need to be written to local storage. In this embodiment, the initial active shard file is determined through hash calculation. The device generates a log in real time, including: device ID, timestamp, and log level, which are concatenated into a hash input value. A lightweight hash function, such as CRC32, is used to accelerate the calculation of the hash value of the input value. A preset total number of shards and a target shard index are used. If the log is the first log in the current log stream, the shard file corresponding to the shard index is directly determined as the initial active shard. The system checks the remaining space of each shard and selects the target shard index with the largest remaining space as the initial active shard. The initial active shard information, including the index, starting address, remaining space threshold, and write pointer, is recorded in non-volatile storage. Through hash calculation and dynamic strategies, intelligent determination of the initial active shard file is achieved, supporting both the first log hash and the largest remaining space modes, ensuring balanced log writing and storage efficiency.

[0036] Please refer to Figure 2 To ensure log continuity and greatly improve the efficiency of troubleshooting and system analysis, the following steps are also included when viewing the complete log: S3: Traverse the log directory to extract all files that conform to the segment naming rules, parse and extract the segment sequence number of each file, sort the segment sequence numbers according to the size of the segment file, and output the contents of each segment file in turn to obtain a complete log file with continuous time.

[0037] Specifically, when it is necessary to view the complete historical operation logs of an embedded device for fault diagnosis or system behavior analysis, the system performs the following steps to achieve intelligent reorganization of log fragments: First, the system traverses the log directory in the embedded device's local storage, filters all log fragment files according to predefined fragment naming rules, and automatically ignores other files that do not conform to the naming rules, ensuring that only valid log data blocks are processed. For each identified fragment file, the system parses its file header or embedded metadata block. This metadata records a key piece of information—the fragment sequence number. This sequence number is generated by the system when logs are written to a fragment and is used to identify the logical order of log records within that fragment. The system associates the parsed fragment sequence number with the actual size of each fragment file. Subsequently, the system sorts the fragments in ascending or descending order according to file size.

[0038] The core logic of the aforementioned intelligent reassembly lies in the fact that, under a circular buffer or sequential write mechanism, the size of a log file is usually proportional to the accumulated time of the log records; a larger file often means that it contains earlier log time ranges or covers a longer period. Through this size-based sorting, the system can indirectly reconstruct the temporal flow of the log data. The system strictly follows the sorted order, sequentially reading the entire content of each fragment file. Then, the system sequentially concatenates these content blocks into a continuous byte stream or text stream. Finally, the system outputs or generates a complete, time-sequential log file. The log records in this file are arranged strictly in chronological order, providing maintenance personnel with a clear and coherent view of system activity. Through the above steps, this embodiment achieves efficient and accurate reassembly of fragmented logs. This method ensures that even after the logs are automatically segmented and stored, users can still obtain a complete log view arranged in chronological order, greatly improving the efficiency and accuracy of troubleshooting, security auditing, and system performance analysis.

[0039] When it is necessary to securely export log files, the method further includes the following steps: S4, obtaining a call request to export log files; if the call request does not contain a key parameter, encrypting and compressing the log file using a preset internal key, generating and returning a first download URL; if the call request contains a key parameter, verifying whether the key parameter is correct; if the key parameter is correct, compressing the log file without encryption, generating and returning a second download URL; if the key parameter is incorrect, performing the same operation as when no key parameter is contained, and returning the first download URL.

[0040] Specifically, when secure log export is required, encryption is determined by the key parameter, and an encrypted or compressed URL is returned to ensure data security. Specifically: When a user initiates an export request through the management interface without a key, the system triggers the following operations: The complete log file is encrypted using a preset internal key, then compressed using a compression algorithm to generate an encrypted compressed package; a first download URL is generated for the compressed package, containing a timestamp and file hash, and returned to the user. The logs must be decrypted using the internal key to view them.

[0041] In another scenario, the user initiates an export request with the correct key. The system compares the requested key with the preset external key. If the verification is successful, the log file is compressed but not encrypted, generating a compressed package. A second download URL is generated for the compressed package, which the user can download and directly decompress to view the plaintext log.

[0042] Another scenario is that a user initiates a request but the key is incorrect, causing a key comparison failure. This triggers the same encryption and compression process as in the "no key" scenario, returning a first URL: generating an encrypted compressed package and a first download URL. The user needs to decrypt it using an internal key, thus preventing unauthorized access. This embodiment dynamically controls the encryption strategy for log export through key parameters, balancing data security and access convenience, and providing a flexible and reliable solution for the secure management of embedded device logs.

[0043] Please refer to Figure 3 This application provides an embedded device log management system for executing the aforementioned embedded device log management method. The system includes: a log filtering module, a routing decision module, a local fragmented writing and dynamic management module, a temporary buffer management module, a log reassembly module, a log export module, and a non-volatile storage module. Each module is connected via an on-chip bus or communication interface to collaboratively implement log filtering routing, fragmented writing, rule generation, complete log viewing, and secure export. The log filtering module receives the log stream output in real time from the embedded system, parses each log tag line by line, extracts the tag content and log metadata, and outputs the parsed log information to the routing decision module. The routing decision module makes decisions on the parsed logs based on preset routing rules: if the log tag matches the routing rule, the log is routed to the local sharding and dynamic management module or the cloud interface module; if the log tag does not match the routing rule, the log is written to the temporary buffer of the temporary buffer management module; if the log has no predefined prefix tag, it is directly discarded and the discard event is recorded to the non-volatile storage module. The local shard writing and dynamic management module receives logs routed to local storage, extracts key log fields to generate hash input, calculates hash values ​​using a lightweight hash function, takes the modulo of the preset total number of shards N to obtain the target shard index, determines the initial active shard file, writes the logs to that file, and records the active shard index, shard physical address, remaining space threshold, and write pointer in the shard mapping table of the non-volatile storage module. It also monitors the remaining space of the active shard in real time. When the remaining space is less than or equal to the threshold, it switches to the next shard with sufficient remaining space as the new active shard according to hash order. If all shards have insufficient remaining space, it locates the earliest write shard with the smallest write pointer, overwrites its earliest log data, and updates the write pointer. The shard mapping table is updated synchronously in real time. The temporary buffer management module is used to store logs routed to the temporary buffer and count the number of times the same predefined prefix label appears within a preset time. If the number exceeds the threshold, a temporary routing rule is generated and updated to the routing table of the routing decision module. At the same time, the log of the label is re-output to the routing decision module for secondary routing. For logs whose number of occurrences of the label within the preset time does not exceed the threshold, they are marked as abnormal logs and stored in the abnormal log partition of the non-volatile storage module for subsequent analysis. The non-volatile storage module is used to store metadata and log files required for system operation, including: routing table, sharding mapping table, temporary buffer log, exception log partition and exported log index.

[0044] The log reassembly module is used to traverse the log directory and extract files that conform to the segmentation naming rules when a request to view the complete log is received; parse the sequence number within each segment file and sort them according to the size of the segment file; read the contents of each segment file in sequence and concatenate them according to the sequence number to generate a complete log file with continuous time, and output it to the user interface or debugging interface.

[0045] The log export module, upon receiving a secure log export request, retrieves the request's call parameters and determines whether a key parameter is included. If no key parameter is included, it encrypts and compresses the complete log file using a preset internal key, generating an encrypted compressed package and returning the first download URL. If a key parameter is included, it verifies the key's correctness: if correct, it only compresses the file and generates a second download URL; if incorrect, it performs the same encryption and compression operation as when no key was included and returns the first download URL. After exporting, it updates the export log index in the non-volatile storage module, recording the export time, file hash, and access permissions.

[0046] For storing logs in the cloud, the log management system also includes: The log receiving module, deployed in the cloud, is used to receive logs uploaded by embedded devices through an encrypted channel and connects to a message queue to achieve traffic peak shaving and valley filling.

[0047] The tiered storage module includes a hot data storage unit for configuring a time-series database and memory cache to store high-value logs from the past 7 days, supporting millisecond-level queries; a warm data storage unit for configuring a distributed NoSQL database and columnar storage files to store historical logs from 3 months to 1 year, supporting complex aggregation analysis; and a cold data storage unit for configuring cloud object storage or tape library to store low-frequency logs from more than 1 year, supporting lifecycle management and on-demand unfreezing.

[0048] The data processing engine integrates stream processing, batch processing, and rules engine to perform log cleaning, standardization, and quality verification.

[0049] The security audit module includes an encryption unit for encrypting stored logs using AES-256, with the key managed by KMS; it also includes an access control unit that manages user operation permissions based on the RBAC model and supports two-way digital certificate authentication; and an audit log unit for recording all log management operations and uploading them to the consortium blockchain via a hash chain to ensure immutability.

[0050] This application also provides a specific embodiment of a log management method based on log tag filtering and fragmented round-robin writing. Its core processes include log filtering and distribution, fragmented round-robin storage, complete log reassembly, and secure export. Specifically, this method is implemented on an embedded device such as an OpenWrt router, primarily executed by a background shell script (hereinafter referred to as the "log filtering script"), and interacts with the system's standard log service logread and application layer bus ubus.

[0051] The key parameters are defined as follows: MAX_FLASH_FILE_SIZE: The maximum total size of log files on flash memory.

[0052] NUM_PART: The number of log shards.

[0053] PART_SIZE: The size threshold for a single log shard, MAX_FLASH_FILE_SIZE / NUM_PART.

[0054] Log shard naming convention: pratFlash_n_m.log. Where n is the original shard number, representing a fixed storage slot; m is the renamed sequence number, representing the chronological order of log records, with larger m values ​​indicating newer logs.

[0055] The method steps in this embodiment include: Step one, tag-based log filtering and distribution, specifically includes: 1. Applications such as C++, LUA, and Shell output logs by calling standard syslog or logger interfaces. To be recognized by the system, they must include a predefined special string prefix when setting log tags, such as JYTL_TOFL_ for routing to flash memory and JYTL_TORA_ for routing to memory. Log filtering scripts pipe all log streams from the system log reading tool `logread -f` in real time.

[0056] 2. The script scans the log stream line by line, checking each line of log data for a predefined tag prefix through string matching.

[0057] 3. If a match is found, such as if JYTL_TOFL_ is included, the log line is appended to the currently active flash log shard file. Otherwise, it can be discarded or processed according to other rules.

[0058] Step two, based on circular buffer-based fragmented round-robin writing, specifically includes: 1. During system initialization, the `get_current_log_file()` function searches for the existing fragment file with the largest `m` value and uses it as the currently active write file. If there are no fragments, the initial file `pratFlash_0_0.log` is created.

[0059] During the log writing process, the script periodically checks the size of the currently active shard file, for example, after writing every 50 log entries.

[0060] 2. When the file size exceeds the preset PART_SIZE threshold, the rotate_log_files() function is triggered to perform rotation.

[0061] The rotation logic is as follows: a. Parse the currently filled filename, such as pratFlash_n_m.log, and obtain its n and m values; b. Calculate the sequence number m_next for the next slice = (m + 1) % NUM_PART; c. Construct a new activity file name, with the n value remaining unchanged and the m value updated to m_next, i.e., pratFlash_n_${m_next}.log; d. Subsequent logs will be written directly to this new shard file.

[0062] This process achieves circular overwriting, meaning that after the value of m completes a full cycle, the new log will automatically overwrite the segment file containing the oldest log data. This ensures a constant total log size without moving or rewriting a large amount of data. This method eliminates the need for frequent erasure and rewriting of flash memory blocks, thus increasing the lifespan of the device hardware.

[0063] Step 3, the orderly reorganization of the complete log, specifically includes: 1. To view the complete, time-sorted logs, call the `get_full_log()` function. This function first iterates through all files in the log directory that conform to the naming convention `pratFlash_*_*.log`.

[0064] 2. Parse each filename and extract its sequence number m. Sort all files strictly in ascending order according to the numerical value of m.

[0065] 3. Following the sorted file list, use commands such as cat to output the contents of each file segment sequentially, thus obtaining a complete log with continuous time.

[0066] Step four, secure export based on the key, specifically includes: 1. The system provides an ubus interface, such as luci.getLogFileUrl, for the web frontend or other services to call to obtain logs.

[0067] 2. This interface accepts an optional key parameter.

[0068] The default (no key) call is: `ubus call luci getLogFileUrl`. The interface first performs step three to generate a complete log, then encrypts and compresses it using tools such as OpenSSL and a preset internal key, and finally returns a download URL pointing to the encrypted compressed package.

[0069] Calling with a key: `ubus call luci getLogFileUrl '{"key":"XWCyyds778"}'`. The interface first verifies whether the passed key is correct.

[0070] If the key is correct, the interface will only compress the complete log without encryption and return a download URL pointing to the unencrypted compressed package.

[0071] If the key is incorrect, the behavior is the same as the default call, returning the URL of the encrypted compressed file.

[0072] Users can download the file corresponding to the URL through their browser. Encrypted files need to be manually decrypted using the corresponding key via the openssl command.

[0073] It's important to note that, to significantly reduce the number of write operations to the flash memory, the system doesn't immediately flush each log entry to flash. Instead, it concatenates multiple log entries generated within a short period into a larger data block in memory before writing them all at once. This consolidates a large number of small I / O requests into a smaller number of large I / O requests.

[0074] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.

[0075] It should be noted that the embodiments described above are only some embodiments of the present invention, and not all embodiments. The singular forms "a," "described," and "it" used in the embodiments and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

Claims

1. An embedded device log management method, characterized in that, Including steps: S1 scans the real-time log stream output by the embedded system line by line, parses the predefined prefix label of each log line and filters the logs according to the routing rules. If the predefined prefix label of the log matches the routing rule, the log is routed to the cloud or local storage according to the routing rule; if the predefined prefix label of the log does not match the routing rule, the log is routed to a temporary buffer; if the log has no predefined prefix label, the log is discarded. For logs routed to local storage, the following steps are performed: S2a splits the logs routed to local storage into multiple independent files, determines the initial active shard file and writes it to all independent files; during the writing process, it monitors the size of the active shard file in real time to determine whether to trigger dynamic switching of the active shard and circular overlay. For logs routed to a temporary buffer, the following steps are performed: S2b: For logs routed to the temporary buffer, if the number of times the same predefined prefix label appears exceeds the threshold within a predetermined time, a temporary routing rule is generated and added to the routing table, and the logs containing the predefined prefix label are returned to the execution step S1; logs routed to the temporary buffer whose number of times the predefined prefix label appears within the predetermined time does not exceed the threshold are marked as abnormal logs.

2. The method according to claim 1, characterized in that, The predefined prefix tag includes a dynamic anti-tampering verification field. When constructing the tag, a lightweight hash value is calculated based on the original values ​​of all fixed fields within the tag, and this hash value is embedded at the end of the tag as an anti-tampering field.

3. The method according to claim 1, characterized in that, The routing rules adopt a dynamic strategy, which allows routing rules to be dynamically loaded, modified, or deleted during system runtime through configuration files or management interfaces based on the real-time status of the device or external commands. The routing rules are dynamically configurable, multi-dimensional routing rules that support pattern matching of prefix labels using two types of wildcards. The first wildcard matches one label field, while the second wildcard can match zero or more consecutive label fields.

4. The method according to claim 1, characterized in that, The method for determining the initial active fragment file in step S2a specifically includes: Extract key fields from the log to be written and generate hash input values; A lightweight hash function is used to calculate the log hash value from the hash input value; the target shard index is obtained by taking the log hash value modulo the preset total number of shards N. Based on the log writing order, the shard file corresponding to the target shard index of the first log is determined as the initial active shard file; or based on the remaining space size of each shard file, the shard file with the largest remaining space is selected as the initial active shard file. Record the index, starting address, remaining space threshold, and current write pointer position of the initial active fragment file.

5. The method according to claim 1, characterized in that, The method for triggering dynamic switching of active fragments and ring coverage in step S2a specifically includes: During the writing process, the remaining space of the active shard file is read in real time, and the remaining space threshold of the current active shard is obtained through the shard mapping table; When the remaining space of the active shard file is less than or equal to the remaining space threshold, the active shard is dynamically switched. The target shard index for switching is determined by hash order: based on the current active shard index, the next shard index is calculated, and the shard corresponding to the next shard index with a remaining space greater than the remaining space threshold is selected as the new active shard. When the remaining space of all shards is less than or equal to the remaining space threshold, a circular overwrite is triggered; based on the write pointers of each shard in the shard mapping table, the earliest write shard with the smallest write pointer is located as the overwrite target shard; new logs are written to the write pointer position of the overwrite target shard, overwriting the earliest written log data, and the write pointer of the shard is updated. After a switch or overwrite operation is completed, the active shard index, remaining space of each shard, and write pointer information in the shard mapping table are updated in real time to ensure consistency of state.

6. The method according to claim 1, characterized in that, When you need to view the complete log, the following steps are also included: S3: Traverse the log directory to extract all files that conform to the segment naming rules, parse and extract the segment sequence number of each file, sort the segment sequence numbers according to the size of the segment file, and output the contents of each segment file in turn to obtain a complete log file with continuous time.

7. The method according to claim 6, characterized in that, When it is necessary to safely export log files, the following steps are also included: S4, obtain the call request to export the log file. If the call request does not contain a key parameter, encrypt and compress the log file using a preset internal key, generate and return the first download URL; If the call request contains a key parameter, then verify whether the key parameter is correct; if the key parameter is correct, compress the log file without encrypting it, generate and return a second download URL; If the key parameter is incorrect, perform the same operation as if no key parameter is included, and return the first download URL.

8. An embedded device log management system, characterized in that, It includes: a log filtering module, a routing decision module, a local shard writing and dynamic management module, a temporary buffer management module, a log reassembly module, a log export module, and a non-volatile storage module; Each module is connected via an on-chip bus or communication interface to collaboratively achieve log filtering and routing, fragmented writing, rule generation, complete log viewing, and secure export. The log filtering module receives the log stream output in real time from the embedded system, parses each log tag line by line, extracts the tag content and log metadata, and outputs the parsed log information to the routing decision module. The routing decision module makes decisions on the parsed logs based on preset routing rules: if the log tag matches the routing rule, the log is routed to the local sharding and writing local dynamic management module or cloud interface module; if the log tag does not match the routing rule, the log is written to the temporary buffer of the temporary buffer management module; if the log has no predefined prefix tag, it is directly discarded and the discard event is recorded to the non-volatile storage module. The local shard writing and dynamic management module receives logs routed to the local dynamic management module for storage, extracts key log fields to generate hash input, calculates hash values ​​using a lightweight hash function, takes the modulo of the preset total number of shards N to obtain the target shard index, determines the initial active shard file, writes the logs to that file, and records the active shard index, shard physical address, remaining space threshold, and write pointer into the shard mapping table of the non-volatile storage module. It also monitors the remaining space of the active shard in real time; when the remaining space is less than or equal to the threshold, it switches to the next shard with sufficient remaining space as the new active shard according to hash order; if all shards have insufficient remaining space, it locates the earliest written shard with the smallest write pointer, overwrites its earliest log data, and updates the write pointer; and it synchronously updates the shard mapping table in real time. The temporary buffer management module is used to store logs routed to the temporary buffer and count the number of times the same predefined prefix label appears within a preset time. If the number exceeds the threshold, a temporary routing rule is generated and updated to the routing table of the routing decision module. At the same time, the log of the label is re-output to the routing decision module for secondary routing. For logs whose number of occurrences of the label within the preset time does not exceed the threshold, they are marked as abnormal logs and stored in the abnormal log partition of the non-volatile storage module for subsequent analysis. The non-volatile storage module is used to store metadata and log files required for system operation, including: routing table, sharding mapping table, temporary buffer log, exception log partition and exported log index.

9. The system according to claim 8, characterized in that, The log reassembly module is used to traverse the log directory and extract files that conform to the segmentation naming rules when a request to view the complete log is received; parse the sequence number within each segment file and sort them according to the size of the segment file; read the contents of each segment file in sequence, and concatenate them according to the sequence number to generate a complete log file with continuous time, and output it to the user interface or debugging interface.

10. The system according to claim 9, characterized in that, The log export module is used to obtain the call parameters of the export request when a secure log export request is received, and determine whether a key parameter is included: if no key parameter is included, the module uses a preset internal key to encrypt and compress the complete log file, generate an encrypted compressed package, and return the first download URL. If a key parameter is included, verify the key's correctness: if correct, only compress the file and generate the second download URL; if incorrect, perform the same encryption and compression operation as if no key was included, and return the first download URL. After the export is complete, update the export log index of the non-volatile storage module to record the export time, file hash, and access permissions.

Images