Cyclic overlay storage method for embedded system logs

Abstract

The invention discloses a cyclic coverage storage method for logs of an embedded system, which belongs to the technical field of computer data processing, and comprises the following steps: responding to a condition that a log write-in space of a nonvolatile memory meets a trigger condition, starting a coverage process and determining old log data to be covered; extracting core semantic features of the old log data by using a dynamic feature extraction model, and generating a structured value abstract; embedding the abstract into new log data to form composite log data, and writing the composite log data into a memory to cover old logs; after a query request is received, information reconstruction is carried out based on the current log data and the embedded abstract thereof, and a query response result is generated; and generating feedback information according to the response result, and performing adaptive adjustment on the feature weight parameter of the dynamic feature extraction model by using the feedback information. According to the method, a dynamic feature extraction and value abstract embedding mechanism is adopted, and query feedback optimization in the coverage process is combined, so that cyclic coverage storage of log data can be realized in a limited storage space.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and in particular to a method for cyclic overwrite storage of embedded system logs. Background Technology

[0002] Embedded systems are computer systems that perform specific, dedicated functions. Their hardware and software resources are typically severely limited, and they are widely used in fields such as communications, automotive electronics, and industrial control. System logs are an indispensable part of embedded systems. They record system status information, critical events, and exceptions in a time-series format, providing crucial information for developers to diagnose problems, analyze performance, and perform post-incident troubleshooting. These systems typically use non-volatile memory, such as flash memory, as the persistent storage medium for logs.

[0003] In related technologies, Chinese invention patent CN112685373B discloses an embedded system log processing method, apparatus, system, and medium, comprising: obtaining a log recording request, wherein the log recording request includes the file number to be recorded in the source code file where the log record is located, the line number to be recorded in the source code file where the log record is located, and log parameters; extracting the file number to be recorded, the line number to be recorded, and the log parameters from the log recording request and storing them in SRAM memory; and generating log information corresponding to the log recording request based on the file number to be recorded, the line number to be recorded, the log parameters, and a pre-generated formatted file when a log reading request corresponding to the log recording request is obtained. This enables fast log processing in embedded systems, reduces the impact of log recording on normal data processing, and ensures the data processing efficiency of embedded systems.

[0004] However, the aforementioned existing technical solutions have the following technical shortcomings. First, their storage space management is inefficient and cannot effectively handle long-term operation scenarios: Existing solutions focus on rapid logging and formatting, but do not address the recycling mechanism of log storage space. In embedded devices with limited storage space, as the system runs for a long time, log data accumulates until the storage space is full, preventing subsequent critical logs from being written and severely impacting the continuity of fault diagnosis. Second, their log overwriting strategies are simple, leading to the loss of high-value historical information: When storage space is exhausted and old logs need to be overwritten, existing technologies typically employ simple static strategies such as first-in, first-out (FIFO), which cannot distinguish the value differences of log content. This may result in logs recording important system anomalies or critical events being overwritten prematurely, while a large amount of redundant routine information is retained, causing the permanent loss of historical information with high diagnostic value and weakening the log system's post-analysis capabilities. Third, they lack the ability to retain and reconstruct historical information: Once logs are overwritten, the information they contain is completely lost. They lack the ability to extract and retain core semantic features from overwritten logs, and cannot reconstruct historical event chains based on limited information during subsequent queries, making it difficult to support root cause analysis of complex, intermittent faults spanning different time periods. Rigid log retention strategies lack adaptive optimization: Existing log management strategies are static and cannot adjust themselves based on actual query feedback and system operating status. They cannot learn which types of log information are more valuable for subsequent diagnostics, thus failing to dynamically optimize log retention priorities and resulting in inefficient storage space utilization. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a method for cyclic overlay storage of embedded system logs. It employs a dynamic feature extraction and value summary embedding mechanism, combined with query feedback optimization during the overlay process, enabling cyclic overlay storage of log data within a limited storage space.

[0006] The above objectives can be achieved through the following approach:

[0007] A method for cyclic overwrite storage of embedded system logs includes: initiating an overwrite process and determining the old log data to be overwritten in response to a trigger condition being met in the log write space of non-volatile memory; extracting the core semantic features of the old log data using a dynamic feature extraction model to generate a structured value summary; embedding the summary into new log data to form composite log data and writing it into memory to overwrite the old logs; upon receiving a query request, reconstructing information based on the current log data and its embedded summary to generate a query response result; generating feedback information based on the response result, and adaptively adjusting the feature weight parameters of the dynamic feature extraction model using the feedback information.

[0008] Optionally, the step of using the dynamic feature extraction model to extract features from the old log data to be covered and generating a structured value summary corresponding to the core semantics of the old log data includes: parsing the old log data to be covered to obtain its log level and event type; calculating the frequency of occurrence of the event type within a system-defined time window and identifying the set of associated modules related to the old log data; obtaining the number of historical queries associated with the old log data, and combining the log level, the event type, the frequency of occurrence, the size of the set of associated modules, and the number of historical queries to form a multi-dimensional feature vector; inputting the multi-dimensional feature vector into the dynamic feature extraction model, and generating a structured value summary through the weighted calculation of the model.

[0009] Optionally, generating a structured value summary further includes: determining the summary type by combining and mapping the log level and the event type; and generating a key feature vector by encoding the occurrence frequency, the size of the associated module set, and the number of historical queries by the dynamic feature extraction model.

[0010] Optionally, embedding the structured value summary into the new log data to be written to form composite log data carrying historical value information includes: allocating an extended data area for storing metadata to the new log data to be written; parsing the inherited value summary already carried by the old log data to be overwritten from the extended data area; merging the newly generated structured value summary with the inherited value summary to form a current value summary chain containing historical information; encapsulating the current value summary chain into a metadata data package and writing it into the extended data area to form composite log data carrying historical value information.

[0011] Optionally, the information reconstruction based on the log data currently stored in the non-volatile memory and its embedded structured value summary includes: batch extracting the current value summary chain within each extended data area from multiple composite log data related to the time range of the query request; performing reverse backtracking and correlation analysis on the extracted multiple current value summary chains to reconstruct a high-value event correlation graph representing the correlation of historical events; and generating a final response result for responding to the query request based on the high-value event correlation graph and the obtained complete log content currently stored.

[0012] Optionally, the step of generating feedback information for model optimization based on the query response result and adaptively adjusting the internal feature weight parameters of the dynamic feature extraction model using the feedback information includes: evaluating the effectiveness of the query response result and obtaining query success rate and positioning efficiency indicators to quantify the query effect; comparing the query success rate and positioning efficiency indicators with preset performance expectation thresholds to generate a model optimization signal; and adjusting the internal feature weight parameters corresponding to the relevant features that lead to query success or failure in the dynamic feature extraction model according to the polarity and intensity of the model optimization signal.

[0013] Optionally, determining the old log data to be overwritten from the non-volatile memory includes: scanning the log storage area of ​​the non-volatile memory to identify all complete log entries; obtaining and applying a primary generation filtering rule containing storage duration conditions to filter out a new generation log set from the log entries; obtaining the historical value score corresponding to each log entry in the new generation log set, and selecting log entries with lower historical value scores as the old log data to be overwritten.

[0014] Optionally, before selecting log entries with lower historical value scores as the old log data to be overwritten, the method further includes: obtaining a physical block erase count distribution map that records the erase counts of each physical block of the non-volatile memory; mapping log entries in the young generation log set to their stored physical blocks, and based on the physical block erase count distribution map, filtering out log entries stored on physical blocks with lower current erase counts to form a candidate overwrite set; and within the scope of the candidate overwrite set, performing the operation of obtaining the historical value score and selecting log entries with lower historical value scores.

[0015] Optionally, the log writing space meeting the trigger condition includes: continuously monitoring the remaining contiguous available space of the non-volatile memory; obtaining the estimated size of the new log data to be written; and determining that the trigger condition is met when the remaining contiguous available space is less than the product of the estimated size and the system-defined safety factor.

[0016] Based on the same inventive concept, this invention also provides a cyclic overwrite storage system for embedded system logs. The system includes: an overwrite triggering and old log determination module, used to initiate an overwrite process and determine the old log data to be overwritten from the non-volatile memory in response to the log write space of the non-volatile memory meeting a trigger condition; a value summary generation module, used to acquire a dynamic feature extraction model for evaluating the value of log data, and use the dynamic feature extraction model to extract features from the old log data to be overwritten, generating a structured value summary corresponding to the core semantics of the old log data; and a composite log writing and overwrite module, used to write the structured value summary to the old log data. The structured value summary is embedded into the new log data to be written, forming composite log data carrying historical value information, and the composite log data is written into the non-volatile memory to overwrite the old log data; the query processing and information reconstruction module is used to receive query requests for historical logs, and to reconstruct information based on the log data currently stored in the non-volatile memory and its embedded structured value summary, generating query response results; the model feedback and adaptive optimization module is used to generate feedback information for model optimization based on the query response results, and to adaptively adjust the internal feature weight parameters of the dynamic feature extraction model using the feedback information.

[0017] Compared with the prior art, the present invention has the following advantages:

[0018] This invention constructs a chain of intergenerational inheritance of historical information by extracting a structured value summary from old logs to be overwritten and embedding it into new logs. Even if the original logs are physically overwritten, their core value information continues to circulate and be preserved within the storage space. This enables the system to reconstruct a sequence of key historical events spanning multiple storage cycles based on limited current log data, enhancing the ability to trace the root causes of chronic and complex failures and overcoming the limitation of complete loss of historical information in traditional circular storage.

[0019] This invention establishes a closed-loop adaptive optimization mechanism based on query feedback. By evaluating the query effect of reconstructing historical information, the system can quantify the effectiveness of the value summary and generate model optimization signals accordingly, adaptively adjusting the internal parameters of the dynamic feature extraction model. This enables the evaluation criteria for log value to continuously learn and evolve, automatically adapting to changes in system operating status and the focus of maintenance personnel, achieving intelligent and personalized log retention strategies, and improving the utilization efficiency of storage resources.

[0020] This invention considers both information value and the health of the physical storage medium when selecting logs to be overwritten. By prioritizing logs with lower historical value scores and fewer write cycles to their corresponding physical blocks, this method not only ensures the retention of high-value information but also achieves wear leveling for non-volatile memory. This strategy combines software-level data management with hardware-level lifetime management, improving the performance of the logging system while effectively extending the lifespan of the embedded device's storage hardware.

[0021] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a method for cyclically overwriting and storing embedded system logs according to an embodiment of the present invention.

[0024] Figure 2 This is a reconstructed heatmap of the event association map according to an embodiment of the present invention.

[0025] Figure 3 This is the feature weight adaptive adjustment graph of an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the structure of an embedded system log cyclic overwrite storage system according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Reference Figure 1One embodiment of the present invention proposes a cyclic overlay storage method for embedded system logs. It adopts a dynamic feature extraction and value summary embedding mechanism, combined with query feedback optimization during the overlay process, which can realize cyclic overlay storage of log data within a limited storage space.

[0029] The method described in this embodiment specifically includes:

[0030] S1. In response to the log write space of the non-volatile memory meeting the trigger condition, the overwrite process is started, and the old log data to be overwritten is determined from the non-volatile memory.

[0031] Optionally, determining the old log data to be overwritten from the non-volatile memory includes:

[0032] Scan the log storage area of ​​the non-volatile memory to identify all complete log entries;

[0033] Obtain and apply the initial generation filtering rules containing storage duration conditions to filter out the new generation log set from the log entries;

[0034] Obtain the historical value score corresponding to each log entry in the new generation log set, and select the log entries with lower historical value scores as the old log data to be overwritten.

[0035] Specifically, the process involves a comprehensive scan of the log storage area of ​​non-volatile memory (NVMemory) and identification of all log entries that remain physically intact. NVOMemory refers to a type of computer storage medium that retains its stored information even after power is off, such as flash memory commonly used in embedded systems. During the identification process, a set of initial filtering rules, including storage duration criteria, is acquired and applied to filter out the so-called young generation log set from a massive number of log entries. This young generation log set refers to the group of log entries that have been stored for some time, are eligible to be overwritten, but whose intrinsic value has not yet been evaluated.

[0036] For each log entry within this set, its corresponding historical value score is further obtained, serving as the quantitative basis for the final decision on whether to cover it. To obtain this score, the log data is parsed to extract its log level and event type. The log level defines the importance of the log, such as urgent, error, or general alarm. Subsequently, the frequency of occurrence of this event type within a preset time window is calculated, and the associated set of modules is identified. Furthermore, the historical query count of this log entry by technicians or automated diagnostic tools is extracted from the historical records.

[0037] The dynamic feature extraction model performs dimensional alignment and normalization on the log level values, event frequency, size of the associated module set, and number of historical queries, and then calculates a comprehensive historical value score through weighted calculation. During this process, log entries with lower scores are prioritized and marked as older log data to be covered.

[0038] For example, suppose an embedded device's flash memory has reached a trigger threshold, and the system needs to free up space to write new sensor data. The system identifies two candidate logs, Log A and Log B, through scanning. Log A records a routine system heartbeat, its log level is defined as the lowest, it occurred frequently in the past hour, the associated module involves only a single timer, and it has never been queried during storage. After evaluation, Log A receives a very low historical value score due to its high information redundancy and lack of diagnostic reference value. In contrast, Log B records an occasional communication handshake failure. Although it occurred earlier, its log level is error-level, and it is associated with multiple core modules such as the protocol stack and RF driver. It has been retrieved three times by maintenance tools in the past. Because Log B contains crucial fault tracing clues, its historical value score is higher than Log A. In this case, the system will precisely choose to overwrite Log A, thus reclaiming storage space while ensuring that high-potentially-valuable information like Log B is not lost due to a simple first-in, first-out rule.

[0039] Optionally, the process of selecting log entries with lower historical value scores as the old log data to be overwritten further includes:

[0040] Obtain a physical block erase / write count distribution map that records the erase / write counts of each physical block of the non-volatile memory;

[0041] Log entries in the young generation log set are mapped to their stored physical blocks, and based on the physical block erase / write count distribution map, log entries stored on physical blocks with a low current erase / write count are selected to form a candidate overwrite set.

[0042] Within the candidate coverage set, the operation of obtaining the historical value score and selecting log entries with lower historical value scores is performed.

[0043] Specifically, the physical wear and tear of non-volatile memory (NVMemory) is globally assessed, and a physical block erase / write cycle distribution map, recording the wear and tear status of each physical partition of the NNVMemory, is acquired in real time. NNVMemory refers to hardware media that can persistently retain data after power loss, and a physical block is the smallest physical unit for data erasure and write operations on this media. This distribution map, in the form of a digital archive, meticulously records the cumulative erase / write cycles experienced by each physical block since its inception. Subsequently, log entries that meet the initial screening criteria for storage duration and thus the coverage conditions are associated and mapped with their specific physical locations. By comparing the distribution map, logs located on physical blocks with high cumulative erase / write cycles and poor health are preferentially excluded, retaining only log entries stored on physical blocks with low erase / write cycles and healthy status. This constructs a refined candidate coverage set at the hardware level.

[0044] Within the candidate coverage set, a dynamic feature extraction model is activated to perform the core operation of obtaining a historical value score. This score is a quantitative indicator for evaluating whether logs have long-term retention value. The model deeply analyzes various multi-dimensional features of candidate logs, including log level score, normalized frequency of occurrence, size of the associated module set, and normalized number of historical queries. The log level score reflects the severity of the event, the frequency of occurrence reflects the redundancy of information, the size of the associated module represents the breadth of the affected systems, and the number of historical queries reflects the reference value of the data in actual operation and maintenance. These normalized, dimensionless feature values ​​are used as input and weighted according to the currently preset business weight parameters. This process transforms the originally messy raw text information into a directly comparable numerical value, namely the historical value score, thus providing an objective and scientific basis for subsequent selection.

[0045] After obtaining the historical value scores of each log entry in the set, the process selects log entries with lower historical value scores. By sorting the scores from low to high or comparing them with preset value thresholds, log entries with scores at the bottom of the range are precisely identified. These low-scoring logs typically represent highly repetitive routine heartbeats, daily fluctuations in non-core modules, or low-value data that has never been monitored by operations personnel. These entries are formally identified as old log data to be overwritten, and the physical space they occupy is released to store newly generated composite log data. In this way, while ensuring a balance in hardware wear and tear, storage resources are optimized in terms of information content, laying the foundation for the subsequent retention of high-value information.

[0046] For example, suppose the memory of an embedded industrial gateway is about to be full. The system initially screens out logs A and B. The system queries the distribution map and finds that the block containing log A has been erased 100 times, classifying it as a healthy block, while the block containing log B has been erased 5000 times, approaching the physical limit. To protect hardware lifespan, the system directly excludes log B and adds log A to the candidate coverage set. The model then scores log A. Analysis reveals that log A records a normal temperature sensor reading once per second, resulting in an extremely low log grade score. Its extremely high frequency of occurrence leads to a near-perfect normalized score, and it is associated with only a single sensor module, with zero historical queries. After weighted calculation, log A, due to its extremely high redundancy and extremely low importance, ultimately receives a low historical value score. Because its score is far lower than other abnormal logs in the set involving communication link switching, the system ultimately selects log A as the target to be covered and allocates its physical space to the newly generated composite log in the subsequent write process.

[0047] Optionally, the log write space meeting the triggering conditions includes:

[0048] Continuously monitor the remaining contiguous available space of the non-volatile memory;

[0049] Get the estimated size of the new log data to be written;

[0050] When the remaining continuous available space is less than the product of the estimated size and the system-defined safety factor, it is determined that the trigger condition is met.

[0051] Specifically, during the operation of an embedded system, the space status of non-volatile memory is continuously monitored through the underlying driver interface. Non-volatile memory refers to storage media that can retain data even after power loss, and it is usually divided into multiple physical sectors for management. The unoccupied portions of these sectors are counted in real time to determine the remaining contiguous available space.

[0052] At the same time, the estimated size of the new log data to be written is obtained. This value is a predicted value calculated by analyzing the message length of the current logs to be stored and taking into account the concurrent log increments that may occur in the short term.

[0053] To ensure absolute security of storage operations and prevent space overflow during writes, a trigger condition determination mechanism is introduced. This mechanism determines whether to initiate the overwrite process by comparing the current available space with the estimated demand, and its logic follows the following determination formula:

[0054] ,

[0055] In this formula, Represents the remaining contiguous available space in non-volatile memory. This represents the estimated size of the new log data to be written. This represents the system-defined safety factor. The method for obtaining this information is to call the free list of the storage manager and read the total amount of consecutive physical addresses in real time. The method to obtain it is to calculate the sum of all bytes in the current log buffer. This is a dimensionless constant preset in the system configuration file, used to compensate for the additional overhead incurred by the file system due to index updates and metadata storage. Since both sides of the formula are physical quantities representing storage capacity, with units of bytes and consistent dimensions, the calculation process conforms to objective physical laws. When the formula is true, the triggering condition is met, and the subsequent old log cleanup and compound log writing process is started.

[0056] For example, suppose that the non-volatile memory of an industrial-grade embedded controller currently has contiguous available space remaining. The value is 1024 bytes. At this point, due to the system detecting abnormal motor speeds, a series of new logs were generated that needed to be saved immediately. The system calculated the estimated size of this new log data. The system's default security level is 600 bytes. The coefficient is 2.0, which takes into account the overhead of the structured value digest that must be generated synchronously due to writing new logs, as well as the redundant space in the file system log header. During the logical judgment, the product of the estimated size and the safety factor is calculated, i.e., 600 bytes multiplied by 2.0, resulting in 1200 bytes. Since the current remaining contiguous available space of 1024 bytes is less than the calculated safety requirement of 1200 bytes, the judgment formula is valid. The system then determines that the trigger condition has been met and immediately initiates the overwrite process, freeing up space by deleting the old log with the lowest score, thereby ensuring that this set of critical exception logs can be safely and completely written to memory.

[0057] S2. Obtain a dynamic feature extraction model for evaluating the value of log data, and use the dynamic feature extraction model to extract features from the old log data to be covered, generating a structured value summary corresponding to the core semantics of the old log data.

[0058] Optionally, the step of using the dynamic feature extraction model to extract features from the old log data to be covered and generating a structured value summary corresponding to the core semantics of the old log data includes:

[0059] Parse the old log data to be overwritten to obtain its log level and event type;

[0060] Calculate the frequency of occurrence of the event type within the system-defined time window, and identify the set of associated modules related to the old log data;

[0061] Obtain the historical query count associated with the old log data, and combine the log level, the event type, the occurrence frequency, the size of the associated module set, and the historical query count to form a multi-dimensional feature vector;

[0062] The multidimensional feature vector is input into the dynamic feature extraction model, and a structured value summary is generated through the weighted calculation of the model.

[0063] Specifically, the old log data is parsed by a parser to extract the log level and event type. The log level represents the importance of the log, such as urgent, error, or alarm; the event type corresponds to the specific business logic or functional module of the system.

[0064] The frequency of the event type to which the log belongs within a preset time window is calculated. For example, for events like "database connection timeout," the number of times it occurred in the past hour is counted. High-frequency events usually represent routine states or periodic operations, and their unique value as individual log entries is lower; while occasional, low-frequency events often point to potential anomalies or special problems, and therefore have higher retention value. Simultaneously, the set of modules associated with the event is identified. For example, a "memory allocation failure" log may be associated with multiple modules such as "task scheduler," "network protocol stack," and "cache management." The more modules associated and the higher the core status of these modules in the system, the wider the impact of the event and the more prominent its importance. By combining the two key dimensions of "event frequency" and "associated module set," a dynamic and quantitative "information value score" can be generated for each log entry. This rating will directly determine which logs should be prioritized for retention and which can be safely overwritten when log storage space is tight. This will maximize the preservation of historical information most valuable for fault diagnosis, system analysis, and security auditing within limited storage space, achieving a fundamental shift from "simple cyclic overwriting" to "intelligent retention based on information value".

[0065] The log level, event type, frequency of occurrence, size of the associated module set, and number of historical queries mentioned above are dimensionally aligned and normalized to form a multidimensional feature vector. To quantify the process of generating a structured value summary, a value weight calculation model is introduced. The calculation logic of this model is as follows:

[0066] ,

[0067] In the above formula, Numerical representation of the final generated structured value summary; The quantified log level score is obtained by applying a preset weight mapping to the log levels; The normalized frequency of occurrence; The size of the associated module set is obtained by counting the number of unique modules in the set and then scaling it linearly. This represents the normalized number of historical queries. These are the weight parameters assigned to each dimension by the dynamic feature extraction model, and all weights are dimensionless coefficients to ensure that the calculation results conform to objective logical laws.

[0068] The multidimensional feature vector is input into a dynamic feature extraction model, which generates a structured value summary reflecting the core semantics through weighted calculation logic. This step performs a quantitative evaluation of the log's value. The multidimensional feature vector, containing log level, event frequency, number of associated modules, and number of historical queries, is used as input. The dynamic feature extraction model carries a set of adjustable weight parameters. The model performs weighted calculations on each dimension of the vector to obtain a quantified value score. This score is combined with semantic information such as the log's event type, and finally encapsulated into a uniformly formatted structured value summary. This summary concisely represents the core value of the original log.

[0069] For example, suppose the embedded system has triggered a log overwrite process, and the system has identified an old log entry to be overwritten, with a log level of "Error" and a corresponding quantization value. The value was 0.9. Analysis revealed that the log event type was "network timeout," and this event occurred 5 times within the past 10-minute time window. The frequency of occurrence after normalization was... The value is 0.5. Simultaneously, this event is associated with both the "communication module" and the "protocol stack module," with the associated module set size being... The value is scaled to 0.2. A search of historical records revealed that this log entry was retrieved 3 times by technical personnel; the normalized query count is... The value is 0.6. At this point, the preset weight parameters in the dynamic feature extraction model are respectively... Substituting these parameters into the calculation formula, the value summary value is obtained as follows: The system then encapsulates this value along with a specific event type encoding into a structured value summary, which is embedded into the new log data to be written. In this way, even if the original old log is physically overwritten, its core "high-value" characteristics are still preserved through the extended area of ​​the new log, verifying the method's ability to balance information retention and space reclamation.

[0070] Optionally, the generation of the structured value summary further includes:

[0071] The summary type is determined by combining and mapping the log level and the event type.

[0072] The dynamic feature extraction model encodes the frequency of occurrence, the size of the associated module set, and the number of historical queries to generate a key feature vector.

[0073] Specifically, the summary type is determined and key feature vectors are generated. The summary type is determined by combining and mapping the parsed log level and event type. The mapping relationship is pre-stored in the system's configuration table. Through logical table lookup operations, discrete level information is combined with specific event identifiers to output a unique summary type label representing the semantic category of the log.

[0074] A dynamic feature extraction model encodes the frequency of occurrence, the size of the associated module set, and the number of historical queries to generate key feature vectors. To ensure that the encoded features have uniform comparability and conform to logical patterns, an encoding mapping function is introduced:

[0075] ,

[0076] In the formula, The magnitude or overall strength index of the generated key feature vectors; The normalized frequency of occurrence is obtained by the ratio of the count value per unit time to the maximum preset frequency of the system. The normalized size of the associated module set is obtained by the ratio of the actual number of associated modules to the total number of modules in the system. The normalized historical query count is obtained by the ratio of the query count of a single log entry to the maximum historical query count. These are dimensionless weighting coefficients, whose values ​​are adaptively adjusted by the dynamic feature extraction model based on the current monitoring focus of the system, and satisfy the following conditions: .because All values ​​are normalized ratios, and all coefficients are dimensionless. Therefore, the dimensions of all terms in this formula are consistent, which conforms to the laws of physics and logic.

[0077] For example, suppose the system is processing an old log entry with a log level of 2 and an event type of storage overflow. Through a combination mapping function, the system matches an exception-type digest. Then, in the feature encoding stage, the frequency of occurrence of this event is determined. The calculated value is 0.8, representing the size of the associated module set. The calculated value is 0.4, representing the number of historical queries. The calculated value is 0.6. If the dynamic feature extraction model then dynamically adjusts the weight parameters based on recent feedback from frequent storage problems in the system... Substituting the values ​​into the formula yields... Finally, the system outputs a structured value summary containing the summary type label "anomaly class" and a key feature vector with a magnitude of 0.66.

[0078] S3. Embed the structured value summary into the new log data to be written to form composite log data carrying historical value information, and write the composite log data into the non-volatile memory to overwrite the old log data;

[0079] Optionally, embedding the structured value summary into the new log data to be written, forming composite log data carrying historical value information, includes:

[0080] Allocate an extended data area for storing metadata for the new log data to be written;

[0081] Parse the inherited value summary already carried by the old log data to be overwritten from the extended data area;

[0082] The newly generated structured value summary is merged with the inherited value summary to form a current value summary chain containing historical information;

[0083] The current value summary chain is encapsulated into a meta data packet and written into the extended data area to form composite log data carrying historical value information.

[0084] Specifically, when creating composite log data that carries historically valuable information, a space allocation operation is first performed to allocate an extended data area specifically for storing metadata for the new log data to be written. This process begins with the system allocating a fixed-length extended data area for each new log data entry to be written. In engineering, this extended data area is typically a reserved byte array within the log data structure, such as a 32-byte or 64-byte metadata space, the location and size of which are fixed in the system log format definition.

[0085] The process involves accessing the identified old log data to be overwritten in non-volatile memory and extracting its inherited value digest from the extended data area using a parsing algorithm. After confirming the old log data to be overwritten, the extended data area of ​​the old log data is read from the same offset at its data storage location and parsed according to a predefined format to extract the inherited value digest. This inherited value digest is essentially a chained collection of value information from previous generations of overwritten logs. If the old log data to be overwritten is being written for the first time and has not overwritten any historical logs, its extended data area may be empty or contain a specific initial marker; in this case, the parsed inherited value digest will be empty.

[0086] To achieve intergenerational transfer of valuable information, the system logically merges the latest structured value summary generated by the dynamic feature extraction model with the extracted inherited value summary, forming a current value summary chain containing historical information from multiple generations. During the merging process, a summary chain length control function is introduced to ensure reasonable storage density.

[0087] ,

[0088] In the formula, Represents the total byte size of the current value summary chain generated after the merger; The number of basic bytes used for the newly generated structured value summary is determined by the summary type; The number of historical summary entries contained in the inherited value summary; For each historical summary, the number of bytes with a fixed step size after data compression is [number], and [number] bytes. and All units are bytes, ensuring that the weighted calculation of the formula conforms to logical rules.

[0089] The current value summary chain is encapsulated into a standardized metadata packet and written entirely into the extended data area of ​​the new log data, thereby generating composite log data and completing the overwriting storage of the old log data. The "current value summary chain," which merges the old log summary with the latest summary, is standardized and encapsulated into a uniformly formatted metadata packet; this metadata packet is then written entirely into the extended data area reserved for the new log; this composite log data, containing both the original content of the new log and the historical value summary, is written to storage, directly overwriting the old log data. The entire process, while overwriting the old data, extracts its core value and transfers it to the new data, achieving the cyclical use of storage space and the continuous retention of key information.

[0090] For example, suppose the system is preparing to write a new log entry about sensor sampling and plans to overwrite an old memory management log entry. The system first demarcates a 128-byte extension data area at the end of the new log entry. Next, the system parses a inherited value digest from the extended data area of ​​the old memory management log. This digest records an earlier file system exception and is known to contain two historical entries. The value is 2. At this point, the system generates the latest structured value summary based on the size of the current old logs. It is 16 bytes, while the system-defined compression history step size is... It is 8 bytes. The total size of the current value digest chain is calculated according to the length control formula. Bytes. The system encapsulates this 32-byte digest chain into a meta-data packet and fills it into the extended data area of ​​the new log.

[0091] S4. Receive a query request for historical logs, and reconstruct information based on the log data currently stored in the non-volatile memory and its embedded structured value summary to generate a query response result;

[0092] Optionally, the information reconstruction based on the log data currently stored in the non-volatile memory and its embedded structured value summary includes:

[0093] From multiple composite log data related to the time range of the query request, the current value summary chain within each extended data area is extracted in batches;

[0094] By performing reverse backtracking and correlation analysis on the extracted multiple current value summary chains, a high-value event correlation map representing the correlation of historical events is reconstructed;

[0095] Based on the high-value event association graph and the obtained complete log content currently stored, a final response result is generated to respond to the query request.

[0096] Specifically, when reconstructing information based on currently stored log data and its embedded structured value summaries, the system first locates and extracts multiple related composite log data from non-volatile memory in batches, according to the time range covered by the received query request. The system then uses a parsing algorithm to enter the extended data area of ​​each composite log data and completely extract the current value summary chain stored within it.

[0097] The reconstruction engine performs reverse backtracking and correlation analysis on multiple extracted current value summary chains. Reverse backtracking refers to recovering historical feature information from the nearest to the furthest along the inheritance relationships in the summary chain, while correlation analysis seeks causal logic by identifying similar event type labels or key feature vectors in different summary chains. To quantitatively evaluate the confidence level of the reconstructed high-value event correlation graph, a correlation strength calculation model is introduced:

[0098] ,

[0099] In the formula, A measure of the overall association strength of a high-value event association graph; The feature similarity between each pair of related events is obtained by comparing the difference in the magnitude of key feature vectors in the summary chain and then normalizing it. The dimensionless weights, generated based on the time decay coefficient, are obtained by calculating the span of historical events from the current time point and performing an exponential decay mapping, ensuring that the weight of recent associations is higher than that of distant associations. Because... and All are dimensionless numerical values, and the index obtained by summing their products is... Following logical patterns, the reconstructed heatmap of the event correlation graph is as follows: Figure 2 As shown.

[0100] Information is integrated to generate the final response. It combines the historical event context revealed by the newly constructed high-value event correlation graph with the still-existing, uncovered complete log content stored in non-volatile memory. The graph provides context, and detailed log entries related to graph nodes are extracted from the complete log as supporting evidence. For example, if the graph shows "network outage" causing "service unavailability," both summary events, along with specific error messages related to "service unavailability" in the current log, will be presented to the user, generating a final response to the query request that is supported by both macro-level causal analysis and micro-level details.

[0101] For example, suppose a technician initiates a query request regarding the cause of a system crash, with the time span limited to the past 24 hours. The system first retrieves 10 composite log entries from non-volatile memory that fall within this time period, and then obtains 10 current value digest chains from its extended data area. The most recent digest chain indicates that the current event is a system restart, and its inherited digest chain traces back to a kernel memory thrashing that occurred 2 hours prior. The reconstruction engine identifies a high degree of similarity between the key feature vector of the memory thrashing event and a driver load failure event recorded in an earlier digest chain. The value is 0.9. At this point, the system calculates the decay weight based on the time span for these two related events. The values ​​are 0.8 and 0.6 respectively. Substituting these values ​​into the correlation strength formula yields... Based on this strong correlation, the system successfully reconstructed a high-value event correlation graph showing that driver anomalies led to memory thrashing and subsequently a restart, thus providing operations and maintenance personnel with a complete fault evolution chain even though the original logs had been physically overwritten.

[0102] S5. Generate feedback information for model optimization based on the query response result, and use the feedback information to adaptively adjust the internal feature weight parameters of the dynamic feature extraction model.

[0103] Optionally, the step of generating feedback information for model optimization based on the query response result, and adaptively adjusting the internal feature weight parameters of the dynamic feature extraction model using the feedback information, includes:

[0104] Evaluate the effectiveness of the query response results and obtain query success rate and location efficiency indicators to quantify the query effect;

[0105] The query success rate metric and the location efficiency metric are compared with a preset performance expectation threshold to generate a model optimization signal;

[0106] Based on the polarity and intensity of the optimized signal, the internal feature weight parameters corresponding to the relevant features that lead to query success or failure in the dynamic feature extraction model are adjusted.

[0107] Specifically, by evaluating the effectiveness of query response results, query success rate and location efficiency metrics are obtained to quantify query effectiveness. The query success rate refers to the proportion of queries that successfully reconstruct the complete event logic using structured value summaries; the location efficiency metric refers to the average time cost required to reconstruct a high-value event association graph.

[0108] These two metrics are compared with the system's preset performance expectation thresholds. If the actual metrics are lower than the thresholds, a signal indicating the need for positive model optimization is generated; otherwise, a signal indicating maintenance or negative weakening is generated. To precisely adjust the model, a weight update calculation model is introduced as follows:

[0109] ,

[0110] In the above formula, This represents the adjustment variable for the internal feature weight parameters in the dynamic feature extraction model; The preset learning rate coefficient is a dimensionless fine-tuning constant. The normalized comprehensive performance score is obtained by weighted mapping of the query success rate index and the location efficiency index, and the value ranges from zero to one, representing the response performance of the current system. The sensitivity factor for relevant features represents the degree to which a specific feature contributes to the success or failure of a query. It is obtained through statistical analysis of historical query records. Since all variables in the formula are dimensionless ratios or coefficients, the calculation process conforms to mathematical logic.

[0111] Based on the polarity and intensity of the optimized signal, i.e., based on the calculated adjustment variables, the corresponding internal feature weight parameters in the dynamic feature extraction model are adaptively adjusted. The adaptive adjustment of feature weights is as follows: Figure 3As shown, the system continuously monitors the utilization efficiency of historical logs during operation, especially query results used for fault diagnosis or behavioral analysis based on the logs. By quantitatively evaluating the success, efficiency, and accuracy of the query results, a model optimization signal is generated. This signal not only indicates the direction of optimization but also quantifies the optimization intensity. Specifically, the adjustment variable corresponding to each feature is calculated based on the polarity and intensity of the signal. Finally, based on this adjustment variable, the corresponding internal feature weight parameters in the dynamic feature extraction model are adaptively adjusted. For example, if multiple successful queries rely on the feature of "number of associated modules," the weight ratio of this feature in the value assessment model will be automatically and appropriately increased; conversely, if a feature is not effectively utilized for a long time, its weight will gradually decrease. Through this continuous closed loop of "usage feedback → signal generation → parameter adjustment," the log retention strategy can continuously self-optimize in actual operation, gradually approaching the optimal feature combination, thereby maximizing the retention of log information value within limited storage space.

[0112] For example, suppose the system successfully reconstructed the event graph through the summary chain 70 times out of the last 100 historical log queries, resulting in a query success rate of 0.7. Simultaneously, the system monitors that the average location efficiency, after normalization, is 0.6. The system calculates the current overall performance score according to preset rules. The value is 0.65. If the system's performance expectation threshold is set to 0.8, the system determines that optimization is needed because the overall performance score is below the threshold. Set the learning rate coefficient. The value was 0.1, and after tracing the source, it was found that the query failures were mostly due to the low weight of the feature "size of the associated module set" in the dynamic feature extraction model. Therefore, the corresponding relevant feature sensitivity factor was determined. The value is 1.2. Substituting this into the formula, we obtain the adjustment variable. The system then increases the internal feature weight parameters for the related module dimension in the dynamic feature extraction model by 0.042.

[0113] Based on the same inventive concept, such as Figure 4 As shown, the present invention also provides a circular overwrite storage system for embedded system logs, the system comprising:

[0114] The overwrite triggering and old log determination module is used to initiate the overwrite process in response to the log write space of the non-volatile memory meeting the triggering condition, and to determine the old log data to be overwritten from the non-volatile memory.

[0115] The value summary generation module is used to obtain a dynamic feature extraction model for evaluating the value of log data, and to use the dynamic feature extraction model to extract features from the old log data to be covered, generating a structured value summary corresponding to the core semantics of the old log data.

[0116] The composite log writing and overwriting module is used to embed the structured value summary into the new log data to be written, forming composite log data carrying historical value information, and write the composite log data into the non-volatile memory to overwrite the old log data;

[0117] The query processing and information reconstruction module is used to receive query requests for historical logs, and to reconstruct information based on the log data currently stored in the non-volatile memory and its embedded structured value summary, and generate query response results.

[0118] The model feedback and adaptive optimization module is used to generate feedback information for model optimization based on the query response results, and to adaptively adjust the internal feature weight parameters of the dynamic feature extraction model using the feedback information.

[0119] It should be noted that the electrical connections between the various units described above do not necessarily represent direct or indirect connections. Any indirect connection method can be applied to the embodiments of the present invention as long as it achieves the purpose of the present invention. The above descriptions are merely exemplary embodiments of the present invention and should not be construed as limiting the scope of the present invention.

[0120] All equivalent changes and modifications made in accordance with the teachings of this invention are still within the scope of this invention. Those skilled in the art will readily conceive of other embodiments of this invention upon considering the specification and the disclosure of practical truth. This application is intended to cover any variations, uses, or adaptations of this invention that follow the general principles of this invention and include common knowledge or conventional techniques in the art not described herein.

Claims

1. A method for cyclically overwriting and storing embedded system logs, characterized in that, The method includes: In response to the log write space of the non-volatile memory meeting the trigger condition, the overwrite process is initiated, and the old log data to be overwritten is determined from the non-volatile memory; A dynamic feature extraction model for evaluating the value of log data is obtained, and the dynamic feature extraction model is used to extract features from the old log data to be covered, generating a structured value summary corresponding to the core semantics of the old log data. The structured value summary is embedded into the new log data to be written, forming composite log data carrying historical value information, and the composite log data is written into the non-volatile memory to overwrite the old log data; Receive query requests for historical logs, and reconstruct information based on the log data currently stored in the non-volatile memory and its embedded structured value summary to generate query response results; Feedback information for model optimization is generated based on the query response results, and the internal feature weight parameters of the dynamic feature extraction model are adaptively adjusted using the feedback information.

2. The method for cyclically overwriting and storing embedded system logs according to claim 1, characterized in that, The step of using the dynamic feature extraction model to extract features from the old log data to be covered, and generating a structured value summary corresponding to the core semantics of the old log data, includes: Parse the old log data to be overwritten to obtain its log level and event type; Calculate the frequency of occurrence of the event type within the system-defined time window, and identify the set of associated modules related to the old log data; Obtain the historical query count associated with the old log data, and combine the log level, the event type, the occurrence frequency, the size of the associated module set, and the historical query count to form a multi-dimensional feature vector; The multidimensional feature vector is input into the dynamic feature extraction model, and a structured value summary is generated through the weighted calculation of the model.

3. The method for cyclically overwriting and storing embedded system logs according to claim 2, characterized in that, The generation of structured value summaries also includes: The summary type is determined by combining and mapping the log level and the event type. The dynamic feature extraction model encodes the frequency of occurrence, the size of the associated module set, and the number of historical queries to generate a key feature vector.

4. The method for cyclically overwriting and storing embedded system logs according to claim 1, characterized in that, The step of embedding the structured value summary into the new log data to be written, forming composite log data carrying historical value information, includes: Allocate an extended data area for storing metadata for the new log data to be written; Parse the inherited value summary already carried by the old log data to be overwritten from the extended data area; The newly generated structured value summary is merged with the inherited value summary to form a current value summary chain containing historical information; The current value summary chain is encapsulated into a meta-data packet and written into the extended data area to form composite log data carrying historical value information.

5. The method for cyclically overwriting and storing embedded system logs according to claim 4, characterized in that, The information reconstruction based on the log data currently stored in the non-volatile memory and its embedded structured value summary includes: From multiple composite log data related to the time range of the query request, the current value summary chain within each extended data area is extracted in batches; By performing reverse backtracking and correlation analysis on the extracted multiple current value summary chains, a high-value event correlation map representing the correlation of historical events is reconstructed; Based on the high-value event association graph and the obtained complete log content currently stored, a final response result is generated to respond to the query request.

6. The method for cyclically overwriting and storing embedded system logs according to claim 1, characterized in that, The step of generating feedback information for model optimization based on the query response result, and adaptively adjusting the internal feature weight parameters of the dynamic feature extraction model using the feedback information, includes: Evaluate the effectiveness of the query response results and obtain query success rate and location efficiency indicators to quantify the query effect; The query success rate metric and the location efficiency metric are compared with a preset performance expectation threshold to generate a model optimization signal; Based on the polarity and intensity of the optimized signal, the internal feature weight parameters corresponding to the relevant features that lead to query success or failure in the dynamic feature extraction model are adjusted.

7. The method for cyclically overwriting and storing embedded system logs according to claim 1, characterized in that, The process of determining the old log data to be overwritten from the non-volatile memory includes: Scan the log storage area of ​​the non-volatile memory to identify all complete log entries; Obtain and apply the initial generation filtering rules containing storage duration conditions to filter out the new generation log set from the log entries; Obtain the historical value score corresponding to each log entry in the new generation log set, and select the log entries with lower historical value scores as the old log data to be overwritten.

8. The method for cyclically overwriting and storing embedded system logs according to claim 7, characterized in that, The process of selecting log entries with lower historical value scores as the old log data to be overwritten also includes: Obtain a physical block erase / write count distribution map that records the erase / write counts of each physical block of the non-volatile memory; Log entries in the young generation log set are mapped to their stored physical blocks, and based on the physical block erase / write count distribution map, log entries stored on physical blocks with a low current erase / write count are selected to form a candidate overwrite set. Within the candidate coverage set, the operation of obtaining the historical value score and selecting log entries with lower historical value scores is performed.

9. The method for cyclically overwriting and storing embedded system logs according to claim 1, characterized in that, The log write space meets the triggering conditions including: Continuously monitor the remaining contiguous available space of the non-volatile memory; Get the estimated size of the new log data to be written; When the remaining continuous available space is less than the product of the estimated size and the system-defined safety factor, it is determined that the trigger condition is met.

10. A circular overwrite storage system for embedded system logs, applied to the circular overwrite storage method for embedded system logs as described in any one of claims 1-9, characterized in that, The system includes: The overwrite triggering and old log determination module is used to initiate the overwrite process in response to the log write space of the non-volatile memory meeting the triggering condition, and to determine the old log data to be overwritten from the non-volatile memory. The value summary generation module is used to obtain a dynamic feature extraction model for evaluating the value of log data, and to use the dynamic feature extraction model to extract features from the old log data to be covered, generating a structured value summary corresponding to the core semantics of the old log data. The composite log writing and overwriting module is used to embed the structured value summary into the new log data to be written, forming composite log data carrying historical value information, and write the composite log data into the non-volatile memory to overwrite the old log data; The query processing and information reconstruction module is used to receive query requests for historical logs, and to reconstruct information based on the log data currently stored in the non-volatile memory and its embedded structured value summary, and generate query response results. The model feedback and adaptive optimization module is used to generate feedback information for model optimization based on the query response results, and to adaptively adjust the internal feature weight parameters of the dynamic feature extraction model using the feedback information.

Images