Intelligent converged terminal and method for efficient compression of logs thereof, medium

Abstract

The present application relates to the technical field of terminal log compression, and specifically provides a kind of intelligent fusion terminal and its log efficient compression method, medium, comprising: the running log of each research business APP of intelligent fusion terminal is collected in real time, and the running log is classified and archived to obtain log file;Log file is washed and pretreated, and based on the occurrence frequency of fixed field in pretreated log, establish terminal log special feature dictionary;Based on terminal log special feature dictionary, two-stage linkage compression is executed to pretreated log, to obtain compressed log file;Two-stage linkage compression includes one-stage redundancy elimination and two-stage coding compression, and the coding dictionary of two-stage coding compression adopts the offset storage mode based on preset reference coding dictionary;Compressed log file is stored in terminal local, and periodic log cycle coverage mechanism is established, and high priority log is preferentially retained when cycle coverage. Adapt to the high redundancy characteristics of terminal log, solve the problem of insufficient terminal storage resources and log overflow loss.

Description

Technical Field

[0001] This invention belongs to the field of terminal log compression technology, and particularly relates to an intelligent fusion terminal and its log compression method and medium. Background Technology

[0002] Intelligent converged terminals are core edge devices in the power Internet of Things, smart homes, and edge computing networks. They integrate multiple business processing apps and can perform various business functions such as data collection, device management and control, protocol parsing, status monitoring, and fault alarms. During operation, they continuously generate massive amounts of structured and semi-structured operation logs. Log data is the core basis for terminal fault tracing, business optimization, status monitoring, and compliance auditing. Currently, all provinces in China have issued mandatory regulations for intelligent converged terminals, requiring devices to retain at least 7 days of complete operation logs locally to ensure that terminal operation is traceable and verifiable.

[0003] However, existing intelligent converged terminal log storage systems have shortcomings: First, there is a significant contradiction between local terminal storage resources and log storage requirements. Intelligent converged terminals are lightweight edge devices with extremely limited local disk storage and hardware computing power. The native log storage threshold for various business apps on the terminal is typically 5MB per file (this value is a standard custom configuration by industry manufacturers). In a 7-day full log cycle storage scenario, the total disk space occupied by logs from a manufacturer's self-developed business apps can reach 3GB. Long-term storage of massive logs will significantly encroach on the terminal's business operation storage space, easily leading to problems such as terminal lag, log storage overflow, forced overwriting of old logs, and loss of critical operation and maintenance logs. Second, existing terminal log processing mostly uses general compression algorithms, which suffer from low compression ratios, poor adaptability, high compression time, and high terminal computing power consumption. In scenarios where manufacturers typically customize single file sizes, existing general compression solutions cannot achieve high compression storage effects for a single log file or the total log.

[0004] In summary, existing solutions cannot achieve ultra-high compression ratio storage of log data with low computing power consumption, and are unable to meet the three core requirements of stable operation of terminal services, complete log retention, and efficient utilization of storage resources. This has become a key technical bottleneck restricting the large-scale and stable operation and maintenance of intelligent converged terminals. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention provides a method for efficient compression of intelligent fusion terminal logs, comprising the following steps: Step S1: Collect the operation logs of the self-developed business APPs of the intelligent converged terminal in real time, and classify and archive the operation logs to obtain log files; Step S2: Clean and preprocess the log file, and establish a terminal log-specific feature dictionary based on the frequency of occurrence of fixed fields in the preprocessed log; the terminal log-specific feature dictionary includes the mapping relationship between the fixed fields whose recurrence frequency exceeds a preset threshold and unique short codes; Step S3: Based on the terminal log-specific feature dictionary, perform two-level linkage compression on the preprocessed log to obtain a compressed log file; the two-level linkage compression includes first-level redundancy elimination and second-level encoding compression, and the encoding dictionary of the second-level encoding compression adopts an offset storage method based on a preset reference encoding dictionary. Step S4: Store the compressed log file locally on the terminal and establish a periodic log cyclic overwrite mechanism, prioritizing the retention of high-priority logs during cyclic overwrite.

[0006] Specifically, step S3 includes: S3.1: The first-level redundancy elimination adopts the adaptive window LZ77 algorithm: the adaptive dynamic window LZ77 algorithm is adopted to dynamically adjust the window size according to the redundancy of the log text, prioritize matching the high-frequency fixed fields in the terminal log exclusive feature dictionary, and replace the successfully matched duplicate strings with offset and length to obtain the first-level compressed data; after completing the adaptive window LZ77 first-level compression, the window parameters used in this compression are recorded. S3.2: The secondary encoding compression adopts an adaptive weighted Huffman algorithm: real-time frequency statistics are performed on the log bytes of the primary compressed data. Based on the real-time occurrence frequency of bytes, the equivalent weight is obtained by multiplying the log priority weighting coefficient. An adaptive weighted Huffman coding tree is constructed, and higher priority log bytes are allocated relatively shorter coding lengths. The dynamic change information relative to the preset baseline coding dictionary is stored using a dictionary offset storage method to obtain the compressed log file. When the secondary encoding compression in S3.2 generates the compressed log file, the window parameters and coding dictionary offset data are written into the header of the compressed log file. The coding dictionary offset data includes coding differences and change mapping entries.

[0007] Based on the above scheme, the step S2 of establishing a terminal log-specific feature dictionary includes: Traverse the preprocessed log files and count the frequency of occurrence of fixed fields in the logs. The fixed fields include at least timestamp, device number, business instruction, running status and fault code. Filter fixed fields that appear more than a preset threshold, establish a mapping relationship between the fixed fields and unique short codes, and obtain the terminal log-specific feature dictionary.

[0008] Based on the above scheme, step S3.1 specifically includes: Set up a dynamic sliding window range. For high-frequency fixed fields in the logs, shrink to the minimum value of the window range. For continuous and repetitive redundant text fragments, automatically enlarge to the maximum value of the window range. For regular business logs with variable field lengths and moderate redundancy, adaptively select the window within the sliding window range. Based on historical log data from the terminal, a terminal log-specific feature dictionary is preloaded, and high-frequency fields within the dictionary are prioritized for matching during the sliding window matching process. For successfully matched duplicate strings, the original text is replaced with an identifier consisting of offset and length. For independent fixed fields without matching, the original encoding is retained, thus obtaining the first-level compressed data.

[0009] Based on the above scheme, step S3.2 specifically includes: Real-time frequency statistics are performed on the log bytes in the first-level compressed data. Log bytes that appear more frequently than a preset threshold are classified as high-frequency identifier bytes, and the rest are classified as low-frequency valid data bytes. The high-frequency identifier bytes are offset identifiers and fault code abbreviation bytes that appear frequently after the first-level compression. The equivalent weight is obtained by multiplying the real-time occurrence frequency of the byte by the log priority weighting coefficient, and a shorter encoding length is allocated to high-priority log bytes under the same real-time frequency. The dictionary offset storage method is adopted, which only stores the encoding difference and change mapping entries that change dynamically relative to the preset base encoding dictionary. When the encoding change is lower than the change threshold, the base encoding dictionary is directly reused.

[0010] Specifically, the step S1 of classifying and archiving the runtime logs to obtain log files includes: Set a single log file storage capacity threshold for each business app; The operation logs are classified and archived according to any one or more of the following: APP business type, log generation time, and log priority, and the operation logs are written to the current log file. When the current log file reaches the single log file storage capacity threshold, the log file is automatically archived and a new log file is created.

[0011] Based on the above scheme, the method further includes: Step S5: During decompression, the window parameters and encoding dictionary offset data from the header of the compressed log file are read. First, the first-level compressed data is restored using Huffman decoding. Then, based on the window parameters, the original log text is restored using LZ77 reverse matching. Finally, the integrity of the restored log data is verified. The encoding dictionary offset data includes encoding differences and change mapping entries.

[0012] Preferably, the cleaning preprocessing includes: removing invalid null characters, redundant newline characters, and duplicate heartbeat fields from the logs, while retaining the core valid fields such as timestamp, device number, business instructions, operating status, and fault code.

[0013] On the other hand, the present invention also provides an intelligent fusion terminal that employs the log high-efficiency compression method described above, comprising: a log collection module for real-time collection of the operation logs of each research and development business APP; a compression processing module for compressing the logs; and a local storage module for storing the compressed log data.

[0014] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method for efficient compression of intelligent fusion terminal logs as described above.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention significantly improves the log compression ratio and reduces the local disk usage of the terminal by constructing a terminal log-specific feature dictionary and performing two-level linkage compression based on the dictionary. This enables lightweight edge devices to meet log retention requirements with limited storage resources, avoiding log overflow, forced overwriting of old logs, and loss of critical operation and maintenance logs. 2. Adopting the adaptive dynamic window LZ77 algorithm and the adaptive weighted Huffman algorithm, the compression efficiency is improved while the matching and encoding operations are simplified, the terminal computing power consumption is reduced, and lightweight, imperceptible compression is achieved, avoiding interference with core business operations such as data acquisition, device management, and protocol interaction; 3. A pre-set reference encoding dictionary plus offset storage method is adopted, storing only the dynamic change information relative to the reference encoding dictionary. Its purpose is to: reduce the storage overhead of the encoding dictionary attached to the compressed file, avoid dictionary usage offsetting compression gain, and further improve the effective compression ratio and storage resource utilization; 4. Store compressed logs locally on the terminal and establish a log loop overwrite mechanism, prioritizing the retention of high-priority fault logs and audit logs; use reverse two-level decoding and perform integrity verification during decompression; ensure that critical logs are not lost and can be fully and losslessly recovered, thereby improving the traceability of terminal operation and operational compliance. Attached Figure Description

[0016] Figure 1 This is a flowchart of the overall method for efficient log compression of the present invention; Figure 2 This is a flowchart of the two-stage compression process of the present invention; Figure 3 This is a flowchart of the lossless decompression process for logs in this invention. Detailed Implementation

[0017] The invention will be further described below with reference to specific embodiments. The following embodiments are for illustrative purposes only and do not constitute a limitation on the scope of protection of the invention. The parameters 5M, 3G, 350K, 300M, 64B-1024B, CRC-32, etc., appearing in the embodiments are exemplary parameters, and those skilled in the art can adjust them according to the actual terminal resources, log characteristics, and compliance requirements.

[0018] This invention provides a method for efficient compression of logs from intelligent fusion terminals, such as... Figure 1 As shown, the method includes: Step S1: Collect the operation logs of the self-developed business APPs of the intelligent converged terminal in real time, and classify and archive the operation logs to obtain log files; The log files obtained by categorizing and archiving the runtime logs include: Set a single log file storage capacity threshold for each business app; the standard industry configuration for the single log file storage capacity threshold is 5M, which can be customized according to actual needs; Operational logs are categorized and archived based on one or more of the following: APP business type, log generation time, and log priority. These logs are then written to the current log file, with each log priority being marked. This categorization and archiving method groups logs with similar attributes into the same file, preparing for subsequent redundancy compression. Furthermore, marking log priorities provides a basis for eviction during subsequent cyclic storage. Log priorities are automatically marked by the business APP according to the event type during the log generation phase, and are automatically written to the priority field when a log is generated, requiring no manual configuration.

[0019] When the current log file reaches the single log file storage capacity threshold, the log file is automatically archived, and a new log file is created. This single file size limit prevents excessive compression of large log files from overloading terminal memory and computing power.

[0020] Step S1 targets logs from manufacturer-developed apps. Logs from non-manufactured apps are not included in the efficient log compression method of this invention. Existing compression methods such as original storage or lightweight compression can be selectively retained.

[0021] Step S2: Clean and preprocess the log files, and establish a terminal log-specific feature dictionary based on the frequency of occurrence of fixed fields in the preprocessed logs; this terminal log-specific feature dictionary is used to represent the mapping relationship between fixed fields whose frequency of occurrence exceeds a preset threshold and unique short codes; The cleaning preprocessing includes: removing invalid null characters, redundant newline characters, and duplicate heartbeat fields from the logs, while retaining core valid fields such as timestamps, device numbers, business instructions, operating status, and fault codes.

[0022] Establishing a terminal log-specific feature dictionary includes: Traverse the preprocessed log files and count the frequency of occurrence of fixed fields in the logs. Among them, fixed fields are a subset of core effective fields, and fixed fields are fields with business traceability value among core effective fields such as timestamp, device number, business instructions, running status, and fault code. Filter high-frequency fixed fields that appear more than a preset threshold, assign a unique short code to each high-frequency fixed field that is much shorter than the original field, establish a mapping relationship between the high-frequency fixed fields and the short codes, and obtain a terminal log-specific feature dictionary.

[0023] The terminal log-specific feature dictionary can be continuously updated based on newly generated logs, providing data support for subsequent two-level linkage compression.

[0024] The log format of the self-developed APP of the intelligent converged terminal is fixed. The timestamp field repeats regularly, and fields such as device number, business instructions, and fault codes are reused frequently. The redundancy of ordinary text is much higher than that of general files. Based on this, the existing LZ77 algorithm and Huffman algorithm are optimized to adapt to the high redundancy characteristics of terminal logs.

[0025] Step S3: Based on the terminal log-specific feature dictionary, perform two-level linkage compression on the preprocessed log to obtain a compressed log file; the two-level linkage compression includes first-level redundancy elimination and second-level encoding compression, and the encoding dictionary of the second-level encoding compression adopts an offset storage method based on a preset reference encoding dictionary; like Figure 2 As shown, step S3 includes: S3.1: First-level redundancy elimination uses the adaptive window LZ77 algorithm: The adaptive dynamic window LZ77 algorithm is used, automatically and dynamically adjusting the window size based on the redundancy of the log text. It prioritizes matching high-frequency fixed fields in the terminal log's unique feature dictionary, replacing successfully matched duplicate strings with offsets and lengths to obtain first-level compressed data. After completing the first-level compression using the adaptive window LZ77 algorithm, the window parameters used in this compression are recorded.

[0026] According to one specific implementation, step S3.1 includes: A dynamic sliding window range of 64B-1024B is set. For high-frequency fixed fields in terminal logs, the window is reduced to the lower limit of the sliding window range (64B). High-frequency fixed fields include device number and fault code. For continuously repeating redundant text segments, the window is automatically enlarged to the upper limit of the range (1024B). For other regular business logs with variable field lengths and moderate redundancy, the window size is adaptively selected within the 64B-1024B sliding window range. Here, "high frequency" refers to the total frequency of a field in the entire log (global statistics), while "redundancy" refers to the degree of continuous repetition of text in a local byte stream (local continuity), representing two different judgment criteria. According to a specific embodiment, a device number appearing repeatedly in multiple logs is considered a high-frequency field; a log segment continuously repeating "normal, normal, normal" indicates a high degree of continuous repetition; regular business logs have neither globally high-frequency fixed fields nor large segments of continuously repeating text, with fields of varying lengths, indicating moderate redundancy.

[0027] Furthermore, based on the terminal's historical log data, specifically the historical 7-day log data in this embodiment, a terminal log-specific feature dictionary is preloaded. This dictionary contains high-frequency fixed fields such as device number, service instructions, timestamp, operating status, and fault codes. During the sliding window matching process, high-frequency fixed fields in the dictionary are matched first. For successfully matched duplicate strings, the original text is replaced with an identifier consisting of offset and length. For independent fixed fields without matching, the original encoding is retained, resulting in first-level compressed data.

[0028] By using a minimalist identifier replacement, only 2-4 bytes of identifier space can replace tens to hundreds of bytes of repetitive log content. For independent, fixed fields without matching identifiers, the original encoding is preserved to avoid data distortion. In practice, after first-level LZ77 compression, terminal log redundancy can be reduced by more than 85%.

[0029] To address the issue of low matching efficiency caused by the fixed window size of the traditional LZ77 algorithm failing to adapt to the varying lengths of repeated fields in terminal logs, the above steps utilize an adaptive dynamic window to adapt to different types of fields, achieving a balance between matching efficiency and computational power consumption, and significantly improving redundancy elimination efficiency. Furthermore, by using a terminal log-specific feature dictionary for high-frequency field matching, there is no need to traverse the entire text in real time, greatly reducing matching computational power consumption and improving compression speed.

[0030] S3.2: The second-level encoding compression adopts an adaptive weighted Huffman algorithm: taking the first-level compressed data output from S3.1 as input, the original fixed fields and text fragments at the business level have been converted into the underlying binary log byte stream, and the second-level Huffman compression is performed at the byte level; based on the log byte stream composed of the first-level compressed data, the real-time occurrence frequency of the log bytes is counted; based on the real-time occurrence frequency of the bytes and the log priority weighting coefficient, the equivalent weight is obtained by multiplying the log byte occurrence frequency and the log priority weighting coefficient, and an adaptive weighted Huffman coding tree is constructed to allocate a relatively shorter coding length to high-priority log bytes, and the dynamic change information relative to the preset baseline coding dictionary is stored in the dictionary offset storage method to obtain the compressed log file.

[0031] According to one specific implementation, step S3.2 includes: Real-time frequency statistics are performed on the log bytes in the first-level compressed data. The first-level compressed data output in step S3.1 is a structured byte stream. Each compressed segment is preceded by a control identifier byte to mark the current segment type. The first-level compressed data byte stream is read sequentially, the control identifier byte is read, and the subsequent corresponding valid data bytes are read according to the segment length indicated by the control identifier byte, thereby distinguishing and parsing the control identifier byte and the valid data bytes corresponding to the original log. Log bytes with a frequency higher than a preset threshold are classified as high-frequency identifier bytes, and the rest are classified as low-frequency valid data bytes. The high-frequency identifier bytes are abbreviations of offset identifiers, fault codes, etc. that appear frequently after the terminal log is compressed.

[0032] Based on the real-time occurrence frequency of bytes and log priority weighting coefficients, equivalent weights are obtained through multiplication operations, and a Huffman coding tree is constructed. Each byte is a leaf node, and the equivalent weight is the node weight. The two nodes with the smallest weights are repeatedly selected and merged. The weight of the new node after merging is the sum of the weights of the two nodes, until the Huffman coding tree is completed. The branches of the Huffman coding tree are assigned binary codes to obtain the variable-length codes of each byte. The higher the equivalent weight of a byte, the closer it is to the root node in the coding tree, and the shorter its code length.

[0033] Under the same real-time frequency, a shorter encoding length is allocated to high-priority log bytes; the high-priority log bytes of this invention correspond to the bytes of core business logs such as fault logs and audit logs; the low-priority logs correspond to ordinary operation logs.

[0034] It's important to note that, given the same real-time frequency of bytes, fault / audit log bytes are multiplied by a higher priority weighting factor, resulting in a larger effective weight. During Huffman coding tree construction, leaf nodes with higher effective weights are closer to the root node and allocated shorter binary encoding lengths. Therefore, under the same frequency, high-priority log bytes occupy fewer bytes after compression, resulting in greater compression. Regular operation logs are not subject to priority weighting and rely solely on their natural byte frequency for standard Huffman coding, achieving maximum compression. Fault and audit logs, however, have their encoding further shortened by multiplying their original real-time byte frequency by a log priority weighting factor, achieving differentiated compression.

[0035] A dictionary offset storage method is adopted. The terminal is pre-loaded with a baseline encoding dictionary pre-trained based on the terminal's historical log samples. During operation, only the encoding differences and change mapping entries that dynamically change relative to this baseline encoding dictionary are stored. When the encoding change is less than the change threshold, the baseline encoding dictionary is directly reused. At the same time, window parameters and encoding dictionary offset data are written to the header of the compressed log file. The encoding dictionary offset data includes encoding differences and change mapping entries.

[0036] The variable threshold can be set according to the actual situation. If it is lower than the variable threshold, there is no need to store the difference, which greatly reduces the storage space occupied by the dictionary along with the compressed file and avoids the dictionary space offsetting the compression gain.

[0037] The baseline encoding dictionary is a pre-trained basic encoding table based on historical log samples from the terminal, storing preset encodings corresponding to high-frequency basic bytes in the power terminal log scenario. During the compression operation phase, the terminal does not need to store the complete encoding tree / encoding table, but only stores the encoding differences and change mapping entries that have changed relative to the baseline encoding dictionary, thereby reducing the storage overhead of the dictionary.

[0038] By increasing the equivalent weight of the corresponding bytes in fault logs and audit logs, we can maximize the compression of ordinary operation log data while ensuring that the core log data remains intact, thus achieving differentiated and precise compression.

[0039] After the above two-stage linkage compression, taking the manufacturer's standard 5M single file configuration as an example, a single original log file can be stably compressed to 350K±20K, with a compression ratio of about 14:1; 7 days of full 3G self-developed APP logs can be compressed to 300M±20M. This method has a significant effect on the compression of logs for the manufacturer's self-developed APP.

[0040] Step S4: Store the compressed log file on the terminal's local disk and establish a periodic log overwrite mechanism. When log overwrite is triggered, high-priority logs are retained first.

[0041] According to one specific implementation method, after 7 days, the earliest ordinary logs are automatically overwritten, while high-priority fault logs and audit logs are retained to ensure storage compliance.

[0042] Furthermore, in order to achieve lossless decompression of compressed data, such as Figure 3 As shown, the method of the present invention further includes step S5, in which, during decompression, the window parameters (LZ77 sliding window size) and encoding dictionary offset data (encoding difference and change mapping entries) of the compressed log file header are first read. A preset baseline encoding dictionary is reused in combination with the offset data to restore the complete encoding table. First-level compressed data is restored through Huffman decoding, and the original log text is restored through LZ77 reverse matching. The restored log data is then subjected to integrity verification. After integrity verification, the complete original log file is output. According to a specific embodiment, integrity verification is achieved by adding a lightweight CRC-32 verification mechanism. A log check code is generated during compression and embedded at the end of the compressed file. After decompression, data integrity is automatically verified to prevent log loss, garbled characters, and distortion.

[0043] This invention employs a lossless compression link consisting of a first-level adaptive window LZ77 and a second-level adaptive weighted Huffman. The compression process does not discard or truncate any original log bit information, and the encoding process is completely reversible. The decoding end can completely restore the original log byte stream based on the baseline encoding dictionary and change entries, thus ensuring that core log data such as fault and audit logs remain lossless.

[0044] To further illustrate the compression effect of this invention on intelligent converged terminal log scenarios, a comparative test was conducted using a 7-day self-developed business log dataset collected from a real terminal. The test hardware used was the native ARM embedded hardware of the intelligent converged terminal. The total amount of original log data was 3GB, with a single log file size of 5MB. The log samples included typical terminal log types such as fault alarm logs, business instruction logs, and periodic heartbeat logs. This invention was compared with commonly used terminal compression methods such as LZ4, Snappy, gzip (zlib-6), and standard DEFLATE (conventional LZ77+Huffman, without a terminal-specific feature dictionary, adaptive dynamic window, or log priority weighting).

[0045] Test results show that lightweight compression algorithms like LZ4 and Snappy have low CPU usage and fast compression speed, but their compression ratio is only 2.0-2.1:1 for structured log scenarios in power terminals, resulting in limited storage savings. While gzip and the standard DEFLATE general-purpose LZ77+Huffman scheme offer improved compression ratios, they have high CPU usage and long compression times, making them prone to preempting computing resources from business applications on intelligent converged terminals with limited computing power. The proposed solution achieves a compression ratio of 10-14:1, with compression CPU overhead between lightweight algorithms and gzip, balancing high compression ratios with the computing power constraints of embedded terminals. Therefore, the overall performance of this invention is achieved through the combined and collaborative use of a terminal log-specific feature dictionary, adaptive dynamic window LZ77, log priority-weighted Huffman, and dictionary offset storage.

[0046] Based on the same technical concept, the present invention also provides an intelligent fusion terminal, which adopts the log high-efficiency compression method described above, including: a log collection module for collecting the operation logs of each research and development business APP in real time; a compression processing module for compressing the logs; and a local storage module for storing the compressed log data.

[0047] Furthermore, the method for efficient compression of intelligent converged terminal logs according to the present invention can be recorded in a computer-readable storage medium. Specifically, according to the present invention, a computer-readable storage medium storing computer-executable instructions can be provided, which, when executed by a processor, causes the processor to execute the method for efficient compression of intelligent converged terminal logs as described above.

[0048] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of terminals, methods, and computer-readable storage media according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code containing at least one executable instruction for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0049] In general, various exemplary embodiments of the present invention can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of the present invention are illustrated or described as block diagrams, flowcharts, or represented using certain other images, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or certain combinations thereof.

[0050] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0051] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for efficient compression of intelligent fusion terminal logs, characterized in that, Includes the following steps: Step S1: Collect the operation logs of the self-developed business APPs of the intelligent converged terminal in real time, and classify and archive the operation logs to obtain log files; Step S2: Clean and preprocess the log file, and establish a terminal log-specific feature dictionary based on the frequency of occurrence of fixed fields in the preprocessed log. The terminal log-specific feature dictionary includes a mapping relationship between fixed fields that appear more than a preset threshold and unique short codes. Step S3: Based on the terminal log-specific feature dictionary, perform two-level linkage compression on the preprocessed log to obtain a compressed log file; The dual-level linkage compression includes a first-level redundancy elimination and a second-level encoding compression. The encoding dictionary for the second-level encoding compression adopts an offset storage method based on a preset reference encoding dictionary. Step S4: Store the compressed log file locally on the terminal and establish a periodic log cyclic overwrite mechanism, prioritizing the retention of high-priority logs during cyclic overwrite.

2. The method for efficient compression of intelligent fusion terminal logs according to claim 1, characterized in that, Step S3 includes: S3.1: The first-level redundancy elimination adopts the adaptive window LZ77 algorithm: the adaptive dynamic window LZ77 algorithm is adopted to dynamically adjust the window size according to the redundancy of the log text, prioritize matching the high-frequency fixed fields in the terminal log exclusive feature dictionary, and replace the successfully matched duplicate strings with offset and length to obtain the first-level compressed data; after completing the adaptive window LZ77 first-level compression, the window parameters used in this compression are recorded. S3.2: The secondary encoding compression adopts an adaptive weighted Huffman algorithm: real-time frequency statistics are performed on the log bytes of the primary compressed data. Based on the real-time occurrence frequency of the bytes, the equivalent weight is obtained by multiplying the log priority weighting coefficient. An adaptive weighted Huffman coding tree is constructed. Higher priority log bytes are allocated relatively shorter coding lengths. The dynamic change information of the coding dictionary relative to the preset baseline coding dictionary is stored using a dictionary offset storage method to obtain the compressed log file. The window parameters and coding dictionary offset data are stored in the header of the compressed log file.

3. The method for efficient compression of intelligent fusion terminal logs according to claim 1, characterized in that, The step S2 of establishing a terminal log-specific feature dictionary includes: Traverse the preprocessed log files and count the frequency of occurrence of fixed fields in the logs. The fixed fields include at least timestamp, device number, business instruction, running status and fault code. Filter fixed fields that appear more than a preset threshold, establish a mapping relationship between the fixed fields and unique short codes, and obtain the terminal log-specific feature dictionary.

4. The method for efficient compression of intelligent fusion terminal logs according to claim 2, characterized in that, Step S3.1 specifically includes: Set up a dynamic sliding window range. For high-frequency fixed fields in the logs, shrink to the minimum value of the window range. For continuous and repetitive redundant text fragments, automatically enlarge to the maximum value of the window range. For regular business logs with variable field lengths and moderate redundancy, adaptively select the window within the sliding window range. Based on historical log data from the terminal, a terminal log-specific feature dictionary is preloaded, and high-frequency fields within the dictionary are prioritized for matching during the sliding window matching process. For successfully matched duplicate strings, the original text is replaced with an identifier consisting of offset and length. For independent fixed fields without matching, the original encoding is retained, thus obtaining the first-level compressed data.

5. The method for efficient compression of intelligent fusion terminal logs according to claim 2, characterized in that, Step S3.2 specifically includes: Real-time frequency statistics are performed on the log bytes in the first-level compressed data. Log bytes that appear more frequently than a preset threshold are classified as high-frequency identifier bytes, and the rest are classified as low-frequency valid data bytes. The high-frequency identifier bytes are offset identifiers and fault code abbreviation bytes that appear frequently after the first-level compression. The equivalent weight is obtained by multiplying the real-time occurrence frequency of the byte by the log priority weighting coefficient, and a shorter encoding length is allocated to high-priority log bytes under the same real-time frequency. The dictionary offset storage method is adopted, which only stores the encoding difference and change mapping entries that change dynamically relative to the preset base encoding dictionary. When the encoding change is lower than the change threshold, the base encoding dictionary is directly reused; the encoding difference and change mapping entries constitute the encoding dictionary offset data.

6. The method for efficient compression of intelligent fusion terminal logs according to claim 1, characterized in that, In step S1, the log files obtained by classifying and archiving the runtime logs include: Set a single log file storage capacity threshold for each business app; The operation logs are classified and archived according to any one or more of the following: APP business type, log generation time, and log priority, and the operation logs are written to the current log file. When the current log file reaches the single log file storage capacity threshold, the log file is automatically archived and a new log file is created.

7. The method for efficient compression of intelligent fusion terminal logs according to claim 2, characterized in that, The method further includes: Step S5: During decompression, read the window parameters and encoding dictionary offset data from the header of the compressed log file. First, restore the first-level compressed data using Huffman decoding. Then, restore the original log text using LZ77 reverse matching based on the window parameters. Perform integrity verification on the restored log data. The encoding dictionary offset data includes encoding differences and change mapping entries.

8. The method for efficient compression of intelligent fusion terminal logs according to claim 1, characterized in that, The cleaning preprocessing includes: removing invalid null characters, redundant newline characters, and duplicate heartbeat fields from the logs, while retaining the core valid fields such as timestamp, device number, business instructions, operating status, and fault codes.

9. A smart converged terminal, characterized in that, The method for efficient log compression as described in any one of claims 1-8 includes: a log collection module for collecting the operation logs of each research and development business APP in real time; a compression processing module for compressing the logs; and a local storage module for storing the compressed log data.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method for efficient compression of intelligent fusion terminal logs as described in any one of claims 1-8.

Images