Log collection method, system and device based on MQTT protocol and medium

Abstract

The application provides a log collection method, system, device and medium based on an MQTT protocol, relates to the technical field of data collection, and comprises the following steps: acquiring a plurality of original log data; storing each original log data to a ring buffer area according to a non-blocking coverage strategy, wherein the ring buffer area comprises a preset number of continuous storage blocks; reading a plurality of target log data from the ring buffer area; performing hierarchical filtering on each target log data according to a preset filtering rule to obtain a plurality of filtered log data; performing binary encoding compression on each filtered log data to obtain a plurality of compressed log data; storing each compressed log data in a preset memory pool based on a dynamic memory allocation strategy; and transmitting each compressed log data in the preset memory pool to a remote server based on a lightweight processed MQTT protocol. The application reduces the memory overhead required for log collection while improving the log transmission efficiency and transmission stability.

Description

Technical Field

[0001] This application relates to the field of data acquisition technology, and more specifically, to a log acquisition method, system, device, and medium based on the MQTT protocol. Background Technology

[0002] Traditional log collection methods based on the Message Queuing Telemetry Transport (MQTT) protocol rely on a complete Transmission Control Protocol / Internet Protocol (TCP / IP) stack, and incur significant memory overhead during the collection process, failing to meet the limited memory requirements of embedded devices. Therefore, proposing a lightweight log collection method based on the MQTT protocol has become a pressing technical challenge. Summary of the Invention

[0003] In view of this, the purpose of this application is to overcome the shortcomings of the prior art and provide a log collection method, system, device, and medium based on the MQTT protocol. This application provides the following technical solution:

[0004] Firstly, this application provides a log collection method based on the MQTT protocol, applied to an embedded device, the method comprising:

[0005] Multiple raw log data are acquired, and each raw log data is stored in a circular buffer area according to a non-blocking overwrite strategy. The circular buffer area includes a preset number of contiguous storage blocks.

[0006] Multiple target log data are read from the circular buffer area, and each target log data is filtered hierarchically according to a preset filtering rule to obtain multiple filtered log data.

[0007] Each of the filtered log data is compressed using binary encoding to obtain multiple compressed log data sets.

[0008] Based on a dynamic memory allocation strategy, the compressed log data is stored in a preset memory pool;

[0009] Based on the lightweight MQTT protocol, the compressed log data in the preset memory pool is transmitted to a remote server.

[0010] In one embodiment, reading multiple target log data from the circular buffer region includes:

[0011] Determine whether the read pointer of the circular buffer area is equal to the write pointer. If so, wait for the original log data to be written to the storage block pointed to by the write pointer.

[0012] If not, then read the storage block pointed to by the read pointer to obtain the original log data;

[0013] The original log data is subjected to integrity verification, and the original log data that passes the verification is determined as the target log data.

[0014] In one embodiment, the target log data includes: a prefix identifier, a path identifier, and a log field, and the preset filtering rules include:

[0015] The log level of the target log data is obtained by querying the preset log level mapping table based on the prefix identifier.

[0016] If the log level is lower than the preset retention level, then the target log data is filtered.

[0017] If the log level is not lower than the preset retention level, the path identifier is matched with the disabled path identifier in the preset tree structure. If the identifier matches successfully, the target log data is filtered.

[0018] If the identifier match fails, the log field is input into a preset regular expression engine for regular expression matching. If the regular expression match succeeds, the target log data is filtered.

[0019] If the regular expression match fails, the target log data will be determined as the filtered log data.

[0020] In one embodiment, the number of the multiple filtered log data is N, and the binary encoding compression is performed on each of the filtered log data to obtain multiple compressed log data, including:

[0021] The i-th filtered log data includes: the i-th prefix identifier and the i-th log field, 1≤i≤N;

[0022] According to the preset encoding rules, the i-th prefix identifier is converted into an M-bit binary enumeration value, where M is a positive integer and M≥1;

[0023] For the i-th log field, a variable-length encoding algorithm is used for compression to obtain the corresponding variable-length encoded bitstream;

[0024] The M-bit binary enumeration value corresponding to the i-th prefix identifier is concatenated with the variable-length encoded bitstream corresponding to the i-th log field to obtain the i-th compressed log data.

[0025] In one embodiment, storing each of the compressed log data in a preset memory pool based on a dynamic memory allocation strategy includes:

[0026] Based on the data length of the i-th compressed log, a target memory block is determined from the preset memory pool. The target memory block is the smallest memory block with a size not less than the data length.

[0027] Write the i-th compressed log into the target memory block.

[0028] In one embodiment, the MQTT protocol is lightweighted, including:

[0029] The message header fields of the MQTT protocol are subject to preset simplification processing, which includes removing preset reserved bits and preset identifier bits;

[0030] The topic field of the MQTT protocol is processed by binary encoding.

[0031] In one embodiment, before transmitting the compressed log data from the preset memory pool to the remote server based on the lightweight MQTT protocol, the method further includes:

[0032] Determine whether the total amount of each compressed log data in the preset memory pool has reached a preset quantity threshold;

[0033] Alternatively, check if the time elapsed since the last transmission exceeds the preset period.

[0034] Secondly, this application provides a log collection system based on the MQTT protocol, the system comprising:

[0035] The log acquisition module is used to acquire multiple raw log data and store each raw log data into a circular buffer area according to a non-blocking overwrite strategy. The circular buffer area includes a preset number of consecutive storage blocks.

[0036] The filtering module is used to read multiple target log data from the ring buffer area, perform hierarchical filtering on each target log data according to preset filtering rules, and obtain multiple filtered log data.

[0037] The compression encoding module is used to perform binary encoding compression on each of the filtered log data to obtain multiple compressed log data.

[0038] The pre-storage module is used to store each of the compressed log data in a preset memory pool based on a dynamic memory allocation strategy;

[0039] The transmission module is used to transmit the compressed log data in the preset memory pool to a remote server based on the lightweight MQTT protocol.

[0040] Thirdly, this application provides an embedded device, including a memory and a processor, wherein the memory stores a computer program, and the computer program executes the log collection method based on the MQTT protocol described in the first aspect when running on the processor.

[0041] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the log collection method based on the MQTT protocol described in the first aspect.

[0042] The log collection method, system, device, and medium provided in this application acquire multiple raw log data sets, store each raw log data set in a circular buffer area according to a non-blocking overlay strategy, the circular buffer area including a preset number of contiguous storage blocks; read multiple target log data sets from the circular buffer area, perform hierarchical filtering on each target log data set according to preset filtering rules, and obtain multiple filtered log data sets; perform binary encoding compression on each of the filtered log data sets to obtain multiple compressed log data sets; store each compressed log data set in a preset memory pool based on a dynamic memory allocation strategy; and transmit each compressed log data set in the preset memory pool to a remote server based on a lightweight MQTT protocol, thereby reducing the memory overhead required for log collection while improving log transmission efficiency and stability.

[0043] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart illustrating a log collection method based on the MQTT protocol provided in an embodiment of this application is shown.

[0046] Figure 2 This paper illustrates a schematic diagram of the structure of a log collection system based on the MQTT protocol provided in an embodiment of this application.

[0047] Figure 3 A schematic diagram of the structure of an embedded device provided in an embodiment of this application is shown.

[0048] Explanation of key component symbols:

[0049] 200 - Log collection system based on MQTT protocol; 210 - Log collection module; 220 - Filtering module; 230 - Compression encoding module; 240 - Pre-storage module; 250 - Transmission module; 300 - Embedded device; 301 - Transceiver; 302 - Processor; 303 - Memory. Detailed Implementation

[0050] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0051] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the template description is for the purpose of describing particular embodiments only and is not intended to limit the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0053] Example 1

[0054] In current technologies, mainstream embedded devices typically have 2-4GB of memory, and the budget for their log collection modules needs to be controlled within 10MB. Traditional log collection methods based on the MQTT protocol rely on a complete TCP / IP stack, generally exceeding the memory usage budget and failing to meet current requirements for limited memory. For further information, please refer to [link to relevant documentation / reference]. Figure 1 This application provides a log collection method based on the MQTT protocol, which is applied to an embedded device. The method includes steps S110 to S150.

[0055] Step S110: Obtain multiple raw log data and store each raw log data in a circular buffer area according to a non-blocking overwrite strategy. The circular buffer area includes a preset number of consecutive storage blocks.

[0056] In this embodiment, multiple raw log data generated during the operation of the embedded device are acquired, and a pre-set fixed-capacity (e.g., 3MB) circular buffer area is initialized. The circular buffer area is configured as contiguous physical pages to ensure Direct Memory Access (DMA) efficiency and is divided into multiple pre-set contiguous storage blocks, such as 1024 contiguous storage blocks of 3072 bytes each. The physical address of the storage block is quickly located using bitwise operations, and the block index uses circular counting to achieve automatic wraparound upon overflow.

[0057] When the raw log data is stored in the circular buffer, dual-pointer synchronization of the write and read pointers is implemented based on ARMv8 atomic operations. The write pointer is updated atomically using __atomic_add_fetch. After the raw log data is copied to the corresponding storage block without locks, the ready_flag bitmap is updated with __ATOMIC_RELEASE memory order to mark the data as valid. When the write pointer + 1 equals the read pointer (i.e., when the circular buffer is full), a non-blocking overwrite strategy is executed to discard the oldest log and increment the discard counter. At the same time, a two-level hardware fault tolerance mechanism ensures data reliability—when the number of consecutively discarded blocks is greater than or equal to a preset number, such as 10, a watchdog hardware reset is triggered. Pointer anomaly detection and data rollback are achieved by verifying the magic_num, crc32 checksum, and timestamp in the storage block header.

[0058] Its beneficial effects are as follows: compared with the traditional double buffering scheme, it saves more memory; through lock-free operation and bitwise operation optimization, it avoids read and write competition, significantly reduces instruction overhead; the non-blocking overwrite strategy ensures that high-concurrency log writing does not block device operation; it not only meets the memory budget of embedded device log collection, but also realizes the combination of continuous storage and hardware-level fault tolerance mechanism, effectively reducing the risk of original log loss, and providing a complete and reliable data source for subsequent log processing.

[0059] Step S120: Read multiple target log data from the circular buffer area, and perform hierarchical filtering on each target log data according to preset filtering rules to obtain multiple filtered log data.

[0060] In this embodiment, during the data reading phase, the read and write pointer states of the circular buffer area are first verified using ARMv8 atomic operations. If they are equal, the system waits for new raw log data to be written. If they are not equal, bitwise operations are performed (block addr = base_addr + (read pointer & 0x3FF)). 3072) Locate the storage block pointed to by the read pointer, complete the data integrity verification through magic_num check, incremental check of CRC32 instruction and protocol identifier check, and determine the original log data that passes the verification as the target log data.

[0061] The filtering phase relies on a five-level log filtering engine with a Trie tree structure. First, it quickly matches the prefix identifier of the target log data using a bitmask, queries the preset log level mapping table to obtain the corresponding log level, and filters target log data that is lower than the preset retention level. For target log data that meets the level requirements, it matches the path identifier with the preset disabled path using a cache row aligned red-black tree and hash compression technology. If the match is successful, it is filtered. The remaining target log data is then processed by a preset regular expression engine that integrates the second-generation regular expression engine (RE2) and deterministic finite automaton state compression (DFA) (compression rate 62%) to filter target log data that successfully matches the regular expression. The final retained data is the filtered log data.

[0062] Its beneficial effects are as follows: it ensures the reliability of target log data through pointer verification and multi-level integrity verification, preventing invalid target log data from entering subsequent processes; the five-level filtering engine adopts a hybrid structure of Trie tree and red-black tree, combined with ARMv8 instructions to accelerate hash calculation, making the runtime memory of 100,000 rules only 9.6MB, which is 80% lower than traditional solutions; the hierarchical filtering mechanism accurately filters key logs, reducing unnecessary data transmission and storage overhead, while the RE2 algorithm and DFA compression technology balance filtering accuracy and efficiency, with a matching latency as low as 8.2μs / login (10,000 rules scenario), and the fragmentation rate controlled within 10%, which is suitable for the memory constraints of embedded devices and meets the real-time processing needs of industrial scenarios.

[0063] In one embodiment, reading multiple target log data from the circular buffer area includes: determining whether the read pointer of the circular buffer area is equal to the write pointer; if so, waiting for the original log data to be written to the storage block pointed to by the write pointer; if not, reading the storage block pointed to by the read pointer to obtain the original log data; performing integrity verification on the original log data, and determining the original log data that passes the verification as the target log data.

[0064] In this embodiment, the read pointer and write pointer status of the circular buffer area are first obtained and compared in real time through ARMv8 atomic operations. If the two values ​​are equal, it means that the buffer is in an idle state, and it enters a waiting state until new raw log data is written to the storage block pointed to by the write pointer; if the two values ​​are not equal, the bitwise operation (blockaddr = base_addr + (read pointer & 0x3FF) is used. 3072) Quickly locate the storage block pointed to by the read pointer, read the original log data in the block, and then start a three-level integrity verification process: first, verify the magic_num and ready_flag bitmap in the storage block header to confirm the validity of the data, then use the ARMv8 CRC32 instruction (__builtin_arm_crc32d) to perform incremental CRC calculation to verify the integrity of the data, and finally check the protocol identifier to ensure that the data format is compliant. The original log data that passes all three levels of verification is finally determined as the target log data.

[0065] Its beneficial effects are as follows: by comparing the read pointer and write pointer states in real time, the occupancy status of the circular buffer can be determined, avoiding resource waste caused by invalid read operations; the bitwise operation positioning method eliminates the need for additional modulo operations, significantly improving the efficiency of storage block access; the three-level integrity verification mechanism based on magic_num, CRC32 checksum and protocol identifier, combined with ARMv8 hardware instruction acceleration, not only ensures the integrity and reliability of the target log data, but also controls the time of a single verification to the microsecond level, adapting to the real-time requirements of embedded devices.

[0066] In one embodiment, the target log data includes: a prefix identifier, a path identifier, and a log field; the preset filtering rule includes: querying a preset log level mapping table based on the prefix identifier to obtain the log level of the target log data; if the log level is lower than a preset retention level, then filtering the target log data.

[0067] If the log level is not lower than the preset retention level, the path identifier is matched with the disabled path identifier in the preset tree structure. If the identifier matches successfully, the target log data is filtered. If the identifier does not match successfully, the log field is input into the preset regular expression engine for regular expression matching. If the regular expression matches successfully, the target log data is filtered. If the regular expression matches unsuccessfully, the target log data is determined as the filtered log data.

[0068] In this embodiment, the target log data includes a prefix identifier, a path identifier, and a log field. The corresponding preset filtering rules are implemented using a three-level progressive verification logic: First, the prefix identifier of the target log data is extracted, and the preset log level mapping table is queried to determine the log level. If the log level is lower than the preset retention level, the target log data is directly filtered. If the log level meets the retention requirements, the path identifier is matched with the disabled path identifier in the preset tree structure based on a cache line aligned red-black tree and Trie tree hybrid structure. The path segment hash calculation is accelerated by using the Jenkins hash algorithm combined with ARMv8 CRC32 instructions (reducing the single time from 120ns to 28ns) to quickly locate the tree node. If the identifier matches successfully, the data is filtered. If the path identifier does not match the disabled rule, the log field is input into a preset regular expression engine that integrates the RE2 algorithm and DFA state compression, and the character class is merged (e.g., [a-zA-Z] is converted to 0x1A). Optimization techniques such as bitmap and transfer table hashing are used to perform regular expression matching. If a match is successful, the data is filtered. Only when none of the three levels of verification triggers filtering is the target log data determined as the filtered log data.

[0069] Its beneficial effects are as follows: the three-level filtering process is progressive, firstly quickly filtering core data by log level, then accurately excluding invalid path logs by path identification, and finally intercepting illegal fields by regular expression matching, which greatly reduces the amount of data to be processed in the later stages; relying on the hybrid structure of red-black trees and Trie trees and hardware instruction acceleration, the runtime memory for 100,000 rules is reduced from 48MB to 9.6MB, the matching latency is as low as 8.2μs / rule (10,000 rules scenario), and the fragmentation rate is controlled within 10%; the combination of RE2 algorithm and DFA state compression not only ensures the accuracy of regular expression matching, but also significantly reduces memory usage, adapting to the memory budget of embedded devices within 300, while reducing network transmission and server storage overhead by accurately filtering non-critical and illegal logs, thus improving the efficiency and security of the overall log processing chain.

[0070] Step S130: Perform binary encoding compression on each of the filtered log data to obtain multiple compressed log data.

[0071] In this embodiment, the prefix identifier and log fields included in each log data are compressed using binary encoding to obtain compressed log data corresponding to each log data. Compared to filtered log data in JSON format, it requires less memory for parsing.

[0072] In one embodiment, the number of the multiple filtered log data is N. The step of performing binary encoding compression on each of the filtered log data to obtain multiple compressed log data includes: the i-th filtered log data includes: an i-th prefix identifier and an i-th log field, 1 ≤ i ≤ N; according to a preset encoding rule, the i-th prefix identifier is converted into an M-bit binary enumeration value, where M is a positive integer and M ≥ 1; for the i-th log field, a variable-length encoding algorithm is used for compression to obtain the corresponding variable-length encoded bitstream; the M-bit binary enumeration value corresponding to the i-th prefix identifier is concatenated with the variable-length encoded bitstream corresponding to the i-th log field to obtain the i-th compressed log data.

[0073] In this embodiment, the encoding and compression process for N filtered log data is as follows: For the i-th filtered log (1≤i≤N), the i-th prefix identifier (corresponding to discrete information such as log level) and the i-th log field (corresponding to specific log content) are respectively encoded using a differentiated encoding strategy. First, according to the preset encoding rules (such as the mapping table between log level and 4-bit binary enumeration value: DEBUG corresponds to 0x0, INFO corresponds to 0x1, etc.), the i-th prefix identifier is converted into an M-bit (M≥1, preferably 4 bits) binary enumeration value. The conversion is achieved quickly through bit field operations (such as #define LOG_LEVEL_MASK (level)((level&0xF)<<4)), which greatly reduces the identifier storage occupation (from the original 4-7 bytes to 0.5 bytes).

[0074] For the i-th log field, the Varint encoding formula is used (value=(byte&0x7F)<<(7)). The basic compression is performed on the M-bit binary enumeration value (where n is the number of byte order shifts). For negative data, ZigZag optimization ((n<<1)^(n>>31)) is used to avoid sign bit interference, and finally a variable-length encoded bit stream is generated. Finally, the M-bit binary enumeration value and the variable-length encoded bit stream are concatenated according to the preset format to form the i-th compressed log data.

[0075] Its beneficial effects are as follows: the fixed-length enumeration conversion of the prefix identifier enables fast decoding and low memory usage, while the variable-length encoding algorithm of the log field maximizes the compression of data volume. The combination of the two significantly reduces the storage overhead after encoding a single log entry; there is no additional dynamic memory allocation during the encoding process, and the encoding time is controlled in the microsecond level through predefined structure and bit operation optimization. Moreover, the concatenated binary data is naturally adapted to the MQTT message format (3072-byte block size adapts to the maximum MQTT message length), avoiding the redundant overhead of traditional encoding methods and further reducing the resource consumption of embedded devices.

[0076] Step S140: Based on the dynamic memory allocation strategy, store each of the compressed log data in a preset memory pool.

[0077] In this embodiment, the memory pool is first divided into multiple levels (from the smallest 256B unit to the largest 1MB block) according to a binary rule. A two-level bitmap structure, global and local, is used to manage the idle status of memory blocks at each level. Available blocks are quickly located using the __builtin_ctz instruction. For each compressed log data, the smallest memory block in the memory pool that is not smaller than its actual data length is matched according to its actual data length. After allocation, the block status is marked by atomic operations. After data storage is completed, when the memory pool occupancy rate reaches a preset threshold, a recursive merging mechanism is triggered to reclaim idle blocks.

[0078] Its beneficial effects are: memory consumption is reduced by 75% in idle state compared with traditional solutions, memory consumption is reduced by 84% under peak load, and the peak fragmentation rate is reduced; the two-level bitmap design reduces the memory block lookup complexity from O(n) to O(1), the __builtin_ctz instruction improves the efficiency of free block lookup, adapts to the memory budget of embedded devices, and dynamically adapts to compressed log data of different lengths to avoid memory waste and ensure the efficiency and stability of log storage process.

[0079] In one embodiment, storing each compressed log data in a preset memory pool based on a dynamic memory allocation strategy includes: determining a target memory block from the preset memory pool based on the data length of the i-th compressed log, wherein the target memory block is the smallest memory block with a size not less than the data length; and writing the i-th compressed log into the target memory block.

[0080] In this embodiment, the data length of the i-th compressed log is first read. Using a multi-level bitmap hierarchical index (from 256B to 1MB) of the memory pool, free memory blocks at each level are matched sequentially. The smallest memory block with a size not less than the data length is selected as the target memory block. The size of this target memory block is chosen based on the characteristics of log data in embedded scenarios. The 4x Cortex-M cache line alignment design with a 256B minimum unit improves data read / write efficiency. After determining the target memory block, address continuity is verified using the IS_BUDDY macro to ensure the memory block meets alignment requirements. Then, the bitmap status is updated to mark the block as occupied using an atomic operation (__sync_fetch_and_or). Finally, the i-th compressed log data is written to the target memory block without locking, completing the storage.

[0081] Its beneficial effects are as follows: the minimum fit memory block selection mechanism minimizes memory waste, and combined with the 256B minimum unit design, the fragmentation rate is only 8% in a scenario of processing 1000 logs; atomic operations ensure storage security in a multi-threaded environment and avoid data competition; address continuity verification and cache line alignment design further improve data read and write speed, ensuring that the storage time of each compressed log is controlled at the microsecond level, which not only meets the resource constraints of embedded devices, but also ensures the efficiency and orderliness of storing a large number of compressed logs.

[0082] Step S150: Based on the lightweight MQTT protocol, transmit each of the compressed log data in the preset memory pool to a remote server.

[0083] In this embodiment, a simplified version of the MQTT protocol stack is first loaded, retaining only essential fields such as Client ID, Clean Session, and Keep Alive, while removing unnecessary fields such as Will Topic / Will Message. The CONNECT message is reduced from 128B to 40B, and only QoS level 0 transmission is implemented to avoid redundant acknowledgment mechanisms. During transmission, compressed log data is read from a preset memory pool, encapsulated in 3072-byte blocks (adapting to the maximum MQTT message length), and sent through a long connection established by the MQTT Broker. Combined with a 5-minute long heartbeat interval (300 seconds), network overhead is reduced, and only 0.1MB of session context is rebuilt during reconnection.

[0084] Its beneficial effects are as follows: compared with traditional MQTT protocols (such as Eclipse Paho), it reduces protocol header overhead by 68%; QoS level 0 transmission reduces real-time processing latency from 200-500ms to 8-12ms, meeting the 10ms response requirements of industrial control; long heartbeat intervals reduce network interaction by 30%; the fast reconnection mechanism shortens recovery time from 8-12 seconds to 0.3 seconds; the overall transmission process has controllable memory usage, perfectly adapting to the memory budget of embedded devices, while ensuring the stability and real-time performance of data transmission.

[0085] In one embodiment, the MQTT protocol is lightweighted by: performing a preset simplification process on the header fields of the MQTT protocol, wherein the preset simplification process includes: removing preset reserved bits and preset identifier bits; and performing binary encoding on the subject field of the MQTT protocol.

[0086] Regarding the simplification of message header fields, firstly, based on the necessity judgment criteria for industrial IoT scenarios, essential fields such as ClientID, Clean Session, and Keep Alive are selected, while unnecessary fields such as Will Topic / Will Message (accounting for 60% of the original fields) are removed. Then, the 8-bit Connect Flags are compressed to 3 valid bits (only the CleanSession and Keep Alive related bits are retained). The unnecessary fields are quickly identified and skipped in parsing using bitmasking technology (WILL_FLAG_MASK=0x04). At the same time, the Protocol Level field is hard-coded to 0x04 (using #define MQTT_PROTOCOL_LEVEL 0x04) to reduce 1B of transmission overhead. Regarding the optimization of the topic field, the same binary encoding strategy as the log field (Varint+ZigZag combined encoding) is adopted to convert the text-formatted topic field into a compact binary stream, avoiding redundant bytes in text transmission.

[0087] Its beneficial effects are as follows: the simplified message header reduces the CONNECT message size by 68% (from 128B to 40B), and the protocol stack memory usage is reduced from more than 6MB in the traditional solution to 0.8MB; the binary encoding of the subject field further compresses the amount of transmitted data, reduces network bandwidth usage, and the simplified protocol parsing logic is simpler, reducing parsing time from 15ms to 2ms. Combined with QoS level 0 transmission, it greatly improves transmission efficiency and adapts to the resource constraints and real-time requirements of embedded devices.

[0088] In one embodiment, before transmitting the compressed log data in the preset memory pool to the remote server based on the lightweight MQTT protocol, the method further includes: determining whether the total amount of the compressed log data in the preset memory pool reaches a preset quantity threshold; or whether the time elapsed since the last transmission exceeds a preset period.

[0089] In this embodiment, a threshold for the quantity of compressed log data and a threshold for the period time are preset. Before transmission, the total amount of compressed log data in the preset memory pool is read through atomic operations to determine whether the preset quantity threshold has been reached. At the same time, the last transmission completion time is recorded using a nanosecond-level timestamp, and the difference between the current time and the last transmission time is calculated to determine whether the preset period time has been exceeded. As long as any condition is met, the transmission process is immediately triggered. If none of the conditions are met, the process continues to wait until the triggering conditions are met.

[0090] Its beneficial effects are as follows: Compared with the fixed transmission cycle of the traditional solution, the dynamic dual-condition triggering shortens the transmission interval to 500ms, significantly improving the real-time performance of data transmission; the quantity threshold triggering avoids excessive memory usage caused by data backlog in the memory pool, and the periodic time triggering prevents the waste of network resources caused by frequent transmission when the data volume is too small. The combination of the two results in each transmission occupying only 0.3MB of temporary memory, reducing the peak network bandwidth utilization by 42%, which not only ensures the memory stability of the embedded device, but also optimizes the network transmission efficiency, achieving a balance between resource consumption and real-time transmission performance.

[0091] The log collection method based on the MQTT protocol provided in this application acquires multiple raw log data sets and stores them in a circular buffer area according to a non-blocking overlay strategy. The circular buffer area includes a preset number of contiguous storage blocks. Multiple target log data sets are read from the circular buffer area and subjected to hierarchical filtering according to preset filtering rules to obtain multiple filtered log data sets. Each filtered log data set is then compressed using binary encoding to obtain multiple compressed log data sets. Based on a dynamic memory allocation strategy, each compressed log data set is stored in a preset memory pool. Finally, based on a lightweight MQTT protocol, the compressed log data sets in the preset memory pool are transmitted to a remote server. This reduces the memory overhead required for log collection while improving log transmission efficiency and stability.

[0092] Example 2

[0093] In addition, please see Figure 2 This application also provides a log collection system 200 based on the MQTT protocol, comprising:

[0094] Log acquisition module 210 is used to acquire multiple raw log data and store each raw log data in a circular buffer area according to a non-blocking overwrite strategy. The circular buffer area includes a preset number of consecutive storage blocks.

[0095] The filtering module 220 is used to read multiple target log data from the annular buffer area, perform hierarchical filtering on each target log data according to preset filtering rules, and obtain multiple filtered log data.

[0096] The compression encoding module 230 is used to perform binary encoding compression on each of the filtered log data to obtain multiple compressed log data;

[0097] The pre-storage module 240 is used to store each of the compressed log data in a preset memory pool based on a dynamic memory allocation strategy;

[0098] The transmission module 250 is used to transmit the compressed log data in the preset memory pool to a remote server based on the lightweight MQTT protocol.

[0099] The log collection system 200 based on the MQTT protocol provided in this application embodiment can execute the log collection method based on the MQTT protocol provided in the above method embodiment 1. To avoid repetition, it will not be described again here.

[0100] The log collection system 200 based on the MQTT protocol provided in this application embodiment reduces the memory overhead required for log collection while improving log transmission efficiency and stability.

[0101] Example 3

[0102] Furthermore, this embodiment of the invention provides an embedded device 300, including a memory 303 and a processor 302. The memory 303 stores a computer program, and the computer program executes the log collection method based on the MQTT protocol provided in Embodiment 1 when it runs on the processor 302.

[0103] For details, please see Figure 3 The embedded device 300 includes a transceiver 301, a bus interface, and a processor 302. The processor 302 is used to acquire multiple raw log data, store each raw log data in a circular buffer area according to a non-blocking overlay strategy, the circular buffer area including a preset number of contiguous storage blocks; read multiple target log data from the circular buffer area, perform hierarchical filtering on each target log data according to a preset filtering rule to obtain multiple filtered log data; perform binary encoding compression on each of the filtered log data to obtain multiple compressed log data; store each compressed log data in a preset memory pool based on a dynamic memory allocation strategy; and transmit each compressed log data in the preset memory pool to a remote server based on a lightweight MQTT protocol.

[0104] In this embodiment of the application, the embedded device 300 further includes a memory 303. Figure 3In this context, the bus architecture can include any number of interconnected buses and bridges, specifically connecting various circuits of one or more processors 302 (represented by processor 302) and memory 303 (represented by memory 303). The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 301 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 302 is responsible for managing the bus architecture and general processing, and the memory 303 can store data used by the processor 302 during operation.

[0105] The embedded device 300 provided in this embodiment of the invention can execute the log collection method based on the MQTT protocol provided in the above method embodiment 1. To avoid repetition, it will not be described again here.

[0106] Example 4

[0107] Furthermore, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the log collection method based on the MQTT protocol provided in Embodiment 1.

[0108] In this embodiment, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0109] The computer-readable storage medium provided in this embodiment can implement the log collection method based on the MQTT protocol provided in Embodiment 1. To avoid repetition, it will not be described again here.

[0110] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.

[0111] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0112] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A log collection method based on the MQTT protocol, characterized in that, Applied to embedded devices, the method includes: Multiple raw log data are acquired, and each raw log data is stored in a circular buffer area according to a non-blocking overwrite strategy. The circular buffer area includes a preset number of contiguous storage blocks. Multiple target log data are read from the circular buffer area, and each target log data is filtered hierarchically according to a preset filtering rule to obtain multiple filtered log data. Each of the filtered log data is compressed using binary encoding to obtain multiple compressed log data sets. Based on a dynamic memory allocation strategy, the compressed log data is stored in a preset memory pool; Based on the lightweight MQTT protocol, the compressed log data in the preset memory pool is transmitted to a remote server; The number of the multiple filtered log data is N. Each of the filtered log data is subjected to binary encoding compression to obtain multiple compressed log data, including: The i-th filtered log data includes: the i-th prefix identifier and the i-th log field, 1≤i≤N; According to the preset encoding rules, the i-th prefix identifier is converted into an M-bit binary enumeration value, where M is a positive integer and M≥1; For the i-th log field, a variable-length encoding algorithm is used for compression to obtain the corresponding variable-length encoded bitstream; The M-bit binary enumeration value corresponding to the i-th prefix identifier is concatenated with the variable-length encoded bitstream corresponding to the i-th log field to obtain the i-th compressed log data.

2. The log collection method based on the MQTT protocol according to claim 1, characterized in that, The step of reading multiple target log data from the circular buffer area includes: Determine whether the read pointer of the circular buffer area is equal to the write pointer. If so, wait for the original log data to be written to the storage block pointed to by the write pointer. If not, then read the storage block pointed to by the read pointer to obtain the original log data; The original log data is subjected to integrity verification, and the original log data that passes the verification is determined as the target log data.

3. The log collection method based on the MQTT protocol according to claim 2, characterized in that, The target log data includes: a prefix identifier, a path identifier, and log fields; the preset filtering rules include: The log level of the target log data is obtained by querying the preset log level mapping table based on the prefix identifier. If the log level is lower than the preset retention level, then the target log data is filtered. If the log level is not lower than the preset retention level, the path identifier is matched with the disabled path identifier in the preset tree structure. If the identifier matches successfully, the target log data is filtered. If the identifier match fails, the log field is input into a preset regular expression engine for regular expression matching. If the regular expression match succeeds, the target log data is filtered. If the regular expression match fails, the target log data will be determined as the filtered log data.

4. The log collection method based on the MQTT protocol according to claim 1, characterized in that, The dynamic memory allocation strategy, which stores each compressed log data in a preset memory pool, includes: Based on the data length of the i-th compressed log, a target memory block is determined from the preset memory pool. The target memory block is the smallest memory block with a size not less than the data length. Write the i-th compressed log into the target memory block.

5. The log collection method based on the MQTT protocol according to claim 1, characterized in that, Lightweighting the MQTT protocol includes: The message header fields of the MQTT protocol are subject to preset simplification processing, which includes removing preset reserved bits and preset identifier bits; The topic field of the MQTT protocol is processed by binary encoding.

6. The log collection method based on the MQTT protocol according to claim 5, characterized in that, Before transmitting the compressed log data from the preset memory pool to the remote server using the lightweight MQTT protocol, the process further includes: Determine whether the total amount of each compressed log data in the preset memory pool has reached a preset quantity threshold; Alternatively, check if the time elapsed since the last transmission exceeds the preset period.

7. A log collection system based on the MQTT protocol, characterized in that, The system includes: The log acquisition module is used to acquire multiple raw log data and store each raw log data into a circular buffer area according to a non-blocking overwrite strategy. The circular buffer area includes a preset number of consecutive storage blocks. The filtering module is used to read multiple target log data from the ring buffer area, perform hierarchical filtering on each target log data according to preset filtering rules, and obtain multiple filtered log data. The compression encoding module is used to perform binary encoding compression on each of the filtered log data to obtain multiple compressed log data. The pre-storage module is used to store each of the compressed log data in a preset memory pool based on a dynamic memory allocation strategy; The transmission module is used to transmit the compressed log data in the preset memory pool to a remote server based on the lightweight MQTT protocol. The number of the multiple filtered log data is N. Each of the filtered log data is subjected to binary encoding compression to obtain multiple compressed log data, including: The i-th filtered log data includes: the i-th prefix identifier and the i-th log field, 1≤i≤N; According to the preset encoding rules, the i-th prefix identifier is converted into an M-bit binary enumeration value, where M is a positive integer and M≥1; For the i-th log field, a variable-length encoding algorithm is used for compression to obtain the corresponding variable-length encoded bitstream; The M-bit binary enumeration value corresponding to the i-th prefix identifier is concatenated with the variable-length encoded bitstream corresponding to the i-th log field to obtain the i-th compressed log data.

8. An embedded device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and the computer program, when run on the processor, executes the log collection method based on the MQTT protocol as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the log collection method based on the MQTT protocol as described in any one of claims 1-6.

Images