Mosquitto log management system based on ARM (Advanced RISC Machines) processor

Abstract

The invention provides a Mosquitto log management system based on an ARM processor, and the system comprises a log generation end which is used for generating log data and controlling a sending behavior through a condition triggering mechanism; the log storage end is used for storing the log data generated by the log generation end; the user control end is used for sending a log viewing request and a storage level adjusting instruction; the log transmission module is used for transmitting log data to the log storage end or the user control end and transmitting a control instruction to the log generation end; the ARM processor is used for executing log data processing and analysis and coordinating operation of each module; and the log generation end only sends key log data to the log storage end when not receiving a request of the user control end, and sends full log data to the user control end when receiving a log viewing request of the user control end. According to the method, the flexibility and the resource utilization rate of the log system can be improved, the storage cost and the system load are reduced, and the expandability and the stability of the system are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of computer log management, and more particularly to a Mosquitto log management system based on an ARM processor. Background Art

[0002] In today's embedded systems and IoT applications, log management is a vital tool for system operation monitoring, troubleshooting, and performance optimization. Traditional log management systems are typically based on general-purpose processor architectures and employ centralized log storage and synchronous transmission. When processing large amounts of log data, these systems often face problems such as high resource usage, low transmission efficiency, and high storage costs. For example, the real-time transmission and storage of full log data may lead to network bandwidth bottlenecks and insufficient storage space, while the loss of critical log data may affect the timeliness and accuracy of troubleshooting. In addition, existing systems lack a flexible resource scheduling mechanism when faced with dynamically changing system loads, making it difficult to effectively balance the performance requirements of log generation, transmission, and storage.

[0003] In the process of implementing the embodiments of the present invention, the inventors found that there are at least the following problems or defects in the prior art: the existing log management system lacks flexibility in log generation strategy and cannot dynamically adjust the log storage level according to user needs; the log transmission efficiency is low, especially in a heterogeneous network environment, and cannot effectively handle the concurrent transmission of large-scale log data; the system resource utilization rate is low, and computing resources cannot be dynamically allocated according to the real-time load, resulting in a decrease in system performance under high load conditions and serious waste of resources under low load conditions. Summary of the Invention

[0004] The present invention provides a Mosquitto log management system based on an ARM processor, comprising:

[0005] The log generation end is used to generate log data and control the sending behavior through the conditional trigger mechanism;

[0006] A log storage terminal, used to store log data generated by the log generation terminal;

[0007] User control terminal, used to send log viewing requests and storage level adjustment instructions;

[0008] A log transmission module, used to transmit log data to the log storage end or the user control end, and transmit control instructions to the log generation end;

[0009] ARM processor, used to perform log data processing and analysis and coordinate the operation of various modules;

[0010] The log generating end only sends key log data to the log storing end when no request is received from the user controlling end, and sends full log data to the user controlling end when a log viewing request is received from the user controlling end.

[0011] Furthermore, the conditional triggering mechanism specifically includes:

[0012] When the log transmission module receives a log viewing request sent by the user control end, it sends a trigger instruction to the log generation end;

[0013] After the log generation end responds to the trigger instruction, the currently generated log data is transmitted to the log transmission module in full mode.

[0014] Furthermore, the user control terminal includes:

[0015] Real-time log interface, used to display the full amount of log data forwarded by the log transmission module;

[0016] The storage level adjustment unit is used to generate a control instruction containing a target storage level and send it to the log generation end through the log transmission module.

[0017] Furthermore, the log generation end includes:

[0018] A log grading unit, configured to dynamically adjust the log generation strategy based on received storage level control instructions;

[0019] The log filtering unit is used to filter log data that is lower than the current storage level.

[0020] Furthermore, the log storage terminal includes:

[0021] Storage medium selection module, used to switch between database storage mode and file system storage mode;

[0022] The log index module is used to create timestamp indexes and event type indexes for stored log data.

[0023] Furthermore, the ARM processor includes:

[0024] Multi-core parallel processing unit for distributed processing of log data streams;

[0025] The resource scheduling module is used to dynamically allocate computing resources to each module based on the current system load.

[0026] Furthermore, the log transmission module includes:

[0027] The control command channel uses a dedicated message queue to transmit the control commands issued by the user control terminal;

[0028] The data return channel uses multiplexing technology to transmit data streams from different log generation ends in parallel.

[0029] Furthermore, the key log data includes:

[0030] Device operation status abnormality logs, communication connection failure logs, protocol parsing error logs, and storage space alarm logs.

[0031] Furthermore, the full log data includes:

[0032] Full records of equipment operating status, complete payload of communication data packets, protocol interaction timing information, and detailed resource usage data.

[0033] Furthermore, an asynchronous transmission mechanism is adopted between the log generation end and the log transmission module, specifically including:

[0034] Set up a ring buffer to temporarily store log data to be transmitted;

[0035] Use batch packaging to transmit log data packets;

[0036] Implement flow control algorithm to dynamically adjust transmission rate.

[0037] According to the above-mentioned embodiments of the present invention, there are at least the following beneficial effects: First, by introducing a conditional trigger mechanism and a log grading unit, the present invention can dynamically adjust the log generation strategy according to user needs, and transmit the full amount of log data only when necessary, thereby effectively reducing the resource consumption of log storage and transmission while ensuring the integrity of key log data. At the same time, the use of asynchronous transmission mechanism and multiplexing technology can significantly improve the transmission efficiency of log data, avoiding network congestion and transmission delay problems caused by excessive log data volume. In addition, the multi-core parallel processing unit and resource scheduling module in the ARM processor can dynamically allocate computing resources according to the system load, further improving the overall performance and resource utilization of the system.

[0038] Secondly, the present invention also has significant advantages in terms of user experience and system flexibility. The real-time log interface and storage level adjustment unit on the user control side provide users with an intuitive operating platform, allowing users to view the full amount of log data or adjust the log storage level at any time according to actual needs, thereby enhancing the operability and adaptability of the system. At the same time, the storage medium selection module and log index module on the log storage side provide flexible storage mode switching and efficient data retrieval functions, further improving the practicality and ease of use of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily apparent by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present invention are shown by way of example and not limitation, in which:

[0040] Figure 1 This is the data flow diagram when there is no user viewing the log in real time in this system;

[0041] Figure 2 Request message flow chart for users of this system to view real-time logs;

[0042] Figure 3 This is a flow chart of the full log data when users of this system view real-time logs;

[0043] Figure 4 Message flow chart for requesting to adjust the storage log level for users of this system;

[0044] Figure 5 Log data flow diagram after adjusting the storage log level for users of this system. DETAILED DESCRIPTION

[0045] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0046] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0047] It should be noted that any number of elements in the drawings is for illustration only and not for limitation, and any naming is only for distinction and does not have any limiting meaning.

[0048] A Mosquitto log management system based on an ARM processor includes:

[0049] The log generation end is used to generate log data and control the sending behavior through the conditional trigger mechanism;

[0050] A log storage terminal, used to store log data generated by the log generation terminal;

[0051] User control terminal, used to send log viewing requests and storage level adjustment instructions;

[0052] A log transmission module, used to transmit log data to the log storage end or the user control end, and transmit control instructions to the log generation end;

[0053] ARM processor, used to perform log data processing and analysis and coordinate the operation of various modules;

[0054] The log generating end only sends key log data to the log storing end when no request is received from the user controlling end, and sends full log data to the user controlling end when a log viewing request is received from the user controlling end.

[0055] It should be noted that the present invention relates to a Mosquitto log management system based on an ARM processor. The system includes a log generation end, a log storage end, a user control end, a log transmission module and an ARM processor. The log generation end is used to generate log data and control the sending behavior of the log data through a conditional trigger mechanism. The log storage end is used to store the log data generated by the log generation end. The user control end is used to send log viewing requests and storage level adjustment instructions. The log transmission module is used to transmit log data to the log storage end or the user control end, and transmit control instructions to the log generation end. The ARM processor is used to perform log data processing and analysis, and coordinate the operation of each module. When the log generation end does not receive a request from the user control end, it only sends key log data to the log storage end; when it receives a log viewing request from the user control end, it sends the full amount of log data to the user control end.

[0056] Specifically, the log generation end includes a log generation module and a conditional trigger mechanism. The log generation module can generate different types of log data according to the operating status of the system, such as device operating status logs, communication connection logs, etc. The conditional trigger mechanism is used to control the sending behavior of log data, and can decide whether to send log data according to preset conditions or trigger events. The log storage end can use a database or file system as a storage medium to store the log data generated by the log generation end. The user control end includes a user interface through which the user can send log viewing requests and storage level adjustment instructions. The log transmission module can use a message queue or other transmission protocol to transmit log data and control instructions between modules. The ARM processor can perform log data processing and analysis tasks, and coordinate the operation of each module to ensure the efficient operation of the system.

[0057] More specifically, the log generation end can be set to generate different levels of log data at different stages of system operation, for example, only generate key log data during normal operation, and generate full log data during troubleshooting. The conditional trigger mechanism can trigger the sending of log data based on a variety of methods such as time intervals, event triggers or user instructions. The log storage end can select a suitable storage strategy based on the different storage media, such as using an indexing mechanism in database mode to improve query efficiency, and using a hierarchical storage strategy in file system mode to optimize storage space. The user interface of the user control end can provide a variety of operation options, and users can easily view log data, adjust storage levels and set trigger conditions. The log transmission module can use multiplexing technology to achieve parallel transmission of log data and improve transmission efficiency. The ARM processor can dynamically adjust the resource allocation of each module according to the system load to ensure that the system runs efficiently while achieving optimal resource utilization.

[0058] In some embodiments, the conditional triggering mechanism specifically includes:

[0059] When the log transmission module receives a log viewing request sent by the user control end, it sends a trigger instruction to the log generation end;

[0060] After the log generation end responds to the trigger instruction, the currently generated log data is transmitted to the log transmission module in full mode.

[0061] It should be noted that the conditional trigger mechanism in the present invention is one of the core components of the log management system. When a user sends a log viewing request through the user control terminal, the log transmission module will receive the request and send a trigger instruction to the log generation terminal. After receiving the trigger instruction, the log generation terminal will transfer the currently generated log data to the log transmission module in full mode. The conditional trigger mechanism here refers to a response mechanism based on specific events or conditions, which is used to control the transmission mode of log data. It can dynamically adjust the transmission strategy of log data according to the user's real-time needs, thereby achieving efficient and flexible log management.

[0062] Specifically, the working principle of the conditional trigger mechanism can be further explained as follows: After receiving the log viewing request sent by the user control end, the log transmission module will parse the request and generate a corresponding trigger instruction. The trigger instruction is a signal used to notify the log generation end to change the log transmission mode. After receiving the trigger instruction, the log generation end will switch its log transmission mode from the default key log data transmission mode to the full log data transmission mode. In this process, the generation of the trigger instruction can be parameterized according to the specific content of the user request, such as specifying the log type, time range or log of a specific event to be viewed. When responding to the trigger instruction, the log generation end will classify and package the currently generated log data according to the preset rules to ensure that the full log data can be transmitted to the user control end in a complete and accurate manner.

[0063] Preferably, the conditional trigger mechanism can be further refined and optimized. For example, the trigger instruction can contain more detailed parameters, such as the priority of the log data, the transmission rate limit or the data format requirements, etc., to meet the needs of different user scenarios. In addition, when responding to the trigger instruction, the log generation end can adopt a multi-threaded or asynchronous processing mechanism to improve the processing efficiency of the log data. At the same time, in order to improve the fault tolerance and stability of the system, a heartbeat detection mechanism can be set between the log transmission module and the log generation end to ensure the reliable transmission of the trigger instruction. In an alternative solution, the conditional trigger mechanism can also be combined with an artificial intelligence algorithm to automatically predict and adjust the log transmission mode according to the usage of historical log data and user behavior patterns, thereby further improving the intelligence level of the system.

[0064] In some embodiments, the user control terminal includes:

[0065] Real-time log interface, used to display the full amount of log data forwarded by the log transmission module;

[0066] The storage level adjustment unit is used to generate a control instruction containing a target storage level and send it to the log generation end through the log transmission module.

[0067] It should be noted that the user control terminal is a key component in the log management system of the present invention for realizing interaction between the user and the system. It includes a real-time log interface and a storage level adjustment unit, which are used to display log data and generate control instructions. The real-time log interface is a visual window for users to view log data. It can display the full amount of log data forwarded by the log transmission module in real time, helping users to quickly understand the system operation status. The storage level adjustment unit is used to generate control instructions containing the target storage level and send them to the log generation terminal through the log transmission module, thereby realizing dynamic adjustment of the log generation strategy. This design allows users to flexibly adjust the operating mode of the log system according to actual needs, enhancing the operability and adaptability of the system.

[0068] Specifically, the real-time log interface of the user control end can adopt a graphical interface design and support multiple display modes, such as displaying logs in chronological order, displaying them by event type, or displaying log data in a table. Users can quickly locate the log content of interest through the filtering function in the interface. The storage level adjustment unit can generate control instructions based on user input, where the storage level can be set to multiple levels, such as critical, warning, information, and debug, and each level corresponds to a different range of log data. Users can select the target storage level through a drop-down menu or input box, and click the send button to send the control instruction to the log generation end through the log transmission module. In addition, the real-time log interface can also set the refresh frequency, such as once per second or once per minute, to meet the different real-time requirements of users.

[0069] Preferably, the real-time log interface on the user control side can further optimize the user experience. For example, a search function for log data can be added, allowing users to enter keywords or regular expressions for quick searches. Furthermore, the interface can support log data export, allowing users to save log data as text files or Excel spreadsheets for subsequent analysis. In the storage level adjustment unit, more granular storage level settings can be introduced, such as allowing users to customize storage levels or set storage level priorities.

[0070] Furthermore, to improve the system's interactive efficiency, the user control terminal can establish a persistent connection with the log transmission module to reduce transmission delays in control commands. Alternatively, the storage level adjustment unit can be integrated with user rights management to assign different storage level adjustment permissions based on user roles, further enhancing the system's security and flexibility.

[0071] In some embodiments, the log generation end includes:

[0072] A log grading unit, configured to dynamically adjust the log generation strategy based on received storage level control instructions;

[0073] The log filtering unit is used to filter log data that is lower than the current storage level.

[0074] It should be noted that the log generation terminal plays a core role in the log management system of the present invention and includes a log grading unit and a log filtering unit. The log grading unit is used to dynamically adjust the log generation strategy based on the received storage level control instructions, thereby achieving refined management of log data generation. The log filtering unit is used to filter log data below the current storage level, ensuring that the system only processes and stores log information that meets user needs. This design enables the log generation terminal to flexibly respond to users' storage requirements for log data while optimizing the efficiency of system resource utilization.

[0075] Specifically, the log grading unit can dynamically adjust the level of log generation based on the control instructions sent by the user through the storage level adjustment unit. For example, the control instructions can specify the generation of log data of different levels such as critical, warning, information or debugging. The log filtering unit will filter out log data that does not meet the requirements based on the currently set storage level. For example, if the storage level is set to warning, the log filtering unit will filter out log data of information and debugging levels, and only retain log data of critical and warning levels. In addition, the log grading unit can also set the format and content of the log data, such as adding timestamps, event types or device identification information to facilitate subsequent log analysis and storage.

[0076] Preferably, the log grading unit can further refine its functionality, for example by introducing more storage level options, such as severe and normal, to meet the needs of different user scenarios. Furthermore, the log filtering unit can incorporate intelligent algorithms, such as machine learning-based anomaly detection algorithms, to automatically identify and filter out potentially useless log data, further improving system efficiency.

[0077] Furthermore, the log generator can set up a log data caching mechanism, for example, caching a certain amount of log data locally and uploading it in batches to the log storage end when network conditions permit. Alternatively, the log filtering unit can support user-defined filtering rules, such as allowing users to filter by specific keywords, event types, or time ranges, thereby achieving more personalized log management capabilities.

[0078] In some embodiments, the log storage terminal includes:

[0079] Storage medium selection module, used to switch between database storage mode and file system storage mode;

[0080] The log index module is used to create timestamp indexes and event type indexes for stored log data.

[0081] It should be noted that the log storage terminal is an important component of the log management system of the present invention, and its function is to achieve efficient storage and rapid retrieval of log data. The log storage terminal includes a storage medium selection module and a log indexing module. The storage medium selection module is used to switch between database storage mode and file system storage mode to adapt to different storage needs and performance requirements. The log indexing module is used to establish a timestamp index and an event type index for the stored log data, thereby improving the retrieval efficiency of the log data. This design enables the log storage terminal to flexibly cope with different usage scenarios while ensuring rapid access and efficient management of log data.

[0082] Specifically, the storage medium selection module can switch between database storage mode and file system storage mode according to the user's configuration or the system's performance requirements. The database storage mode is generally suitable for scenarios that require frequent queries and complex data operations, such as database systems that support SQL queries; while the file system storage mode is suitable for scenarios that require large-scale data storage and fast writing, such as distributed file systems. Users can set the switching conditions of the storage mode through the system interface or configuration file, such as selecting based on factors such as the amount of log data, query frequency, or system load. The log indexing module enables users to quickly locate log records within a specific time range or a specific event type by adding timestamp indexes and event type indexes to log data. The establishment of indexes can be based on key information such as the generation time of log data, event type, device identification, etc., thereby achieving efficient data retrieval.

[0083] Preferably, the storage medium selection module can further optimize its functions, such as introducing an intelligent switching mechanism to automatically select the optimal storage mode based on real-time system performance indicators (such as disk I / O utilization, CPU load, etc.). In addition, the log indexing module can support more types of indexes, such as keyword indexes or geographic location indexes, to meet complex query requirements. As an alternative, the log storage end can combine data compression technology to compress the stored log data to save storage space and improve storage efficiency. At the same time, the log indexing module can support dynamic update functions, allowing users to add or modify index fields in real time during the storage process, thereby further enhancing the flexibility and scalability of the system.

[0084] In some embodiments, the ARM processor comprises:

[0085] Multi-core parallel processing unit for distributed processing of log data streams;

[0086] The resource scheduling module is used to dynamically allocate computing resources to each module based on the current system load.

[0087] It should be noted that the ARM processor plays a core processing and coordination role in the log management system of the present invention. The ARM processor includes a multi-core parallel processing unit and a resource scheduling module, wherein the multi-core parallel processing unit is used to perform distributed processing on the log data stream, which can effectively improve the processing efficiency of the log data; the resource scheduling module is used to dynamically allocate the computing resources of each module according to the current system load to ensure the efficient operation of the system. This design enables the ARM processor to give full play to the advantages of its multi-core architecture, and at the same time, flexibly respond to different system load conditions through the resource scheduling module, thereby improving the performance and stability of the entire log management system.

[0088] Specifically, the multi-core parallel processing unit can distribute the log data stream to different processing cores for parallel processing according to the type of log data and processing requirements. For example, key log data can be distributed to high-performance cores for real-time processing, while the full amount of log data can be distributed to other cores for batch processing. The resource scheduling module can dynamically adjust the computing resources allocated to each module according to the real-time situation of the system load. For example, when the system detects that the load on the log generation end is high, the resource scheduling module can prioritize the allocation of more computing resources to the log generation end to ensure timely processing of log data. In addition, the ARM processor can also support multiple operating systems and programming languages, providing flexibility for system development and expansion.

[0089] Preferably, the multi-core parallel processing unit of the ARM processor can further optimize its task allocation strategy, such as introducing a load balancing algorithm to ensure that the load is evenly distributed among the processing cores, avoiding situations where some cores are overloaded while other cores are idle. The resource scheduling module can be combined with a machine learning algorithm to predict future system load conditions based on historical load data, and make resource allocation adjustments in advance, thereby further improving the system's response speed and resource utilization. As an alternative, the ARM processor can support hardware acceleration functions, such as using a dedicated coprocessor to quickly parse and compress log data, thereby further improving the system's processing efficiency. In addition, the resource scheduling module can also support user-defined resource allocation strategies, allowing users to flexibly configure according to specific application scenarios and needs.

[0090] In some embodiments, the log transmission module includes:

[0091] The control command channel uses a dedicated message queue to transmit the control commands issued by the user control terminal;

[0092] The data return channel uses multiplexing technology to transmit data streams from different log generation ends in parallel.

[0093] It should be noted that the log transmission module is responsible for the transmission of log data and control instructions in the log management system of the present invention. This module includes a control instruction channel and a data return channel, wherein the control instruction channel adopts a dedicated message queue to transmit the control instruction sent by the user control terminal to ensure the reliability and transmission efficiency of the instruction; the data return channel adopts multiplexing technology to transmit the data stream of different log generation terminals in parallel, to improve the efficiency and stability of data transmission. This design enables the log transmission module to efficiently process a large amount of log data and control instructions, while ensuring the real-time performance and reliability of the system.

[0094] Specifically, the control instruction channel is implemented through a dedicated message queue. The message queue is an efficient communication mechanism that ensures the reliable transmission of control instructions between the user control end and the log generation end. The message queue can set parameters such as queue length, priority, and timeout according to the actual needs of the system to meet the transmission requirements in different scenarios. The data return channel uses multiplexing technology, which allows multiple data streams to be transmitted in parallel on the same physical channel, thereby significantly improving data transmission efficiency. For example, log data generated by different log generation ends can be transmitted through time division or multi-channel multiplexing to reduce transmission delay and improve bandwidth utilization.

[0095] Preferably, the message queue of the control instruction channel can be further optimized, for example, by introducing a priority scheduling mechanism to assign different priorities according to the urgency of the control instructions, ensuring that high-priority instructions are processed first. The multiplexing technology of the data return channel can be combined with the flow control algorithm to dynamically adjust the transmission rate of each data stream to avoid network congestion caused by bursts of traffic in some data streams. In addition, the log transmission module can also support data encryption to ensure the security of log data during transmission. As an alternative, the data return channel can adopt a distributed transmission architecture to disperse log data to multiple nodes for transmission, further improving the system's fault tolerance and scalability.

[0096] In some embodiments, the key log data includes:

[0097] Device operation status abnormality logs, communication connection failure logs, protocol parsing error logs, and storage space alarm logs.

[0098] It should be noted that critical log data, in the log management system of this invention, refers to log information that is of significant value for system operation and troubleshooting. This data includes logs of abnormal device operation, communication connection failures, protocol parsing errors, and storage space warnings. The definition of critical log data ensures that only information critical to system operation and maintenance is transmitted and stored when no user access request is received. This optimizes the use of storage resources and network bandwidth, while ensuring that critical information can be quickly retrieved for troubleshooting and resolution when necessary.

[0099] Specifically, the device operation status abnormality log refers to abnormal status information detected by the system during operation, such as hardware failure, system crash or resource exhaustion. The communication connection failure log records the situation where communication between devices is interrupted or the connection cannot be established, which may involve network failure or configuration error. The protocol parsing error log reflects the parsing failure event caused by protocol mismatch or data format error during data transmission. The storage space alarm log is used to prompt that the system storage space is insufficient and needs to be cleaned up or the storage capacity expanded in time. The generation of these log data can be achieved through the system's built-in detection mechanism, such as setting threshold trigger conditions. When the system status or resource usage exceeds the preset threshold, the corresponding key log data is automatically generated. In addition, the recording format of key log data can include information such as timestamp, event type, device identification and error description to facilitate subsequent analysis and problem location.

[0100] Preferably, the generation of key log data can be combined with a real-time monitoring module to continuously monitor the system's operating status. For example, a scheduled task can be set to regularly check system resource usage, and key log data can be generated immediately if an anomaly is detected. In addition, the system can introduce intelligent analysis algorithms to perform preliminary analysis of key log data, extract core information, and generate summaries so that users can quickly understand the severity and scope of the problem. As an alternative, key log data can be combined with an alarm mechanism. When a serious anomaly is detected, not only will a log be generated, but operations and maintenance personnel can also be notified in real time via SMS, email, or system notifications, thereby further improving the system's fault response capabilities.

[0101] In some embodiments, the full log data includes:

[0102] Full records of equipment operating status, complete payload of communication data packets, protocol interaction timing information, and detailed resource usage data.

[0103] It's important to note that full log data refers to all detailed log information generated during system operation. This information covers a complete record of every aspect, including device operating status, communication interactions, protocol execution, and resource usage. The definition of full log data is intended to provide users with a comprehensive view of system operations, enabling them to obtain the richest possible information when conducting in-depth analysis, troubleshooting, or performance optimization. Compared to key log data, full log data has a higher information density and more comprehensive coverage, but it also requires more storage space and transmission bandwidth.

[0104] Specifically, the full log data includes a complete record of the device's operating status, that is, all status information generated by the system during operation, such as CPU usage, memory usage, disk I / O operations, etc.; the complete payload of the communication data packet, which records all data content transmitted during communication between devices, including complete information of requests and responses; protocol interaction timing information, which records the interaction process of the communication protocol in detail, including timestamps, protocol status changes, etc.; and resource usage details, covering the allocation, release and use of system resources. The generation of these log data can be achieved through the system's built-in logging module. For example, during system operation, each operation and event is recorded in real time and stored in a preset format. The format of the log data can include fields such as timestamp, event type, device identification, data content, etc., for subsequent query and analysis.

[0105] Preferably, the generation of full log data can be combined with data compression technology to reduce storage space usage and transmission bandwidth requirements. For example, a lossless compression algorithm can be used to compress log data while retaining all original information. In addition, the system can set up a segmented storage mechanism for log data to store full log data in segments according to time or size for easy management and query. As an alternative, full log data can be combined with a data backup mechanism to regularly back up log data to an external storage device or cloud storage to prevent data loss and improve data security. At the same time, in order to improve the readability and ease of use of log data, a log parsing tool can be introduced to format the full log data, extract key information and generate a visual report to facilitate users to quickly understand and analyze.

[0106] In some embodiments, an asynchronous transmission mechanism is used between the log generation terminal and the log transmission module, specifically including:

[0107] Set up a ring buffer to temporarily store log data to be transmitted;

[0108] Use batch packaging to transmit log data packets;

[0109] Implement flow control algorithm to dynamically adjust transmission rate.

[0110] It should be noted that the asynchronous transmission mechanism between the log generation end and the log transmission module is an important technical means for optimizing data transmission efficiency and system performance in the log management system of the present invention. This mechanism temporarily stores log data to be transmitted by arranging a ring buffer, adopts batch packaging mode to transmit log data packets, and implements a flow control algorithm to dynamically adjust the transmission rate, thereby realizing efficient and stable log data transmission. This design can avoid network congestion and system performance degradation caused by instant transmission when the log generation end produces a large amount of data, while ensuring that data can be transferred to the log transmission module in a timely and complete manner.

[0111] Specifically, the ring buffer is a first-in-first-out (FIFO) data storage structure used to temporarily store log data to be transmitted generated by the log generation end. Its size can be set according to the system's memory resources and the log data generation rate, for example, set to 1MB or larger to meet the needs of different scenarios. Batch packaging refers to combining multiple log data items into one data packet for transmission, which can reduce the overhead during the transmission process and improve transmission efficiency. The size of the data packet can be optimized according to the network bandwidth and transmission delay, for example, set to between 1KB and 10KB. The flow control algorithm dynamically adjusts the data transmission rate according to the real-time status of the network, for example, reducing the transmission rate when network congestion is detected, and increasing the transmission rate when the network status is good, thereby ensuring the stability and reliability of data transmission.

[0112] Preferably, the size of the ring buffer can be dynamically adjusted according to the actual load of the system, for example, automatically expanding the capacity when the log data generation rate is high to avoid data loss. At the same time, the batch packaging method can introduce data compression technology to further reduce the size of the data packet and improve transmission efficiency. The flow control algorithm can be combined with machine learning technology to predict the optimal transmission rate based on historical data and real-time network status, thereby achieving more intelligent flow control. As an alternative, a multi-level buffer mechanism can be introduced to store log data in buffers of different levels according to priority, giving priority to transmitting high-priority log data to meet the needs of different scenarios. In addition, a data verification mechanism can be combined to ensure the integrity and accuracy of the data during transmission.

[0113] The above-mentioned various embodiments of the present invention have the following beneficial effects: First, the present invention can make full use of the efficient computing power and low power consumption characteristics of the ARM processor through its design to achieve efficient generation, transmission and storage of log data. The log generation end adopts a conditional trigger mechanism and an asynchronous transmission mechanism, which can only send key log data when no user request is received, thereby reducing unnecessary data transmission, reducing system load and network bandwidth occupancy. When a user request is received, it can quickly switch to the full log data transmission mode to ensure the integrity and timeliness of the log data.

[0114] Secondly, the design of the user control terminal and the log storage terminal can further enhance the flexibility and operability of the system. The user control terminal can send log viewing requests and storage level adjustment instructions, display the full amount of log data through the real-time log interface, and dynamically adjust the log generation strategy through the storage level adjustment unit. The log storage terminal can switch between the database storage mode and the file system storage mode, and establish a timestamp index and an event type index through the log index module to achieve efficient data retrieval and management. In addition, the multi-core parallel processing unit and the resource scheduling module in the ARM processor can dynamically allocate computing resources according to the system load, further enhancing the overall performance and resource utilization of the system. Through these designs, the present invention can significantly improve the efficiency, flexibility and stability of the log management system.

[0115] like Figure 1 As shown in the figure, when no user is viewing logs in real time, the log generation end sends only critical log data to the log storage end. Critical log data includes abnormal device operating status, communication connection failures, protocol parsing errors, and storage space alarms. The log storage end is responsible for storing this critical log data and can switch between database storage mode and file system storage mode through the storage medium selection module. In addition, the log indexing module creates timestamp and event type indexes for stored log data to facilitate subsequent retrieval and analysis.

[0116] like Figure 2 As shown in the figure. When the user control terminal sends a log viewing request, the log transmission module receives the request and sends a trigger instruction to the log generation terminal. After the log generation terminal responds to the trigger instruction, it transmits the currently generated log data to the log transmission module in full mode. The user control terminal includes a real-time log interface for displaying the full log data forwarded by the log transmission module. The full log data includes a complete record of device operating status, the complete payload of communication data packets, protocol interaction timing information, and detailed resource usage data.

[0117] like Figure 3 As shown in the figure, after receiving a log viewing request from the user control terminal, the log generation terminal sends the full log data to the user control terminal via the log transmission module. The log transmission module includes a control instruction channel that uses a dedicated message queue to transmit control instructions issued by the user control terminal. Furthermore, the data return channel uses multiplexing technology to transmit data streams from different log generation terminals in parallel, ensuring efficient transmission of the full log data.

[0118] like Figure 4As shown in the figure, the user control end includes a storage level adjustment unit, which generates a control instruction containing the target storage level and sends it to the log generation end via the log transmission module. The log generation end also includes a log grading unit, which dynamically adjusts the log generation strategy based on the received storage level control instruction. Furthermore, the log filtering unit filters log data below the current storage level, ensuring that the log generation end only generates and sends log data that meets the user-specified storage level.

[0119] like Figure 5 As shown in the figure, the log generation end dynamically adjusts the log generation strategy based on the user's adjusted storage level through the log grading unit and uses the log filtering unit to filter log data below the current storage level. The adjusted log data is transmitted to the log storage end for storage via the log transmission module. The log storage end switches between database storage mode and file system storage mode through the storage medium selection module and uses the log indexing module to establish timestamp indexes and event type indexes for the stored log data.

[0120] Furthermore, the storage medium of the embodiment of the present application stores program instructions that can implement all the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, or a tablet.

[0121] The above descriptions are merely some preferred embodiments of the present invention and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by mutually replacing the above-mentioned features with (but not limited to) technical features having similar functions disclosed in the embodiments of the present invention.

Claims

1. A Mosquitto log management system based on ARM processor, characterized in that: include: The log generation end is used to generate log data and control the sending behavior through the conditional trigger mechanism; A log storage terminal, used to store log data generated by the log generation terminal; User control terminal, used to send log viewing requests and storage level adjustment instructions; A log transmission module, used to transmit log data to the log storage end or the user control end, and transmit control instructions to the log generation end; ARM processor, used to perform log data processing and analysis and coordinate the operation of various modules; The log generating end only sends key log data to the log storing end when no request is received from the user controlling end, and sends full log data to the user controlling end when a log viewing request is received from the user controlling end.

2. The system according to claim 1, wherein: The conditional triggering mechanism specifically includes: When the log transmission module receives a log viewing request sent by the user control end, it sends a trigger instruction to the log generation end; After the log generation end responds to the trigger instruction, the currently generated log data is transmitted to the log transmission module in full mode.

3. The system according to claim 1, wherein: The user control terminal includes: Real-time log interface, used to display the full amount of log data forwarded by the log transmission module; The storage level adjustment unit is used to generate a control instruction containing a target storage level and send it to the log generation end through the log transmission module.

4. The system according to claim 3, characterized in that The log generation end includes: A log grading unit, configured to dynamically adjust the log generation strategy based on received storage level control instructions; The log filtering unit is used to filter log data that is lower than the current storage level.

5. The system according to claim 1, wherein: The log storage end includes: Storage medium selection module, used to switch between database storage mode and file system storage mode; The log index module is used to create timestamp indexes and event type indexes for stored log data.

6. The system according to claim 1, wherein: The ARM processor comprises: Multi-core parallel processing unit for distributed processing of log data streams; The resource scheduling module is used to dynamically allocate computing resources to each module based on the current system load.

7. The system according to claim 1, wherein: The log transmission module includes: The control command channel uses a dedicated message queue to transmit the control commands issued by the user control terminal; The data return channel uses multiplexing technology to transmit data streams from different log generation ends in parallel.

8. The system according to claim 1, wherein: The key log data includes: Device operation status abnormality logs, communication connection failure logs, protocol parsing error logs, and storage space alarm logs.

9. The system according to claim 1, wherein: The full log data includes: Full records of equipment operating status, complete payload of communication data packets, protocol interaction timing information, and detailed resource usage data.

10. The system according to claim 1, wherein: The asynchronous transmission mechanism is adopted between the log generation end and the log transmission module, specifically including: Set up a ring buffer to temporarily store log data to be transmitted; Use batch packaging to transmit log data packets; Implement flow control algorithm to dynamically adjust transmission rate.

Images