A centerless architecture log collection method for an industrial internet environment

Abstract

The application discloses a kind of centerless architecture log collection methods for industrial internet environment, belong to industrial internet log management technical field.The method includes: obtaining the ability characteristic information broadcasted in each equipment of industrial internet, and based on ability characteristic information, establish equipment capability table;Based on equipment capability table, according to the preset ability evaluation rule, the comprehensive ability score of each equipment is calculated, and according to the comprehensive ability score, log convergence node election is executed, and the log convergence node is determined;By non-log convergence node, local log data is structured and processed, and the processed log data is sent to log convergence node;By log convergence node, the log data received is collected and processed, and log classification, index construction and hierarchical storage are sequentially executed;Real-time acquisition of load data and link data of log convergence node, combined with load data and link data, identify the health state information of log convergence node.

Description

Technical Field

[0001] This application relates to the field of industrial internet log management technology, and more specifically, to a decentralized log collection method for industrial internet environments. Background Technology

[0002] With the rapid development of Industrial Internet technology, a large number of industrial devices, sensors, and edge nodes have been connected to a unified network environment, forming a highly heterogeneous, large-scale, and dynamically changing distributed system. In such systems, to achieve functions such as equipment operation status monitoring, fault diagnosis, performance analysis, and security auditing, it is usually necessary to continuously collect, aggregate, and analyze log data generated by various devices.

[0003] Existing industrial internet log collection methods mostly employ a centralized architecture, relying on fixed log servers or centralized aggregation nodes to uniformly receive and process logs from across the network. However, in the complex network environment of the industrial internet, this centralized architecture has significant limitations. On the one hand, a single central node is prone to becoming a performance bottleneck, easily leading to processing delays or even data loss in high-concurrency log writing scenarios. On the other hand, if a central node fails, the entire log collection chain will be interrupted, affecting the system's availability and reliability. Furthermore, industrial internet nodes typically exhibit dynamic joining, leaving, and frequent changes in resource status, making it difficult for traditional fixed aggregation architectures to adapt to changes in node capabilities in a timely manner, resulting in uneven resource scheduling and poor system scalability.

[0004] On the other hand, while some existing distributed log collection schemes introduce multi-node collaboration mechanisms, they often lack unified capability evaluation standards during node election and coordination, which can easily lead to unreasonable selection of aggregation nodes and thus affect log processing efficiency. Furthermore, during log transmission, the mechanisms for handling network anomalies, node failures, and data migration are inadequate, easily resulting in log loss or duplicate processing, reducing overall data consistency and reliability.

[0005] In summary, how to achieve dynamic election and adaptive switching of log aggregation nodes in the complex and dynamic environment of the Industrial Internet, ensure the continuity and consistency of log data during transmission and processing, and improve the stability and scalability of the log collection system has become an urgent technical problem to be solved. Summary of the Invention

[0006] To overcome a series of shortcomings in existing technologies, the purpose of this application is to provide a decentralized log collection method for industrial internet environments, comprising the following steps: Obtain the capability characteristic information broadcast by each device within the Industrial Internet, and establish a device capability table based on the capability characteristic information; Based on the equipment capability table, calculate the comprehensive capability score of each equipment according to the preset capability evaluation rules, and perform log aggregation node election based on the comprehensive capability score to determine the log aggregation node; The non-log aggregation node performs structured processing on the local log data and then sends the processed log data to the log aggregation node. The log aggregation node collects and processes the received log data, and sequentially performs log classification, index construction, and hierarchical storage. Continuously monitor the operational status of log aggregation nodes, collect load and link data of log aggregation nodes in real time, and identify the health status information of log aggregation nodes by combining load and link data.

[0007] In some embodiments, the method for obtaining the capability characteristic information broadcast by each device in the Industrial Internet is as follows: Collect computing resource information, storage resource information, network resource information, and device stability information of various devices within the Industrial Internet to obtain device capability characteristic data; The equipment capability characteristic data is standardized and encapsulated according to a preset data format to form a capability characteristic data package; According to the preset broadcast cycle, capability feature data packets are sent through the Industrial Internet broadcast protocol. Each device receives and parses capability feature data packets sent by other devices within the Industrial Internet to obtain the capability feature information of the corresponding device.

[0008] In some embodiments, the method for calculating the comprehensive capability score of each device according to preset capability assessment rules is as follows: The capability characteristic information of each device in the Industrial Internet is normalized to obtain normalized computing resource information, storage resource information, network resource information, and device stability information. The computing power sub-score is obtained based on the normalized computing resource information; The storage capacity sub-score is obtained based on the normalized storage resource information; The network capability sub-scores are obtained based on the normalized network resource information; The stability sub-score is obtained based on the normalized equipment stability information; The scores for computing capability, storage capability, network capability, and stability are weighted and summed according to preset evaluation weights to obtain the comprehensive capability score for each device.

[0009] In some embodiments, the method for electing log aggregation nodes based on comprehensive capability scores is as follows: The devices are ranked according to their overall capability scores, and the device with the highest overall capability score is selected as the candidate aggregation node. Determine whether the score difference between the top-ranked devices in terms of overall capability score is less than a preset arbitration threshold; When the score difference is less than the preset arbitration threshold, the devices are sorted in a secondary order based on their MAC addresses, and the devices with smaller MAC address values ​​are identified as candidate aggregation nodes. Compare the identified candidate aggregation nodes with the current log aggregation node; When the candidate aggregation node is inconsistent with the current log aggregation node, the observation process corresponding to the preset debouncing window is started, and the election calculation is continuously performed during the observation process. Determine whether multiple consecutive election results in the observation process all point to the same candidate convergence node; When multiple consecutive election results point to the same candidate aggregation node, the candidate aggregation node is determined as the new log aggregation node and node switching is performed; When the election results are inconsistent in multiple consecutive elections, the current log aggregation node will continue to provide log aggregation services.

[0010] In some embodiments, the method for identifying the health status information of log aggregation nodes by combining load data and link data is as follows: Obtain the load data and link data corresponding to the log aggregation node, and extract the running status indicators; The operational status indicators are evaluated based on preset threshold rules to determine the basic health status of the log aggregation node; The trend of the operating status indicators within a preset time window is analyzed to determine whether there is a continuous deterioration trend; When there is a continuous deterioration trend, the risk level of the basic health status is adjusted. The health status information of the log aggregation node is generated based on the revised risk level.

[0011] In some embodiments, the decentralized architecture log collection method further includes the following steps: The system determines whether the log aggregation node has failed based on its health status information: if a failure is detected, the device capability table is updated and the log aggregation node election is re-executed to determine a new log aggregation node to take over the log collection task; if the system is detected as operating normally, the current log collection task continues to be executed.

[0012] In some embodiments, after a newly determined log aggregation node is declared elected, before it formally receives log data from non-log aggregation nodes, the following steps are also included: Once a new log aggregation node is selected, it broadcasts a takeover readiness notification to the entire network and receives log sequence number range information from each non-log aggregation node. Receive log sequence number range information from each non-log aggregation node and aggregate it to generate log backlog distribution information for the entire network; A point-to-point communication link is established based on the log backlog distribution information. When the original log aggregation node is reachable, log data that has not been indexed and hierarchically stored in its memory buffer is obtained. When the original log aggregation node is unreachable, the corresponding sequence number breakpoint is recorded and the missing interval is marked. Initialize the log receiving queue, log aggregation and processing module, and multi-level index structure based on the log sequence number range and backlog distribution information; Upon completion of the broadcast takeover notification, each non-log aggregation node retransmits log data to the new log aggregation node in descending order of its local cache backlog, and resumes real-time log transmission.

[0013] In some embodiments, the decentralized architecture log collection method further includes the following steps: Continuously monitor changes in the capability characteristics of each device within the Industrial Internet; when the change in the capability characteristics of any device meets the preset reselection conditions, update the device capability table and re-execute the log aggregation node election; if the election result changes, migrate the log data to be processed to the newly determined log aggregation node to complete the log collection service switch; if the election result does not change, maintain the current log aggregation node to continue executing the log collection task.

[0014] In some embodiments, a method for continuously monitoring changes in the capability characteristic information of various devices within the Industrial Internet is as follows: Each node continuously collects local device capability feature information through local resource probes based on a preset basic sampling interval, and generates current capability feature data; The current capability feature data is compared field by field with the capability feature information from the last broadcast to obtain the change range of each field. Compare the magnitude of change in each field with the corresponding preset sensitivity threshold, and mark the fields that have changed; When the number of changed fields reaches the preset trigger threshold or any key field changes, non-periodic active broadcasting is triggered to send the updated capability feature information to the entire network. If the triggering condition is not met, the current state is maintained and the capability feature information collection and comparison process is repeated in the next sampling period.

[0015] In some embodiments, the method for migrating log data to be processed to a newly determined log aggregation node is as follows: After receiving notification of changes in the election results, the original log aggregation node stops receiving new log data sent by non-log aggregation nodes and continues to process the log data to be processed in its local receiving queue until the queue is empty. Log data that has been formatted and collected but has not yet been indexed and hierarchically stored is exported from the memory buffer, packaged into a data migration package, and sent to the new log aggregation node through a point-to-point transmission channel. Synchronize the currently confirmed maximum log sequence number to the new log aggregation node for the purpose of identifying and filtering duplicate log data; After completing the log data migration and sequence number synchronization, a switchover completion notification is broadcast to the entire network, enabling the new log aggregation node to take over the log collection responsibilities. After receiving the switchover completion notification, each non-loggathering node updates the target for subsequent log data transmission to the new loggathering node and completes the log collection path switchover.

[0016] Compared with existing technologies, this application has the following advantages: In the decentralized architecture of the industrial internet, by using a dynamic election mechanism for log aggregation nodes based on comprehensive equipment capability assessment, and combined with node health monitoring and adaptive fault switching, the continuity, stability and scalability of the log collection process are uniformly improved. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a decentralized log collection method for an industrial internet environment disclosed in an embodiment of this application.

[0018] Figure 2 This is a flowchart of the log aggregation node election process in this application embodiment.

[0019] Figure 3 This is a flowchart of log processing and hierarchical storage in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some embodiments of this invention, but not all embodiments.

[0021] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The embodiments and directional terms described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0023] like Figure 1 As shown, a decentralized log collection method for an industrial internet environment includes the following steps: Obtain the capability characteristic information broadcast by each device within the Industrial Internet, and establish a device capability table based on the capability characteristic information; Based on the equipment capability table, calculate the comprehensive capability score of each equipment according to the preset capability evaluation rules, and perform log aggregation node election based on the comprehensive capability score to determine the log aggregation node; The non-log aggregation node performs structured processing on the local log data and then sends the processed log data to the log aggregation node. The log aggregation node collects and processes the received log data, and sequentially performs log classification, index construction, and hierarchical storage. Continuously monitor the operational status of log aggregation nodes, collect load data and link data of log aggregation nodes in real time, and identify the health status information of log aggregation nodes by combining load data and link data. The system determines whether the log aggregation node has failed based on its health status information: if a failure is detected, the device capability table is updated and the log aggregation node election is re-executed to determine a new log aggregation node to take over the log collection task; if the system is determined to be operating normally, the current log collection task continues to be executed. Continuously monitor changes in the capability characteristics of each device within the Industrial Internet; when the change in the capability characteristics of any device meets the preset reselection conditions, update the device capability table and re-execute the log aggregation node election; if the election result changes, migrate the log data to be processed to the newly determined log aggregation node to complete the log collection service switch; if the election result does not change, maintain the current log aggregation node to continue executing the log collection task.

[0024] The decentralized log collection method described in this application efficiently constructs a device capability table through dynamic capability feature perception and adaptive broadcasting mechanisms, ensuring real-time and accurate monitoring of device status within the Industrial Internet. Through multi-dimensional comprehensive capability assessment and intelligent election algorithms, it achieves dynamic election and rapid fault switching of log aggregation nodes, guaranteeing high availability of the log collection service. Furthermore, it performs structured processing, classification, indexing, and hierarchical storage of log data, combined with node health monitoring and adaptive resource adjustment, to achieve efficient log aggregation and reliable management. This significantly improves the real-time performance, stability, and scalability of log collection in the Industrial Internet environment, meeting the log management needs of large-scale distributed industrial scenarios.

[0025] Furthermore, the method for obtaining the capability characteristic information broadcast by each device within the Industrial Internet is as follows: Collect computing resource information, storage resource information, network resource information, and device stability information of various devices within the Industrial Internet to obtain device capability characteristic data; The equipment capability characteristic data is standardized and encapsulated according to a preset data format to form a capability characteristic data package; According to the preset broadcast cycle, capability feature data packets are sent through the Industrial Internet broadcast protocol. Each device receives and parses capability feature data packets sent by other devices within the Industrial Internet to obtain the capability feature information of the corresponding device.

[0026] The method for acquiring capability characteristic information described in this application achieves a comprehensive and accurate characterization of equipment capability characteristics through multi-dimensional resource information collection and standardized encapsulation; it ensures efficient dissemination and synchronization of capability characteristic information within the industrial internet by using a preset broadcast period and industrial internet broadcast protocol; based on this, each device parses the received capability characteristic data packets, providing reliable data support for subsequent equipment capability table construction and comprehensive capability assessment, thereby ensuring the real-time nature and accuracy of equipment capability information and laying a solid foundation for centralized architecture log collection.

[0027] Furthermore, the computing resource information includes the number of CPU cores, CPU clock speed range, current CPU utilization, total memory capacity, and currently available memory capacity; The storage resource information includes total disk capacity, current available capacity, disk read / write speed, and disk health status identifier; The network resource information includes current access bandwidth, network interface throughput, average round-trip latency with other nodes in the industrial internet, and data packet loss rate. The equipment stability information includes the equipment's continuous operating time, the frequency of historical failures, and the current operating temperature.

[0028] The resource information definition method described in this application constructs a comprehensive and systematic equipment capability assessment index system by clearly defining the specific content of key resource information such as computing, storage, network, and stability. The information in each dimension complements each other, accurately reflecting the actual operating status and performance level of the equipment. On this basis, it provides a clear and standardized basis for the collection of equipment capability characteristic data and subsequent comprehensive capability assessment, ensuring the scientific nature and accuracy of capability assessment, thereby guaranteeing the rationality and reliability of log aggregation node election.

[0029] Furthermore, the preset broadcast period is a dynamically adjusted period, which is adaptively adjusted by each device based on its local resource status, specifically including the following rules: When the device's CPU utilization is below 40% and the network interface load is below 30%, the broadcast period remains at the base value T0, where T0 is the default broadcast interval duration configured according to the scale of the industrial internet. When the CPU utilization rate is between 40% and 70% or the network interface load is between 30% and 60%, the broadcast period is adjusted to 1.5 times T0; When the CPU utilization rate is higher than 70% or the network interface load is higher than 60%, the broadcast period is adjusted to twice T0. When the remaining disk capacity of the device is lower than the preset low water level threshold, or the operating temperature in the device stability information exceeds the preset high temperature warning value, the device will add a degradation flag bit to the corresponding data packet when broadcasting capability feature information to indicate that the device is in a resource-constrained state. When other devices receive and update the device capability table, if they detect the downgrade flag, they will reduce the overall capability score of the corresponding device based on a preset reduction coefficient, thereby reducing the probability of the device being selected as the aggregation node in the subsequent election process.

[0030] The dynamic broadcast cycle adjustment method described in this application, through adaptive cycle adjustment based on the local resource status of the device, maintains high-frequency broadcasting when resources are abundant to ensure timely updates of capability characteristic information, and appropriately extends the broadcast cycle when resources are scarce to reduce device resource consumption; by adding a degradation flag and a comprehensive capability score reduction mechanism, it accurately identifies resource-constrained devices and avoids them being over-allocated log aggregation tasks; on this basis, it achieves a dynamic balance between device resource utilization and information synchronization, ensuring the real-time performance and accuracy of the device capability table, and improving the rationality and resource adaptability of log aggregation node election.

[0031] Furthermore, the method for calculating the comprehensive capability score of each device according to the preset capability assessment rules is as follows: The capability characteristic information of each device in the Industrial Internet is normalized to obtain normalized computing resource information, storage resource information, network resource information, and device stability information. The computing power sub-score is obtained based on the normalized computing resource information; The storage capacity sub-score is obtained based on the normalized storage resource information; The network capability sub-scores are obtained based on the normalized network resource information; The stability sub-score is obtained based on the normalized equipment stability information; The scores for computing capability, storage capability, network capability, and stability are weighted and summed according to preset evaluation weights to obtain the comprehensive capability score for each device.

[0032] The comprehensive capability score calculation method described in this application normalizes multi-dimensional capability feature information to eliminate the dimensional differences between different indicators and achieve unified comparability of data. By calculating the scores of each item and weighting and summing them with preset weights, it scientifically integrates key capabilities such as computing, storage, network, and stability to comprehensively and objectively reflect the overall performance of the equipment. On this basis, it provides accurate capability quantification basis for the election of log aggregation nodes, ensuring the rationality and reliability of the election results and guaranteeing the efficient and stable operation of the log collection service.

[0033] Furthermore, such as Figure 2 As shown, the method for electing log aggregation nodes based on comprehensive capability scores is as follows: The devices are ranked according to their overall capability scores, and the device with the highest overall capability score is selected as the candidate aggregation node. Determine whether the score difference between the top-ranked devices in terms of overall capability score is less than a preset arbitration threshold; When the score difference is less than the preset arbitration threshold, the devices are sorted in a secondary order based on their MAC addresses, and the devices with smaller MAC address values ​​are identified as candidate aggregation nodes. Compare the identified candidate aggregation nodes with the current log aggregation node; When the candidate aggregation node is inconsistent with the current log aggregation node, the observation process corresponding to the preset debouncing window is started, and the election calculation is continuously performed during the observation process. Determine whether multiple consecutive election results in the observation process all point to the same candidate convergence node; When multiple consecutive election results point to the same candidate aggregation node, the candidate aggregation node is determined as the new log aggregation node and node switching is performed; When the election results are inconsistent in multiple consecutive elections, the current log aggregation node will continue to provide log aggregation services.

[0034] The log aggregation node election method described in this application determines candidate aggregation nodes by ranking them according to their comprehensive capability scores. It combines a preset arbitration threshold with a secondary ranking based on MAC addresses to avoid election uncertainty caused by similar scores. By initiating a debouncing window observation process, it continuously monitors the stability of the election results to prevent frequent node switching caused by temporary resource fluctuations. On this basis, it achieves stable election and smooth switching of log aggregation nodes, ensuring the continuity and reliability of log collection services and reducing the risk of business interruption caused by node switching.

[0035] Furthermore, such as Figure 3 As shown, the method for structuring local log data by non-log aggregation nodes is as follows: Obtain raw log data generated by non-log aggregation nodes; The raw log data is formatted to obtain standardized log data. Perform field parsing on standardized log data to extract preset log field information; Structured log data is generated based on preset log field information and according to a predefined structured log pattern; The structured log data is compressed and encoded to obtain compressed log data blocks; Integrity check codes are generated based on compressed log data blocks, and the compressed log data blocks and integrity check codes are encapsulated to form log transmission data packets. The log transmission data packets are sent to the log aggregation node according to the preset batch sending strategy.

[0036] The log data structuring method described in this application achieves standardization and structuring of log data by normalizing the format and parsing the fields, facilitating subsequent classification, indexing, and storage. Through compression encoding and integrity check code generation, it reduces the bandwidth consumption of log data transmission, ensuring the integrity and reliability of the log data. By using a preset batch sending strategy, it optimizes log data transmission efficiency and reduces network interaction overhead. Furthermore, it ensures efficient and stable transmission of structured log data from non-log aggregation nodes to the log aggregation node, laying a solid foundation for subsequent log aggregation processing.

[0037] Furthermore, the non-log aggregation node is configured with a local persistent log cache queue. When a network connection interruption, communication timeout, or acknowledgment failure is detected with the current log aggregation node, the non-log aggregation node suspends log transmission and writes the log transmission data packets to be sent into the local persistent log cache queue in the order of their generation. The local persistent log cache queue allocates storage priorities according to log levels, with ERROR and FATAL level logs having higher priorities than INFO and DEBUG level logs. When the cache queue reaches a preset capacity threshold, log transmission data packets are evicted in order of priority from low to high to ensure the effective retention of critical fault logs. After the connection is restored, the non-log aggregation node sends a cache synchronization request to the log aggregation node and obtains the latest acknowledgment log sequence number returned by the log aggregation node. Based on the log sequence number, the non-log aggregation node identifies the log transmission data packets that have not been acknowledged and retransmits them in the original generation order. After the retransmission is completed, the normal log transmission process resumes.

[0038] The local persistent log caching mechanism described in this application automatically triggers log caching through network anomaly detection to avoid log data loss due to network interruption; it prioritizes the reliable storage of critical fault logs through priority allocation based on log level and a tiered eviction strategy under capacity threshold; and it ensures the complete resending and sequential consistency of unconfirmed log data through cache synchronization and sequence number retransmission mechanisms after connection recovery. On this basis, it realizes reliable caching and recovery transmission of log data in network anomaly scenarios, ensuring the continuity and integrity of log collection services and effectively improving the fault tolerance of the system.

[0039] Furthermore, the method for log classification performed by the log aggregation node is as follows: Obtain structured log data; Load source-based classification rules, level-based classification rules, and content-based classification rules; The source of structured log data is identified based on the source dimension classification rules to obtain the source classification results; The structured log data is divided into levels according to the level dimension classification rules to obtain the level classification results; When the classification results meet the preset alarm conditions, a corresponding alarm record is generated; Semantic matching of log message content in structured log data is performed based on content dimension classification rules to obtain content classification results; Log classification results are established based on source classification results, level classification results, content classification results, and alarm records.

[0040] The log classification method described in this application accurately identifies and classifies structured log data from multiple perspectives, including source, level, and content, by loading multi-dimensional classification rules; it automatically triggers alarm record generation through preset alarm conditions, enabling rapid early warning of key abnormal logs; it establishes a complete log classification system by comprehensively considering multi-dimensional classification results, providing a clear classification basis for subsequent index construction and hierarchical storage; and on this basis, it achieves refined classification management of log data, improves log retrieval efficiency and anomaly monitoring capabilities, and meets the diverse log analysis needs in the industrial internet environment.

[0041] Furthermore, the method for index construction performed by the log aggregation node is as follows: The log aggregation node establishes a time-series index mapping relationship between time information and log entry identifier, an inverted index mapping relationship between category tags and log entry identifier, and a full-text index mapping relationship between keywords and log entry identifier based on the log classification results. After the log data is collected and written, the storage location information of the corresponding log entry in the storage medium is obtained, and the storage location information is associated with the corresponding index item to form a log index system that includes time-series index, category inverted index and full-text index; Perform incremental updates and maintenance on the index entries corresponding to newly added logs, and enable retrieval access to the corresponding log entries after the index update is completed.

[0042] The index construction method described in this application achieves multi-dimensional and rapid retrieval of log data by establishing time-series, inverted, and full-text multi-dimensional index mapping relationships; it ensures accurate correspondence between the index and actual data by associating log entry storage location information, thereby improving retrieval efficiency; and it guarantees the real-time performance and integrity of the index system through an incremental update maintenance mechanism, avoiding the performance overhead of full updates. On this basis, it constructs an efficient and flexible log index system that supports rapid location and query of log data, significantly improving the response speed and accuracy of log retrieval and meeting the retrieval needs of large-scale log data.

[0043] Furthermore, such as Figure 3 As shown, the method for performing hierarchical storage by the log aggregation node is as follows: Log data that has been collected and written is stored in the hot storage layer and the corresponding index structure is retained; Monitor the retention status of log data in the hot storage layer, and migrate the log data to the warm storage layer when the hot layer migration conditions are met, while performing data compression and index optimization. Monitor the retention status of log data in the warm storage layer, and migrate the log data to the cold storage layer when the warm layer migration conditions are met, while performing archive encoding processing; The capacity limit and retention threshold of each storage layer are dynamically adjusted according to preset configuration parameters to achieve hierarchical storage management of log data.

[0044] The hierarchical storage method described in this application divides log data into three storage levels—hot, warm, and cold—based on access frequency. Combined with dynamic monitoring and migration condition judgment, it achieves intelligent hierarchical migration of log data. Through data compression, index optimization, and archiving encoding, it optimizes the space utilization and access efficiency of different storage levels. By dynamically adjusting the capacity and retention threshold of each storage level through preset configuration parameters, it ensures the rational allocation of storage resources. Based on this, it achieves efficient storage and cost optimization of log data, balancing data access performance and storage costs, and meeting the long-term storage and fast access needs of large-scale log data in the industrial internet environment.

[0045] Furthermore, the method for real-time collection of load data and link data from log aggregation nodes is as follows: Within the log aggregation node, load data such as CPU utilization, memory usage, disk I / O status, log receiving queue backlog, and log processing latency are collected according to a preset sampling period. Each non-log aggregation node sends a liveness probe data packet to the log aggregation node, and obtains link data such as link round-trip time, data packet arrival rate, and number of consecutive acknowledgment failures based on the liveness probe results; The link data is reported along with the log data to the log aggregation node; The log aggregation node aggregates and stores the collected load data and received link data to form operational status data.

[0046] The load data and link data collection methods described in this application collect multi-dimensional load data within the log aggregation node at a preset sampling period to comprehensively reflect the node's resource consumption and processing pressure; obtain link data by sending liveness detection data packets from non-log aggregation nodes to accurately grasp the communication quality between nodes; and form a complete operational status dataset by reporting and aggregating the link data along with the logs; based on this, real-time and comprehensive data support is provided for the identification of the health status and fault determination of the log aggregation node, ensuring accurate monitoring and rapid response of the node's operational status.

[0047] Furthermore, the method for identifying the health status information of log aggregation nodes by combining load data and link data is as follows: Obtain the load data and link data corresponding to the log aggregation node, and extract the running status indicators; The operational status indicators are evaluated based on preset threshold rules to determine the basic health status of the log aggregation node; The trend of the operating status indicators within a preset time window is analyzed to determine whether there is a continuous deterioration trend; When there is a continuous deterioration trend, the risk level of the basic health status is adjusted. The health status information of the log aggregation node is generated based on the revised risk level.

[0048] The health status identification method described in this application extracts operational status indicators from load and link data, combines them with preset threshold rules to assess the basic health status of nodes, and achieves rapid preliminary judgment. By analyzing the changing trends of operational status indicators within a preset time window, it identifies potential deterioration risks and performs risk correction on the basic health status. By generating corrected health status information, it accurately reflects the actual operational risks of nodes. On this basis, it achieves dynamic and accurate identification of the health status of log aggregation nodes, providing a reliable basis for fault diagnosis and node switching, and ensuring the stable operation of log collection services.

[0049] Furthermore, the method for updating the device capability table and re-executing the log aggregation node election is as follows: When a log aggregation node is determined to have failed, the fault status information is broadcast and the entire network is triggered to enter a fast reselection process. Each non-log aggregation node responds to the fault status information, stops sending log data to the original log aggregation node, and writes the unsent or unacknowledged log data to the local persistent cache; Each node updates its local device capability table, marks the original log aggregation node as faulty and offline, and obtains the comprehensive capability score information of the remaining valid nodes based on the updated device capability table. Each node performs a fast election calculation based on its locally cached capability feature data to determine new candidate log aggregation nodes and assess the consistency of the election results. If there is any inconsistency, a unique candidate aggregation node is determined through lightweight negotiation and preset MAC address arbitration rules. The system establishes a log receiving relationship with a designated node as the new log aggregation node and restores the log aggregation service, while controlling the node switching process to complete within a preset maximum switching latency threshold.

[0050] The fault reselection method described in this application triggers a rapid reselection process across the entire network through fault status information broadcasting, ensuring timely fault response; it prevents log data loss by having non-log aggregation nodes stop sending data to the faulty node and cache unacknowledged logs; it marks the faulty node by updating the device capability table and quickly performs election calculations based on locally cached data, combining lightweight negotiation and MAC arbitration to ensure the uniqueness of the election results; it ensures rapid service switching by controlling node switching to be completed within a preset time delay; and on this basis, it achieves rapid and reliable switching after a log aggregation node failure, minimizing the impact of the fault on the log collection service and ensuring service continuity.

[0051] Furthermore, after being declared elected, the newly determined log aggregation node performs a takeover initialization step before officially receiving log data from non-log aggregation nodes: Once a new log aggregation node is selected, it broadcasts a takeover readiness notification to the entire network and receives log sequence number range information from each non-log aggregation node. Receive log sequence number range information from each non-log aggregation node and aggregate it to generate log backlog distribution information for the entire network; A point-to-point communication link is established based on the log backlog distribution information. When the original log aggregation node is reachable, log data that has not been indexed and hierarchically stored in its memory buffer is obtained. When the original log aggregation node is unreachable, the corresponding sequence number breakpoint is recorded and the missing interval is marked. Initialize the log receiving queue, log aggregation and processing module, and multi-level index structure based on the log sequence number range and backlog distribution information; Upon completion of the broadcast takeover notification, each non-log aggregation node retransmits log data to the new log aggregation node in descending order of its local cache backlog, and resumes real-time log transmission.

[0052] The takeover initialization method described in this application collects backlog information from the entire network through broadcast takeover readiness notifications to accurately grasp the distribution of logs to be processed; it obtains unprocessed logs or marked missing intervals from the original nodes by establishing point-to-point links to ensure the integrity of log data; it initializes the receiving queue and processing module based on the backlog information to ensure that the new node has complete log processing capabilities; it guides non-log aggregation nodes to retransmit in an orderly manner according to the backlog volume through broadcast takeover completion notifications to restore real-time transmission; on this basis, it achieves smooth takeover of the new log aggregation node and seamless connection of log data, avoiding log loss and processing chaos during the takeover process, and ensuring a smooth transition of log collection services.

[0053] Furthermore, the method for continuously monitoring changes in the capability characteristics of various devices within the Industrial Internet is as follows: Each node continuously collects local device capability feature information through local resource probes based on a preset basic sampling interval, and generates current capability feature data; The current capability feature data is compared field by field with the capability feature information from the last broadcast to obtain the change range of each field. Compare the magnitude of change in each field with the corresponding preset sensitivity threshold, and mark the fields that have changed; When the number of changed fields reaches the preset trigger threshold or any key field changes, non-periodic active broadcasting is triggered to send the updated capability feature information to the entire network. If the triggering condition is not met, the current state is maintained and the capability feature information collection and comparison process is repeated in the next sampling period.

[0054] The method for monitoring changes in the capability characteristics of the aforementioned devices in this application continuously collects device capability characteristic data at preset sampling intervals to achieve routine monitoring of device status; it accurately identifies changes in capability characteristics through field-by-field comparison and sensitivity threshold judgment; it controls non-periodic active broadcasting through trigger conditions to ensure timely synchronization of key changes across the entire network, avoiding resource consumption caused by invalid broadcasts; it maintains the repeated execution of the sampling cycle when not triggered to ensure the continuity of monitoring; on this basis, it achieves dynamic and accurate monitoring and efficient synchronization of device capability characteristic information, providing timely triggering basis for updating the device capability table and reselecting nodes, and ensuring the dynamic adaptability of the decentralized architecture.

[0055] Furthermore, the preset reselection conditions include the following independent judgment rules, which are executed in parallel and do not affect each other. When any rule is satisfied, the device capability table update and re-election process are triggered: The first type is the absolute threshold change rule, which is triggered when the change in the comprehensive ability score of any node compared to the value of the previous election exceeds a preset percentage threshold. The second category is the relative ranking change rule, which is triggered when the ranking of the comprehensive capability scores of each node in the device capability table changes compared with the previous election results, and the current aggregation node is no longer ranked first. It does not respond to ranking changes between non-log aggregation nodes that do not affect the ranking of the current aggregation node. The third category is the new node addition rule, which is triggered when a new device that is not recorded in the device capability table appears in the industrial internet and the capability characteristic information of the new device remains stable online for at least two consecutive broadcast cycles. The fourth category is node exit rules, which are triggered immediately when any node broadcasts capability feature information carrying an active exit flag.

[0056] The aforementioned method for setting preset reselection conditions in this application comprehensively covers key scenarios such as changes in device capabilities and dynamic additions and removals of nodes by setting four independent judgment rules: changes in absolute thresholds, changes in relative rankings, addition of new nodes, and removal of nodes. Each rule is executed in parallel without affecting the others, ensuring that the reselection process is initiated in a timely manner when any triggering condition is met. Through targeted rule design, key changes affecting the rationality of log aggregation nodes are accurately captured, avoiding invalid reselections. On this basis, the device capability table update and node reselection are accurately triggered, ensuring that the log aggregation node is always the device with the best overall capabilities, thereby improving the rationality and stability of the log collection service.

[0057] Furthermore, the method for migrating the log data to be processed to the newly determined log aggregation node is as follows: After receiving notification of changes in the election results, the original log aggregation node stops receiving new log data sent by non-log aggregation nodes and continues to process the log data to be processed in its local receiving queue until the queue is empty. Log data that has been formatted and collected but has not yet been indexed and hierarchically stored is exported from the memory buffer, packaged into a data migration package, and sent to the new log aggregation node through a point-to-point transmission channel. Synchronize the currently confirmed maximum log sequence number to the new log aggregation node for the purpose of identifying and filtering duplicate log data; After completing the log data migration and sequence number synchronization, a switchover completion notification is broadcast to the entire network, enabling the new log aggregation node to take over the log collection responsibilities. After receiving the switchover completion notification, each non-loggathering node updates the target for subsequent log data transmission to the new loggathering node and completes the log collection path switchover.

[0058] The log data migration method described in this application ensures complete processing of pending logs by stopping the original node from receiving new logs and processing its local queue; it ensures no data loss by exporting and encapsulating unprocessed log data for migration; it identifies duplicate data by synchronizing confirmed sequence numbers to avoid duplicate processing; and it guides non-log aggregation nodes to update their sending targets through broadcast switching notifications, thus completing the path switch. Based on this, it achieves complete and orderly migration of log data during node switching, ensuring seamless integration of log collection services and ensuring the continuity and consistency of log data after the new node takes over.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A decentralized log collection method for an industrial internet environment, characterized in that, Includes the following steps: Obtain the capability characteristic information broadcast by each device within the Industrial Internet, and establish a device capability table based on the capability characteristic information; Based on the equipment capability table, calculate the comprehensive capability score of each equipment according to the preset capability evaluation rules, and perform log aggregation node election based on the comprehensive capability score to determine the log aggregation node; The non-log aggregation node performs structured processing on the local log data and then sends the processed log data to the log aggregation node. The log aggregation node collects and processes the received log data, and sequentially performs log classification, index construction, and hierarchical storage. Continuously monitor the operational status of log aggregation nodes, collect load and link data of log aggregation nodes in real time, and identify the health status information of log aggregation nodes by combining load and link data.

2. The decentralized log collection method according to claim 1, characterized in that, The method for obtaining the capability characteristic information broadcast by each device within the Industrial Internet is as follows: Collect computing resource information, storage resource information, network resource information, and device stability information of various devices within the Industrial Internet to obtain device capability characteristic data; The equipment capability characteristic data is standardized and encapsulated according to a preset data format to form a capability characteristic data package; According to the preset broadcast cycle, capability feature data packets are sent through the Industrial Internet broadcast protocol. Each device receives and parses capability feature data packets sent by other devices within the Industrial Internet to obtain the capability feature information of the corresponding device.

3. The decentralized log collection method according to claim 2, characterized in that, The method for calculating the comprehensive capability score of each device according to the preset capability assessment rules is as follows: The capability characteristic information of each device in the Industrial Internet is normalized to obtain normalized computing resource information, storage resource information, network resource information, and device stability information. The computing power sub-score is obtained based on the normalized computing resource information; The storage capacity sub-score is obtained based on the normalized storage resource information; The network capability sub-scores are obtained based on the normalized network resource information; The stability sub-score is obtained based on the normalized equipment stability information; The scores for computing capability, storage capability, network capability, and stability are weighted and summed according to preset evaluation weights to obtain the comprehensive capability score for each device.

4. The decentralized log collection method according to claim 3, characterized in that, The method for electing the aggregation node based on the comprehensive capability score is as follows: The devices are ranked according to their overall capability scores, and the device with the highest overall capability score is selected as the candidate aggregation node. Determine whether the score difference between the top-ranked devices in terms of overall capability score is less than a preset arbitration threshold; When the score difference is less than the preset arbitration threshold, the devices are sorted in a secondary order based on their MAC addresses, and the devices with smaller MAC address values ​​are identified as candidate aggregation nodes. Compare the identified candidate aggregation nodes with the current log aggregation node; When the candidate aggregation node is inconsistent with the current log aggregation node, the observation process corresponding to the preset debouncing window is started, and the election calculation is continuously performed during the observation process. Determine whether multiple consecutive election results in the observation process all point to the same candidate convergence node; When multiple consecutive election results point to the same candidate aggregation node, the candidate aggregation node is determined as the new log aggregation node and node switching is performed; When the election results are inconsistent in multiple consecutive elections, the current log aggregation node will continue to provide log aggregation services.

5. The decentralized log collection method according to claim 1, characterized in that, The method for identifying the health status information of log aggregation nodes by combining load data and link data is as follows: Obtain the load data and link data corresponding to the log aggregation node, and extract the running status indicators; The operational status indicators are evaluated based on preset threshold rules to determine the basic health status of the log aggregation node; The trend of the operating status indicators within a preset time window is analyzed to determine whether there is a continuous deterioration trend; When there is a continuous deterioration trend, the risk level of the basic health status is adjusted. The health status information of the log aggregation node is generated based on the revised risk level.

6. The decentralized log collection method according to claim 1, characterized in that, The decentralized architecture log collection method also includes the following steps: The system determines whether the log aggregation node has failed based on health status information. When a failure is detected, the device capability table is updated and the log aggregation node election is re-executed to determine a new log aggregation node to take over the log collection task. If the system is determined to be running normally, continue executing the current log collection task.

7. The decentralized log collection method according to claim 6, characterized in that, After a newly selected log aggregation node is declared elected, before it officially receives log data from non-log aggregation nodes, the following steps are also included: Once a new log aggregation node is selected, it broadcasts a takeover readiness notification to the entire network and receives log sequence number range information from each non-log aggregation node. Receive log sequence number range information from each non-log aggregation node and aggregate it to generate log backlog distribution information for the entire network; A point-to-point communication link is established based on the log backlog distribution information. When the original log aggregation node is reachable, log data that has not been indexed and hierarchically stored in its memory buffer is obtained. When the original log aggregation node is unreachable, the corresponding sequence number breakpoint is recorded and the missing interval is marked. Initialize the log receiving queue, log aggregation and processing module, and multi-level index structure based on the log sequence number range and backlog distribution information; Upon completion of the broadcast takeover notification, each non-log aggregation node retransmits log data to the new log aggregation node in descending order of its local cache backlog, and resumes real-time log transmission.

8. The decentralized log collection method according to claim 6, characterized in that, The decentralized architecture log collection method also includes the following steps: Continuously monitor changes in the capability characteristics of each device within the Industrial Internet; when the change in the capability characteristics of any device meets the preset reselection conditions, update the device capability table and re-execute the log aggregation node election; If the election results change, the log data to be processed will be migrated to the newly determined log aggregation node to complete the log collection service switch. If the election results remain unchanged, the current log aggregation node will continue to perform the log collection task.

9. The decentralized log collection method according to claim 8, characterized in that, The method for continuously monitoring changes in the capability characteristics of various devices within the Industrial Internet is as follows: Each node continuously collects local device capability feature information through local resource probes based on a preset basic sampling interval, and generates current capability feature data; The current capability feature data is compared field by field with the capability feature information from the last broadcast to obtain the change range of each field. Compare the magnitude of change in each field with the corresponding preset sensitivity threshold, and mark the fields that have changed; When the number of changed fields reaches the preset trigger threshold or any key field changes, non-periodic active broadcasting is triggered to send the updated capability feature information to the entire network. If the triggering condition is not met, the current state is maintained and the capability feature information collection and comparison process is repeated in the next sampling period.

10. The decentralized log collection method according to claim 8, characterized in that, The method for migrating log data to be processed to the newly determined log aggregation node is as follows: After receiving notification of changes in the election results, the original log aggregation node stops receiving new log data sent by non-log aggregation nodes and continues to process the log data to be processed in its local receiving queue until the queue is empty. Log data that has been formatted and collected but has not yet been indexed and hierarchically stored is exported from the memory buffer, packaged into a data migration package, and sent to the new log aggregation node through a point-to-point transmission channel. Synchronize the currently confirmed maximum log sequence number to the new log aggregation node for the purpose of identifying and filtering duplicate log data; After completing the log data migration and sequence number synchronization, a switchover completion notification is broadcast to the entire network, enabling the new log aggregation node to take over the log collection responsibilities. After receiving the switchover completion notification, each non-loggathering node updates the target for subsequent log data transmission to the new loggathering node and completes the log collection path switchover.

Images