Multi-source voucher data acquisition method under distributed proxy framework

Abstract

The invention discloses a multi-source voucher data acquisition method under a distributed proxy framework, and the method comprises the steps: constructing an acquisition system through cluster arrangement of lightweight probes, extracting key information of an application layer through unpacking processing, executing deep grabbing only for a key high-risk window, and avoiding the influence on the original business performance due to the occupation of excessive resources; and meanwhile, the type and version of new online equipment can be automatically identified, and the dynamic expansion of the acquisition range is realized. According to the method, by integrating an operation and maintenance platform asset management library and a corresponding data access interface, network view data collected by a probe is supplemented, and the collection dimension is perfected; and the data acquisition integrity is guaranteed through a local cache and breakpoint resume mechanism, the security of the data in a transmission link is ensured in combination with an encryption transmission mode, and finally efficient, safe and low-interference acquisition of the multi-source voucher data is realized.

Description

Technical Field

[0001] This invention belongs to the field of network security and data acquisition technology, and relates to a method for acquiring multi-source credential data under a distributed proxy framework. Background Technology

[0002] Currently, multi-source data acquisition systems generally adopt a single-agent or centralized acquisition architecture, obtaining data from different system interfaces, logs, and databases through a unified acquisition node. This type of architecture has the following problems:

[0003] Poor scalability: The central node is under too much pressure and cannot effectively support the collection of multi-source heterogeneous and high-concurrency data.

[0004] Insufficient real-time performance: Each acquisition channel is executed serially, and network latency or node abnormalities can easily cause overall delays.

[0005] Weak security: Credential information is mostly transmitted and stored in a centralized manner, posing a risk of leakage, and lacks end-side isolation and authorization mechanisms.

[0006] Difficult to configure dynamically: Adding new data sources or collection strategies requires manual intervention and cannot achieve adaptive expansion. Summary of the Invention

[0007] This invention aims to provide a multi-source credential data collection method under a distributed proxy framework. By using a distributed proxy mechanism, the method achieves autonomy at the collection end and coordination at the center, thereby improving data collection efficiency and system scalability while ensuring security.

[0008] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:

[0009] A method for collecting multi-source credential data under a distributed proxy framework includes the following operations:

[0010] 1) Deploy collection probes in a cluster manner on multiple nodes of the target network to form multiple collection agent nodes, which then form a distributed data collection network with the central control node;

[0011] 2) The probe adopts a local traffic mirroring and differentiated sampling strategy to collect multi-source credential data for trust calculation and risk identification from the nearest source. The probe has a high-risk business window identification rule, which only captures data from the identified key high-risk windows and performs regular data collection on non-high-risk windows.

[0012] The captured multi-source credential data is unpacked to extract key application layer information.

[0013] 3) The lightweight probe automatically identifies the type and version information of newly launched devices through periodic cross-validation;

[0014] 4) After field extraction and formatting are performed locally, the data collected by each acquisition agent node of the distributed data acquisition network is encrypted and transmitted.

[0015] 5) The lightweight probe is equipped with a local cache module. If the network is interrupted during transmission, the collected data will be temporarily stored locally and resumed to the message bus after the network is restored.

[0016] 6) The data analysis center receives data transmitted from each node via the message bus;

[0017] The data analysis center is also connected to the asset management library of the operation and maintenance platform through an access interface. It obtains management data sources through the data access interface and integrates them with the underlying real-time data collected by probes.

[0018] Furthermore, the probe is deployed in the form of a containerized or service process, which includes:

[0019] Device-side probes, deployed on host or security devices, collect data including operating status, performance indicators, and alarm information;

[0020] Network-side probes are deployed at core switches or network egress nodes to collect data including network traffic and 5-tuple information; the 5-tuple information includes IP address, source port, destination IP address, destination port, and transmission protocol.

[0021] System-side probes are deployed on business servers or log servers to collect data including system logs and application-layer interaction information.

[0022] All probes are registered through the registration center, and the central node uniformly issues collection strategies and credentials.

[0023] Furthermore, each data acquisition agent node registers with the central control node upon startup and periodically reports its heartbeat and resource usage status. The central control node monitors the status of each node in real time.

[0024] When a node is detected to be overloaded or a new node is added online, the central control node redistributes tasks to achieve dynamic balancing. If a node is disconnected or in an abnormal state, the central control node marks it as offline and transfers its data collection tasks to a backup or standby node in the same area.

[0025] The distributed data acquisition network uses NTP or PTP for unified time synchronization; each node probe synchronizes its clock with the central time source upon startup and periodically calibrates it during operation; when data is reported, the probes of the acquisition agent node use the local synchronized clock to generate an acquisition timestamp, and the data analysis center performs another unified time difference correction at the receiving end, so that the time accuracy error of the entire network data is less than the millisecond level.

[0026] Furthermore, among the data collected by the probe, data related to identity and device is used for subject-object identification; data related to behavior and traffic is used for dynamic trust modeling; data related to logs and alarms is used for risk verification and tracing; and data related to configuration is used to assist in risk propagation analysis and strategy generation.

[0027] The probes are deployed in nodes of the distributed power system network. The probes are configured with multi-protocol compatible modules to support the parsing and adaptation of IEC61850, DNP3, MQTT, RESTfulAPI, and LoRaWAN protocols.

[0028] Furthermore, the central control node schedules each agent node through methods including module trimming mechanism, sampling frequency limiting and frequency limiting strategy, resource self-monitoring mechanism, and remote control, so as to keep the resource utilization rate of the probe within the preset threshold range;

[0029] The module pruning mechanism is as follows: the probe loads the acquisition plugin on demand, and only the acquisition plugins required by the current strategy are enabled;

[0030] The sampling and frequency limiting strategy is as follows: high-frequency data, including traffic and logs, is sampled and uploaded in batches.

[0031] Resource self-monitoring mechanism: The probe periodically detects its own CPU and memory usage, and automatically reduces the collection frequency or suspends some tasks when the threshold is exceeded;

[0032] Remote control: The analysis center can remotely adjust the granularity and frequency of data collection through the central control node, so that the probes of different devices operate within the set resource threshold range.

[0033] Furthermore, the probe employs a local traffic mirroring and differentiated sampling strategy as follows:

[0034] The service traffic copy is generated by mirroring the network interface card port or replicating the virtual switching layer. The probe only parses the copy data and does not participate in the original service forwarding. The sampling rate is dynamically adjusted according to the data type, and high-frequency repetitive data is sampled at a low frequency.

[0035] The system collects all key business interaction data and parses key fields of the network layer, transport layer, and application layer in the mirrored traffic, supporting dynamic granular adjustment of the parsing depth. The key business is determined through a three-fold logic of pre-screening with identifiers, precise verification of the matching pool, and dynamic adjustment and adaptation: first, the business code and priority label in the data stream are used for preliminary screening; then, a second verification is performed with the key business matching pool containing features such as storage business ID and device range; finally, the identifiers and matching pool contents are dynamically updated as the business iterates.

[0036] Furthermore, the identification rules for the high-risk window are issued by the data analysis center, and include:

[0037] Time-based behavioral characteristic rules: Identify high-risk periods based on the temporal distribution patterns of historical attacks or anomalies;

[0038] Traffic anomaly rules: Risk windows are determined based on characteristics including traffic surges, abnormal growth in connection counts, and surges in access frequency. The thresholds for traffic anomaly rules include: instantaneous traffic increase exceeding 200% of the baseline value, concurrent connection count surge exceeding 300% of the historical average, and access frequency per unit time exceeding a preset threshold.

[0039] Event triggering rules: triggered based on attack, unauthorized access, or authentication failure events reported by upstream security devices; event triggering rules receive event data from upstream devices through standardized interfaces, including attack type, event level, and scope of impact;

[0040] Policy association rules: Set trigger conditions for specific commands or frame features of key business nodes; policy association rules match preset key business operation features by parsing business protocol fields;

[0041] Self-learning rules: High-risk behavior patterns are summarized from historical samples and the identification rules are dynamically updated. The self-learning rules use machine learning algorithms, including clustering / classification algorithms, to train the time, traffic, and protocol characteristics of historical high-risk events and generate a dynamic identification model.

[0042] Furthermore, the process of automatically identifying the type and version information of newly launched devices includes: if a device appears for the first time in the traffic or ARP list collected by the probe, but there is no matching record in the historical asset database or local cache, it is determined to be a newly launched device; at the same time, the data analysis center detects the communication behavior characteristics and manufacturer identification of the device to supplement the initial asset attributes;

[0043] Once a new device or asset change is detected, the system sends an update event to the analysis center via a message queue.

[0044] The analysis center detects API calls that update CMDB or local asset database records.

[0045] Furthermore, the preliminary field extraction and formatting is a lightweight preprocessing of the collected raw traffic or log data, mainly including field extraction, structuring, desensitization, and encoding unification;

[0046] The field extraction involves extracting key fields related to trust calculation and risk analysis from the raw data packets or logs. Key fields include timestamp, source / destination IP, port, protocol, session ID, and event type.

[0047] The formatting process involves converting the extracted fields into a standardized structure and desensitizing or hashing sensitive information.

[0048] The encrypted transmission includes encrypting the data using a TLS encrypted channel before transmission; in the event of network anomalies or when the data analysis center is unreachable, the probe node temporarily writes the collected data into a local cache queue; after the fault is recovered, the cached data will be automatically retransmitted to the message bus.

[0049] The local cache module adopts a circular queue storage structure. When the cached data reaches the preset capacity limit, the old data is overwritten according to the timestamp priority, and the cached data is marked with an integrity verification identifier.

[0050] When the cached data reaches the preset capacity limit, the oldest cached data is automatically cleaned up according to the first-in-first-out principle, and an alarm is generated and reported to the central control node at the same time. If the connection is restored, the probe will send the latest data that has not been uploaded first, and asynchronously re-upload the remaining part to ensure that the critical data is uploaded first. Historical data that exceeds the cache limit will be summarized before being cleaned up.

[0051] Furthermore, the data analysis center periodically synchronizes management data sources, including device lists, IP address allocations, asset attributes, and network mapping information, by calling access interfaces provided by the operation and maintenance platform, CMDB, and / or DHCP server, including APIs, database views, and Syslog interfaces.

[0052] The data analysis center manages the data and collects the underlying real-time data through probes. By associating and matching key fields including IP, MAC, and asset number, the context information of the network session is completed, and a closed-loop association from traffic behavior to asset identity is achieved.

[0053] Compared with the prior art, the present invention has the following beneficial technical effects:

[0054] This invention constructs a data collection system using cluster-deployed probes. The probes prioritize capturing data from multiple sources, extracting key application-layer information after unpacking, and possess multi-protocol compatibility to adapt to different scenarios. A differentiated strategy is employed during collection, performing deep crawling only on key high-risk windows to avoid excessive resource consumption and impacting existing business performance. It also supports automatic identification of the type and version of newly deployed devices, enabling dynamic expansion of the collection scope. Furthermore, by integrating the asset management library and corresponding data access interfaces of the operations and maintenance platform, network-perspective data collected by the probes is supplemented, improving the collection dimensions. Local caching and breakpoint resumption mechanisms ensure data integrity during collection, while encrypted transmission ensures data security during transmission, ultimately achieving efficient, secure, and low-interference collection of multi-source credential data.

[0055] After adopting a distributed proxy architecture, the average concurrency of data collection increased by more than 3 times, and the overall system throughput was significantly improved; the credential hierarchical authorization mechanism effectively reduced the risk of sensitive credentials being exposed, achieving secure and compliant data collection; asynchronous transmission enabled the system to maintain task continuity and data integrity even under node failures or network fluctuations.

[0056] The sampling frequency and parsing rules of all probes are uniformly defined by the center, eliminating the need to operate each probe individually. Dynamic synchronization is achieved by pushing update notifications through the registry center (such as Zookeeper). Probes listen in real time or periodically pull new strategies. By listening to the node status of the registry center (such as load and online status), task allocation or migration is triggered, and synchronization and effectiveness are achieved automatically without manual intervention. The cluster uses NTP (Network Time Protocol) or PTP (Precision Time Protocol) for unified time synchronization, significantly reducing manual configuration and maintenance costs. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the data acquisition architecture of the present invention.

[0058] Figure 2 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0060] See Figures 1-2 A method for collecting multi-source credential data under a distributed proxy framework, comprising the following operations:

[0061] 1) Deploy collection probes in a cluster manner on multiple nodes of the target network to form multiple collection agent nodes, which then form a distributed data collection network with the central control node;

[0062] 2) The probe adopts a local traffic mirroring and differentiated sampling strategy to collect multi-source credential data for trust calculation and risk identification from the nearest source. The probe has a high-risk business window identification rule, which only captures data from the identified key high-risk windows and performs regular data collection on non-high-risk windows.

[0063] The captured multi-source credential data is unpacked to extract key application layer information.

[0064] 3) The lightweight probe automatically identifies the type and version information of newly launched devices through periodic cross-validation;

[0065] 4) After field extraction and formatting are performed locally, the data collected by each acquisition agent node of the distributed data acquisition network is encrypted and transmitted.

[0066] 5) The lightweight probe is equipped with a local cache module. If the network is interrupted during transmission, the collected data will be temporarily stored locally and resumed to the message bus after the network is restored.

[0067] 6) The data analysis center receives data transmitted from each node via the message bus;

[0068] The data analysis center is also connected to the asset management library of the operation and maintenance platform through an access interface. It obtains management data sources through the data access interface and integrates them with the underlying real-time data collected by probes.

[0069] Furthermore, the probe is deployed in the form of a containerized or service process, which includes:

[0070] Device-side probes, deployed on host or security devices, collect data including operating status, performance indicators, and alarm information;

[0071] Network-side probes are deployed at core switches or network egress nodes to collect data including network traffic and 5-tuple information; the 5-tuple information includes IP address, source port, destination IP address, destination port, and transmission protocol.

[0072] System-side probes are deployed on business servers or log servers to collect data including system logs and application-layer interaction information.

[0073] All probes are registered through the registration center, and the central node uniformly issues collection strategies and credentials to achieve a distributed, controllable, and scalable collection system.

[0074] Specifically, the lightweight probe uses a software probe deployment method, which achieves localized data collection by installing a lightweight collection agent on the target network or host node.

[0075] The probes are deployed in the form of containers or service processes, eliminating the need for manual wiring. They can be configured simply by going online on a server or border node with the necessary access permissions.

[0076] Furthermore, the lightweight probe captures multi-source credential data, including: network 5-tuple information, whereby the network 5-tuple includes the source IP address, source port, destination IP address, destination port, and transport protocol; this data collectively contributes to trust calculation and risk identification.

[0077] Identity and device data are used for subject and object identification;

[0078] Behavioral and traffic data are used for dynamic trust modeling;

[0079] Log and alert data are used for risk verification and tracing.

[0080] Configuration data is used to assist in risk propagation analysis and strategy generation.

[0081] Overall, they constitute a complete data support chain of perception, analysis, decision-making, and optimization.

[0082] Furthermore, each data acquisition agent node registers with the central control node upon startup and periodically reports its heartbeat and resource usage status. The central control node monitors the status of each node in real time.

[0083] The central control node schedules each agent node through methods including module pruning mechanism, sampling frequency limiting and frequency limiting strategy, resource self-monitoring mechanism, and remote control, so as to keep the resource utilization rate of the probe within the preset threshold range.

[0084] The module pruning mechanism is as follows: the probe loads the acquisition plugin on demand, and only the acquisition plugins required by the current strategy are enabled;

[0085] The sampling and frequency limiting strategy is as follows: high-frequency data, including traffic and logs, is sampled and uploaded in batches.

[0086] Resource self-monitoring mechanism: The probe periodically detects its own CPU and memory usage, and automatically reduces the collection frequency or suspends some tasks when the threshold is exceeded;

[0087] Remote control: The analysis center can remotely adjust the granularity and frequency of data collection through the central control node, so that the probes of different devices operate within the set resource threshold range.

[0088] Specifically, when a node is detected to be overloaded or a new node is added online, the central control node redistributes tasks to achieve dynamic balancing; if a node is disconnected or in an abnormal state, the central control node marks it as offline and transfers its data collection tasks to a backup or standby node in the same area.

[0089] The distributed data acquisition network uses NTP or PTP for unified time synchronization; each node probe synchronizes its clock with the central time source upon startup and periodically calibrates it during operation; when data is reported, the probes of the acquisition agent node use the local synchronized clock to generate an acquisition timestamp, and the data analysis center performs another unified time difference correction at the receiving end, so that the time accuracy error of the entire network data is less than the millisecond level.

[0090] This invention deploys lightweight probes in a cluster at key nodes of the target network to form a distributed data acquisition network, which performs tasks such as data forwarding, request processing, and resource scheduling, while solving problems such as single point of failure, excessive load, and cross-environment communication.

[0091] A distributed proxy framework refers to a system structure consisting of a central control node and multiple data collection proxy nodes. The central node is responsible for task scheduling and policy distribution, while each proxy node is deployed near a different data source, independently executes data collection tasks, and sends the results back.

[0092] This approach allows for distributed data collection, avoiding single-point pressure and improving collection efficiency and system reliability. For example, in a power grid system, probes can simultaneously access Modbus protocol devices (such as electricity meters) in substations, OPC UA protocol devices (such as inverters) in new energy power plants, and HTTP protocol business systems (such as scheduling platforms) on cloud platforms through a distributed agent architecture. Data is collected in parallel from these three types of data sources without the need to deploy separate collection tools for each protocol.

[0093] The sampling frequency of node probes can be customized (e.g., 1 second / time for core devices, 5 minutes / time for ordinary devices), and the parsed fields can be customized (e.g., retaining key fields such as "power value" and "fault code"). Configuration commands are sent to the probes in real time. When the probe detects a new device (e.g., connecting to a new photovoltaic power station), it automatically matches the preset collection strategy for the same type of device, without the need for manual reconfiguration; it automatically degrades the parsing granularity during peak traffic and recovers it after the load decreases.

[0094] The following is an example of probe deployment:

[0095] The probes are deployed in the nodes of the distributed power system network. The probes are configured with multi-protocol compatible modules to support the parsing and adaptation of IEC61850, DNP3, MQTT, RESTfulAPI, and LoRaWAN protocols.

[0096] The cluster approach for deploying lightweight probes at key nodes of the target network specifically includes: deploying device-side probes on hosts (physical machines / virtual machines) and security devices (firewalls / intrusion detection devices); deploying network-side probes on core switches and network egress nodes (routers / gateways); and deploying system-side probes on business servers (application servers / middleware servers) and log servers.

[0097] A high-risk business window identification rule is preset. The lightweight probe only performs in-depth data capture on the identified key high-risk windows, and performs regular data collection on non-high-risk windows to avoid affecting the original business performance.

[0098] The lightweight probe automatically identifies the type and version information of newly launched devices;

[0099] The lightweight probe is equipped with a local caching module, which temporarily stores the collected data locally when the network is interrupted, and resumes the transmission after the network is restored.

[0100] After initial field extraction and formatting on the local machine, encrypted transmission is used to encrypt the data transmission process.

[0101] The encrypted transmission includes encrypting the data using a TLS encrypted channel.

[0102] Preliminary field extraction and formatting refers to the lightweight preprocessing of raw traffic or log data collected by the probe before data reporting. This mainly includes field extraction, structuring, desensitization, and encoding standardization.

[0103] Specifically:

[0104] Field extraction: Extract key fields from raw data packets or logs, such as timestamp, source / destination IP, port, protocol, session ID, event type, etc., retaining only the core content related to trust calculation and risk analysis.

[0105] Formatting process: The extracted fields are uniformly converted into a standardized structure (such as JSON or custom KV format), and sensitive information is de-identified or hashed so that it can be uniformly accessed and parsed by the subsequent message bus.

[0106] This processing method applies to all collected data, but will automatically differentiate based on business strategies:

[0107] For ordinary data, only the basic fields are extracted;

[0108] For data from key sessions or high-risk windows, we will perform deeper field parsing and structuring to support subsequent trust calculations and source tracing analysis.

[0109] Regarding encrypted channels:

[0110] After initial extraction and formatting, the data is pushed to a real-time message bus (such as Kafka) via a TLS encrypted channel.

[0111] The TLS channel primarily ensures transmission security. The probe does not compress or analyze the content before encryption to maintain data integrity and real-time transmission.

[0112] The probe employs a local traffic mirroring and differentiated sampling strategy, including:

[0113] The service traffic replica is generated by mirroring the network interface card port or replicating the virtual switching layer. The probe only parses the replica data and does not participate in the original service forwarding. The sampling rate is dynamically adjusted according to the data type, sampling high-frequency repetitive data at low frequency and collecting all key service interaction data. Key fields of the network layer, transport layer and application layer are parsed in the mirrored traffic, and the dynamic granularity of the parsing depth is supported.

[0114] The key business is determined through a three-tiered logic: pre-screening using identifiers, precise verification using a matching pool, and dynamic adjustment and adaptation. First, initial screening is performed using specific identifiers such as business codes and priority tags in the data stream. Then, a secondary verification is conducted using a key business matching pool with features such as stored business IDs and device ranges. Finally, the identifiers and matching pool content are dynamically updated as the power grid business iterates, balancing identification efficiency, accuracy, and adaptability.

[0115] Specifically, regarding node status awareness and dynamic task allocation, each probe node automatically registers with a registry center (such as Zookeeper or etcd) upon startup and periodically reports heartbeats and resource usage (CPU, memory, task load, etc.). The scheduling service monitors the status of each node in real time. When it detects that a node is overloaded or a new node comes online, it automatically redistributes tasks to achieve dynamic balance. If a node becomes disconnected or its status is abnormal, the system immediately marks it as "offline" and transfers its collection tasks to a backup or standby node in the same region to ensure uninterrupted collection.

[0116] The clock synchronization and timestamp consistency are achieved by using NTP (Network Time Protocol) or PTP (Precision Time Protocol) for unified time synchronization in the cluster. All probe nodes synchronize their clocks with the central time source upon startup and periodically (e.g., every 5 minutes) during operation. When reporting data, the probes use their local synchronized clocks to generate collection timestamps, and the center performs a unified time difference correction at the receiving end to ensure that the time accuracy error of the entire network data is less than the millisecond level.

[0117] The identification rules for key high-risk windows are dynamically determined based on the following rules:

[0118] Time-based behavioral characteristic rules: Based on the time distribution patterns of historical attacks or abnormal events, identify high-risk periods (such as late at night, holidays, and system switching periods);

[0119] Traffic anomaly rules: Risk windows are determined based on characteristics such as sudden increases in traffic, abnormal growth in the number of connections, and a surge in access frequency;

[0120] Event triggering rules: Triggered based on attack, unauthorized access, or authentication failure events reported by upstream security devices (firewall / IDS / threat intelligence center);

[0121] Policy association rules: Set trigger conditions for specific command or frame characteristics of key business nodes (scheduling instruction issuance / core control signal transmission);

[0122] Self-learning rules: High-risk behavior patterns are summarized from historical samples and the identification rules are dynamically updated.

[0123] The thresholds for determining abnormal traffic rules include: a sudden increase in traffic exceeding 200% of the baseline value, a sudden increase in the number of concurrent connections exceeding 300% of the historical average, and an access frequency exceeding a preset threshold per unit time.

[0124] The event triggering rules receive event data from upstream devices through a standardized interface, including attack type (such as DDoS / port scan), event level (high / medium risk), and scope of impact.

[0125] The policy association rules match preset key business operation features by parsing business protocol fields (such as Modbus function codes / OPC UA instruction types).

[0126] The self-learning rules use machine learning algorithms (such as clustering / classification algorithms) to train the time, traffic, and protocol characteristics of historical high-risk events to generate a dynamic identification model;

[0127] The probe is only responsible for collecting data and sending it to the backend. The backend model then uses the trained rules to identify risks in the data, and the identification results are fed back as needed.

[0128] Specifically, a default verification cycle of 5-10 minutes is set, which can be adjusted according to business scale and real-time requirements, with a minimum support of 1 minute; a short cycle is configured for high-frequency asset change scenarios, and a long cycle is configured for core production network segments; multiple sources of data, such as CMDB, DHCP, ARP tables, and probe collection results, are periodically retrieved; cross-comparison of multiple sources of data is performed based on key fields such as IP, MAC, hostname, and device ID; when anomalies such as new MAC-IP mapping combinations, changes in asset attributes, or traffic from unregistered devices are detected, they are marked as asset changes.

[0129] The process of automatically identifying the type and version information of newly launched devices includes: if a device appears for the first time in the traffic or ARP list collected by the probe, but there is no matching record in the historical asset database (CMDB or local cache), it is determined to be a newly launched device.

[0130] At the same time, the system will detect its communication behavior characteristics, manufacturer identification and other information, and automatically supplement the initial asset attributes.

[0131] Once a new device or asset change is detected, the system sends an "update event" to the asset center via a message queue (Kafka);

[0132] The asset management module calls the interface to update CMDB or local asset database records, including device status, IP, MAC, network domain, etc.

[0133] Once synchronization is complete, the data will be automatically distributed to all relevant subsystems (such as risk assessment and strategy configuration modules) to maintain data consistency.

[0134] Specifically, the probe node has a local caching module, which temporarily writes the collected data into the local cache queue when the network is abnormal or the center is unreachable; after the fault is recovered, the cached data will be automatically retransmitted to the message bus to ensure that the data is not lost; the system log records the fault and recovery time of the node, which is convenient for subsequent auditing and location.

[0135] The local caching module adopts a circular queue storage structure. When the cached data reaches the preset capacity limit, older data is overwritten according to timestamp priority, and the cached data carries an integrity verification mark. The probe is designed with a local cache protection mechanism to ensure that data is not lost when the network is abnormal or the center is temporarily unreachable. The cache limit handling strategy is as follows: when the cached data reaches the preset capacity limit, the system automatically cleans up the oldest cached data according to the "first-in, first-out (FIFO)" principle and simultaneously generates an alarm and reports it to the central control node. If the center restores the connection, the probe prioritizes sending the latest data that has not yet been uploaded, and asynchronously re-uploads the remaining part in the background to ensure that critical data is uploaded first. Historical data exceeding the cache limit is summarized before being cleaned up for subsequent statistics and anomaly tracing.

[0136] The default cache limit is 10% to 20% of the available local disk space or about 500MB to 2GB (which can be adjusted according to device performance); a larger threshold can be set for high-performance servers, while a smaller cache area is used for resource-constrained terminals (such as RTUs and monitoring and control devices); the threshold parameters are uniformly issued by the center and can also be dynamically adjusted according to the probe's operating status.

[0137] Furthermore, the data analysis center is connected to the asset management database of the operations and maintenance platform via an access interface. This interface allows the center to obtain basic asset information, supplementing the network perspective data collected by probes and achieving data fusion. Probes are primarily responsible for collecting underlying real-time data such as network traffic and logs, while the operations and maintenance platform, CMDB, DHCP / ARP, and other management data sources are typically accessed via interfaces.

[0138] Specifically, the data analysis center periodically synchronizes management data sources, including device lists, IP address allocations, asset attributes, and network mapping information, by calling access interfaces provided by the operation and maintenance platform, CMDB, and / or DHCP server, including APIs, database views, and Syslog interfaces.

[0139] The data analysis center manages the data and collects the underlying real-time data through probes. By associating and matching key fields including IP, MAC, and asset number, the context information of the network session is completed, and a closed-loop association from traffic behavior to asset identity is achieved.

[0140] The following provides an example of the interface.

[0141] Interface integration method: The system periodically synchronizes device list, IP address allocation, asset attributes and network mapping information by calling APIs, database views or Syslog interfaces provided by the operation and maintenance platform, CMDB, DHCP server, etc.

[0142] Data fusion: The data access module on the central side associates and matches this management data with the network perspective data collected by the probe (such as through key fields such as IP, MAC, and asset number) to complete the context information of the network session.

[0143] This integration allows the system to not only see "who is communicating in the network," but also "which device is communicating, which service it belongs to, and who is responsible for it," thus achieving a closed-loop association from traffic behavior to asset identity.

[0144] Specific implementation examples are given below.

[0145] Example 1

[0146] like Figure 2 As shown, the probe is compatible with mainstream industrial and general protocols such as SNMP and Modbus, and can be uniformly connected to devices from different manufacturers to achieve multi-domain data collection. It reads data through standard protocols to avoid modifying the original system, and at the same time obtains multi-dimensional information such as operating status and alarms to improve integrity. When adding new devices, only the corresponding plugin needs to be loaded to expand. It can also use the built-in authentication, encryption and other security mechanisms of the protocol to ensure transmission security.

[0147] The method for collecting multi-source credential data under a distributed proxy framework includes the following operations:

[0148] 101) Lightweight probes are deployed in a cluster at key nodes of the target network to form a distributed data acquisition network; the cluster method deploys lightweight probes, with different probes deployed at different locations;

[0149] Deploy device-side probes on hosts (physical machines / virtual machines) and security devices (firewalls / intrusion detection devices); deploy network-side probes on core switches and network egress nodes (routers / gateways); and deploy system-side probes on business servers (application servers / middleware servers) and log servers.

[0150] Among them, device-side probes collect operating status, performance indicators, and alarm information; network-side probes collect network traffic and quintuple information; and system-side probes collect system logs and application layer interaction information.

[0151] The central node ensures that the probe's resource utilization rate remains within a preset threshold range by loading modules on demand, limiting sampling frequency, self-monitoring resources, and remotely adjusting policies.

[0152] Specifically, the probe adopts a local traffic mirroring and differentiated sampling strategy. It generates a copy of the business traffic by mirroring the network interface card port or replicating the virtual switching layer. The probe only parses the copy data and does not participate in the original business forwarding. It dynamically adjusts the sampling rate according to the data type, samples high-frequency repetitive data at low frequency, and collects all key business interaction data. It parses key fields of the network layer, transport layer and application layer in the mirrored traffic and supports dynamic granular adjustment of the parsing depth.

[0153] Each probe node automatically registers with the central node (usually a cluster of 3 or more odd-numbered nodes, physically isolated from the probe nodes, and in the same intranet environment as backend components such as scheduling services and system management centers to ensure low-latency communication and high availability) upon startup, such as Zookeeper or etcd, and periodically reports heartbeats and resource usage (CPU, memory, task load, etc.).

[0154] The central node provides node status data and performs dynamic scheduling based on this data. It monitors the status of each node in real time and automatically redistributes tasks when it detects excessive node load or the addition of a new node. When the scheduling service detects excessive node load or the addition of a new node, it dynamically balances tasks according to the following logic:

[0155] Load threshold trigger: A preset load threshold (such as CPU utilization of 80% or the number of tasks exceeding the limit) is set. When a node exceeds the threshold, the scheduling service marks it as overloaded.

[0156] Task priority sorting: From the tasks on overloaded nodes, filter out non-core tasks that can be migrated (such as low-frequency status collection) and retain core tasks (such as key business voucher collection).

[0157] Target node filtering: Query the registry center for nodes with low load (e.g., CPU utilization <50%), in the same region (to reduce data transmission latency), and with matching resources (e.g., sufficient memory) (including newly added nodes).

[0158] Task migration execution: The scheduling service sends a "task migration instruction" to the overloaded node, requiring it to stop the specified task and upload the incomplete intermediate data.

[0159] At the same time, a task allocation instruction is sent to the target node, which includes information such as task type, collection frequency, and data source address. The target node starts the task and synchronizes historical configurations from the registry center.

[0160] State synchronization: The scheduling service updates the task allocation results to the registry center to ensure that the states of all nodes are consistent and to achieve dynamic balance;

[0161] If a node becomes disconnected or its status is abnormal, the system will immediately mark it as "offline" and transfer its data collection task to a backup or standby node in the same area to ensure that data collection is not interrupted.

[0162] Specifically, for clock synchronization and timestamp consistency, the cluster uses NTP (Network Time Protocol) or PTP (Precision Time Protocol) for unified time synchronization; all probe nodes synchronize their clocks from the central time source upon startup and periodically (e.g., every 5 minutes) during operation; when reporting data, the probes use their local synchronized clocks to generate collection timestamps, and the center performs a unified time difference correction at the receiving end to ensure that the time accuracy error of the entire network data is less than the millisecond level.

[0163] 102) The probe adopts local traffic mirroring and differentiated sampling strategies to avoid impacting the performance of the original business, and only performs in-depth capture of the characteristics of key sessions or high-risk time windows.

[0164] The service traffic replica is generated by mirroring the network interface card port or replicating the virtual switching layer. The probe only parses the replica data and does not participate in the original service forwarding. The sampling rate is dynamically adjusted according to the data type, sampling high-frequency repetitive data at low frequency and collecting all key service interaction data. Key fields of the network layer, transport layer and application layer are parsed in the mirrored traffic, and the dynamic granularity of the parsing depth is supported.

[0165] Specifically, the implementation of the local traffic mirroring strategy involves configuring the mirroring source port and target port on the border device, or setting mirroring rules in the virtualization environment through the vSwitch / Docker network plugin.

[0166] Specifically, the dynamic adjustment of the differentiated sampling strategy is based on the following criteria: protocol type (e.g., high-frequency sampling for OPC UA transmitting critical data, and low-frequency sampling for Modbus transmitting ordinary data), traffic characteristics (frequency reduction to preserve the channel when the data volume exceeds the threshold), equipment level (full sampling for core equipment, and on-demand sampling for ordinary equipment), and business time period (peak / off-peak, focusing on critical data during peak hours, and supplementing data collection during off-peak hours).

[0167] Specifically, the fields captured in the deep crawl include: IP / port / protocol at the network layer, TCP / UDP session state at the transport layer, and OPC UA session ID, TLS certificate digest, HTTP headers, and Modbus function codes at the application layer.

[0168] Specifically, the probe has a built-in granularity adjustment mechanism that runs with the probe and does not require additional independent deployment. The probe monitors local CPU / memory usage and traffic peaks in real time. When the peak exceeds the preset threshold, it automatically shuts down field parsing and session tracking functions and only counts basic data such as packet length and transmission rate.

[0169] When device resources are strained or traffic is at its peak, the system automatically downgrades from "deep parsing (field level / session level)" to "shallow statistics (packet length / rate)" mode.

[0170] Table 1 lists the types of probes and their corresponding details.

[0171] Table 1 Three Types of Software Probes

[0172]

[0173]

[0174] 103) Multi-source voucher data is captured using the lightweight probe; Table 2 lists the classification and function of multi-source voucher data.

[0175] Table 2. Classification and Function of Multi-Source Voucher Data

[0176]

[0177]

[0178] 104) After preliminary field extraction and formatting on the local machine, encrypted transmission is used to encrypt the transmission process of the collected data;

[0179] The preliminary field extraction and formatting refers to the lightweight preprocessing of the raw traffic or log data collected by the probe before data reporting, which mainly includes field extraction, structuring, desensitization, and encoding unification.

[0180] The encrypted transmission includes encrypting the data using a TLS encrypted channel.

[0181] 105) Lightweight probes are configured with a local cache module to temporarily store collected data locally when the network is interrupted, and resume transmission from the breakpoint after the network is restored.

[0182] The probe node has a local caching module, which temporarily writes the collected data into the local cache queue when the network is abnormal or the center is unreachable; after the fault is recovered, the cached data will be automatically retransmitted to the message bus to ensure that the data is not lost; the system log records the fault and recovery time of the node, which is convenient for subsequent auditing and location.

[0183] The local caching module adopts a circular queue storage structure. When the cached data reaches the preset capacity limit, older data is overwritten according to timestamp priority, and the cached data carries an integrity verification mark. The probe is designed with a local cache protection mechanism to ensure that data is not lost when the network is abnormal or the center is temporarily unreachable. The cache limit handling strategy is as follows: when the cached data reaches the preset capacity limit, the system automatically cleans up the oldest cached data according to the "first-in, first-out (FIFO)" principle and simultaneously generates an alarm and reports it to the central control node; if the center restores the connection, the probe prioritizes sending the latest data that has not yet been uploaded and asynchronously re-uploads the remaining data.

[0184] After the latest data is transmitted, the remaining old data (such as device status logs from the early stages of the disconnection and non-critical historical behavior credentials) will not occupy the current main transmission channel, but will be transmitted slowly in the background in an asynchronous manner. Asynchronous transmission here can be understood as "not disturbing the main line of work"—the probe can collect and transmit new data normally, while the operation of retransmitting old data is quietly performed in the background, without affecting each other, ensuring that neither real-time business is delayed nor historical data is missed, thus ensuring that critical data is uploaded first; historical data exceeding the cache limit will be summarized before being cleaned up for subsequent statistics and anomaly tracing.

[0185] The default cache limit is 10% to 20% of the available local disk space or about 500MB to 2GB (which can be adjusted according to device performance); a larger threshold can be set for high-performance servers, while a smaller cache area is used for resource-constrained terminals (such as RTUs and monitoring and control devices); the threshold parameters are uniformly issued by the center and can also be dynamically adjusted according to the probe's operating status.

[0186] 106) Establish a data access interface with the asset management database of the operation and maintenance platform, obtain basic asset information through the interface, supplement the network perspective data collected by the probe, and realize data fusion.

[0187] The data access interface for establishing and maintaining the asset management database of the operation and maintenance platform includes the new power system periodically synchronizing equipment lists, IP address allocations, asset attributes and network mapping information by calling APIs, database views or Syslog interfaces provided by the operation and maintenance platform, CMDB, DHCP server, etc.

[0188] The central data access module associates and matches this management data with the network perspective data collected by the probe (such as through key fields such as IP, MAC, and asset number) to complete the context information of the network session.

[0189] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for collecting multi-source credential data under a distributed agent framework, characterized in that, The method comprises the following operations: 1) deploying collection probes in a cluster mode on multiple nodes of a target network to form multiple collection agent nodes, and then forming a distributed data collection network together with a central control node; 2) the probes collect multi-source credential data for trust calculation and risk identification by using local traffic mirroring and differential sampling strategies, wherein the probes are provided with high-risk business window identification rules, and only the data of identified key high-risk windows are captured, and the data of non-high-risk windows are collected regularly; The captured multi-source credential data is unpacked and processed to extract application layer key information; 3) the lightweight probes automatically identify the type and version information of newly online devices through periodic cross-validation; 4) after field extraction and formatting, the collected data of the distributed data collection network are transmitted by encryption by the collection agent nodes; 5) the lightweight probes are configured with a local cache module, and if the network is interrupted during transmission, the collected data are temporarily stored locally, and then the breakpoint transmission is performed to the message bus after the network is restored; 6) the data analysis center receives the data transmitted by each node through the message bus; The data analysis center is also connected to the asset management library of the operation and maintenance platform through an access interface, and obtains management data sources through the data access interface.

2. The method of claim 1, wherein the distributed agent framework is a framework of a mobile device. The probes are deployed in the form of containers or service processes, and comprise: Device-side probes deployed on hosts or security devices to collect data including running status, performance indicators, and alarm information; Network-side probes deployed on core switches or network exit nodes to collect data including network traffic and five-tuple information; the five-tuple information includes IP address, source port, target IP address, target port, and transmission protocol; System-side probes deployed on business servers or log servers to collect data including system logs and application layer interaction information. All probes are registered through a registration center, and the central node uniformly issues collection strategies and credentials.

3. The method of claim 1 or 2, wherein the method comprises: Each collection agent node registers with the central control node when starting, and regularly reports heartbeat and resource usage, and the central control node monitors the states of the nodes in real time; When detecting that the load of a node is too high or a new node is online, the central control node reassigns tasks to achieve dynamic balance; if a node is offline or abnormal, the central control node marks it as offline, and transfers its collection tasks to a standby node or a standby node in the same region; The distributed data collection network uses NTP or PTP for unified time synchronization; the node probes synchronize the clock from the central time source when starting, and periodically calibrate during running; when data is reported, the probes of the collection agent nodes generate collection time stamps using the local synchronized clock, and the data analysis center performs unified time difference correction at the receiving end, and the time precision error of the whole network is less than milliseconds.

4. The method of claim 2, wherein the method comprises: The data collected by the probes includes identity and device data for subject-object identification; behavior and traffic data for dynamic trust modeling; log and alarm data for risk verification and tracing; configuration data for assisting risk propagation analysis and strategy generation; The probe is deployed in the nodes of a distributed power system network, and the probe is configured with a multi-protocol compatible module to support parsing and adaptation of IEC61850, DNP3, MQTT, RESTful API, and LoRaWAN protocols.

5. The method of claim 1 or 2, wherein, The central control node schedules the proxy nodes through a mechanism including module tailoring, sampling and frequency limiting strategies, resource self-monitoring mechanisms, and remote regulation, so that the resource occupancy rate of the probe is maintained within a preset threshold range. The module tailoring mechanism: the probe loads collection plug-ins on demand, and only enables the collection plug-ins required by the current strategy. The sampling and frequency limiting strategy: for high-frequency data including traffic and logs, batch sampling and uploading are performed. The resource self-monitoring mechanism: the probe periodically detects its CPU and memory occupancy, and automatically reduces the collection frequency or suspends part of the tasks when the threshold is exceeded. Remote regulation: the analysis center remotely adjusts the collection granularity and frequency through the central control node, so that different device probes operate within the set resource threshold range.

6. The method of claim 1, wherein the distributed agent framework is a framework of a mobile device. The probe uses local traffic mirroring and differential sampling strategies, which are as follows: Through network card port mirroring or virtual switch layer replication, the probe only analyzes the replica data and does not participate in the forwarding of the original service, and dynamically adjusts the sampling rate according to the data type, and low-frequency sampling is performed on high-frequency repeated data. Full collection of key business interaction data is performed, network layer, transport layer and application layer key fields are analyzed in the mirrored traffic, and dynamic granularity adjustment of the analysis depth is supported; the key business is determined through three logical determinations of identification pre-filtering, matching pool accurate verification, and dynamic adjustment and adaptation: first, the business code and priority tag in the data stream are used to complete preliminary screening, then the key business matching pool storing characteristics such as business ID and device range is used for secondary verification, and finally the identification and matching pool content are dynamically updated with business iteration.

7. The method of claim 1, wherein the distributed agent framework is a framework of a mobile device. The identification rules of the high-risk window are issued by the data analysis center, which include: Time behavior characteristic rules: based on the time distribution law of historical attacks or abnormal events, high-risk periods are identified; Traffic anomaly rules: risk windows are determined according to characteristics including traffic surge, abnormal increase in connection number, and access frequency surge; the determination threshold of the traffic anomaly rules includes: traffic instantaneous increase of more than 200% of the baseline value, concurrent connection number surge of more than 300% of the historical average, and access frequency per unit time exceeding the preset threshold; Event triggering rules: triggered based on attack, illegal access or authentication failure events reported by upstream security devices; the event triggering rules receive event data of upstream devices through a standardized interface, including attack type, event level and impact range; Strategy association rules: trigger conditions are set for specific commands or frame characteristics of key business nodes; the strategy association rules match preset key business operation characteristics by analyzing business protocol fields; Self-learning rules: high-risk behavior patterns are induced from historical samples, and the identification rules are dynamically updated; the self-learning rules use machine learning algorithms including clustering / classification algorithms to train the time, traffic, and protocol characteristics of historical high-risk events, and generate a dynamic identification model.

8. The method of claim 1, wherein the distributed agent framework is a framework of a mobile device. The automatic identification of new online device type and version information includes: if the device first appears in the traffic or ARP list collected by the probe, but there is no matching record in the historical asset library or local cache, it is judged as a new online device; at the same time, the data analysis center detects the communication behavior characteristics and manufacturer identification of the device, and supplements the initial asset attributes; Once new devices or asset changes are identified, the system sends update events to the analysis center detection through the message queue; The analysis center detection calls the interface to update the CMDB or local asset library record.

9. The method of claim 1, wherein the distributed agent framework is a framework of a mobile device. The preliminary field extraction and formatting is a lightweight preprocessing of the collected raw traffic or log data, mainly including field extraction, structuring, desensitization and coding unification; The field extraction: extracts the key fields related to trust calculation and risk analysis from the original data packet or log; The key fields include timestamp, source / destination IP, port, protocol, session ID, and event type. The formatting process: converts the extracted fields into a standardized structure, and performs desensitization or hash processing on sensitive information. The encrypted transmission includes encrypting the data using a TLS encrypted channel, and then transmitting; when the network is abnormal or the data analysis center is unreachable, the probe node temporarily writes the collected data to the local cache queue; After fault recovery, the cache data is automatically retransmitted to the message bus; The local cache module uses a ring queue storage structure, when the cache data reaches the preset upper limit of the capacity, the old data is overwritten according to the timestamp priority, and the cache data has a complete integrity verification identifier; When the cache data reaches the preset upper limit of the capacity, the earliest cache data is automatically cleaned up according to the first-in-first-out principle, and an alarm is generated and reported to the center control node at the same time; if the connection is restored, the probe sends the latest data that has not been uploaded first, and asynchronously supplements the remaining part, ensuring that critical data is uploaded first; before cleaning up, the historical data exceeding the cache upper limit will be recorded in summary.

10. The method of claim 1, wherein the distributed agent framework is a framework of a mobile device. The data analysis center periodically synchronizes the management data sources including device list, IP address allocation, asset attributes and network mapping information by calling the access interfaces provided by the operation and maintenance platform, CMDB and / or DHCP server, including API, database view and Syslog interface; The data analysis center management data and the probe collected bottom layer real-time data are associated and matched through key fields including IP, MAC and asset number, so as to complete the context information of network session, realize the closed loop association from traffic behavior to asset identity.

Images