Information system integrated service data intelligent processing and storage system

Abstract

The invention provides an intelligent processing and storage system for integrated service data of an information system. The intelligent processing and storage system consists of an edge center, a protocol bridge and a hierarchical storage array, the marginal center outputs an original message through a differential hardware bus, and the original message is pushed to the hierarchical storage array for disk falling after semantic conversion is completed by a protocol bridge; the marginal center captures byte-level data in real time in a cross-protocol heterogeneous scene and converges the byte-level data into an original message, the protocol bridge converts the original message into a standard message according to a storage write-in contract, and the hierarchical storage array persistently stores the standard message. The real-time performance, the accuracy and the storage resource utilization rate of data processing can be improved, and the data redundancy and the transmission cost can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, and more particularly, to an information system integration service data intelligent processing and storage system. BACKGROUND

[0002] In today's digital era, information system integration services are facing the challenges of explosive growth of data volume and diversification of data sources. Traditional data processing and storage systems usually adopt centralized architecture, collecting data and transmitting it to the central server for processing and storage. Although this architecture can meet the demand of data storage to some extent, it often encounters performance bottlenecks when facing large-scale and high-concurrency data input, resulting in increased data processing delay and inability to respond to business needs in a timely manner. In addition, the centralized architecture lacks support for data heterogeneity, making it difficult to adapt to different protocols and formats of data access, greatly increasing the complexity of data integration.

[0003] Existing data processing systems mostly use a single transmission protocol and fixed message format during data transmission, lacking deep analysis and conversion capabilities for data semantics. This makes it difficult to ensure data compatibility and consistency in cross-protocol heterogeneous integration scenarios, and easily leads to problems such as data loss, errors or duplicate storage. At the same time, traditional storage systems mostly use a single storage medium, which cannot perform hierarchical storage according to data access frequency and importance, resulting in low utilization of storage resources and difficulty in efficient management and long-term preservation of data.

[0004] In the implementation of the embodiments of the present application, there are at least the following problems or defects in the prior art: first, the data processing efficiency is low, which cannot meet the real-time processing needs of high concurrency and large-scale data; second, the support for data heterogeneity is insufficient, making it difficult to realize cross-protocol data integration; third, the reliability and accuracy of data transmission need to be improved, and problems such as data loss or duplicate storage are prone to occur; fourth, the utilization rate of storage resources is low, lacking an efficient hierarchical storage mechanism, and unable to optimize storage according to data characteristics; fifth, lacking intelligent data management and scheduling strategies, unable to dynamically adjust data processing and storage processes according to system load and data characteristics. SUMMARY

[0005] The present application provides an information system integration service data intelligent processing and storage system, comprising: an edge hub, a protocol bridge, and a hierarchical storage array; The edge hub includes a differential hardware bus for data service output, and the original message output by the differential hardware bus is processed by the protocol bridge and then written to disk by the hierarchical storage array; The edge hub communicates with the protocol bridge through the differential hardware bus, and the protocol bridge communicates with the hierarchical storage array; The edge hub has a real-time stream capture function, and triggers a lightweight collection agent based on a policy engine to perform byte-level grabbing in a cross-protocol heterogeneous integration scenario, and aggregates data collected by the collection agent to obtain original messages; When the edge hub outputs high-rate original messages through the differential hardware bus, the edge hub transmits the original messages to the protocol bridge through the differential hardware bus; The protocol bridge performs semantic conversion on the received original messages, and pushes the normative messages converted semantically to the hierarchical storage array; the semantic conversion includes: converting the original messages into normative messages adapted to a write contract based on the write contract supported by the hierarchical storage array; The hierarchical storage array persists the received normative messages.

[0006] Further, when the edge hub detects continuous original messages with the same transaction identifier and a timestamp interval less than a set time threshold, the edge hub only marks the first appearing original message as valid and transmits it; Subsequent original messages with the same transaction identifier are marked as redundant and cached in an internal ring buffer; When the number of redundant messages with the same transaction identifier in the ring buffer reaches a preset number threshold, or the cache duration exceeds a set timeout window, all redundant messages are discarded.

[0007] Further, the edge hub dynamically selects original messages that meet matching conditions according to a first compression parameter group matching rule declared by the hierarchical storage array; If the original message compression parameters do not match the first compression parameter group, the original message is converted into a matched compression format according to a preset degradation strategy before being transmitted to the protocol bridge; A cyclic redundancy check is performed before transmission, and when the check fails, an exponential backoff retransmission mechanism is triggered, and when the number of retransmissions reaches a set upper limit, the message metadata is written into a fault log and marked as an unrecoverable error.

[0008] Further, the protocol bridge maintains a compression parameter mapping table, and when the second compression parameter group of the original message does not match the declaration of the hierarchical storage array, the compression format is converted step by step according to the priority order of the mapping table until a match is found; An irreversible hash value of the normative message is generated during the conversion process, and the hash value is bound to the normative message as a unique data identifier and pushed; If the number of consecutive conversion failures exceeds a set retry upper limit, the format is rolled back to lossless compression and an alarm event is generated.

[0009] Further, the hierarchical storage array performs hash value comparison operation before persisting: If the hash value exists in the deduplication index, increment the reference count of the associated data block and skip storage; If the hash value does not exist, store the canonical message in row-column hybrid format and register the hash value to the deduplication index; When the reference count value exceeds a set split threshold, create a data block copy and reset the reference count.

[0010] Further, the edge hub monitors the hierarchical state of the hierarchical storage array in real time; When the archive layer data is detected and the policy engine issues a migration instruction, perform the migration operation: First, calculate the migration cost index, which is equal to the amount of data to be migrated multiplied by the current network delay coefficient; If the cost index is less than or equal to the preset upper threshold, immediately start the migration; If the cost index is greater than the preset upper threshold, extend the migration delay time by the ratio of the cost index to the upper threshold, and periodically reevaluate the cost index during the delay period.

[0011] Further, the policy engine performs load prediction operation before issuing the migration instruction: Get the central processor utilization and memory occupancy of the edge hub; If the central processor utilization is lower than the set processor threshold and the memory occupancy is lower than the set memory threshold, immediately confirm the migration instruction; If any of the indicators exceeds the threshold, reduce the number of simultaneous migration tasks by the over-limit ratio until the system load drops to the safe threshold range.

[0012] Further, the lightweight collection agent's crawling window adjustment method includes: Initialize the crawling window to the maximum window size; Measure the network round-trip time in real time; If the current round-trip time is measured for several times continuously exceeds the set multiple of the historical average time delay, dynamically reduce the window size according to the ratio of the current window size to the round-trip time; When the round-trip time returns to the normal range and the duration of stability exceeds the set stability threshold, gradually increase the window size until the maximum window size.

[0013] Further, the event sensor performs event classification logic includes: Extract the event feature vector, which contains the time stamp, protocol type and data volume dimensions; Calculate the similarity score of the vector with the nearest neighbor sample in the historical event library; If the similarity score is greater than or equal to the set classification threshold, it is marked as a regular event and assigned a base priority; If the similarity score is less than the set classification threshold, it is marked as an abnormal event, and the priority is raised based on the difference between the score and the threshold.

[0014] Further, the rate limiting control method of the traffic shaping probe node comprises: Statistical current instantaneous throughput; Calculate the bandwidth change slope per unit time; If the absolute value of the slope is greater than the set alert slope and continuously satisfies the over-limit period, a rate limiting instruction is generated, and the new rate is equal to the current rate multiplied by one minus the ratio of the absolute value of the slope to the alert slope; After rate limiting, periodically sample the throughput fluctuation rate, and if the fluctuation rate is lower than the set fluctuation threshold for multiple consecutive times, gradually restore the original rate by a fixed step.

[0015] The above embodiments according to the present application have at least the following beneficial effects: 1. Through the real-time stream capture function of the edge hub and the semantic conversion capability of the protocol bridge, the system can efficiently process cross-protocol heterogeneous data, solve the compatibility problem caused by protocol differences in traditional system data integration, improve the accuracy and consistency of data processing, and ensure seamless transmission and processing of data in different protocol environments.

[0016] 2. The intelligent filtering and caching mechanism of the edge hub for high code rate raw packets effectively avoids data redundant storage, reduces storage cost, and improves the overall performance of the system. By dynamically selecting compression parameters and cyclic redundancy check mechanism, the system further optimizes the reliability and stability of data transmission, reduces errors and losses in the data transmission process.

[0017] 3. The persistent storage strategy and hash value comparison mechanism of the hierarchical storage array realize efficient deduplication and block storage of data, and improve the utilization rate of storage resources. At the same time, the system dynamically adjusts the storage strategy according to the data characteristics, such as data migration and load prediction, further optimizes the storage efficiency and system performance, and solves the problems of low resource utilization and performance bottleneck of traditional storage systems. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and other objects, features and advantages of the exemplary embodiments of the present application will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: Figure 1 The structural diagram of the information system integration service data intelligent processing and storage system provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] The technical solutions in the present application will be described clearly and completely in the present application combined with the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all. The components of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. It should be noted that: similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0020] As Figure 1 shown, the present application proposes an information system integration service data intelligent processing storage system, comprising: an edge hub 101, a protocol bridge 102 and a tiered storage array 103; The edge hub includes a differential hardware bus for data service output. The original message output by the differential hardware bus is processed by the protocol bridge and written to disk by the tiered storage array; The edge hub communicates with the protocol bridge through the differential hardware bus, and the protocol bridge communicates with the tiered storage array; The edge hub has a real-time stream capture function. In a cross-protocol heterogeneous integration scenario, a lightweight collection agent is triggered based on a policy engine to perform byte-level grabbing, and the data collected by the collection agent is aggregated to obtain original messages; When the edge hub outputs high-rate original messages on the differential hardware bus, it sends the original messages to the protocol bridge through the differential hardware bus; The protocol bridge performs semantic conversion on the received original messages and pushes the normalized messages after semantic conversion to the tiered storage array; the semantic conversion includes: converting the original messages into normalized messages adapted to the write contract supported by the tiered storage array; the tiered storage array persists the received normalized messages.

[0021] The application realizes data near-end processing and protocol-independent conversion through an edge computing architecture, combines a dynamic optimization mechanism of layered storage, and constructs an intelligent data processing system with high throughput, low delay and strong compatibility. The system uses a differential hardware bus to ensure the stability of high code rate data transmission, realizes cross-protocol semantic level conversion through a protocol bridge, and realizes on-demand allocation of storage resources through a layered storage array, effectively solving the performance bottleneck and heterogeneous protocol compatibility problem of traditional centralized architecture.

[0022] Specifically, the edge hub communicates with the protocol bridge through the differential hardware bus, and the protocol bridge communicates with the layered storage array. The edge hub has a real-time stream capture function, and in a cross-protocol heterogeneous integration scene, a lightweight collection agent is triggered based on a policy engine to perform byte-level grabbing, and the data grabbed by the collection agent is aggregated to obtain original messages. When the differential hardware bus needs to output high code rate original messages, the edge hub sends the original messages to the protocol bridge through the differential hardware bus.

[0023] After receiving the original messages, the protocol bridge performs semantic conversion operations. The core of semantic conversion is to convert the original messages into standard messages adapted to the write contract supported by the layered storage array based on the write contract. This step ensures the uniformity and compatibility of the data format. After conversion, the protocol bridge pushes the standard messages to the layered storage array.

[0024] The layered storage array is responsible for persisting the received standard messages. Through this layered architecture design, the system realizes a complete processing link from data collection, protocol conversion to final storage. The use of the differential hardware bus improves the stability and anti-interference ability of data transmission, and is particularly suitable for high-speed data transmission scenarios from the edge to the storage. Byte-level grabbing of the lightweight collection agent avoids introducing too much protocol parsing burden in the data receiving stage, improving the compatibility of the system to heterogeneous protocols.

[0025] The semantic conversion mechanism of the protocol bridge is the key link of the entire system, which solves the adaptation problem between the original messages and the write contract of the storage layer. By uniformly converting original messages of different protocols and formats into standard messages that meet the write contract of the storage array, the system greatly improves the efficiency of data processing and storage.

[0026] The design of the layered storage array allows the system to optimize storage according to the characteristics and access frequency of data, improving the utilization rate of storage resources. The entire system is designed modularly and functionally, effectively reducing the coupling degree between components, improving the scalability and maintainability of the system.

[0027] It should be noted that the edge hub refers to a computing node deployed near the data source for performing data aggregation and preprocessing, which can be implemented by an embedded device or an industrial gateway with a multi-core processor, and the near-end processing reduces the data transmission delay. The differential hardware bus refers to a data channel using differential signal transmission technology, which can be implemented by LVDS or PCIe interface, and the anti-interference design ensures the stability of high code rate data transmission.

[0028] The protocol bridge refers to a middleware component that implements conversion between different communication protocols, which can be implemented by a protocol analysis library and a conversion engine, and the semantic mapping eliminates compatibility problems between heterogeneous protocols. The layered storage array refers to a hybrid storage system composed of multiple storage media, which can be implemented by a combination of SSD and HDD architecture, and the hot and cold data layering improves the utilization rate of storage resources. The real-time stream capture function refers to a technology for uninterrupted monitoring and interception of network traffic, which can be implemented by a zero-copy packet capture library, and the bypass listening mechanism avoids affecting the performance of the business system. The strategy engine refers to a software module with built-in rule management and decision logic, which can be implemented by a rule engine based on the Rete algorithm, and the dynamic strategy adjustment optimizes the data collection efficiency. The lightweight collection agent refers to a micro data capture program deployed at the data source end, which can be implemented by a multi-threaded daemon process, and the low resource occupation ensures the overall performance of the system.

[0029] Byte-level capture refers to a technology for parsing network packets byte by byte, which can be implemented by raw socket programming, and fine-grained parsing ensures data integrity. High code rate raw packets refer to data streams with a transmission rate exceeding the conventional threshold, which can be marked with timestamps and sequence numbers, and traffic shaping ensures transmission order. Semantic conversion refers to the process of reconstructing the format of raw data according to the requirements of the target system, which can be implemented by XSLT or custom conversion scripts, and structure mapping enables cross-protocol compatibility. Write contract refers to the data format specification exposed by the storage system, which can be implemented by JSON Schema or Protobuf protocol, and format constraints ensure storage consistency. Standardized packets refer to data units that meet the requirements of the target storage format, which can be implemented by TLV encoding or fixed-length structures, and standardization improves storage processing efficiency. Persistence refers to the operation of permanently writing data to non-volatile storage media, which can be implemented by asynchronous I / O and write cache mechanisms, and batch submission reduces disk load.

[0030] As a preferred embodiment, the scheme of the present application is implemented as follows: The edge hub is equipped with high-performance processors and large-capacity memories to support real-time stream capture and data processing. The differential hardware bus uses low-voltage differential signal technology to support transmission rates of up to 10 Gbps. The lightweight collection agent is deployed on the edge device and uses zero-copy technology to achieve efficient byte-level data capture.

[0031] The policy engine triggers the collection agent based on preset rules, including data source type, collection frequency, and priority parameters. The data collected by the collection agent is quickly aggregated to the edge hub through memory mapping technology.

[0032] The protocol bridge uses a multi-threaded parallel processing architecture, with each thread responsible for semantic conversion of a specific protocol. The semantic conversion module includes a protocol parsing engine and a contract adapter. The protocol parsing engine supports common industrial protocols such as Modbus, MQTT, OPC UA, etc. The contract adapter maintains a write contract library for the storage array and dynamically selects the optimal conversion strategy.

[0033] The hierarchical storage array uses a hybrid storage architecture, including a cache layer, a hot data layer, and a cold data layer. The write contract defines the storage format, compression algorithm, and index structure of the data. The storage array implements automatic migration of data between different levels through intelligent scheduling algorithms.

[0034] When the system is running, the edge hub continuously monitors the data source and triggers the collection agent when a data stream is detected. The collected raw messages are transmitted to the protocol bridge through the differential hardware bus. The protocol bridge parses the raw messages, extracts the key fields, and then reconstructs them into standard messages according to the storage contract requirements. The standard messages are pushed to the hierarchical storage array, first written to the cache, and then migrated to the appropriate storage layer according to the access pattern.

[0035] The application further proposes that when consecutive raw messages with the same transaction identifier and a timestamp interval less than a set time threshold are detected, only the first occurrence of the raw message is marked as valid and delivered; subsequent raw messages with the same transaction identifier are marked as redundant and cached in an internal ring buffer; when the number of redundant messages with the same transaction identifier in the ring buffer reaches a preset number threshold, or the cache duration exceeds a set timeout window, all redundant messages are discarded.

[0036] The transaction identifier detection module obtains the transaction identifier by analyzing the message header field, and the timestamp interval calculation unit uses a sliding window algorithm to calculate the time difference between adjacent messages. The ring buffer uses a fixed capacity design, and each storage unit records the transaction identifier, timestamp, and message content. The preset number threshold is dynamically adjusted according to the buffer capacity and transaction concurrency, and the timeout window is set to a millisecond-level time range.

[0037] Specifically, the transaction identification detection module extracts the transaction identifier when capturing continuous messages, and the timestamp interval calculation unit compares whether the time difference of adjacent messages is lower than a threshold. If the condition is met, the first message is marked as valid and transmitted to the protocol bridge, and the subsequent messages are marked as redundant and written into the ring buffer. The buffer management module monitors the number of messages corresponding to each transaction identification and the cache time in real time, and when the number reaches the threshold or the time exceeds the window, a batch deletion operation is triggered. For example, when the redundant messages of a certain transaction identification in the buffer accumulate to 100 or are retained for more than 500 milliseconds, the batch of messages is uniformly cleared. This mechanism filters redundant data streams, reduces invalid data processing and storage operations, and avoids the risk of buffer overflow. As a preferred embodiment, the scheme of the present application is implemented as follows: when the edge hub detects continuous original messages with the same transaction identification and a timestamp interval less than 10 milliseconds, only the first original message is marked as valid and transmitted. Subsequent original messages with the same transaction identification are marked as redundant and cached in the internal ring buffer. When the number of redundant messages with the same transaction identification in the ring buffer reaches 100, or the cache duration exceeds 5 seconds, all redundant messages are uniformly discarded.

[0038] Specifically, the edge hub maintains a transaction identification mapping table, which records the timestamp of the most recent occurrence of each transaction identification. When a new original message is received, its transaction identification and timestamp are first extracted. Then the mapping table is queried, and if the transaction identification exists and the timestamp interval between the new message and the recorded timestamp is less than 10 milliseconds, the new message is marked as redundant. Otherwise, the new message is marked as valid, the timestamp of the transaction identification in the mapping table is updated, and the message is transmitted to the protocol bridge through the differential hardware bus.

[0039] For messages marked as redundant, the edge hub writes them into the internal ring buffer. The ring buffer manages redundant messages in groups according to transaction identification. Each group of redundant messages is set with a counter and a timer. When the number of redundant messages in a group reaches 100 or the timer exceeds 5 seconds since the first redundant message was written, a discard operation is triggered. The discard operation clears all redundant messages in the group and resets the counter and timer.

[0040] The present application further proposes that the edge hub dynamically selects original messages that meet the matching conditions according to the first compression parameter group matching rule declared by the hierarchical storage array; if the original message compression parameters do not match the first compression parameter group, the message is converted to a matching compression format according to a pre-set degradation strategy before being transmitted to the protocol bridge; a cyclic redundancy check is performed before transmission, and when the check fails, an exponential backoff retransmission mechanism is triggered, and when the number of retransmissions reaches a set upper limit, the message metadata is written into a fault log and marked as an unrecoverable error.

[0041] The dynamic selection process generates a matching rule template by analyzing the compression parameter group declared by the hierarchical storage array, including compression algorithm type, compression level, and block size parameters. When the compression parameters carried by the original message do not match the template parameters, a downgrade conversion strategy is triggered. The downgrade strategy includes a compression format conversion table, which defines the conversion priority between different compression algorithms, such as converting LZ4 compression to Zstandard compression. The cyclic redundancy check uses the CRC-32 algorithm to generate a check code, which is attached to the end of the message. The exponential backoff retransmission mechanism sets the initial retransmission interval to 200 milliseconds, and each retransmission interval is increased by 1.5 times, with a maximum of 5 retransmissions. The fault log records the message hash value, timestamp, and error code, and stores them in a separate partition.

[0042] Specifically, when the original message enters the edge hub, its compression parameters are first extracted and compared item by item with the first compression parameter group. If the compression algorithm type is GZIP and the storage array requires Snappy, the conversion table is queried according to the downgrade strategy, and the highest compatible LZO algorithm is selected for conversion. The converted message calculates the CRC-32 check value before transmission, and if the check fails, it enters the retransmission queue. The first retransmission is delayed by 200 milliseconds, the second by 300 milliseconds, and so on until the upper limit of 5 times is reached. All retransmission failed message metadata is written into the log, including the original compression parameters, converted parameters, and failure time. This process ensures that incompatible compression data can still be reliably transmitted after format adaptation, while the checksum and retransmission mechanism reduces the risk of data loss, and the log record provides traceable evidence for subsequent fault analysis.

[0043] As a preferred embodiment, the scheme of the application is implemented as follows: The edge hub dynamically selects the original message that meets the matching conditions according to the first compression parameter group matching rule declared by the hierarchical storage array. Specifically, after the edge hub receives the original message, it first checks whether the compression parameters of the message match the first compression parameter group declared by the hierarchical storage array. The matching rule can include compression algorithm type, compression level, and other dimensions.

[0044] If the original message compression parameters do not match the first compression parameter group, they are converted to a matching compression format according to the pre-set downgrade strategy before being delivered to the protocol bridge. The downgrade strategy can include reducing the compression level, switching to a more general compression algorithm, etc. For example, if the original message uses high-level LZMA compression, and the storage array only supports GZIP compression, the edge hub will convert the message to GZIP format.

[0045] A cyclic redundancy check is performed before transmission, and an exponential backoff retransmission mechanism is triggered when the check fails. The cyclic redundancy check is used to ensure the integrity of data during conversion and transmission. If the check fails, the system does not immediately retry, but uses an exponential backoff algorithm, i.e. the interval time of each retry increases exponentially, to avoid network congestion.

[0046] When the number of retransmissions reaches a set upper limit, the packet metadata is written into a fault log and marked as an unrecoverable error. This ensures that the system does not attempt to retransmit failed data indefinitely, while retaining error information for subsequent analysis and processing.

[0047] The application further proposes that the protocol bridge maintain a compression parameter mapping table, and when the second compression parameter set of the original packet does not match the declaration of the tiered storage array, the compression format is converted in priority order according to the mapping table until a match is found; an irreversible hash value of the canonical packet is generated during the conversion process, and the hash value is bound to the canonical packet as a unique data identifier and pushed; if the number of consecutive conversion failures exceeds a set retry upper limit, the system reverts to a lossless compression format and generates an alarm event.

[0048] The compression parameter mapping table contains conversion paths of various compression formats and their priority order, and the priority order is pre-configured according to compression efficiency or resource consumption. The irreversible hash value is obtained by calculating the content of the canonical packet using a pre-set hash algorithm, ensuring the tamper-proof nature of the unique data identifier. The lossless compression format uses a pre-set general compression algorithm, and its compression parameters are compatible with the write contract of the tiered storage array. The alarm event contains the number of conversion failures, the final compression format, and timestamp information, which is used to trigger subsequent maintenance operations.

[0049] Specifically, after the protocol bridge receives the original packet, it first extracts its second compression parameter set and compares it with the parameters declared by the tiered storage array. When the parameters do not match, the compression format is converted in order of priority from high to low according to the mapping table. After each conversion, a hash calculation is performed on the generated canonical packet, and the hash value is bound to the packet. If the number of consecutive conversion attempts reaches a pre-set threshold, the priority conversion process is terminated, and the conversion is completed using a lossless compression format and the packet is pushed. In this process, the binding operation of the hash value ensures that the subsequent storage link can quickly identify duplicate data through the unique identifier, and the fallback mechanism avoids packet loss due to continuous conversion failures. For example, the priority order of the mapping table can be set to first try LZ4 compression, then Zstandard, and finally GZIP; when three consecutive conversions fail, the system reverts to Snappy lossless compression format.

[0050] As a preferred embodiment, the scheme of the application is implemented as follows: The protocol bridge maintains a compression parameter mapping table. When the second compression parameter set of the original message does not match the compression format declared by the layered storage array, the protocol bridge converts the compression format step by step according to the priority order of the mapping table until a match is found. Specifically, the protocol bridge first checks the compression parameter of the received original message, and then compares it with the list of supported compression formats declared by the layered storage array in advance. If a mismatch is found, the protocol bridge queries the compression parameter mapping table maintained internally. The mapping table lists the conversion paths between various compression formats in a preset priority order. The protocol bridge attempts conversion according to the priority from high to low until a compression format compatible with the layered storage array is found.

[0051] An irreversible hash value of the canonical message is generated during the conversion process, and the hash value is bound to the canonical message as a unique data identifier and pushed. Further, after each compression format conversion is completed, the protocol bridge calculates an irreversible hash value of the converted canonical message content. This hash value is generated using a secure hash algorithm to ensure its uniqueness. The protocol bridge binds this hash value as a unique identifier of the data to the converted canonical message, and then pushes it to the layered storage array.

[0052] If the number of consecutive conversion failures exceeds the set retry upper limit, the protocol bridge reverts to a lossless compression format and generates an alarm event. For example, if the protocol bridge fails to find a matching format after trying multiple compression format conversions, and the number of failures reaches the pre-set retry upper limit, the system will adopt a conservative strategy. As a result, the protocol bridge reverts the original message to a common lossless compression format to ensure data integrity. At the same time, the system generates an alarm event to notify the administrator of the continuous conversion failure, so as to intervene manually or adjust the system in time.

[0053] The present application further proposes that the layered storage array performs a hash value comparison operation before persisting: if the hash value exists in the deduplication index, the reference count of the associated data block is incremented and storage is skipped; if the hash value does not exist, the canonical message is stored in a row-column mixed format, and the hash value is registered to the deduplication index; when the reference count value exceeds the set split threshold, a data block copy is created and the reference count is reset.

[0054] The hash value comparison operation uses a Bloom filter to achieve fast retrieval, and the false positive rate of the Bloom filter is set to 0.1%. The reference count is stored using a 32-bit unsigned integer, and the split threshold is set to 65535 times. The row-column mixed format block storage uses an alternating arrangement of Parquet columnar storage and ORC row storage, and the single data block capacity is limited to 128MB. The deduplication index is implemented using an LSM tree structure, the index key is a 160-bit SHA-1 hash value, and the index value includes the data block physical address and the reference count field.

[0055] Specifically, when the protocol bridge pushes a canonical message to the tiered storage array, it first calculates the irreversible hash value corresponding to the message. By querying the deduplication index, it determines whether the hash value already exists. If it does, it locates the associated data block and performs an atomic increment operation on its reference count, skipping the actual storage process. If it does not, it cuts the canonical message into multiple 128MB data blocks in a row-column mixed format, independently compresses each data block, and stores it to the physical medium. At the same time, it writes the hash value and data block address to the deduplication index. When the data block reference count reaches 65535 times, a complete copy of the data block is automatically created, and the original data block reference count is reset to zero. This mechanism achieves data deduplication through the uniqueness of the hash value, reducing the consumption of repeated storage. The row-column mixed storage format balances the performance requirements of batch queries and random access, and the reference count splitting mechanism avoids the risk of counter overflow caused by long-period access.

[0056] As a preferred embodiment, the scheme of the present application is implemented as follows: The tiered storage array performs a hash value comparison operation before persisting. First, the tiered storage array receives a canonical message and its bound hash value from the protocol bridge. Then, the tiered storage array queries the deduplication index it maintains internally to check whether the hash value already exists.

[0057] If the hash value exists in the deduplication index, the tiered storage array locates the data block associated with the hash value and increments the reference count of the data block by one. At this point, the tiered storage array no longer performs the actual storage operation, but directly skips the storage process to complete the current persistence task.

[0058] If the hash value does not exist in the deduplication index, the tiered storage array blocks the canonical message according to the predefined row-column mixed format. Specifically, the canonical message is divided into multiple fixed-size data blocks, each containing a certain number of row data and column data. Then, the tiered storage array stores these data blocks to different storage media or storage locations. At the same time, the tiered storage array registers the hash value in the deduplication index, establishing a mapping relationship between the hash value and the storage location.

[0059] The tiered storage array continuously monitors the reference count of each data block. When the reference count value of a certain data block exceeds the pre-set splitting threshold, the tiered storage array automatically creates a copy of the data block. After the copy is created, the tiered storage array resets the reference count of the original data block to the initial value and allocates the excess reference count to the newly created copy.

[0060] The application further proposes that the edge hub monitors the hierarchical state of the hierarchical storage array in real time; when the archiving layer data is detected and the policy engine issues a migration instruction, the migration operation is performed: first, the migration cost index is calculated, which is equal to the amount of data to be migrated multiplied by the current network delay coefficient; if the cost index is less than or equal to the preset upper threshold, the migration is immediately started; if the cost index is greater than the preset upper threshold, the migration delay time is extended in proportion to the cost index and the upper threshold, and the cost index is periodically re-evaluated during the delay period.

[0061] In the formula, the calculation of the migration cost index is achieved by multiplying the amount of data to be migrated by the real-time network delay coefficient, and the network delay coefficient is dynamically obtained by measuring the ratio of the current network round-trip delay to the reference delay. When the cost index exceeds the preset upper threshold, the extension ratio of the migration delay time is determined by a linear mapping function, for example, when the cost index is twice the threshold, the delay time is extended to twice the base delay time. The interval time for periodically re-evaluating the cost index during the delay period is set to a fixed period or an adaptive adjustment period, which is dynamically shortened or lengthened according to the change rate of historical evaluation results.

[0062] Specifically, after the policy engine issues a migration instruction, the total size of the data block to be migrated and the current network delay parameter are first obtained, and the migration cost index is generated by multiplication operation. The preset upper threshold is set according to the system available bandwidth and processor load, for example, it is set to the product of the data throughput capacity and the delay tolerance. If the calculation result does not exceed the threshold, the migration task is directly added to the execution queue; if the calculation result exceeds the threshold, the migration task is suspended and enters a delay waiting state, and the delay time is positively correlated with the over-limit ratio. During the delay waiting period, the network delay and data volume parameters are re-collected every fixed time window, the cost index is recalculated and compared with the threshold. If the recalculated result falls below the threshold, the migration task is immediately started; if it is still higher than the threshold, the waiting time is continued to be extended until the condition is met. Through the periodic evaluation mechanism, it is ensured that the migration operation is only executed under the condition that the system resources allow, avoiding the occupation of too many resources in the migration process due to sudden network fluctuations or data surges.

[0063] As a preferred embodiment, the scheme of the application is implemented as follows: The edge hub monitors the hierarchical state of the hierarchical storage array in real time. When the archiving layer data is detected and the policy engine issues a migration instruction, the migration operation is performed. First, the migration cost index is calculated, which is equal to the amount of data to be migrated multiplied by the current network delay coefficient. If the cost index is less than or equal to the preset upper threshold, the migration is immediately started. If the cost index is greater than the preset upper threshold, the migration delay time is extended in proportion to the cost index and the upper threshold, and the cost index is periodically re-evaluated during the delay period.

[0064] For example, the edge hub monitors the hierarchical state of the tiered storage array by periodic polling or event triggering. When detecting the data in the archive tier, the policy engine determines whether data migration is needed according to preset rules. If the policy engine issues a migration instruction, the edge hub starts to perform the migration operation.

[0065] The migration operation first calculates the migration cost index. Assuming that the amount of data to be migrated is 100 GB and the current network delay coefficient is 1.5, the migration cost index is 150. If the preset upper threshold is 200, since 150 is less than 200, the system will immediately start the migration process.

[0066] If the calculated migration cost index is 250, which is greater than the preset upper threshold 200, the system will extend the migration delay time. The calculation method of the delay time is: delay time = base delay time * (cost index / upper threshold). Assuming that the base delay time is 10 minutes, the actual delay time is 12.5 minutes. During the 12.5-minute delay period, the system re-evaluates the cost index every certain time, such as 1 minute, until the cost index decreases to an acceptable range or the maximum number of retries is reached.

[0067] The present application further proposes that the policy engine performs a load prediction operation before issuing a migration instruction to obtain the central processor utilization and memory occupancy of the edge hub. If the central processor utilization is lower than the set processor threshold and the memory occupancy is lower than the set memory threshold, the migration instruction is immediately confirmed. If any index exceeds the threshold, the number of simultaneous migration tasks is reduced by an over-limit proportion until the system load decreases to a safe threshold range.

[0068] The central processor utilization and memory occupancy are collected in real time through the operating system interface, the processor threshold is set to 70%, and the memory threshold is set to 80%. The over-limit proportion is calculated as the larger value of the ratio of the central processor utilization exceeding part to the processor threshold or the ratio of the memory occupancy exceeding part to the memory threshold. The migration task number adjustment adopts a linear decreasing algorithm, and the number of each adjustment is equal to the current total number of tasks multiplied by the over-limit proportion. The safe threshold range is defined as the central processor utilization being lower than 60% and the memory occupancy being lower than 70%.

[0069] Specifically, the system periodically acquires the central processor and memory usage data of the edge hub through monitoring the process, and compares them with the preset threshold in real time. When both indicators do not exceed the threshold, the migration instruction is immediately executed to ensure that the migration task is started under the condition of sufficient resources. When any indicator exceeds the threshold, the system dynamically calculates the allowed upper limit of parallel migration tasks according to the overrun degree. For example, when the central processor utilization is 85%, the overrun ratio is (85-70) / 70=21.4%, and the number of parallel migration tasks is reduced by 21.4% at this time. This adjustment process continues until the central processor and memory usage rate falls within the safe range. In this process, the migration tasks that are not allowed to execute enter the waiting queue and wait for resource release for rescheduling. This mechanism effectively balances the resource allocation between data processing tasks and migration tasks, preventing performance degradation caused by system overload.

[0070] As a preferred embodiment, the scheme of the application is implemented as follows: The policy engine performs load prediction before issuing the migration instruction: acquires the central processor utilization and memory occupancy of the edge hub; if the central processor utilization is lower than the set processor threshold and the memory occupancy is lower than the set memory threshold, the migration instruction is immediately confirmed; if any indicator exceeds the threshold, the number of simultaneous migration tasks is reduced by the overrun ratio until the system load falls within the safe threshold range.

[0071] Specifically, the policy engine can acquire the central processor utilization and memory occupancy of the edge hub in real time through the system monitoring interface. For example, the set processor threshold is 80% and the memory threshold is 75%. When it is detected that the central processor utilization is 70% and the memory occupancy is 65%, the policy engine judges that both indicators are lower than the threshold, so the migration instruction is immediately confirmed.

[0072] If the central processor utilization is detected to be 85%, which exceeds the set threshold, the policy engine will calculate the overrun ratio as (85%-80%) / 80%=6.25%. According to this, the policy engine will reduce the number of simultaneous migration tasks by 6.25%. For example, the original plan is to execute 16 migration tasks simultaneously, and now it is adjusted to 15. The policy engine continues to monitor the system load and re-evaluates every certain time interval until the system load falls within the safe threshold range.

[0073] The application further proposes a light-weight collection agent's crawling window adjustment method, including the following steps: initializing the crawling window to the maximum window size; measuring the network round-trip time in real time; if the current round-trip time is measured more than a set multiple of the historical average time delay for several times in a row, then dynamically reduce the window size according to the ratio of the current window size to the round-trip time; when the round-trip time returns to the normal range and the duration of stability exceeds the set stability threshold, gradually increase the window size until the maximum window size.

[0074] The initialization phase sets the fetch window to the maximum size to maximize the initial throughput capacity. The network round-trip time is measured by sending probe packets and recording the response time, and the historical average time is calculated using a sliding window algorithm. When the measured time exceeds 1.5 times the historical average for three consecutive times, the window reduction mechanism is triggered, and the new window size is calculated as the current size multiplied by the ratio of the historical time to the current time. The window recovery phase uses a linear growth strategy, increasing the original size by 10% each time until reaching the initial maximum value. The stability threshold is set to 5 seconds to confirm the persistence of network state recovery.

[0075] Specifically, the fetch window adjustment process is divided into three stages. First, the maximum window size is used in the initial stage to quickly capture data flow. When the network time delay abnormally increases, the window size is reduced to reduce the amount of data sent per unit time, and network congestion is alleviated. For example, if the current window is 1000 bytes and the time delay increases to 2 times the average, the window size will be adjusted to 500 bytes. After the time delay returns to normal, the system continuously monitors the time delay fluctuation within 5 seconds, which does not exceed ±10%, and then starts the window recovery process, which increases by 100 bytes each time until it recovers to 1000 bytes. This dynamic adjustment mechanism balances data throughput efficiency and network transmission stability through real-time feedback control, avoiding performance fluctuations caused by fixed window size.

[0076] As a preferred embodiment, the scheme of the application is implemented as follows: when the lightweight collection agent is deployed in the edge hub, the fetch window is first initialized to the maximum window size, which is determined by the maximum transmission unit of the underlying network interface. The real-time measurement module periodically collects network round-trip time data, with a sampling frequency of ten times per second, and the measurement results are stored in a sliding time window queue for smoothing processing.

[0077] When the current round-trip time is detected to be more than twice the historical average time for three consecutive times, the window adjustment mechanism is triggered. The adjustment algorithm calculates the window reduction factor based on the ratio of the current window size to the round-trip time, where the ratio is obtained by dividing the window size by the current round-trip time in milliseconds. Each reduction operation updates the window size to the original size multiplied by the reduction factor, while setting a minimum window size lower limit to prevent excessive shrinkage.

[0078] When the network round-trip time falls within 1.2 times the historical average time and is stably maintained for more than thirty seconds, the window recovery mechanism is started. The recovery process uses a gradual growth strategy, increasing the current window size by 20% each time until reaching the initial maximum window size. After each size change, the next adjustment is performed after waiting for two round-trip time periods to prevent frequent oscillation.

[0079] The application further proposes that the logic of the event sensor performing event classification includes: extracting an event feature vector, which contains time stamp, protocol type and data volume dimensions; calculating the similarity score of the vector with the nearest neighbor sample in the historical event library; if the similarity score is greater than or equal to the set classification threshold, it is marked as a regular event and assigned a basic priority; if the similarity score is less than the set classification threshold, it is marked as an abnormal event, and the priority is raised based on the difference ratio between the score and the threshold.

[0080] The feature vector is composed of time stamp, protocol type and data volume dimensions, and the event attributes are described by multi-dimensional data. The similarity score calculation uses the Euclidean distance algorithm, and the historical event library stores standardized feature samples. The classification threshold is obtained by training historical data, and the dynamic adjustment period is 24 hours. The abnormal event priority raising ratio is linearly related to the score difference, and the priority is raised by one level for every 10% reduction in the difference.

[0081] Specifically, the event sensor first extracts the time stamp, protocol type and data volume dimensions from the original message to generate a feature vector. After normalization, the vector is matched with the samples stored in the historical event library to calculate the similarity score. When the score is higher than the classification threshold, it is determined as a regular event and assigned a basic priority to enter the processing queue; when the score is lower than the threshold, it is determined as an abnormal event, and the priority raising amplitude is calculated according to the difference between the score and the threshold. For example, when the threshold is 0.8 and the current score is 0.6, the difference ratio is 25%, and the priority is raised by two levels. This mechanism ensures that abnormal events get higher processing weight, while avoiding excessive resource occupation by regular events.

[0082] As a preferred embodiment, the scheme of the application is implemented as follows: The rate limiting control operation of the traffic shaping detection node is realized by a monitoring module deployed at the network interface. The module periodically collects the instantaneous throughput of the current network link, with the collection period set to milliseconds. The calculation of the bandwidth change slope is based on the throughput data of the last five periods in the time series, and the least squares method is used to fit the slope value of the linear change trend. When it is monitored that the absolute value of the fitted slope exceeds the preset alert slope threshold for three consecutive periods, the rate limiting algorithm generates a new transmission rate. The new rate is determined by multiplying the current rate by an adjustment coefficient, and the adjustment coefficient is dynamically calculated by the ratio of the absolute value of the slope to the alert slope. After the rate limiting operation is executed, the throughput fluctuation is continuously monitored, and if the fluctuation amplitude is lower than the preset stability threshold for ten consecutive periods, the original transmission rate is gradually restored by incremental method.

[0083] The application further proposes a rate limiting control method of a traffic shaping detection node, comprising: counting current instantaneous throughput; calculating a bandwidth change slope in a unit time; if the absolute value of the slope is greater than a set warning slope and the over-limit period is continuously met, generating a rate limiting instruction, wherein the new rate is equal to the current rate multiplied by a subtraction of the ratio of the absolute value of the slope and the warning slope; periodically sampling the throughput fluctuation rate after rate limiting, and if the fluctuation rate is lower than a set fluctuation threshold for multiple times, gradually restoring the original rate by a fixed step.

[0084] The instantaneous throughput is counted by a sliding time window, and the window length is set to millisecond level precision to ensure real-time performance. The least square method is used to fit the data trend of the latest five sampling points to calculate the bandwidth change slope, and the warning slope is set to 0.5 Mbps / ms. In the rate limiting instruction generation stage, when the absolute value of the slope exceeds the warning slope and lasts for three sampling periods, the rate adjustment operation is triggered. The calculation of the new rate introduces a proportional factor, for example, when the absolute value of the slope reaches 1.2 times the warning slope, the rate is reduced by 20% of the current rate. The sampling interval of the throughput fluctuation rate is set to 500 ms, and the fluctuation threshold is set to ±2%. When the fluctuation rate is lower than the threshold for three times, the rate is increased by 5% each time until the original level.

[0085] Specifically, the method first captures the current network traffic state through a high-precision time window, and calculates the bandwidth change trend in real time. When it is detected that the bandwidth change rate in a unit time exceeds the preset warning level and lasts for an over-limit period, the rate reduction mechanism is automatically triggered, and the transmission rate is dynamically reduced by the over-limit proportion. This nonlinear adjustment method can quickly suppress burst traffic and avoid resource idling caused by sudden rate drop. After the rate limiting takes effect, the system continuously monitors the network fluctuation, and when it is detected that the traffic tends to be stable, a gradual recovery strategy is used to gradually increase the transmission rate. The whole process controls the adjustment range through quantitative indicators, for example, the warning slope is associated with the rate adjustment factor to ensure that the network state change and the control intensity form a positive feedback. The method realizes the dynamic allocation of bandwidth resources through a closed-loop control mechanism, maintains the transmission efficiency, and effectively suppresses network jitter.

[0086] As a preferred embodiment, the scheme of the application is implemented as follows: the rate limiting control process of the traffic shaping probe node is realized by the cooperation of the data plane and the control plane. The current instantaneous throughput is collected periodically and input into the sliding time window statistics module, and the window time span is set to a fixed sampling period. The calculation of the bandwidth change slope is performed by fitting the throughput change curve in unit time by the least square method, and the absolute value of the slope is determined by the ratio of the covariance to the variance. The warning slope threshold is preset as an empirical value and stored in the configuration file, and the overrun period is recorded by the timer for the duration of continuously meeting the slope overrun condition. When the rate limiting instruction is generated, the new rate calculation module multiplies the current rate by the slope adjustment coefficient, and the adjustment coefficient is determined by the quotient of the absolute value of the slope and the warning slope. In the rate recovery stage, a linear incremental strategy is adopted, and the fixed step size is set to a preset percentage of the original rate. The throughput fluctuation rate is calculated in the sampling period by the standard deviation algorithm, and the fluctuation threshold is dynamically calibrated according to the historical baseline.

[0087] Through the above technical scheme, the bandwidth oscillation problem caused by network traffic mutation is effectively solved, and the smooth transition of the transmission rate is realized through the dynamic rate adjustment mechanism based on the mathematical model. Through the dual judgment conditions of the slope threshold and the overrun period, the false rate limiting operation caused by instantaneous fluctuation is avoided. In the rate recovery stage, the fluctuation rate is monitored to prevent secondary congestion caused by premature rate recovery, and the stability and reliability of network transmission are ensured.

[0088] The above only describes the embodiments of the application and does not limit the protection scope of the application. For those skilled in the art, the application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. An information system integration service data intelligent processing storage system, characterized in that, Comprise: Edge hub, protocol bridge and tiered storage array; The edge hub comprises a differential hardware bus for data service output, the original message output by the differential hardware bus is processed by the protocol bridge and is executed by the tiered storage array to fall disk; The edge hub communicates with the protocol bridge through the differential hardware bus, and the protocol bridge communicates with the tiered storage array; The edge hub has a real-time stream capture function, which triggers a lightweight collection agent based on a policy engine to perform byte-level capture in a cross-protocol heterogeneous integration scene, and gathers the data collected by the collection agent to obtain original messages; When the edge hub outputs high-rate original messages on the differential hardware bus, it sends the original messages to the protocol bridge through the differential hardware bus; The protocol bridge performs semantic conversion on the received original message, and pushes the specification message converted by the semantic conversion to the tiered storage array; The semantic conversion comprises: converting the original message into a specification message adapted to the write contract supported by the tiered storage array; The tiered storage array persists the received specification message.

2. The system of claim 1, wherein: When the edge hub detects continuous original messages with the same transaction identifier and a time stamp interval less than a set time threshold, it only marks the first appearing original message as valid and transmits it; Subsequent original messages with the same transaction identifier are marked as redundant and cached in an internal ring buffer; When the number of redundant messages with the same transaction identifier in the ring buffer reaches a preset number threshold, or the cache duration exceeds a set timeout window, all redundant messages are discarded.

3. The system of claim 1, wherein: The edge hub dynamically selects original messages that meet the matching conditions according to the first compression parameter group matching rule declared by the tiered storage array; If the original message compression parameters do not match the first compression parameter group, the message is converted to a matching compression format according to a preset degradation strategy before being transmitted to the protocol bridge; Before transmission, a cyclic redundancy check is performed, and when the check fails, an exponential backoff retransmission mechanism is triggered. When the number of retransmissions reaches a set upper limit, the message metadata is written to a fault log and marked as an unrecoverable error.

4. The system of claim 1, wherein: The protocol bridge maintains a compression parameter mapping table, and when the second compression parameter group of the original message does not match the declaration of the tiered storage array, it converts the compression format step by step according to the priority order of the mapping table until a match is found; An irreversible hash value of the specification message is generated during the conversion process, and the hash value is used as a unique data identifier and bound to the specification message for pushing; If the number of consecutive conversion failures exceeds a set retry upper limit, the system reverts to lossless compression format and generates an alarm event.

5. The system of claim 4, wherein: The tiered storage array performs a hash value comparison operation before persistence: If the hash value exists in the deduplication index, the reference count of the associated data block is incremented and storage is skipped; If the hash value does not exist, the specification message is stored in a mixed format of rows and columns, and the hash value is registered to the deduplication index; When the reference count value exceeds a set split threshold, a data block copy is created and the reference count is reset.

6. The system of claim 1, wherein: The edge hub monitors the hierarchical state of the tiered storage array in real time; When the archive tier data is detected and the policy engine issues a migration instruction, a migration operation is performed: First, a migration cost index is calculated, which is equal to the amount of data to be migrated multiplied by the current network delay coefficient; If the cost index is less than or equal to the preset upper threshold, the migration is immediately started; If the cost index is greater than the preset upper threshold, the migration delay time is extended by the ratio of the cost index to the upper threshold, and the cost index is periodically re-evaluated during the delay period.

7. The system of claim 6, wherein: The policy engine performs a load prediction operation before issuing a migration instruction: The central processor utilization and memory occupancy of the edge hub are obtained; If the central processor utilization is below the set processor threshold and the memory occupancy is below the set memory threshold, the migration instruction is immediately confirmed; If any of the indicators exceeds the threshold, the number of simultaneous migration tasks is reduced by the exceeding ratio until the system load falls within the safe threshold range.

8. The system of claim 1, wherein: The light-weight collection agent's crawling window adjustment method includes: Initialize the crawling window to the maximum window size; Real-time measurement of network round-trip delay; If the current round-trip delay is measured for multiple times continuously and exceeds the set multiple of the historical average delay, the window size is dynamically reduced according to the ratio of the current window size to the round-trip delay; When the round-trip delay returns to the normal range and remains stable for a duration longer than the set stability threshold, the window size is gradually increased until the maximum window size.

9. The system of claim 8, wherein: The event sensor's event classification logic includes: Extract an event feature vector, which contains time stamp, protocol type, and data volume dimensions; Calculate the similarity score of the vector to the nearest neighbor sample in the historical event library; If the similarity score is greater than or equal to the set classification threshold, it is marked as a regular event and assigned a basic priority; If the similarity score is less than the set classification threshold, it is marked as an abnormal event, and the priority is raised based on the difference ratio between the score and the threshold.

10. The system of claim 9, wherein: The traffic shaping probe node's rate limiting control method includes: Statistical current instantaneous throughput; Calculate the bandwidth change slope per unit time; If the absolute value of the slope is greater than the set alert slope and continues to exceed the threshold for a period of time, a rate limiting instruction is generated, and the new rate is equal to the current rate multiplied by one minus the ratio of the absolute value of the slope to the alert slope; After rate limiting, periodically sample the fluctuation rate, and if the fluctuation rate is lower than the set fluctuation threshold for multiple times continuously, gradually restore the original rate by a fixed step.

Images