Intelligent compression transmission method and system of message body

Abstract

The application provides a smart compression transmission method and system for a message body, which realizes efficient compression and smart transmission of structured messages through technical means such as structured data template compression, type-aware entropy coding, real-time network quality monitoring and dynamic compression strategy decision, and can adaptively adjust the compression strategy according to the dynamic change of the network environment, thereby improving the data transmission efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer network communication technology, and in particular to an intelligent compression and transmission method and system for message bodies, which is mainly used for intelligent compression and efficient transmission of structured message data. Background Technology

[0002] Data compression and transmission technology is a key technology area in computer networks and distributed systems. With the explosive growth of Internet applications, efficient transmission of large amounts of data has become one of the core challenges of modern network systems. Currently, most network application systems use general compression algorithms to compress data before transmission in order to reduce network bandwidth consumption and improve transmission efficiency.

[0003] In existing technologies, common compression techniques mainly include dictionary-based general compression algorithms and entropy coding-based compression algorithms. For example, GZIP uses a combination of the LZ77 algorithm and Huffman coding to achieve data compression; while LZ4 focuses on providing extremely fast compression and decompression speeds, sacrificing some compression ratio for higher processing speed. These algorithms exhibit different characteristics when processing various types of data and are widely used in various network transmission scenarios.

[0004] Current mainstream compression transmission solutions typically employ a static configuration approach to select compression algorithms, meaning that compression parameters and algorithm types are pre-set based on estimated network conditions and data characteristics during system deployment. This approach faces significant challenges when processing structured data (such as JSON and XML) because general-purpose compression algorithms cannot recognize the semantic structure of this data, resulting in low compression efficiency. Furthermore, these algorithms apply the same compression strategy to all data, failing to provide targeted optimization based on data type characteristics.

[0005] However, the above technical solutions have obvious shortcomings: First, general compression algorithms generally have low compression rates for structured data and cannot make full use of the structural characteristics of the data; second, they lack business semantic awareness capabilities and it is difficult to achieve a dynamic balance between compression rate, speed and resource overhead; in addition, statically configured compression strategies cannot adapt to the dynamic changes in network environment and business data, and it is difficult to maintain optimal performance under complex and ever-changing network conditions. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an intelligent compression and transmission method and system for message bodies. This method achieves efficient compression and intelligent transmission of structured messages through techniques such as structured data template compression, type-aware entropy coding, real-time network quality monitoring, and dynamic compression strategy decision-making. It can adaptively adjust the compression strategy according to the dynamic changes in the network environment, thereby improving data transmission efficiency.

[0007] To achieve the above objectives, the present invention provides an intelligent compression and transmission method for message bodies, comprising the following steps:

[0008] The system receives a structured message data stream, converts the structured message data stream into an abstract syntax tree through a parser, traverses each node of the abstract syntax tree using a depth-first search algorithm and records the complete path, identifies the node type of each node and calculates the message structure fingerprint.

[0009] Based on the message structure fingerprint, template matching or new template creation is performed to obtain a template dictionary containing the mapping relationship between field paths and field IDs;

[0010] Based on the template dictionary and the original message data in the structured message data stream, type identification is performed on each data field, encoding method is selected according to data type and encoding operation is performed, and the encoded field data is assembled in order of field ID to obtain type-encoded compressed data;

[0011] The round-trip delay, bandwidth, packet loss rate, and jitter indicators during network transmission are collected. The round-trip delay, bandwidth, packet loss rate, and jitter indicators are standardized and a weighted average formula is applied to obtain a comprehensive network quality score.

[0012] Based on the comprehensive network quality score and historical compression effect data, a compression strategy is dynamically selected using an adaptive learning mechanism, and compression configuration parameters corresponding to the compression strategy are obtained.

[0013] Based on the compression configuration parameters and the template dictionary, the type-encoded compressed data is further compressed, and a template ID and compression algorithm identifier are added to the message header to obtain the compressed message body, which is then sent through the network channel.

[0014] The receiving end receives and parses the message header of the message body to obtain the template ID and the compression algorithm identifier. Based on the template ID, it searches for the corresponding template from the local template library, uses the corresponding decompression algorithm to restore the original message data.

[0015] Optionally, identifying the node type of each node and calculating the message structure fingerprint includes:

[0016] The abstract syntax tree is subjected to node type identification, which distinguishes between basic key-value pairs, arrays and nested objects. The internal element structure of array types is analyzed, and primitive type arrays, object arrays with consistent structure or unstructured mixed arrays are identified. The node type classification results are then output.

[0017] Based on the node type classification results, all path and type pairing information is collected, and the hash value is calculated after sorting the pairing information to obtain the message structure fingerprint;

[0018] The step of performing template matching or creating a new template based on the message structure fingerprint to obtain a template dictionary containing the mapping relationship between field paths and field IDs includes:

[0019] The message structure fingerprint is compared with existing templates in the template library to determine whether there is a matching existing template. If a matching template is found, the ID of the matching template is used. If no match is found, a new matching template is created and a unique matching template ID is assigned.

[0020] The frequency of occurrence of fields in the new matching template is counted, a first ID is assigned to the first frequency field, a second ID is assigned to the second frequency field, and a mapping relationship from field path to field ID is constructed to obtain the template dictionary containing the mapping relationship from field path to field ID.

[0021] Optionally, the data types in the template dictionary and the original message data include integer, floating-point, string, boolean, and null values. Based on the template dictionary and the original message data in the structured message data stream, type identification is performed on each data field, an encoding method is selected according to the data type, and encoding is performed. The encoded field data is then assembled in order of field ID to obtain type-encoded compressed data, including:

[0022] For the integer type data in the template dictionary and the original message data, apply ZigZag transform to map signed integers to unsigned integers, and then perform VarInt variable-length encoding to obtain integer encoded data;

[0023] Perform precision analysis on floating-point data, and select IEEE 754 half-precision or standard double-precision representation according to precision requirements to obtain floating-point encoded data;

[0024] Pattern recognition is performed on string data, date strings are converted into timestamps or numeric strings are converted into actual numeric values ​​and then encoded, and Huffman encoding is applied to ordinary strings to obtain string encoded data;

[0025] Bitmap encoding is used for Boolean data and Null values ​​to pack multiple values ​​into the same byte, resulting in Boolean encoded data.

[0026] Based on the field ID order defined in the template dictionary, the integer encoded data, the floating-point encoded data, the string encoded data, and the boolean encoded data are assembled to obtain the compressed data after the type encoding.

[0027] Optionally, the step of dynamically selecting a compression strategy based on the comprehensive network quality score and historical compression effect data, combined with an adaptive learning mechanism, to obtain compression configuration parameters adapted to the current network conditions includes:

[0028] Based on the comprehensive network quality score, three network quality states are set: a score greater than or equal to 0.8 is set as the first state EXCELLENT, a score between 0.5 and 0.8 is set as the second state GOOD, and a score less than 0.5 is set as the third state POOR.

[0029] The first state configuration uses the LZ4 high-speed low compression ratio algorithm, the second state configuration uses the ZSTD balanced algorithm, and the third state configuration enables template compression and entropy coding to maximize the compression ratio, outputting the initial compression strategy configuration;

[0030] Record a decision snapshot for each compression operation, measure and record the actual compression ratio, compression time, and end-to-end message latency, calculate the comprehensive benefit function, and output the historical data of the compression effect;

[0031] Based on the historical data of the compression effect, a contextual gambling machine model or regression model is constructed to learn the mapping relationship between network indicators and returns. The parameters of the contextual gambling machine model or regression model are updated periodically using the accumulated effect data, and the weights and state thresholds of the network indicators are optimized to obtain the optimized model parameters.

[0032] Based on the current network quality status, the initial compression strategy configuration, and the optimized model parameters, a corresponding compression algorithm and parameter configuration are selected to obtain the compression configuration parameters adapted to the current network conditions.

[0033] Optionally, based on the historical data of the compression effect, a contextual gambling machine model or regression model is constructed to learn the mapping relationship between network indicators and returns. The model parameters are periodically updated using accumulated performance data, and the weights and state thresholds of each indicator are optimized to obtain the optimized model parameters, including:

[0034] Feature extraction is performed on the historical data of compression effect. Network indicator features and compression decision features are extracted as inputs, and comprehensive benefits are extracted as outputs to obtain a training dataset.

[0035] Based on the training dataset, a prediction model is constructed using the contextual gambling machine algorithm or the gradient boosting regression algorithm to learn the mapping relationship from network metrics to compression strategy returns, thereby obtaining an initial prediction model.

[0036] For the initial prediction model, incremental training is performed using newly accumulated effect data at a set period to update the model weight parameters and obtain the updated prediction model.

[0037] Based on the updated prediction model, the weight coefficients of each indicator in the comprehensive network quality score and the threshold parameters of the network quality status are adjusted to obtain the optimized model parameters.

[0038] Optionally, the step of further compressing the type-encoded compressed data according to the compression configuration parameters and the template dictionary, adding a template ID and compression algorithm identifier to the message header to obtain a compressed message body, and sending the compressed message body through a network channel includes:

[0039] The sending end checks whether the receiving end has the template information corresponding to the template dictionary. If the receiving end does not have the template information, the sending end constructs a control message containing complete template content and outputs a template synchronization control message.

[0040] The template synchronization control message is sent to the receiving end and a confirmation reply is awaited. The receiving end stores the template and returns a confirmation, thus establishing template consensus and obtaining the template synchronization status.

[0041] Based on the template synchronization status and the compression configuration parameters, the currently used compression algorithm and its corresponding parameters are determined, and the corresponding compression algorithm is applied to the type-encoded compressed data to compress it, resulting in further compressed data.

[0042] In the further compressed data, the template ID, the compression algorithm identifier, and decompression parameters are added to the message header to obtain the complete compressed message body;

[0043] The complete compressed message body is sent through the network channel.

[0044] Optionally, the step of performing template matching or creating a new template based on the message structure fingerprint to obtain a template dictionary containing the mapping relationship between field paths and field IDs includes:

[0045] The message structure in the message structure fingerprint is converted into a high-dimensional sparse vector representation, where each field path is used as a dimension. Existing fields are assigned weight values ​​according to their data types, and non-existent fields are set to have a dimension value of 0. The message structure vector is then output.

[0046] Multiple hash functions are created using a symbolic random projection method. A set of random vectors is selected from the multiple hash functions. The dot product of the message structure vector and each of the random vectors is calculated, and the sign of the dot product is taken as one bit of the hash value to obtain the hash signature.

[0047] Using a combination of k hash functions as an index, the hash value of the templates stored in the template library is calculated and L hash tables are constructed to form a template hash index with a multi-level LSH index structure, where k and L are positive integers;

[0048] Obtain the hash signature of the new message, search in parallel in the L hash tables of the template hash index for candidate templates with the same or similar hash signatures, calculate the actual similarity between the returned candidate set and the message structure vector, select the best match with a similarity exceeding a threshold, and obtain the matching template ID.

[0049] Based on the matched template ID, if a matching template is found, it is used directly; otherwise, a new matching template is created and a unique template ID is assigned. A mapping relationship between field paths and field IDs is constructed, resulting in the template dictionary containing the mapping relationship between field paths and field IDs.

[0050] Optionally, the step of using a combination of k hash functions as an index to calculate hash values ​​for templates already stored in the template library and constructing L hash tables to form a multi-level LSH index structure for the template hash index includes:

[0051] For the first hash function, select m independent symbolic random projection functions, calculate m hash values ​​for each stored template, and combine the m hash values ​​as the index key of the corresponding template in the first hash table to obtain the first-level hash table, where m is a positive integer;

[0052] For the remaining k-1 hash functions, (k-1)*m independent symbolic random projection functions are generated using different random seeds. The hash value is calculated for each stored template and the corresponding hash table is constructed to obtain a multi-level LSH hash table set, where k is greater than or equal to 2.

[0053] Based on each hash table in the multi-level hash table set, templates with the same index key are stored in the same hash bucket, and a mapping relationship from hash key to template list is established to obtain the template hash index.

[0054] Optionally, the round-trip delay, bandwidth, packet loss rate, and jitter indicators collected during network transmission are standardized and a weighted average formula is applied to obtain a comprehensive network quality score, including:

[0055] Construct a weighted graph of network topology, wherein each node is represented as a vertex in the graph, the communication links between nodes are represented as edges, and the edge weight is the link's RTT or bandwidth;

[0056] The network topology weighted graph is decomposed into n subgraphs by applying the directional low-diameter decomposition algorithm, and the communication path length between any two points in each subgraph is ensured to be no more than O(log n / (log log n)), thus obtaining a set of low-diameter subgraphs, where n is a positive integer;

[0057] For each of the subgraphs in the set of low-diameter subgraphs, select the node with the highest centrality as the main monitoring point, design the aggregation path of the monitoring data to ensure that the data is transmitted along the shortest path within the subgraph, and determine the layout of the hierarchical monitoring points;

[0058] Based on the hierarchical monitoring point layout, network quality assessment is performed independently within each subgraph, and a comprehensive local network quality score is calculated. The main monitoring point of each subgraph transmits the local assessment results to the global coordination node. The global coordination node integrates all local assessment results to generate a global network quality map and outputs local and global scores.

[0059] Based on the local and global scores, the monitoring frequency is dynamically adjusted according to the stability of each subgraph, and the final comprehensive network quality score is determined.

[0060] Optionally, based on the comprehensive network quality score and historical compression performance data, and combined with an adaptive learning mechanism, a compression strategy is dynamically selected to obtain compression configuration parameters corresponding to the compression strategy, including:

[0061] Based on the set of low-diameter subgraphs, a policy propagation tree is constructed in each subgraph. The node with the best computing power and network connectivity in each subgraph is selected as the master decision node. The master decision node serves as the root node of the policy propagation tree, used to optimize propagation delay and construct the policy propagation tree structure.

[0062] Based on the policy propagation tree structure, policy caching is set at different levels, including: the main decision node caches the complete policy model and parameters, the root nodes of subtrees in the policy propagation tree cache simplified policy rules, and the leaf nodes only store the currently applicable specific compressed configuration parameters.

[0063] The design of the consistency protocol ensures that all nodes within the same subgraph adopt a consistent compression strategy under similar network conditions, including: adopting a version number-based strategy synchronization mechanism, exchanging strategy version information between nodes periodically, and automatically synchronizing to the latest version when version differences are found;

[0064] For nodes located at the boundaries of multiple subgraphs, a cross-subgraph coordination strategy is designed to allow boundary nodes to merge compression strategies from different subgraphs. In this strategy, multiple compression configurations are maintained in parallel at each boundary node, and the compression configurations are dynamically switched according to the source subgraph of the message.

[0065] Based on the consistent compression strategy and the cross-subgraph coordination strategy, combined with the comprehensive network quality score and historical compression effect data, the optimal compression strategy is dynamically selected to obtain the compression configuration parameters adapted to the current network conditions.

[0066] Optionally, based on the template dictionary and the original message data in the structured message data stream, type identification is performed on each data field, an encoding method is selected according to the data type and an encoding operation is performed, and the encoded field data is assembled in order of field ID to obtain type-encoded compressed data, including:

[0067] The frequency of each symbol in the input string data is counted. The sliding window technique is used to obtain the symbol frequency in multiple messages. Independent frequency statistics tables are used to maintain different types of data to determine the symbol frequency distribution.

[0068] Based on the symbol frequency distribution, a variety of possible encoding schemes are generated, including Huffman coding, arithmetic coding, or range coding with different parameters. The theoretical compression ratio and coding speed of each encoding scheme are calculated to obtain a candidate set of encoding schemes.

[0069] In the candidate encoding scheme set, the problem of selecting candidate encoding schemes is transformed into a problem of sampling from a discrete distribution. Using an efficient low-variance resampling algorithm, an equally spaced circle of size M is created, and M points are evenly placed on the circle. Each candidate scheme is assigned an arc length proportional to its weight. A starting point is randomly selected from the range of zero to one divided by M. Starting from the starting point, the algorithm moves along the circle with a fixed step size of one divided by M to select the candidate scheme corresponding to each region it falls into, thus obtaining the set of selected encoding schemes, where M is a positive integer.

[0070] Actual encoding tests are performed on the selected encoding scheme set to measure the actual compression ratio and speed. The weights are updated based on the test results to obtain the optimized encoding scheme.

[0071] Based on the optimized encoding scheme, an encoding operation is performed on each data field, and the encoded field data is assembled in order of field ID to obtain the compressed data encoded in the specified type.

[0072] Optionally, in the candidate coding scheme set, the problem of selecting candidate coding schemes is transformed into a problem of sampling from a discrete distribution. Using an efficient, low-variance resampling algorithm, an equally spaced circle of size M is created, and M points are uniformly placed on the circle. Each candidate scheme is assigned an arc length proportional to its weight. A starting point is randomly selected from a range of zero to one divided by M. Starting from the starting point, the algorithm moves along the circle with a fixed step size of one divided by M, selecting the candidate scheme corresponding to each region it falls into, thus obtaining a set of selected coding schemes, including:

[0073] For each candidate scheme in the candidate set of the encoding scheme, a weight value for each candidate scheme is calculated based on its expected performance, wherein the weight value is directly proportional to the compression ratio and inversely proportional to the encoding time. All the weight values ​​are normalized to obtain a normalized weight distribution.

[0074] Based on the normalized weight distribution, a circular structure is created with a total length of 1. The circle is divided into M regions, each region corresponding to a candidate solution. The length of the i-th region is equal to the normalized weight of the i-th candidate solution, thus obtaining the weighted circular structure, where the value of i ranges from 1 to M.

[0075] For the weighted circular structure, a random number is generated as the starting position. The random number is in the range of zero to one divided by M. Starting from the starting position, M sampling points are marked on the circumference in sequence according to a fixed step size of one divided by M to obtain the set of sampling point positions.

[0076] For each sampling point in the set of sampling point locations, determine the specific region where each sampling point falls within the weighted circular structure, add the candidate schemes for the corresponding region to the selected set, and obtain the selected encoding scheme set.

[0077] This invention also provides an intelligent compressed transmission system for message bodies, comprising: an identification and calculation module, a matching module, an encoding module, a weighting module, a compression configuration module, a compression module, and a restoration module. The identification and calculation module receives a structured message data stream, converts it into an abstract syntax tree using a parser, traverses each node of the abstract syntax tree using a depth-first search algorithm and records the complete path, identifies the node type of each node, and calculates a message structure fingerprint. The matching module performs template matching or creates new templates based on the message structure fingerprint to obtain a template dictionary containing field path-to-field ID mappings. The encoding module identifies the type of each data field based on the template dictionary and the original message data in the structured message data stream, selects an encoding method according to the data type, performs encoding operations, and assembles the encoded field data in field ID order to obtain type-encoded compressed data. The weighting module collects round-trip delay, bandwidth, packet loss rate, and jitter metrics during network transmission. The round-trip delay, bandwidth, packet loss rate, and jitter indicators are standardized and a weighted average formula is applied to obtain a comprehensive network quality score. A compression configuration module dynamically selects a compression strategy based on the comprehensive network quality score and historical compression performance data, using an adaptive learning mechanism, and obtains compression configuration parameters corresponding to the compression strategy. A compression module further compresses the type-encoded compressed data according to the compression configuration parameters and the template dictionary, adding a template ID and compression algorithm identifier to the message header to obtain a compressed message body, which is then sent through the network channel. A restoration module receives and parses the message header of the message body to obtain the template ID and compression algorithm identifier, searches for the corresponding template in the local template library based on the template ID, and uses the appropriate decompression algorithm to restore the original message data.

[0078] The technical solution provided by this invention can solve the problems of low compression efficiency of structured data, lack of business semantic awareness, and inability of static configuration to adapt to dynamic changes in the network environment in the prior art. It realizes efficient compression and transmission of structured data, significantly reduces network bandwidth consumption and improves the overall performance of the system while ensuring the reliability of data transmission. Attached Figure Description

[0079] Figure 1 This is a flowchart of the intelligent compression and transmission method for message bodies provided by the present invention;

[0080] Figure 2 This is a structural block diagram of the intelligent compression and transmission system for message bodies provided by the present invention. Detailed Implementation

[0081] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it.

[0082] like Figure 1 As shown, the present invention provides an intelligent compression and transmission method for message bodies, specifically including the following steps:

[0083] S1: Receive structured message data stream, convert the structured message data stream into an abstract syntax tree through a parser, traverse each node of the abstract syntax tree using a depth-first search algorithm and record the complete path, identify the node type of each node and calculate the message structure fingerprint.

[0084] This step begins by receiving the raw structured message data stream, such as JSON or XML format. A corresponding parser (e.g., a JSON parser) is used to convert the structured message into an in-memory Abstract Syntax Tree (AST). Then, a Depth-First Search (DFS) algorithm is used to traverse each node of the AST, recording the complete path from the root node to the current node during the traversal, such as "user.name" or "order.items[].price". Next, the traversed nodes are type-identified, distinguishing between different types of nodes, such as basic key-value pairs, arrays, and nested objects. Array type nodes require special handling; their internal element structure is analyzed to identify whether they are native type arrays, structured object arrays, or unstructured mixed arrays. Finally, all "path:type" pairs are collected, sorted, and their hash values ​​are calculated as the unique fingerprint of the message structure.

[0085] S2: Perform template matching or create a new template based on the message structure fingerprint to obtain a template dictionary containing the mapping relationship between field paths and field IDs.

[0086] In this step, the message structure fingerprint calculated in step S1 is compared with existing templates in the template library to determine if a matching template exists. If a matching template is found, its ID is used directly; otherwise, a new template is created and a unique template ID is assigned. For newly created templates, based on the statistical results of field frequency, short IDs are assigned to high-frequency fields, and long IDs are assigned to low-frequency fields, implementing variable-length integer encoding. Finally, a complete template dictionary containing the mapping relationship between field paths and field IDs is obtained for use in subsequent compression processes.

[0087] S3: Based on the template dictionary and the original message data in the structured message data stream, type identification is performed on each data field, encoding method is selected according to data type and encoding operation is performed, and the encoded field data is assembled in order of field ID to obtain type-encoded compressed data.

[0088] In this step, using a template dictionary and raw message data as input, the original data type (e.g., Number, String, Boolean, Null) of each data field is first identified according to JSON parsing rules. Simultaneously, semantic context analysis is performed based on field names and paths; for example, field names like "id" and "count" indicate integer types, while "price" and "amount" indicate floating-point numbers. Then, based on the identified type information, the optimal encoding method is selected for each data type: for integer data, a ZigZag transformation (mapping signed integers to unsigned integers) is applied before VarInt variable-length encoding; for floating-point data, precision analysis is performed, and IEEE 754 half-precision (16-bit) or standard double-precision (64-bit) representation is selected based on precision requirements; for string types, pattern recognition is performed, converting date strings to timestamps, converting numeric strings to actual numerical values ​​before encoding, and applying Huffman encoding to ordinary strings; for Boolean and Null values, bitmap encoding is used, packing multiple values ​​into the same byte. Finally, based on the field ID order defined in the template dictionary, the encoded field data is assembled sequentially. For optional fields, a bitmap is used to indicate their presence or absence, avoiding redundant encoding of non-existent fields. The final type-aware encoded complete message data is output for subsequent transmission.

[0089] S4: Collect round-trip delay, bandwidth, packet loss rate, and jitter indicators during network transmission. Standardize the round-trip delay, bandwidth, packet loss rate, and jitter indicators and apply a weighted average formula to obtain a comprehensive network quality score.

[0090] In this step, the round-trip time (RTT) is first calculated using the timestamps in the heartbeat or acknowledgment frames of the message queue; the number of acknowledged message bytes is counted within a fixed time window to estimate the current available bandwidth; the current network packet loss rate is inferred using TCP retransmission counts or gaps in application layer sequence numbers; and the standard deviation of the most recent N RTT measurements is calculated to quantify the degree of network jitter. After obtaining the original network metric dataset, normalization is performed, mapping each metric to the [0,1] interval.

[0091] Apply the formula to the RTT metric: ;

[0092] Apply the formula to the bandwidth metric: ;

[0093] The formula for applying the packet loss rate metric is: ;

[0094] Apply the formula to the jitter index: .

[0095] Then, set the weight coefficients for each indicator to reflect their importance in different business scenarios, and apply the weighted average formula:

[0096]

[0097] Among them, W rtt W bandwidth W loss and W jitter These are the weighting coefficients for round-trip delay, bandwidth, packet loss rate, and jitter, respectively, and W... rtt + W bandwidth + W loss + W jitter = 1;

[0098] The calculated scores are smoothed to reduce the impact of instantaneous fluctuations, resulting in the final network. quality_score The value serves as a comprehensive evaluation indicator for network quality.

[0099] S5: Based on the comprehensive network quality score and historical compression effect data, dynamically select a compression strategy using an adaptive learning mechanism, and obtain the compression configuration parameters corresponding to the compression strategy.

[0100] In this step, based on network quality_score Set three network quality states: EXCELLENT (≥0.8), GOOD (0.5-0.8), and POOR (<0.5).

[0101] Configure a corresponding compression strategy for each state:

[0102] The EXCELLENT state uses high-speed, low-compression-ratio algorithms such as LZ4; the GOOD state uses balanced algorithms such as ZSTD; and the POOR state enables template compression + entropy coding to maximize the compression ratio.

[0103] In addition, a state transition hysteresis mechanism needs to be introduced to prevent frequent jumps at threshold edges. For example, upgrading from POOR to GOOD requires a score exceeding 0.55, and downgrading from GOOD to POOR requires a score below 0.45. Record a snapshot of the decision for each compression operation, including the algorithm used, the network_quality_score at that time, and the values ​​of each metric; measure and record the actual compression ratio, compression time, and end-to-end message latency; calculate the comprehensive benefit function.

[0104] ,

[0105] Where parameters , , This reflects the business's emphasis on latency, bandwidth, and CPU. A contextual gambling machine model or regression model is built based on historical datasets to learn the mapping relationship between network metrics and revenue. The model parameters are updated periodically (e.g., hourly) using accumulated performance data, optimizing the weights and state thresholds of each metric. Based on the current network quality status and the optimized model parameters, the corresponding compression algorithm and parameter configuration are selected to obtain compression configuration parameters adapted to the current network conditions.

[0106] S6: Based on the compression configuration parameters and the template dictionary, further compression is performed on the type-encoded compressed data, and a template ID and compression algorithm identifier are added to the message header to obtain the compressed message body, which is then sent through the network channel.

[0107] In this step, the sending end first checks if the receiving end already possesses the template information. If the receiving end may not have the template, the sending end first constructs a control message containing the complete template content (a mapping list from field paths to field IDs); it sends the control message to the receiving end through a reliable channel and waits for an acknowledgment reply; the receiving end stores the template and returns an acknowledgment, establishing template consensus between the two parties and preparing for subsequent transmission of business data using the template. Then, based on the compression strategy configuration parameters selected in S5, the current compression algorithm and parameters are determined. If the strategy indicates the use of template compression + entropy encoding, the template from S2 and type-aware encoding from S3 are applied; if it indicates the use of a general algorithm, the corresponding LZ4 / ZSTD algorithm is applied. Metadata is added to the compressed message header, including the template ID used, the compression algorithm identifier, and necessary decompression parameters. Finally, the compressed message body is sent through the network channel to ensure reliable transmission.

[0108] S7: The receiving end receives and parses the message header of the message body to obtain the template ID and the compression algorithm identifier. Based on the template ID, it searches for the corresponding template from the local template library, uses the corresponding decompression algorithm to restore the original message data.

[0109] In this step, the receiving end first receives the compressed message body, parses the message header metadata, and identifies the template ID and compression algorithm used. Then, it searches for the corresponding template in the local template library based on the template ID; if not found, it triggers a template synchronization request. Finally, it uses the appropriate decompression algorithm to restore the original message content, completing the end-to-end compression and transmission process, and outputs the decompressed original message for use in business logic processing.

[0110] To more intuitively understand the workflow of the embodiments of the present invention, the following uses specific examples to elaborate on steps S1 - S7 in detail.

[0111] For step S1, assume that the distributed system (hereinafter referred to as the system for short) receives an e-commerce order data in JSON format:

[0112] `{"orderId": 12345, "customer": {"id": 1001, "name": "Zhang San", "phone":"13812345678"},

[0113] "items": [{"productId": 2001, "name": "Smartphone", "price": 4999.00, "quantity": 1},

[0114] {"productId": 3002, "name": "Protective Case", "price": 99.00, "quantity":2}],

[0115] "totalAmount": 5197.00, "status": "paid", "createTime": "2023-06-15T10:30:22Z"}`.

[0116] The system first parses this JSON data into an abstract syntax tree, and then traverses each node through depth-first search. During the traversal process, the system records the complete path from the root node to the current node, such as "orderId", "customer.id", "customer.name", "items[].productId", etc. Then, the system performs type recognition on these nodes: recognizes "orderId" as an integer type, "customer" as an object type, "items" as an array type, and further analyzes that the elements of this array are objects with consistent structures. Then the system collects all "path:type" pairs, such as "orderId:Integer", "customer.id:Integer", "items[].price:Float", etc., sorts these paired information in dictionary order and calculates the SHA-256 hash value, obtaining a message structure fingerprint such as "8f4e29b74b1c439899e1f1299b8e1307".

[0117] In step S2, the system compares the message structure fingerprint "8f4e29b74b1c439899e1f1299b8e1307" obtained in the previous step with the template library. Assume the system finds a matching template in the library with the ID "T-1022", which contains the complete field structure of the order data. This template dictionary maps field paths to short field IDs, for example, mapping "orderId" to ID "1", "customer.id" to ID "2", and "items[].productId" to ID "7", etc. Through this mapping, the originally lengthy field names are replaced with short numeric IDs, significantly reducing the size of the message's metadata. If no matching template is found, the system creates a new template, analyzes the frequency of each field, assigns short IDs (such as "1" and "3") to high-frequency fields (such as "orderId" and "status"), and assigns slightly longer IDs to low-frequency fields, ultimately generating a mapping dictionary from field paths to IDs.

[0118] In step S3, the system begins type-aware encoding based on the template dictionary "T-1022" and the original JSON data. For integer fields such as "orderId" (value 12345), the system applies ZigZag encoding to convert it to 24690 (2*12345), and then uses VarInt encoding to encode it into a byte sequence [B6, C1, 01]. For floating-point numbers such as "totalAmount" (value 5197.00), the system analyzes its precision requirements and decides to use IEEE 754 half-precision representation, encoded as 16 bits [65, A9]. For strings such as "customer.name" (value "Zhang San"), the system first checks whether it is a special pattern (such as date or numeric strings). After confirming that it is a normal string, it applies Huffman encoding to compress it, obtaining a more compact binary representation. For Boolean values ​​and multiple optional fields, the system uses bitmap encoding, for example, different bits of a byte can be used to represent the presence and value of multiple Boolean fields. Finally, the system assembles all the encoded field data into a continuous byte stream according to the field ID order defined in the template dictionary, forming type-encoded compressed data.

[0119] In step S4, the system continuously monitors network transmission status. Assume that in the last 5 minutes, the system collected the following network metrics: average round-trip time (RTT) of 120ms, available bandwidth of 8.5Mbps, packet loss rate of 0.8%, and RTT standard deviation (jitter) of 15ms. The system first standardizes these raw metrics: setting the maximum acceptable RTT to 500ms, the maximum expected bandwidth to 10Mbps, the maximum tolerable packet loss rate to 5%, and the maximum tolerable jitter to 50ms. The standardized metric values ​​are as follows:

[0120] ,

[0121] ,

[0122] ,

[0123] .

[0124] Assuming the system assigns weights to each indicator based on business characteristics as follows: RTT weight 0.3, bandwidth weight 0.3, packet loss rate weight 0.2, and jitter weight 0.2, the calculated comprehensive network quality score is: 0.3*0.76 + 0.3*0.85 + 0.2*0.84 + 0.2*0.7 = 0.792. The system also smooths this score, for example, by weighting it with the score from the previous time window at a ratio of 0.7:0.3, to reduce the impact of instantaneous fluctuations.

[0125] In step S5, the system determines the current network state as "GOOD" (score between 0.5 and 0.8) based on a network quality comprehensive score of 0.792 and a preset threshold. For the "GOOD" state, the initial strategy configuration is to use the ZSTD balanced compression algorithm with a compression level of 3 (range 1-9, higher levels result in higher compression ratios but slower speeds). Simultaneously, the system queries a historical compression performance database and finds that under similar network conditions, for structures like order data, using ZSTD level 2 actually yields better overall benefits (bandwidth savings minus additional latency and CPU overhead). Based on this historical experience, the system initiates an adaptive learning mechanism, predicting the benefits of different compression strategies using a contextual gambling machine model. After comprehensive consideration, it decides to adjust to ZSTD level 2 and fine-tunes the weight parameters, placing greater emphasis on latency rather than bandwidth (suitable for this type of interactive scenario). Finally, the system determines the compression configuration parameters: {algorithm: "ZSTD", level: 2, strategy: "fast", window_log: 20, min_match: 4}.

[0126] In step S6, the system first checks whether the receiving end already possesses the template "T-1022". Assuming the system finds that the receiving end synchronized this template 10 minutes ago, retransmission is unnecessary. The system then further compresses the type-encoded data generated in step S3 according to the previously selected compression configuration parameters (ZSTD level 2). During compression, the ZSTD algorithm analyzes data patterns, constructs an optimized compression dictionary, and replaces recurring data sequences with concise codes. After compression, the system adds necessary metadata to the message header: a 4-byte magic number (for protocol identification), a 2-byte template ID "T-1022", a 1-byte compression algorithm identifier "2" (indicating ZSTD), a 1-byte compression level "2", and other necessary decompression parameters. Finally, the system sends the assembled complete message body to the receiving end via a TCP connection. The size of the entire compressed message body is approximately 15% of the original JSON, significantly saving network bandwidth.

[0127] In step S7, after receiving the compressed message body, the receiving end first parses the message header. The 4-byte magic number in the header confirms that this is a message compressed using this system. Then, it extracts information such as the template ID "T-1022", the compression algorithm identifier "2" (ZSTD), and the compression level "2". The receiving end then searches for the template with ID "T-1022" in its local template library. Once found, it obtains the complete mapping relationship between field paths and IDs. The receiving end then uses the ZSTD level 2 decompression algorithm to perform the first stage of decompression on the message body, obtaining the type-encoded data. Next, the receiving end decodes each field data according to the field type and encoding method defined in the template: restoring VarInt-encoded integer fields to their original values, converting IEEE 754 half-precision floating-point numbers to standard double precision, decoding Huffman-encoded strings back to the original text, and so on. Finally, the receiving end reassembles all the decoded fields into a complete JSON object according to the paths defined in the template, restoring the original message data that is completely consistent with the sending end: `{"orderId": 12345, "customer": {...}, ...}`, for further processing by business logic.

[0128] This method not only significantly reduces redundancy in structured data through templating, but also selects the optimal compression strategy for different data types using type-aware encoding and dynamically adjusts compression parameters based on the network environment, ultimately achieving a significant improvement in data transmission efficiency. In practical applications, this method is particularly suitable for scenarios such as API services, microservice communication, and IoT data transmission, and can support larger-scale data exchange needs under limited bandwidth conditions.

[0129] In one embodiment of the present invention, step S1 involves identifying the node type of each node and calculating a message structure fingerprint, including:

[0130] S1.1 Perform node type identification on the abstract syntax tree, distinguish between basic key-value pairs, arrays and nested objects, analyze the internal element structure of array types, identify primitive type arrays, object arrays with consistent structure or unstructured mixed arrays, and output node type classification results;

[0131] In the process of processing structured data, the first step is to accurately identify and classify the data structure, which is the foundation for subsequent template matching and construction. The method proposed in this invention first achieves efficient structure identification and template processing by deeply analyzing the node characteristics of the Abstract Syntax Tree (AST).

[0132] Specifically, when the system receives a structured message (such as JSON data), it first parses it into an Abstract Syntax Tree (AST). During node type identification of the AST, the system meticulously distinguishes between different data structures. For example, for simple key-value pairs like `"userId": 1001`, the system identifies it as a basic key-value pair and records its type as integer; for nested objects like `"address": {"city": "a certain city", "zipCode": 100000}`, the system identifies it as an object type and recursively analyzes its internal structure; and for array types, the system's processing is more complex, requiring further analysis of the structural characteristics of its internal elements.

[0133] In array processing, the system intelligently distinguishes between three main types: primitive type arrays, structured object arrays, and unstructured mixed arrays. For example, `"tags": ["Important", "Urgent", "High Priority"]` is recognized as a primitive type string array; `"items": [{"id": 1, "name": "Product A"}, {"id": 2, "name": "Product B"}]` is recognized as a structured object array where each element has the same structural pattern; and `"mixedData": [42, "Test", {"key": "value"}]` is recognized as an unstructured mixed array where elements have inconsistent types. This detailed classification is crucial for subsequent code optimization because different array types are suited to different compression strategies.

[0134] S1.2 Based on the node type classification results, collect all path and type pairing information, sort the pairing information, calculate the hash value, and obtain the message structure fingerprint;

[0135] After completing node type identification, the system collects all path and type pairing information from the root node to each leaf node. For ordinary object fields, paths are represented in dot-separated form, such as `"user.profile.age"`; for array elements, square brackets are used, such as `"orders[].items[].price"`. The system records the data type corresponding to each path, such as `"user.profile.age: Integer"`, `"orders[].items[].price: Float"`, etc. To ensure the determinism of the generated structural fingerprint, the system sorts these path and type pairing information in lexicographical order, and then uses hash algorithms such as SHA-256 to calculate their hash values, obtaining a unique message structural fingerprint, such as `"a7f9b3c2d1e8f5g6..."`.

[0136] In step S2, template matching or new template creation is performed based on the message structure fingerprint to obtain a template dictionary containing the mapping relationship between field paths and field IDs, including:

[0137] S2.1 Compare the message structure fingerprint with existing templates in the template library to determine if there is a matching existing template. If a matching template is found, the ID of the matching template is used. If there is no match, a new matching template is created and a unique matching template ID is assigned.

[0138] After obtaining the message structure fingerprint, the system enters the template matching phase. The system queries its internally maintained template library to check if an existing template matches the current message structure fingerprint. The template library can be implemented as an efficient key-value store, using the structure fingerprint as the key and the template ID and content as the values. If a matching template is found, the system directly uses the template ID, avoiding the duplication of templates with the same structure and improving the system's space utilization efficiency. For example, suppose the system finds that the structure fingerprint of the current message matches an existing template with the ID "Template-573", then this template can be used directly for subsequent encoding.

[0139] If the system does not find a matching existing template, it will create a new one. The creation process begins by assigning a unique ID to the new template, such as "Template-574". Next, the system analyzes the frequency and importance of each field in the template, typically based on historical data analysis or domain knowledge. This field frequency analysis is crucial for subsequent ID assignment because, during encoding, the IDs of high-frequency fields should be as short as possible to maximize compression efficiency.

[0140] S2.2 Calculate the frequency of occurrence of fields in the new matching template, assign a first ID to the first frequency field, assign a second ID to the second frequency field, and construct a mapping relationship from field path to field ID to obtain the template dictionary containing the mapping relationship from field path to field ID.

[0141] Based on frequency statistics, the system sorts the fields in the template from highest to lowest frequency. For the highest frequency (first frequency) fields, such as order ID or user ID, the system assigns the shortest ID, such as "1" or "2"; for the second most frequent fields, such as order status or payment method, a slightly longer ID is assigned, such as "10" or "11"; and for less frequently used fields, an even longer ID is assigned. This variable-length encoding strategy ensures that frequently used fields occupy the least amount of space during message transmission, thereby improving overall compression efficiency.

[0142] Ultimately, the system completes the mapping relationship from field paths to field IDs, forming a template dictionary. For example, an e-commerce order template might contain the following mappings: `"orderId" -> "1", `"customer.id" -> "2", `"customer.name" -> "3", `"items[].productId" -> "4", `"items[].name" -> "5", `"items[].price" -> "6", `"items[].quantity" -> "7", `"totalAmount" -> "8", `"status" -> "9", etc. This template dictionary will be used in subsequent coding processes and will be persistently stored in the template library for future message encoding reuse.

[0143] Through this sophisticated node type identification and frequency-based template construction method, the system can generate highly optimized templates for different types of structured data, significantly improving data compression rates. This template-based approach is particularly advantageous when processing large volumes of messages with similar structures (such as API responses, log entries, and IoT sensor data), minimizing metadata overhead and transmitting only the actual changing data values, thereby significantly saving network bandwidth and improving transmission efficiency.

[0144] In another embodiment of the present invention, in step S3, the data types in the template dictionary and the original message data include integer type, floating-point type, string type, boolean type, and null value. Based on the template dictionary and the original message data in the structured message data stream, type identification is performed on each data field, an encoding method is selected according to the data type, and an encoding operation is performed. The encoded field data is then assembled according to the field ID order to obtain type-encoded compressed data, including:

[0145] S3.1 Apply ZigZag transform to the integer type data in the template dictionary and the original message data to map signed integers to unsigned integers, and then perform VarInt variable-length encoding to obtain integer encoded data;

[0146] In the process of structured data compression and transmission, encoding optimization for different data types is a key step in improving compression efficiency. The type-aware entropy encoding method proposed in this invention employs specially optimized encoding strategies for different data types, significantly improving encoding efficiency.

[0147] First, for integer data, this invention employs a two-stage encoding method. In the first stage, the system applies ZigZag transformation to map signed integers to unsigned integers. The core idea of ​​this transformation is to alternately map negative and positive numbers, so that smaller negative and positive numbers can be mapped to smaller unsigned integers. For example, 0 maps to 0, -1 to 1, 1 to 2, -2 to 3, and so on. The specific transformation formula is: (n<<1) ^ (n>>31), where n is the original signed integer, << indicates left shift, >> indicates right shift, and ^ indicates XOR operation. In the second stage, the system applies VarInt variable-length encoding to the unsigned integers after the ZigZag transformation. VarInt encoding groups integers into 7-bit groups, each group occupying one byte. The highest bit of the byte is used as a flag bit to indicate whether there are more bytes following. This encoding method allows small integers (such as 1-127) to occupy only one byte, while larger integers (such as 128-16383) occupy two bytes, greatly optimizing the storage space of integers. For example, the value 127 is encoded as a single byte [7F], while the value 128 is encoded as two bytes [80, 01].

[0148] S3.2 Perform precision analysis on floating-point data, and select IEEE 754 half-precision or standard double-precision representation according to precision requirements to obtain floating-point encoded data;

[0149] For floating-point data, the system first analyzes the required precision. It determines the necessary precision by examining historical data or analyzing field semantics (e.g., prices are typically rounded to two decimal places). Based on the precision analysis, the system selects an appropriate floating-point representation format: for scenarios with low precision requirements (e.g., sensor readings, progress percentages), IEEE 754 half-precision (16-bit) can be used; for scenarios requiring higher precision (e.g., financial calculations), standard double-precision (64-bit) is used. For example, for a temperature value of "23.5" with low precision requirements, half-precision can save 50% of storage space, while for monetary amounts accurate to multiple decimal places, double precision is used to ensure accuracy. The system also checks whether the floating-point number can be losslessly converted to an integer (e.g., "100.0" can be represented as the integer 100). If conversion is possible, integer encoding is preferred for higher compression ratios.

[0150] S3.3 performs pattern recognition on string data, converts date strings into timestamps or numeric strings into actual numeric values ​​and then encodes them, applies Huffman coding to ordinary strings, and obtains string encoded data.

[0151] Processing string data is particularly complex because strings may contain various patterns. The system first performs pattern recognition on the string to determine if it conforms to a specific format. For date and time strings (such as "2023-06-15T10:30:22Z"), the system converts them to a Unix timestamp (such as 1686826222) before integer encoding, saving more than 50% of space compared to the original string. For strings that can be parsed as numbers (such as "123.45"), the system converts them to actual numbers and processes them using numeric encoding. For UUIDs or other standard format strings, the system applies specialized encoding algorithms. For ordinary text strings, the system applies Huffman coding, constructing an optimal prefix code based on character frequency, using shorter codes for high-frequency characters and longer codes for low-frequency characters. When processing highly repetitive strings, the system also maintains a dynamic string dictionary, using dictionary indices to replace repeated string fragments, further improving the compression ratio.

[0152] S3.4 uses bitmap encoding for Boolean type data and Null values, packing multiple values ​​into the same byte to obtain Boolean value encoded data;

[0153] For data types like Boolean and Null values, which occupy relatively little space but can be numerous, the system uses bitmap encoding to pack multiple values ​​into the same byte. For example, one byte can store eight Boolean values, each occupying one bit. The system fills the bitmap sequentially according to the order of the Boolean fields defined in the template. For structures with many optional fields (fields may be Null), the system first generates an existence bitmap to mark which fields actually exist, and then only encodes the existing fields, avoiding redundant encoding operations on non-existent fields.

[0154] S3.5 According to the field ID order defined in the template dictionary, the integer encoded data, the floating-point encoded data, the string encoded data, and the boolean encoded data are assembled to obtain the compressed data after type encoding.

[0155] After encoding each data type, the system assembles all the encoded data sequentially according to the field ID order defined in the template dictionary. This assembly process ensures a consistent understanding of the data structure between the sender and receiver. For array types, the system first encodes the array length (using VarInt encoding), and then encodes each element's data sequentially. Finally, the system outputs a fully type-encoded compressed data stream, providing highly optimized input for subsequent general compression algorithms, further improving end-to-end compression efficiency.

[0156] In another embodiment of the present invention, step S5, which involves dynamically selecting a compression strategy based on the comprehensive network quality score and historical compression effect data, combined with an adaptive learning mechanism, to obtain compression configuration parameters adapted to the current network conditions, includes:

[0157] S5.1 Based on the comprehensive network quality score, three network quality states are set: a score greater than or equal to 0.8 is set as the first state EXCELLENT, a score between 0.5 and 0.8 is set as the second state GOOD, and a score less than 0.5 is set as the third state POOR.

[0158] In real-world applications with ever-changing network environments, static compression strategies struggle to adapt to varying network conditions. The dynamic compression strategy decision-making mechanism proposed in this invention adaptively adjusts the compression strategy based on real-time network quality, balancing compression ratio, latency, and resource consumption.

[0159] First, the system establishes a three-level network status classification based on a comprehensive network quality score. When the score is greater than or equal to 0.8, the network is classified as "EXCELLENT," indicating that the current network conditions are good, bandwidth is sufficient, and latency is low. When the score is between 0.5 and 0.8, the network is classified as "GOOD," indicating that the network conditions are average, with some limitations but still able to communicate normally. When the score is below 0.5, the network is classified as "POOR," indicating that the network conditions are poor, facing problems such as limited bandwidth, high latency, or severe packet loss.

[0160] S5.2 The first state configuration uses the LZ4 high-speed low compression ratio algorithm, the second state configuration uses the ZSTD balanced algorithm, and the third state configuration enables template compression and entropy coding to maximize the compression ratio, and outputs the initial compression strategy configuration.

[0161] For these three network states, the system is configured with corresponding initial compression strategies. In the "EXCELLENT" network state, the system prioritizes processing speed, selecting high-speed, low-compression-ratio algorithms such as LZ4. These algorithms are characterized by extremely fast compression and decompression speeds, but relatively low compression ratios, making them suitable for use in good network conditions and minimizing CPU overhead and processing latency. In the "GOOD" network state, the system selects balanced algorithms such as ZSTD, which achieve a good balance between compression ratio and speed, suitable for general network environments. In the "POOR" network state, the system will activate a maximum compression ratio strategy, combining the aforementioned template compression and entropy coding techniques, and may even use computationally intensive but extremely high-compression-ratio algorithms (such as LZMA) to minimize the amount of data transmitted and adapt to harsh network environments.

[0162] S5.3 Records a decision snapshot for each compression operation, measures and records the actual compression ratio, compression time, and end-to-end message latency, calculates the comprehensive benefit function, and outputs the historical data of the compression effect;

[0163] To prevent frequent state switching at threshold boundaries, the system introduces a state transition hysteresis mechanism. For example, upgrading from "POOR" to "GOOD" requires not only a network score exceeding 0.5, but also a score of 0.55; similarly, downgrading from "GOOD" to "POOR" is not triggered when the score just falls below 0.5, but rather when the score falls below 0.45. This hysteresis design effectively avoids frequent policy switching when the network state fluctuates near the critical point, improving system stability.

[0164] S5.4 Based on the historical data of the compression effect, construct a contextual gambling machine model or regression model to learn the mapping relationship between network indicators and returns, periodically update the parameters of the contextual gambling machine model or regression model using the accumulated effect data, optimize the weights and state thresholds of the network indicators, and obtain the optimized model parameters.

[0165] The system records a complete snapshot of each compression decision, including the compression algorithm and parameters selected at that time, the network quality score and its constituent metrics, message size and type, and other contextual information. Simultaneously, the system measures and records the actual effect of each compression operation, including the actual compression ratio (the ratio of the original size to the compressed size), the CPU time consumed during compression and decompression, and the end-to-end transmission latency of the message.

[0166] As the system continues to operate, the accumulated historical data forms a valuable knowledge base. The system uses this data to build machine learning models, such as Contextual Bandit models or Gradient Boosted Regression models, to learn the complex relationship between network metrics and the returns of compression strategies. Contextual Bandit models are particularly well-suited for this scenario because they strike a balance between "exploration" (trying different compression strategies to gather more information) and "exploitation" (selecting the known optimal strategy). The system updates the model with newly accumulated data at predetermined intervals (e.g., hourly or daily), continuously optimizing its understanding of network metrics and its ability to predict returns.

[0167] Through the learning process, the system dynamically adjusts the weights of each indicator in the network quality score. For example, if historical data shows that packet loss rate has a much greater impact on compression effect than bandwidth in a specific business scenario, the weight of the packet loss rate indicator will be increased. Similarly, the system will also optimize network status thresholds, for example, by adjusting the dividing line between "EXCELLENT" and "GOOD" from 0.8 to 0.75 or 0.85 to better adapt to actual business models.

[0168] Furthermore, in one embodiment, in S5.4, the step of constructing a mapping relationship between network indicators and returns based on the historical data of the compression effect, periodically updating the model parameters using accumulated effect data, optimizing the weights of each indicator and the state threshold, and obtaining the optimized model parameters includes:

[0169] Feature extraction is performed on the historical data of compression effect. Network indicator features and compression decision features are extracted as inputs, and comprehensive benefits are extracted as outputs to obtain a training dataset.

[0170] Based on the training dataset, a prediction model is constructed using the contextual gambling machine algorithm or the gradient boosting regression algorithm to learn the mapping relationship from network metrics to compression strategy returns, thereby obtaining an initial prediction model.

[0171] For the initial prediction model, incremental training is performed using newly accumulated effect data at a set period to update the model weight parameters and obtain the updated prediction model.

[0172] Based on the updated prediction model, the weight coefficients of each indicator in the comprehensive network quality score and the threshold parameters of the network quality status are adjusted to obtain the optimized model parameters.

[0173] A core innovation of intelligent compression systems lies in their self-learning capability, which continuously optimizes compression decisions by building mathematical models. This mechanism enables the system to learn from historical compression operations and transform this experience into a more accurate decision-making model, achieving true adaptive optimization.

[0174] First, the system needs to extract valuable features from the accumulated historical data on compression effects. These features are mainly divided into two categories: network indicator features and compression decision features. Network indicator features include raw network measurement data, such as round-trip time (RTT) and its variability, available bandwidth, packet loss rate, and network congestion level; message features, such as message size, structural complexity, and data type distribution; and environmental features, such as device type, time period, and network type (Wi-Fi, 4G, 5G, etc.). Compression decision features record the specific strategy used in each compression operation, including the selected compression algorithm (such as LZ4, ZSTD, LZMA), compression level, window size, buffer settings, and other parameter configurations. Simultaneously, the system uses the comprehensive benefit of each operation as the output label. This benefit value comprehensively considers bandwidth savings, latency increases, and computational resource consumption, calculated using a weighted formula. Through this feature extraction, the system transforms unstructured historical logs into a structured training dataset, laying the foundation for subsequent model training.

[0175] During the model building phase, the system can choose from two main machine learning algorithms: the contextual gambler algorithm or the gradient boosting regression algorithm. The contextual gambler algorithm is particularly well-suited for decision optimization problems. It treats each compression strategy as an "arm," and the system needs to select the optimal "arm" to maximize long-term gains, given the current context (network conditions). This algorithm strikes a balance between "exploration" (trying new strategies to gain more information) and "exploitation" (selecting the known optimal strategy), which is especially important for dynamically changing network environments. For example, the system might implement variants such as LinUCB or Thompson Sampling, which estimate the expected return and uncertainty of each decision based on historical observations and consider both factors when making a selection.

[0176] Another option is gradient boosting regression algorithms (such as XGBoost or LightGBM). These algorithms excel at handling complex nonlinear relationships and can capture the intricate interactions between network metrics and compression parameters. The system constructs multiple regression models, each predicting the expected return under different compression strategies, and then selects the strategy with the highest expected return. The advantage of this approach is that it can directly model continuous return values, rather than just the selection of the optimal strategy, allowing the system to understand the decision space in greater detail. Regardless of the algorithm chosen, the system uses historical data for initial training to generate an initial predictive model capable of predicting the returns of various compression strategies under different network conditions.

[0177] The initial training of the model is only the beginning of the learning process. As the system continues to run, new compression operations generate new performance data, which may reflect changes in the network environment or new data patterns. To maintain the model's timeliness, the system incrementally trains the model at preset intervals (e.g., hourly, daily, or weekly, depending on the amount of data and the rate of business change). Incremental training does not require retraining the entire model; instead, it fine-tunes the existing model using newly accumulated data, updating the model's weight parameters. For example, for a contextual gambling machine model, the system updates the payoff estimates and confidence intervals for each strategy; for a gradient boosting regression model, the system adjusts the tree structure and the predicted values ​​of leaf nodes. This periodic update ensures that the model can adapt to changes in the network environment and the evolution of user behavior, maintaining the accuracy and timeliness of decision-making.

[0178] Model training not only improves the accuracy of compression strategy selection but also provides a basis for adjusting underlying parameters. By analyzing the updated prediction model, the system can identify which network metrics have the greatest impact on compression performance, thereby adjusting the weighting coefficients of each metric in the overall network quality score. For example, if the model shows that in certain scenarios, network jitter has a much greater impact on compression performance than bandwidth, the system will increase the weight of the jitter metric and decrease the weight of the bandwidth metric, making the network quality score more accurately reflect the actual situation. Similarly, the system will also optimize the threshold parameters of network quality status based on model predictions. For example, if data shows that in a network environment with a score of 0.78, using a compression strategy designed for the "EXCELLENT" status is actually more effective, the system will lower the dividing line between "EXCELLENT" and "GOOD" from 0.8 to 0.75. This parameter adjustment is not a simple modification of a single metric but an optimization result derived after comprehensively considering all historical data, enabling it to better adapt to the characteristics of specific business scenarios and network environments.

[0179] Through this complete machine learning process, the system achieves closed-loop optimization of the compression strategy. Each compression operation not only completes the immediate data transmission task but also provides valuable learning material for future decisions, enabling the system to continuously improve itself and achieve increasingly higher transmission efficiency over long-term operation. This data-driven adaptive optimization mechanism is the key innovation of this invention, distinguishing it from traditional static compression schemes, and enabling it to maintain excellent performance in complex and ever-changing real-world network environments.

[0180] S5.5 Based on the current network quality status, the initial compression strategy configuration, and the optimized model parameters, select the corresponding compression algorithm and parameter configuration to obtain the compression configuration parameters adapted to the current network conditions.

[0181] Each time a message needs to be sent, the system comprehensively considers the current network quality status, the initial compression strategy configuration, and the optimized model parameters to make a final compression decision. This decision includes not only choosing which compression algorithm to use, but also specific parameter configurations, such as compression level, window size, hash table size, and other detailed parameters, to achieve optimal overall performance. For example, in some cases, the system may select a modified ZSTD algorithm with smaller parameters in a "GOOD" network state, using a smaller compression window to achieve faster processing speed, based on historical learning results; while in cases of high data repetition, a larger window will be selected to improve the compression ratio.

[0182] Through this adaptive learning mechanism, the system can maintain near-optimal compression transmission efficiency in various complex and ever-changing network environments, which not only meets the business requirements for low latency, but also saves network bandwidth to the maximum extent, while keeping CPU resource consumption within a reasonable range, thus achieving a dynamic balance among the three.

[0183] In another embodiment of the present invention, step S6, which involves further compressing the type-encoded compressed data according to the compression configuration parameters and the template dictionary, adding a template ID and a compression algorithm identifier to the message header to obtain a compressed message body, and sending the compressed message body through a network channel, includes:

[0184] S6.1 The sending end checks whether the receiving end has the template information corresponding to the template dictionary. If the receiving end does not have the template information, the sending end constructs a control message containing complete template content and outputs a template synchronization control message.

[0185] In the actual transmission of structured messages, the sender and receiver need to establish a consensus on the message structure, which requires an efficient template synchronization mechanism. Simultaneously, the system also needs to perform final compression of the encoded data and complete network transmission. This process involves multiple intricate steps that collectively ensure the efficiency and reliability of the compressed transmission.

[0186] First, the sender needs to confirm whether the receiver already possesses the template dictionary used for the current message. To this end, the system maintains a template version management mechanism, recording the template ID and version number last confirmed by each receiver. When a message needs to be sent, the sender queries this management system to check if the target receiver has synchronized the required template. If the query result shows that the receiver does not yet possess the template, the sender will not immediately send the data message. Instead, it will first construct a special control message containing the complete template definition. This control message typically includes: template ID (e.g., "Template-574"), template version number, structure fingerprint hash value, complete field path-to-field ID mapping, and data type definitions for each field. The control message also includes a special identifier to inform the receiver that this is a template synchronization request, not a regular data message.

[0187] S6.2 The template synchronization control message is sent to the receiving end and an acknowledgment reply is awaited. The receiving end stores the template and returns an acknowledgment to establish template consensus and obtain the template synchronization status.

[0188] After the sending end sends this template synchronization control message to the receiving end, it enters a waiting state until it receives an acknowledgment from the receiving end. Upon receiving the template synchronization message, the receiving end parses it and stores it in its local template library. After successful storage, the receiving end returns an acknowledgment message containing the template ID and version number, indicating that the template has been successfully received and saved. Once the sending end receives this acknowledgment message, both parties establish a consensus regarding the template, and the sending end updates the record in its template management system, marking that the receiving end already possesses this template. This template synchronization mechanism ensures correct parsing of subsequent data transmissions and avoids template inconsistencies through version control.

[0189] S6.3 Based on the template synchronization status and the compression configuration parameters, determine the currently used compression algorithm and the corresponding parameters of the compression algorithm, and apply the corresponding compression algorithm to the type-encoded compressed data to obtain further compressed data;

[0190] After confirming the template synchronization status, the system begins preparing for formal data transmission. Based on the compression configuration parameters obtained from the previous decisions, the system determines the specific compression algorithm to be used (e.g., ZSTD) and the corresponding parameter settings (e.g., compression level 2, window size 20, etc.). Then, the system applies these parameter configurations to the compression engine to further compress the previously generated type-encoded compressed data. This step leverages the capabilities of general compression algorithms to identify and eliminate redundant patterns that still exist after type encoding, further improving the compression ratio. For different compression algorithms, the system applies different parameter optimization strategies. For example, for the ZSTD algorithm, the system may dynamically adjust the dictionary size and hash table width based on data characteristics; for the LZ4 algorithm, it may adjust the block size and acceleration parameters, etc.

[0191] S6.4 In the further compressed data, the template ID, the compression algorithm identifier, and the decompression parameters are added to the message header to obtain the complete compressed message body;

[0192] After compression, the system needs to construct a complete message header so that the receiving end can correctly parse the received data. The message header typically includes the following key fields: Magic Number – a fixed value used to identify that this is a message compressed using this system; Protocol Version Number – indicating the encoding protocol version used, supporting protocol upgrades; Template ID – specifying the template dictionary used, such as "Template-574"; Compression Algorithm Identifier – indicating the compression algorithm used (e.g., 1 for LZ4, 2 for ZSTD, etc.); Compression Parameters – containing key parameters required for decompression; Message Length – the length of the compressed message body; Optional Checksum – used to verify message integrity; and other metadata, such as timestamps and priorities. This header information is encoded using a fixed format, typically occupying very little space (tens of bytes), but providing all the information necessary for decoding.

[0193] The system concatenates the header information with the compressed data body to form a complete compressed message body. This message body is a completely self-contained data unit. The receiving end only needs this message body to complete all decompression and parsing operations, without any external dependencies (except for pre-synchronized template information).

[0194] S6.5 The complete compressed message body is sent through the network channel.

[0195] Finally, the system sends the complete message body to the target receiver through an appropriate network channel (such as a TCP connection, WebSocket, HTTP request, etc.). The sending process follows the specifications of the underlying communication protocol, handling possible transmission errors and retry logic.

[0196] In specific scenarios, the system also supports streaming mode. For large messages, the system doesn't need to wait for all data to be compressed before sending; instead, it can divide the message into multiple chunks, each compressed and sent independently. This streaming processing significantly reduces the latency of sending the first byte, improving real-time performance, and is particularly suitable for interactive scenarios requiring rapid response. Each chunk has its own small header containing enough information for independent decoding, but all chunks share the same template ID, ensuring parsing consistency.

[0197] Through this complete template synchronization and message transmission mechanism, the system not only ensures the correct parsing of compressed data but also minimizes metadata overhead, transmitting template information only when necessary. This achieves extremely high bandwidth utilization efficiency in scenarios involving the transmission of a large number of messages with similar structures. Simultaneously, flexible compression algorithm selection and parameter configuration ensure that the system can adapt to various network environments and data characteristics, achieving an optimal balance between bandwidth saving, processing latency, and resource consumption.

[0198] In an optimized embodiment of the present invention, step S2, which involves performing template matching or creating a new template based on the message structure fingerprint to obtain a template dictionary containing the mapping relationship between field paths and field IDs, includes:

[0199] A1. Convert the message structure in the message structure fingerprint into a high-dimensional sparse vector representation, where each field path is used as a dimension. Assign weight values ​​to existing fields according to their data types, and set the dimension value of non-existent fields to 0. Output the message structure vector.

[0200] In large-scale distributed systems, as the template library continues to grow, accurately and efficiently finding matching templates becomes a key challenge. Traditional template matching methods often employ precise hash value comparisons, which, while fast, lack flexibility and cannot effectively handle situations where structures are similar but not identical. This invention proposes an intelligent template matching mechanism based on Locality Sensitive Hash (LSH), enabling similarity search in high-dimensional message structure spaces and significantly improving template reuse and matching efficiency.

[0201] First, the system needs to represent the structural features of the message in a computable mathematical form. The system uses a high-dimensional sparse vector (greater than 3 dimensions) to represent the message structure, where each possible field path corresponds to a dimension in the vector space. For example, in an e-commerce order system, there may be thousands of potential field paths such as "orderId", "customer.name", "customer.address", "items[].id", and "items[].price". For a specific message structure, existing fields are assigned a non-zero weight value in their corresponding dimension based on their data type, while non-existent fields have a value of 0 in their corresponding dimension. The allocation of data type weights follows specific rules; for example, integer types may be assigned a value of 1.0, floating-point types 1.2, string types 1.5, and array types 2.0, etc. These weights reflect the importance of different data types in structural similarity judgment. Through this representation, a complex message structure containing hundreds or thousands of fields is transformed into a high-dimensional sparse vector, making subsequent similarity calculations and template matching possible.

[0202] A2. Multiple hash functions are created using the symbolic random projection method. A set of random vectors is selected from the multiple hash functions. The dot product of the message structure vector and each of the random vectors is calculated, and the sign of the dot product is taken as one bit of the hash value to obtain the hash signature.

[0203] While this vector representation is intuitive, it can lead to dimensionality explosion in real-world systems—common enterprise applications may have tens of thousands of different field paths. Directly calculating similarity in such a high-dimensional space is not only computationally intensive but also requires considerable storage. To address this issue, the system employs a signed random projection method to create multiple hash functions, mapping the high-dimensional vector to a low-dimensional signature space. Specifically, the system randomly generates multiple unit vectors, each with elements following a standard normal distribution. Then, it calculates the dot product of the message structure vector and these random vectors, taking the sign (positive or negative) of the dot product as one bit of the hash value. For example, using 128 random vectors, the final hash signature is a 128-bit binary string, such as "10110101...". This method is based on the theory of locality-sensitive hashing, ensuring that messages with similar structures are mapped to similar hash signatures, while messages with significantly different structures receive significantly different hash signatures.

[0204] A3. Using a combination of k hash functions as an index, calculate the hash value of the templates stored in the template library and construct L hash tables to form a template hash index with a multi-level LSH index structure, where k and L are positive integers;

[0205] A3 includes:

[0206] A31. For the first hash function, select m independent symbolic random projection functions, calculate m hash values ​​for each stored template, and combine the m hash values ​​as the index key of the corresponding template in the first hash table to obtain the first-level hash table, where m is a positive integer;

[0207] A32. For the remaining k-1 hash functions, generate (k-1)*m independent symbolic random projection functions using different random seeds. Calculate the hash value for each stored template and construct the corresponding hash table to obtain a multi-level LSH hash table set, where k is greater than or equal to 2.

[0208] A33. Based on each hash table in the multi-level hash table set, templates with the same index key are stored in the same hash bucket, and a mapping relationship from hash key to template list is established to obtain the template hash index.

[0209] To further improve matching accuracy and efficiency, the system constructs a multi-level LSH index structure. The system uses k different hash function combinations, each consisting of m independent symbolic random projection functions, forming L independent hash tables (usually L=k). This design follows the AND-OR structure of LSH—each hash table internally uses an "AND" relationship between m hash functions (only if all m hash values ​​match are they considered to be in the same hash bucket), while different hash tables are related by an "OR" relationship (if a match is found in any hash table, it is considered a candidate match). This structure ensures high recall while effectively controlling the size of the candidate set, balancing query performance and matching accuracy.

[0210] Specifically, for the first combination of hash functions, the system selects m independent symbolic random projection functions. For each template already stored in the template library, the system calculates m hash values ​​and combines these m values ​​as the index key of that template in the first hash table. For example, if m=4, then a possible index key for a template is "1001", indicating that the dot product is positive in the first random projection, negative in the second and third, and positive in the fourth. The system performs the same calculation on all stored templates to construct a complete first-level hash table.

[0211] For the remaining k-1 hash function combinations, the system generates a new set of random projection functions using different random seeds, ensuring the independence between different hash tables. Through these different random projections, the system constructs k independent hash tables, forming a complete multi-level LSH hash table set. This multi-level structure is the core mechanism enabling the system to efficiently handle "similarity queries"—even if similar structures are mapped to different hash buckets due to the limitations of random projection in a particular hash table, other hash tables may still map them to the same hash bucket, thus ensuring a high recall rate.

[0212] Within each hash table, the system stores templates with the same index key in the same hash bucket, establishing a mapping from hash key to template list. For example, all templates with index key "1001" are placed in bucket "1001" of the first hash table. This bucket-based storage structure allows the system to retrieve all potential matching candidates by simply calculating the hash key of the query structure and directly accessing the corresponding hash bucket during a query, greatly reducing the search space and improving query efficiency.

[0213] A4. Obtain the hash signature of the new message, search in parallel in the L hash tables of the template hash index for candidate templates with the same or similar hash signatures, calculate the actual similarity between the returned candidate set and the message structure vector, select the best match with a similarity exceeding the threshold, and obtain the matching template ID.

[0214] When a new message structure requires template matching, the system first calculates the hash signature of that structure. Similar to index building, the system uses the same k hash functions to calculate k hash values ​​for the structure vector of the new message. Then, the system searches in parallel across the L constructed hash tables for candidate templates with the same hash key. This parallel query strategy fully leverages the multi-core capabilities of modern processors, significantly improving query performance. The query result is a set of candidate templates, which have the same hash key as the query structure in at least one hash table.

[0215] After obtaining the candidate set, the system needs to further calculate the actual similarity between the candidate templates and the query structure to determine the best match. This step typically uses metrics such as cosine similarity or Jaccard similarity. For each candidate template, the system calculates the similarity score between its structure vector and the query vector, and then selects the template with the highest similarity score exceeding a preset threshold (e.g., 0.9) as the final matching result. This precise calculation step ensures the accuracy of the matching, and since LSH has effectively reduced the size of the candidate set (usually only a small fraction of the total number of templates), the cost of this calculation remains within an acceptable range.

[0216] A5. Based on the matched template ID, if a matching template is found, it is used directly; otherwise, a new matching template is created and a unique template ID is assigned. A mapping relationship between field paths and field IDs is constructed to obtain the template dictionary containing the mapping relationship between field paths and field IDs.

[0217] Finally, the system determines the next step based on the matching results. If a matching template with a similarity exceeding the threshold is found, the system will directly use the template's ID and field mapping to continue the subsequent encoding process. If no suitable match is found (all candidate similarities are below the threshold), the system will create a completely new template for the new message structure, assign a unique template ID, establish a mapping between field paths and field IDs, and add the new template to the template library and LSH index for future queries. The system will also periodically rebuild the entire LSH index structure to incorporate newly added templates and maintain query performance.

[0218] This LSH-based template matching mechanism is particularly suitable for handling large-scale, high-dimensional message structure matching problems. Compared to traditional exact hash matching, it can discover templates that are structurally similar but not identical, significantly improving template reuse. For example, in real-world enterprise applications, different API responses may have subtle field differences (such as the presence or absence of certain optional fields). Traditional methods would treat them as completely different structures and create multiple templates, while the LSH matching mechanism of this invention can identify their essential similarities and use the same template for processing, thereby reducing the number of templates and improving the overall compression efficiency of the system.

[0219] In the implementation of multi-level LSH indexes, parameter selection is crucial, directly impacting the system's matching performance and accuracy. The system employs an adaptive parameter adjustment strategy, dynamically adjusting parameters such as m (the number of hash functions in each hash function combination), k (the number of hash function combinations), and the similarity threshold based on the template library size and query performance. For example, when the template library is small, smaller values ​​for m and k can be used; as the template library grows, the system automatically increases these parameters to maintain query accuracy. This adaptive adjustment ensures that the system maintains optimal performance at different scales.

[0220] In practical applications, the system implements a hierarchical caching strategy for the hash table, keeping frequently used templates in memory while cold templates may be moved to disk or lower-level storage. This caching strategy further improves the system's query response speed, enabling near real-time template matching for high-frequency message patterns. The system also implements a template version management mechanism, supporting template evolution and compatibility maintenance, ensuring that the system can correctly process both old and new versions of messages even as the message structure changes over time.

[0221] In summary, the LSH-based template matching mechanism of this invention addresses the core challenge of template management in large-scale distributed systems. Through an intelligent similarity search algorithm, it achieves efficient and accurate template matching, laying a solid foundation for subsequent high-compression-rate encoding. This method not only improves the system's compression performance but also reduces maintenance costs, enabling the system to maintain excellent performance and scalability in complex and ever-changing business environments.

[0222] In another optimized embodiment of the present invention, in step S4, the round-trip delay, bandwidth, packet loss rate, and jitter indicators during the network transmission process are collected. These indicators are then standardized and a weighted average formula is applied to obtain a comprehensive network quality score, including:

[0223] B1. Construct a weighted graph of network topology, wherein each node is represented as a vertex in the graph, the communication links between nodes are represented as edges, and the edge weight is the RTT or bandwidth of the link;

[0224] B2. Apply the directional low-diameter decomposition algorithm to decompose the network topology weighted graph into n subgraphs, and ensure that the communication path length between any two points in each subgraph does not exceed O(log n / (log log n)), to obtain a set of low-diameter subgraphs, where n is a positive integer;

[0225] B3. For each subgraph in the set of low-diameter subgraphs, select the node with the highest centrality as the main monitoring point, design the aggregation path of the monitoring data to ensure that the data is transmitted along the shortest path within the subgraph, and determine the layout of the hierarchical monitoring points;

[0226] B4. Based on the hierarchical monitoring point layout, network quality assessment is performed independently within each subgraph, and a comprehensive local network quality score is calculated. The main monitoring point of each subgraph transmits the local assessment results to the global coordination node. The global coordination node integrates all local assessment results to generate a global network quality map and outputs the local score and the global score.

[0227] B5. Based on the local and global scores, the monitoring frequency is dynamically adjusted according to the stability of each subgraph, and the final comprehensive network quality score is determined.

[0228] The network environment of large-scale distributed systems is often exceptionally complex, spanning multiple data centers, geographical regions, and even the public internet. In such a complex environment, traditional network quality assessment methods based on single-point monitoring struggle to accurately reflect the overall network condition. To address this challenge, this invention proposes an efficient network quality assessment mechanism based on network topology decomposition and hierarchical monitoring, achieving precise network quality awareness and providing a reliable foundation for compression strategy decisions.

[0229] First, the system needs to construct a complete weighted network topology graph. In this graph, each node participating in communication (such as servers, edge devices, etc.) is represented as a vertex, and the communication links between nodes are represented as edges connecting the vertices. The system assigns weights to each edge, which can reflect various network characteristics: the most common is round-trip time (RTT), representing the time required for a data packet to travel from one node to another and return, usually in milliseconds; another important weight is bandwidth, representing the data transmission capacity of the link, usually in Mbps or Gbps. The system periodically performs network probing tasks, such as ping tests, bandwidth tests, or traceroute analysis, to obtain the latest network parameters and update the graph weights. This dynamic updating ensures that the network topology graph accurately reflects the current network state, rather than outdated historical information.

[0230] Because the topology of large-scale networks can be extremely large and complex, directly executing monitoring and decision-making algorithms across the entire network would lead to excessive computational complexity and latency. To address this issue, the system employs a Directed Low-Diameter Decomposition algorithm to decompose the overall network topology into multiple subgraphs. The core objective of this decomposition is to ensure that the communication path length between any two points within each subgraph does not exceed O(log n / (log logn)), where n is the number of nodes. This bound is an important result in graph theory, indicating that within a subgraph, the number of hops between any two nodes remains at a low logarithmic level, ensuring efficient communication within the subgraph. The decomposition algorithm considers edge weights, prioritizing edges with smaller weights (representing higher-quality connections) within the same subgraph, ensuring relative consistency in network quality and topology across each subgraph.

[0231] After completing the network topology decomposition, the system needs to deploy network monitoring points in each subgraph. An ideal monitoring point layout should cover the largest network area with the fewest possible monitoring points, while ensuring the accuracy and timeliness of the monitoring data. The system selects the node with the highest centrality in each subgraph as the primary monitoring point. Centrality is an important concept in network analysis, measuring the importance or influence of a node in the network. The system may use various centrality measures, such as degree centrality (number of connected nodes), betweenness centrality (number of shortest paths through the node), or eigenvector centrality. After selecting the primary monitoring point, the system designs aggregation paths for the monitoring data, ensuring that data is transmitted along the shortest paths within the subgraph, minimizing monitoring overhead, and guaranteeing data timeliness. In addition to the primary monitoring point, the system also selects secondary monitoring points to form a hierarchical monitoring structure. These secondary monitoring points are typically located on the critical paths of the subgraph or in areas where network conditions change frequently.

[0232] Based on the established hierarchical monitoring point layout, the system independently assesses network quality within each subgraph. Each monitoring point periodically collects four key network metrics: Round-Trip Time (RTT) – measures network response speed; lower RTT indicates faster response time. Bandwidth – measures data transmission capacity; higher bandwidth means greater throughput. Packet loss rate – measures network reliability; lower packet loss rate indicates more reliable transmission. Network jitter – measures the degree of latency variation; lower jitter indicates better network stability. These raw metric data are standardized and converted into a unified scoring scale (typically values ​​between 0 and 1), and then a weighted average formula is used to calculate the comprehensive local network quality score. The weighting coefficients reflect the degree of influence of different metrics on application performance; for example, for video streaming applications, bandwidth and packet loss rate may have higher weights, while for command and control applications, RTT may have a higher weight.

[0233] Each subgraph's master monitoring point is responsible for collecting the evaluation results from all monitoring points within its subgraph, performing local aggregation calculations, and obtaining a comprehensive local network quality score. These local scores are then passed to the global coordination node, which is typically a supernode with the strongest computing power and best network connectivity in the entire system. The global coordination node receives the local scores from all subgraphs and, combined with the connectivity between subgraphs, integrates all local evaluation results to generate a complete global network quality map. This quality map includes not only the internal network conditions of each subgraph but also the connectivity quality assessment between subgraphs, forming a complete picture of the entire network. The global coordination node then feeds this global view back to the master monitoring points of each subgraph, enabling each subgraph to understand both its own local network conditions and the overall network environment, forming a two-tiered scoring system—local scores and global scores.

[0234] The system dynamically adjusts the monitoring frequency to balance monitoring accuracy and system overhead. For sub-graphs with stable network conditions, the system gradually reduces the monitoring frequency (e.g., from once per second to once per minute) to reduce unnecessary resource consumption. For sub-graphs with frequent changes or where critical business operations are located, the system maintains a higher monitoring frequency to ensure timely capture of changes in network conditions. The system also sets anomaly detection thresholds. When a certain indicator suddenly exceeds the normal range (e.g., packet loss rate suddenly increases from 2% to 15%), additional monitoring procedures are immediately triggered to ensure timely response to anomalies.

[0235] Through this hierarchical, adaptive network monitoring mechanism, the system can efficiently acquire accurate network quality information. This avoids the enormous overhead of frequent monitoring across the entire network while ensuring monitoring accuracy in key areas, providing a solid data foundation for intelligent decision-making in compression strategies. The comprehensive network quality score, as a crucial input for subsequent compression strategy decisions, directly impacts the overall system performance.

[0236] In step S5, based on the comprehensive network quality score and historical compression effect data, and combined with an adaptive learning mechanism, a compression strategy is dynamically selected, and compression configuration parameters corresponding to the compression strategy are obtained, including:

[0237] C1. Based on the set of low-diameter subgraphs, construct a policy propagation tree in each subgraph, and select the node with the best computing power and network connection in each subgraph as the master decision node, wherein the master decision node serves as the root node of the policy propagation tree, used to optimize propagation delay and construct the policy propagation tree structure;

[0238] C2. Based on the policy propagation tree structure, policy caching is set at different levels, including: the main decision node caches the complete policy model and parameters, the root nodes of subtrees in the policy propagation tree cache simplified policy rules, and the leaf nodes only store the currently applicable specific compressed configuration parameters;

[0239] C3. Design a consistency protocol to ensure that all nodes within the same subgraph adopt a consistent compression strategy under similar network conditions, including: adopting a version number-based strategy synchronization mechanism, exchanging strategy version information between nodes periodically, and automatically synchronizing to the latest version when version differences are found;

[0240] C4. For nodes located at the boundaries of multiple subgraphs, design a cross-subgraph coordination strategy to allow boundary nodes to merge compression strategies from different subgraphs, wherein multiple compression configurations are maintained in parallel in the boundary nodes, and the compression configurations are dynamically switched according to the source subgraph of the message;

[0241] C5. Based on the consistent compression strategy and the cross-subgraph coordination strategy, combined with the comprehensive network quality score and historical compression effect data, dynamically select the optimal compression strategy to obtain the compression configuration parameters adapted to the current network conditions.

[0242] After obtaining an accurate network quality assessment, the system needs to efficiently transform this information into compressed policy decisions and ensure that these decisions can be consistently applied in a distributed environment. To this end, this invention designs a distributed policy decision-making and propagation mechanism based on network topology, achieving efficiency, consistency, and adaptability in decision-making.

[0243] First, the system needs to construct a policy propagation tree based on the previously established set of low-diameter subgraphs. In each subgraph, the system selects a node with optimal computing power and network connectivity as the master decision node. This node typically has high processing power, memory capacity, and network bandwidth, capable of handling complex decision-making tasks. The master decision node, as the root node of the policy propagation tree, is responsible for receiving network quality assessment data, executing policy decision algorithms, generating optimal compression policies, and propagating these policies to other nodes within the subgraph. The system carefully designs the policy propagation path to ensure that the propagation delay from the master decision node to any leaf node is minimized; this is typically achieved by constructing a minimum spanning tree or a shortest path tree. When designing the propagation tree, the system also considers redundant paths to handle node failures or network partitions.

[0244] To reduce policy propagation latency and bandwidth consumption, the system employs a policy caching mechanism at different levels. In this three-layer caching structure, the master decision node stores the complete policy model and parameters, including all parameters of the machine learning model, historical data analysis results, and currently applicable compression configuration options. The root nodes (intermediate nodes) of the subtrees in the policy propagation tree cache simplified policy rules, typically decision trees or rule sets extracted from the complete model. These rules are small in size but still cover most decision-making scenarios. Leaf nodes, on the other hand, only store the specific compression configuration parameters currently applicable, such as the compression algorithm and compression level to be used under the current network conditions. This information is extremely small and easy to apply quickly. This layered caching design ensures both the flexibility of decision-making (the master node possesses complete decision-making capabilities) and the efficient propagation and execution of decisions (leaf nodes only need to apply simple configurations).

[0245] In distributed systems, ensuring that all nodes make consistent decisions under identical conditions is crucial, avoiding unnecessary duplicate compression or decompression errors. To this end, the system employs a dedicated consistency protocol to ensure that all nodes within the same subgraph adopt a consistent compression strategy under similar network conditions. The core mechanism is version-based policy synchronization: each time the master decision node generates a new compression strategy, it assigns an incrementing version number and records the timestamp of strategy generation. All nodes periodically exchange information about the version of their currently used strategy, typically via heartbeat messages. When a node discovers that its strategy version is lower than that of its neighbors, it automatically requests an update from the neighbor with the latest version, achieving automatic synchronization. To handle potential network partitions, the system also implements a conflict detection and resolution mechanism: if two different versions of a strategy claim to be the latest (possibly due to network partitions causing two decision centers to generate strategies independently), the system resolves the conflict according to predetermined rules (e.g., later timestamp or higher version number takes precedence), ensuring eventual consistency.

[0246] A particularly complex situation involves handling nodes located at the boundaries of multiple subgraphs. These boundary nodes may communicate with nodes in multiple subgraphs simultaneously. Simply selecting a strategy for one subgraph can lead to inconsistencies in cross-subgraph communication. To address this issue, the system employs a cross-subgraph coordination strategy. Boundary nodes maintain compressed configurations from all adjacent subgraphs in parallel, forming a set of compressed configurations. When a message needs to be sent, the boundary node dynamically selects the appropriate compressed configuration based on the subgraph containing the target node. For example, when sending a message to a node in subgraph A, the compression strategy of subgraph A is used; when sending to subgraph B, the strategy of subgraph B is switched. This dynamic switching mechanism ensures the efficiency and consistency of cross-subgraph communication, avoiding the additional overhead caused by mismatched compression strategies.

[0247] The system also specifically considers the decision weights of boundary nodes. Since boundary nodes simultaneously acquire network conditions from multiple subgraphs, they typically possess richer network context information. Leveraging this characteristic, the system assigns higher weights to boundary nodes during policy decision-making, enabling their observations to play a greater role in policy formation. Boundary nodes periodically report their observed cross-subgraph network conditions to the master decision-making nodes of all adjacent subgraphs; this information becomes valuable input for optimizing the global policy.

[0248] When making the final compression strategy decision, the system comprehensively considers the previously obtained network quality scores (including both local and global scores) and historical compression performance data. Historical data includes the actual effects of different compression strategies under similar network conditions in the past, such as the achieved compression ratio, the introduced additional latency, and the computational resources consumed. The system uses adaptive learning algorithms to analyze this data and continuously optimize the decision model. In a typical implementation, the system may combine algorithms such as decision trees, Bayesian networks, or reinforcement learning to learn the mapping relationship between network conditions and the optimal compression strategy. Finally, the system outputs a set of optimized compression configuration parameters, including the selected compression algorithm (such as LZ4, ZSTD, or LZMA), the specific compression level, memory usage limits, parallelism settings, and other detailed parameters.

[0249] This network topology-based distributed policy decision-making and propagation mechanism ensures that the system can make efficient, consistent, and adaptable compression decisions in complex distributed environments. Its advantages are particularly evident in rapidly changing or heterogeneous network environments—it allows the system to adopt the most suitable compression strategy in different areas based on local and global network conditions, ensuring both data transmission efficiency and overall system consistency, thus providing a solid decision-making foundation for efficient compressed transmission of structured data.

[0250] In another optimized embodiment of the present invention, step S3, which involves identifying the type of each data field based on the template dictionary and the original message data in the structured message data stream, selecting an encoding method according to the data type and performing an encoding operation, and assembling the encoded field data in order of field ID to obtain type-encoded compressed data, includes:

[0251] D1. Calculate the frequency of each symbol in the input string data, use the sliding window technique to obtain the symbol frequency in multiple messages, maintain independent frequency statistics tables for different types of data, and determine the symbol frequency distribution;

[0252] D2. Based on the symbol frequency distribution, generate a variety of possible encoding schemes, including Huffman coding, arithmetic coding or range coding with different parameters, calculate the theoretical compression ratio and coding speed of each encoding scheme, and obtain a candidate set of encoding schemes;

[0253] D3. In the candidate encoding scheme set, the problem of selecting candidate encoding schemes is transformed into a problem of sampling from a discrete distribution. Using an efficient low-variance resampling algorithm, an equally spaced circle of size M is created. M points are evenly placed on the circle. Each candidate scheme is assigned an arc length proportional to its weight. A starting point is randomly selected from the range of zero to one divided by M. Starting from the starting point, the circle is moved along the circle with a fixed step size of one divided by M to select the candidate scheme corresponding to each region it falls into, thus obtaining the set of selected encoding schemes, where M is a positive integer.

[0254] D3 includes:

[0255] D31. For each candidate scheme in the candidate set of the coding scheme, calculate the weight value of each candidate scheme according to its expected performance, wherein the weight value is directly proportional to the compression ratio and inversely proportional to the coding time, and normalize all the weight values ​​to obtain a normalized weight distribution.

[0256] D32. Based on the normalized weight distribution, create a circular structure with a total length of 1. Divide the circle into M regions, each region corresponding to a candidate solution. The length of the i-th region is equal to the normalized weight of the i-th candidate solution, thus obtaining the weighted circular structure, where the value of i ranges from 1 to M.

[0257] D33. For the weighted circular structure, generate a random number as the starting position. The random number is a number between zero and one divided by M. Starting from the starting position, mark M sampling points on the circumference in sequence according to a fixed step size of one divided by M to obtain the set of sampling point positions.

[0258] D34. For each sampling point in the set of sampling point locations, determine the specific region in which each sampling point falls within the weighted circular structure, add the candidate schemes for the corresponding region to the selected set, and obtain the selected encoding scheme set.

[0259] D4. Perform actual encoding tests on the selected encoding scheme set, measure the actual compression ratio and speed, update the weights based on the test results, and obtain the optimized encoding scheme;

[0260] D5. Based on the optimized encoding scheme, perform encoding operations on each data field, assemble the encoded field data in order of field ID, and obtain the compressed data encoded in the specified type.

[0261] In large-scale distributed systems, the transmission of structured messages consumes a significant amount of network bandwidth. Although the template matching mechanism has eliminated most of the structural redundancy, a large amount of compressible space remains in the actual data of each field in the message. This invention proposes an adaptive type-aware encoding mechanism that selects the optimal encoding method based on the characteristics of different data types and dynamically optimizes the encoding scheme through an innovative sampling algorithm, significantly improving data compression efficiency.

[0262] First, the system needs to perform in-depth analysis of the input message data to establish an accurate symbol frequency model. The system employs a sliding window technique, not only counting the frequency of each symbol in the current message but also tracking statistical data from multiple historical messages to form a more stable and accurate frequency distribution model. This method is particularly suitable for processing data streams with temporal locality, meaning that messages transmitted within a short period often have similar data characteristics. Unlike traditional compression methods, the system maintains independent frequency statistics tables for different types of data fields. For example, string, integer, floating-point, and timestamp types each have dedicated frequency tables. This type-aware frequency statistics allows the encoding to more accurately adapt to various data characteristics. During the statistical process, the system dynamically adjusts the size of the statistical window based on the frequency of data changes. Smaller windows are used for rapidly changing data types to quickly adapt to new distribution characteristics, while larger windows are used for stable data types to obtain a more accurate statistical model.

[0263] Based on the established symbol frequency distribution, the system generates several possible encoding schemes. These schemes include Huffman coding, arithmetic coding, and range coding with different parameters, each with its own advantages. Huffman coding is simple to implement and fast in encoding and decoding, but it is inefficient when processing data with relatively even probability distributions. Arithmetic coding can approach the Shannon entropy limit more closely and is suitable for handling non-integer bit encoding. Range coding is a variant of arithmetic coding, offering better precision control and error recovery capabilities. For each basic encoding scheme, the system generates multiple parameter variations, such as different coding tree depths, block sizes, or precision settings. For example, for Huffman coding, the system generates different tree structure variations; for arithmetic coding, it tries different precision and interval partitioning schemes.

[0264] For each generated encoding scheme, the system calculates its theoretical compression ratio and encoding speed, forming a comprehensive set of candidate schemes. The theoretical compression ratio is calculated based on information entropy theory, that is, given the symbol frequency distribution, calculating the average number of bits required per symbol under ideal conditions. The encoding speed is estimated based on algorithm complexity and historical performance data. This set of candidate schemes may contain dozens or even hundreds of different encoding options, each with a different trade-off between compression ratio and speed.

[0265] In practical applications, trying all candidate schemes is obviously impractical and would lead to huge computational overhead. At the same time, selecting only the theoretically optimal scheme is not robust enough, because theoretical models cannot fully capture all the characteristics of real-world data. To address this problem, the system transforms the candidate coding scheme selection problem into a sampling problem from a discrete distribution and employs an efficient, low-variance resampling algorithm.

[0266] This sampling algorithm first calculates a weight value for each scheme in the candidate set. The weight calculation formula is cleverly designed so that it is directly proportional to the expected compression ratio and inversely proportional to the expected encoding time. This weight design ensures that schemes with higher compression ratios and faster encoding speeds receive a higher selection probability. After calculating the weights of all candidate schemes, the system performs normalization so that the sum of all weights is 1, forming a probability distribution.

[0267] Based on this normalized weight distribution, the system creates a virtual circular structure. This circle has a total length of 1 and is divided into M regions, each corresponding to a candidate coding scheme. Crucially, the arc length of each region is proportional to the normalized weight of the corresponding scheme. For example, if a scheme has a normalized weight of 0.25, then the arc length it occupies on the circle is also 0.25 (i.e., one-quarter of the circle). Thus, schemes with higher weights occupy longer arcs on the circle, and therefore have a greater probability of being selected.

[0268] After constructing the circumference, the system generates a random number within the range [0, 1 / M) as the starting position. Starting from this random position, the system moves along the circumference with a fixed step size of 1 / M, marking M sampling points. This equal-step sampling strategy ensures a uniform distribution of sampling points on the circumference, effectively reducing sampling variance. For each sampling point, the system determines which region of the circumference it falls within and adds the corresponding region's encoding scheme to the selected set.

[0269] This systematic resampling algorithm has several significant advantages over simple random sampling: First, it guarantees exactly M options to be selected, no more and no less; second, it significantly reduces sampling variance, making the selection results more stable and reliable; third, its computational complexity is O(M+N) (where N is the total number of candidate options), which is much more efficient than the traditional roulette wheel sampling O(M*N); finally, it automatically achieves a balance between weights and the number of samples, so options with high weights may be selected multiple times, while options with low weights may not be selected at all.

[0270] For a concrete example, suppose the system has 5 candidate encoding schemes with normalized weights of [0.1, 0.15, 0.4, 0.25, 0.1], and needs to select M=3 schemes for testing. The system creates a circle and divides it into 5 regions with lengths of 0.1, 0.15, 0.4, 0.25, and 0.1. Assuming the randomly generated starting position is 0.05 and the step size is 1 / 3, the positions of the three sampling points are 0.05, 0.38(0.05+0.33), and 0.71(0.38+0.33). Checking which regions these points fall into: 0.05 falls into the first region (corresponding to scheme 1), 0.38 falls into the third region (corresponding to scheme 3), and 0.71 also falls into the third region (again, corresponding to scheme 3). Therefore, the final set of selected schemes is {1, 3, 3}. Note that since option 3 has the highest weight (0.4), it was selected twice, which is exactly the expected behavior of the algorithm—the better option gets more testing opportunities.

[0271] After obtaining the selected set of encoding schemes, the system conducts actual encoding tests on these schemes. Unlike theoretical predictions, actual tests directly use real data samples to accurately measure the compression ratio (the ratio of compressed size to original size) and encoding speed (bytes processed per second) of each scheme. These test results provide direct empirical evidence of the schemes' merits, which is more reliable than theoretical models. Based on the test results, the system updates the weight of each scheme, increasing the weight of better-performing schemes and decreasing the weight of poorly performing schemes. This test-based weight adjustment mechanism ensures that the system can quickly converge to a truly efficient encoding scheme.

[0272] After testing and weight updates, the system ultimately determines an optimized encoding scheme for actual data encoding. For each data field in the message, the system performs the following steps: First, it selects the most suitable encoding method based on the field's data type (e.g., integer, floating-point, string) and specific value characteristics; then, it applies the optimized encoding parameters to encode the field value; finally, it organizes the encoded result along with the field ID into a specific format for correct recognition and parsing by the decoding end. All encoded field data is assembled in order of field ID to form the final type-encoded compressed data.

[0273] In practical implementation, the system tailors specific encoding strategies for different data types. For integer types, the system selects different encoding methods based on the range and distribution characteristics of the values, such as variable-length encoding (efficient for small integers) or differential encoding (efficient for increasing sequences). For floating-point numbers, the system considers precision requirements and value distribution, and may use fixed-point representation or special floating-point compression algorithms. For string types, the system not only considers character frequency distribution but also identifies common patterns and repeating substrings, combining dictionary encoding to improve efficiency. For special types such as timestamps, the system utilizes their specific structural characteristics (such as fixed format and limited range) to implement highly specialized encoding schemes.

[0274] Furthermore, the system implements an adaptive encoding adjustment mechanism. As the data stream changes, the symbol frequency distribution may shift, and the system periodically re-evaluates and updates the encoding scheme. The update frequency is dynamically adjusted based on the rate of data change; faster-changing data streams trigger more frequent updates. To avoid performance fluctuations caused by frequent encoding scheme switching, the system sets a stability threshold, only switching occurs when the expected performance of the new scheme is significantly better than the current scheme (e.g., a compression rate improvement exceeding 5%).

[0275] During the encoding process, the system also records detailed performance metrics, such as the average compression ratio of each type of field, encoding speed, and memory usage. These metrics are not only used for internal optimization but also serve as diagnostic tools, helping system administrators understand compression efficiency and potential optimization opportunities. The system can even generate recommendation reports, suggesting which types of data fields can further improve compression efficiency through structural adjustments or data standardization.

[0276] Through this adaptive type-aware encoding mechanism, the system can automatically select the optimal encoding method based on the characteristics of different data fields, and dynamically optimize encoding parameters through innovative sampling and testing algorithms. This highly adaptive and type-aware approach enables the system to maintain near-theoretical optimal compression efficiency when processing complex and variable structured data, while also meeting the requirements of encoding speed, providing strong support for efficient data transmission.

[0277] In summary, the system's type encoding mechanism, through a deep understanding of data type characteristics and distribution patterns, combined with an innovative encoding scheme selection algorithm, achieves a higher compression ratio than general compression algorithms while maintaining low computational overhead. This intelligent encoding strategy is one of the key components of this invention for achieving efficient structured message transmission, enabling the system to transmit more data with limited bandwidth resources, significantly improving the performance and scalability of distributed systems. Especially in resource-constrained environments such as edge computing and the Internet of Things, the bandwidth savings brought by this efficient compression mechanism are crucial, allowing complex distributed applications to run smoothly in these environments.

[0278] like Figure 2 As shown, the present invention also provides an intelligent compression and transmission system for message bodies, comprising:

[0279] The identification and calculation module is used to receive structured message data streams, convert the structured message data streams into abstract syntax trees through a parser, traverse each node of the abstract syntax tree using a depth-first search algorithm and record the complete path, identify the node type of each node and calculate the message structure fingerprint.

[0280] The matching module is used to perform template matching or create new templates based on the message structure fingerprint to obtain a template dictionary containing the mapping relationship between field paths and field IDs;

[0281] The encoding module is used to identify the type of each data field based on the template dictionary and the original message data in the structured message data stream, select the encoding method according to the data type and perform the encoding operation, and assemble the encoded field data in the order of field ID to obtain the type-encoded compressed data.

[0282] The weighting module is used to collect round-trip delay, bandwidth, packet loss rate and jitter indicators during network transmission, standardize the round-trip delay, bandwidth, packet loss rate and jitter indicators and apply a weighted average formula to obtain a comprehensive network quality score.

[0283] The compression configuration module is used to dynamically select a compression strategy based on the network quality comprehensive score and historical compression effect data, combined with an adaptive learning mechanism, and obtain compression configuration parameters corresponding to the compression strategy.

[0284] The compression module is used to further compress the type-encoded compressed data according to the compression configuration parameters and the template dictionary, add a template ID and compression algorithm identifier to the message header to obtain a compressed message body, and send the compressed message body through the network channel;

[0285] The restoration module is used to receive and parse the message header of the message body to obtain the template ID and the compression algorithm identifier, search for the corresponding template from the local template library according to the template ID, and restore it using the corresponding decompression algorithm to obtain the original message data.

[0286] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for intelligent compression and transmission of message bodies, characterized in that, Includes the following steps: The system receives a structured message data stream, converts the structured message data stream into an abstract syntax tree through a parser, traverses each node of the abstract syntax tree using a depth-first search algorithm and records the complete path, identifies the node type of each node and calculates the message structure fingerprint. Based on the message structure fingerprint, template matching or new template creation is performed to obtain a template dictionary containing the mapping relationship between field paths and field IDs; Based on the template dictionary and the original message data in the structured message data stream, type identification is performed on each data field, encoding method is selected according to data type and encoding operation is performed, and the encoded field data is assembled in order of field ID to obtain type-encoded compressed data; The round-trip delay, bandwidth, packet loss rate, and jitter indicators during network transmission are collected. The round-trip delay, bandwidth, packet loss rate, and jitter indicators are standardized and a weighted average formula is applied to obtain a comprehensive network quality score. Based on the comprehensive network quality score and historical compression effect data, a compression strategy is dynamically selected using an adaptive learning mechanism, and compression configuration parameters corresponding to the compression strategy are obtained. Based on the compression configuration parameters and the template dictionary, the type-encoded compressed data is further compressed, and a template ID and compression algorithm identifier are added to the message header to obtain the compressed message body, which is then sent through the network channel. The receiving end receives and parses the message header of the message body to obtain the template ID and the compression algorithm identifier. Based on the template ID, it searches for the corresponding template from the local template library, uses the corresponding decompression algorithm to restore the original message data.

2. The method according to claim 1, characterized in that, The step of identifying the node type of each node and calculating the message structure fingerprint includes: The abstract syntax tree is subjected to node type identification, which distinguishes between basic key-value pairs, arrays and nested objects. The internal element structure of array types is analyzed, and primitive type arrays, object arrays with consistent structure or unstructured mixed arrays are identified. The node type classification results are then output. Based on the node type classification results, all path and type pairing information is collected, and the hash value is calculated after sorting the pairing information to obtain the message structure fingerprint; The step of performing template matching or creating a new template based on the message structure fingerprint to obtain a template dictionary containing the mapping relationship between field paths and field IDs includes: The message structure fingerprint is compared with existing templates in the template library to determine whether there is a matching existing template. If a matching template is found, the ID of the matching template is used. If no match is found, a new matching template is created and a unique matching template ID is assigned. The frequency of occurrence of fields in the new matching template is counted, a first ID is assigned to the first frequency field, a second ID is assigned to the second frequency field, and a mapping relationship from field path to field ID is constructed to obtain the template dictionary containing the mapping relationship from field path to field ID.

3. The method according to claim 1, characterized in that, The data types in the template dictionary and the original message data include integer, floating-point, string, boolean, and null values. Based on the template dictionary and the original message data in the structured message data stream, each data field is type-identified, an encoding method is selected according to the data type, and encoding is performed. The encoded field data is then assembled in order of field ID to obtain type-encoded compressed data, including: For the integer type data in the template dictionary and the original message data, apply ZigZag transform to map signed integers to unsigned integers, and then perform VarInt variable-length encoding to obtain integer encoded data; Perform precision analysis on floating-point data, and select IEEE 754 half-precision or standard double-precision representation according to precision requirements to obtain floating-point encoded data; Pattern recognition is performed on string data, date strings are converted into timestamps or numeric strings are converted into actual numeric values ​​and then encoded, and Huffman encoding is applied to ordinary strings to obtain string encoded data; Bitmap encoding is used for Boolean data and Null values ​​to pack multiple values ​​into the same byte, resulting in Boolean encoded data; Based on the field ID order defined in the template dictionary, the integer encoded data, the floating-point encoded data, the string encoded data, and the boolean encoded data are assembled to obtain the compressed data after the type encoding.

4. The method according to claim 1, characterized in that, The compression configuration parameters, adapted to the current network conditions, are obtained by dynamically selecting a compression strategy based on the comprehensive network quality score and historical compression performance data, combined with an adaptive learning mechanism. These parameters include: Based on the comprehensive network quality score, three network quality states are set: a score greater than or equal to 0.8 is set as the first state EXCELLENT, a score between 0.5 and 0.8 is set as the second state GOOD, and a score less than 0.5 is set as the third state POOR. The first state configuration uses the LZ4 high-speed low compression ratio algorithm, the second state configuration uses the ZSTD balanced algorithm, and the third state configuration enables template compression and entropy coding to maximize the compression ratio, outputting the initial compression strategy configuration; Record a decision snapshot for each compression operation, measure and record the actual compression ratio, compression time, and end-to-end message latency, calculate the comprehensive benefit function, and output the historical data of the compression effect; Based on the historical data of the compression effect, a contextual gambling machine model or regression model is constructed to learn the mapping relationship between network indicators and returns. The parameters of the contextual gambling machine model or regression model are updated periodically using the accumulated effect data, and the weights and state thresholds of the network indicators are optimized to obtain the optimized model parameters. Based on the current network quality status, the initial compression strategy configuration, and the optimized model parameters, a corresponding compression algorithm and parameter configuration are selected to obtain the compression configuration parameters adapted to the current network conditions.

5. The method according to claim 4, characterized in that, Based on the historical data of the compression effect, a contextual gambling machine model or regression model is constructed to learn the mapping relationship between network indicators and returns. The model parameters are periodically updated using accumulated performance data, and the weights and state thresholds of each indicator are optimized to obtain the optimized model parameters, including: Feature extraction is performed on the historical data of compression effect. Network index features and compression decision features are extracted as inputs, and comprehensive benefits are extracted as outputs to obtain a training dataset. Based on the training dataset, a prediction model is constructed using the contextual gambling machine algorithm or the gradient boosting regression algorithm to learn the mapping relationship from network metrics to compression strategy returns, thereby obtaining an initial prediction model. For the initial prediction model, incremental training is performed using newly accumulated effect data at a set period to update the model weight parameters and obtain the updated prediction model; Based on the updated prediction model, the weight coefficients of each indicator in the comprehensive network quality score and the threshold parameters of the network quality status are adjusted to obtain the optimized model parameters.

6. The method according to claim 1, characterized in that, The step of further compressing the type-encoded compressed data according to the compression configuration parameters and the template dictionary, adding a template ID and compression algorithm identifier to the message header to obtain a compressed message body, and sending the compressed message body through a network channel includes: The sending end checks whether the receiving end has the template information corresponding to the template dictionary. If the receiving end does not have the template information, the sending end constructs a control message containing complete template content and outputs a template synchronization control message. The template synchronization control message is sent to the receiving end and a confirmation reply is awaited. The receiving end stores the template and returns a confirmation, thus establishing template consensus and obtaining the template synchronization status. Based on the template synchronization status and the compression configuration parameters, the currently used compression algorithm and its corresponding parameters are determined, and the corresponding compression algorithm is applied to the type-encoded compressed data to compress it, resulting in further compressed data. In the further compressed data, the template ID, the compression algorithm identifier, and decompression parameters are added to the message header to obtain the complete compressed message body; The complete compressed message body is sent through the network channel.

7. The method according to claim 1, characterized in that, The step of performing template matching or creating a new template based on the message structure fingerprint to obtain a template dictionary containing the mapping relationship between field paths and field IDs includes: The message structure in the message structure fingerprint is converted into a high-dimensional sparse vector representation, where each field path is used as a dimension. Existing fields are assigned weight values ​​according to their data types, and non-existent fields are set to have a dimension value of 0. The message structure vector is then output. Multiple hash functions are created using a symbolic random projection method. A set of random vectors is selected from the multiple hash functions. The dot product of the message structure vector and each of the random vectors is calculated, and the sign of the dot product is taken as one bit of the hash value to obtain the hash signature. Using a combination of k hash functions as an index, the hash value of the templates stored in the template library is calculated and L hash tables are constructed to form a template hash index with a multi-level LSH index structure, where k and L are positive integers; Obtain the hash signature of the new message, search in parallel in the L hash tables of the template hash index for candidate templates with the same or similar hash signatures, calculate the actual similarity between the returned candidate set and the message structure vector, select the best match with a similarity exceeding a threshold, and obtain the matching template ID. Based on the matched template ID, if a matching template is found, it is used directly; otherwise, a new matching template is created and a unique template ID is assigned. A mapping relationship between field paths and field IDs is constructed, resulting in the template dictionary containing the mapping relationship between field paths and field IDs.

8. The method according to claim 7, characterized in that, The process of using a combination of k hash functions as an index to calculate hash values ​​for templates stored in the template library and constructing L hash tables to form a multi-level LSH index structure includes: For the first hash function, select m independent symbolic random projection functions, calculate m hash values ​​for each stored template, and combine the m hash values ​​as the index key of the corresponding template in the first hash table to obtain the first-level hash table, where m is a positive integer; For the remaining k-1 hash functions, (k-1)*m independent symbolic random projection functions are generated using different random seeds. The hash value is calculated for each stored template and the corresponding hash table is constructed to obtain a multi-level LSH hash table set, where k is greater than or equal to 2. Based on each hash table in the LSH multilevel hash table set, templates with the same index key are stored in the same hash bucket, and a mapping relationship from hash key to template list is established to obtain the template hash index.

9. The method according to claim 1, characterized in that, The round-trip delay, bandwidth, packet loss rate, and jitter indicators collected during network transmission are standardized and a weighted average formula is applied to obtain a comprehensive network quality score, including: Construct a weighted graph of network topology, wherein each node is represented as a vertex in the graph, the communication links between nodes are represented as edges, and the edge weight is the link's RTT or bandwidth; The network topology weighted graph is decomposed into n subgraphs by applying the directional low-diameter decomposition algorithm, and the communication path length between any two points in each subgraph is ensured to be no more than O(log n / (log log n)), thus obtaining a set of low-diameter subgraphs, where n is a positive integer; For each of the subgraphs in the set of low-diameter subgraphs, select the node with the highest centrality as the main monitoring point, design the aggregation path of the monitoring data to ensure that the data is transmitted along the shortest path within the subgraph, and determine the layout of the hierarchical monitoring points; Based on the hierarchical monitoring point layout, network quality assessment is performed independently within each subgraph, and a comprehensive local network quality score is calculated. The main monitoring point of each subgraph transmits the local assessment results to the global coordination node. The global coordination node integrates all local assessment results to generate a global network quality map and outputs local and global scores. Based on the local and global scores, the monitoring frequency is dynamically adjusted according to the stability of each subgraph, and the final comprehensive network quality score is determined.

10. The method according to claim 9, characterized in that, Based on the comprehensive network quality score and historical compression performance data, a compression strategy is dynamically selected using an adaptive learning mechanism, and compression configuration parameters corresponding to the compression strategy are obtained, including: Based on the set of low-diameter subgraphs, a policy propagation tree is constructed in each subgraph. The node with the best computing power and network connectivity in each subgraph is selected as the master decision node. The master decision node serves as the root node of the policy propagation tree, used to optimize propagation delay and construct the policy propagation tree structure. Based on the policy propagation tree structure, policy caching is set at different levels, including: the main decision node caches the complete policy model and parameters, the root nodes of subtrees in the policy propagation tree cache simplified policy rules, and the leaf nodes only store the currently applicable specific compressed configuration parameters. The design of the consistency protocol ensures that all nodes within the same subgraph adopt a consistent compression strategy under similar network conditions, including: adopting a version number-based strategy synchronization mechanism, exchanging strategy version information between nodes periodically, and automatically synchronizing to the latest version when version differences are found; For nodes located at the boundaries of multiple subgraphs, a cross-subgraph coordination strategy is designed to allow boundary nodes to merge compression strategies from different subgraphs. In this strategy, multiple compression configurations are maintained in parallel at each boundary node, and the compression configurations are dynamically switched according to the source subgraph of the message. Based on the consistent compression strategy and the cross-subgraph coordination strategy, combined with the comprehensive network quality score and historical compression effect data, the optimal compression strategy is dynamically selected to obtain the compression configuration parameters adapted to the current network conditions.

11. The method according to claim 1, characterized in that, Based on the template dictionary and the original message data in the structured message data stream, the type of each data field is identified, an encoding method is selected according to the data type, and an encoding operation is performed. The encoded field data is then assembled in order of field ID to obtain type-encoded compressed data, including: The frequency of each symbol in the input string data is counted. The sliding window technique is used to obtain the symbol frequency in multiple messages. Independent frequency statistics tables are used to maintain different types of data to determine the symbol frequency distribution. Based on the symbol frequency distribution, a variety of possible encoding schemes are generated, including Huffman coding, arithmetic coding, or range coding with different parameters. The theoretical compression ratio and coding speed of each encoding scheme are calculated to obtain a candidate set of encoding schemes. In the candidate encoding scheme set, the problem of selecting candidate encoding schemes is transformed into a problem of sampling from a discrete distribution. Using an efficient low-variance resampling algorithm, an equally spaced circle of size M is created, and M points are evenly placed on the circle. Each candidate scheme is assigned an arc length proportional to its weight. A starting point is randomly selected from the range of zero to one divided by M. Starting from the starting point, the algorithm moves along the circle with a fixed step size of one divided by M to select the candidate scheme corresponding to each region it falls into, thus obtaining the set of selected encoding schemes, where M is a positive integer. Actual encoding tests are performed on the selected encoding scheme set to measure the actual compression ratio and speed. The weights are updated based on the test results to obtain the optimized encoding scheme. Based on the optimized encoding scheme, an encoding operation is performed on each data field, and the encoded field data is assembled in order of field ID to obtain the compressed data encoded in the specified type.

12. The method according to claim 11, characterized in that, In the candidate coding scheme set, the selection problem of candidate coding schemes is transformed into a sampling problem from a discrete distribution. Using an efficient, low-variance resampling algorithm, an equally spaced circle of size M is created. M points are uniformly placed on the circle. Each candidate scheme is assigned an arc length proportional to its weight. A starting point is randomly selected from the range of zero to one divided by M. Starting from the starting point, the algorithm moves along the circle with a fixed step size of one divided by M, selecting the candidate scheme corresponding to each region it falls into. This results in a set of selected coding schemes, including: For each candidate scheme in the candidate set of the encoding scheme, a weight value for each candidate scheme is calculated based on its expected performance, wherein the weight value is directly proportional to the compression ratio and inversely proportional to the encoding time. All the weight values ​​are normalized to obtain a normalized weight distribution. Based on the normalized weight distribution, a circular structure is created with a total length of 1. The circle is divided into M regions, each region corresponding to a candidate solution. The length of the i-th region is equal to the normalized weight of the i-th candidate solution, thus obtaining the weighted circular structure, where the value of i ranges from 1 to M. For the weighted circular structure, a random number is generated as the starting position. The random number is in the range of zero to one divided by M. Starting from the starting position, M sampling points are marked on the circumference in sequence according to a fixed step size of one divided by M to obtain the set of sampling point positions. For each sampling point in the set of sampling point locations, determine the specific region where each sampling point falls within the weighted circular structure, add the candidate schemes for the corresponding region to the selected set, and obtain the selected encoding scheme set.

13. An intelligent compression and transmission system for message bodies, characterized in that, include: The identification and calculation module is used to receive structured message data streams, convert the structured message data streams into abstract syntax trees through a parser, traverse each node of the abstract syntax tree using a depth-first search algorithm and record the complete path, identify the node type of each node and calculate the message structure fingerprint. The matching module is used to perform template matching or create new templates based on the message structure fingerprint to obtain a template dictionary containing the mapping relationship between field paths and field IDs; The encoding module is used to identify the type of each data field based on the template dictionary and the original message data in the structured message data stream, select the encoding method according to the data type and perform the encoding operation, and assemble the encoded field data in the order of field ID to obtain the type-encoded compressed data. The weighting module is used to collect round-trip delay, bandwidth, packet loss rate and jitter indicators during network transmission, standardize the round-trip delay, bandwidth, packet loss rate and jitter indicators and apply a weighted average formula to obtain a comprehensive network quality score. The compression configuration module is used to dynamically select a compression strategy based on the network quality comprehensive score and historical compression effect data, combined with an adaptive learning mechanism, and to obtain compression configuration parameters corresponding to the compression strategy. The compression module is used to further compress the type-encoded compressed data according to the compression configuration parameters and the template dictionary, add a template ID and compression algorithm identifier to the message header to obtain a compressed message body, and send the compressed message body through the network channel; The restoration module is used to receive and parse the message header of the message body to obtain the template ID and the compression algorithm identifier, search for the corresponding template from the local template library according to the template ID, and restore the original message data using the corresponding decompression algorithm.

Images