Information scheduling methods, data compression and transmission methods, and data decompression methods

Abstract

This application relates to an information scheduling method, a data compression and transmission method, and a data decompression method. The method involves obtaining a locally cached list of popular patterns, which includes interrelated pattern numbers and parser identifiers. This list is broadcast by a central scheduling system, which generates the list by tracking parsers that appear frequently in recent node reports. A target parser matching the original message to be sent is determined from the popular pattern list. A data packet is constructed based on the dynamic variable vector obtained after encoding the original message by the target parser, and the target pattern number corresponding to the target parser. The data packet is then sent to a second node. This method effectively improves the compression rate for data with uncertain structures.

Description

Technical Field

[0001] This application relates to the field of distributed network data transmission, and in particular to an information scheduling method, a data compression and transmission method, and a data decompression method. Background Technology

[0002] Traditional data compression methods mainly include general-purpose lossless compression methods and compression methods based on predefined structures (schemas). General-purpose lossless compression methods, such as Gzip, Zstd, and LZ4, compress data by finding repeating byte sequences in the data stream. Compression methods based on predefined structures, such as JSON, Protobuf, and Avro, require data to strictly adhere to a predefined format or pattern.

[0003] In scenarios such as log transmission and real-time data synchronization, the data stream to be transmitted (such as a large amount of log information) appears unstructured or semi-structured on the surface. However, its internal structure evolves over time or through events, revealing a hidden and complex internal structure that cannot be described using deterministic grammars (such as fixed schemas or regular expressions). Traditional data compression methods struggle to effectively compress this type of data.

[0004] Currently, no effective solution has been proposed for the problem of effectively compressing data with uncertain structures. Summary of the Invention

[0005] Therefore, it is necessary to provide an information scheduling method, a data compression and transmission method, and a data decompression method that can improve the data compression rate for data with uncertain structures, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a data compression and transmission method applied to a first node, the method comprising:

[0007] Obtain a list of popular patterns cached locally. The list of popular patterns includes interrelated pattern numbers and parser identifiers. The list of popular patterns is broadcast by a central scheduling system, which generates the list by tracking parsers that appear relatively frequently in recent reports from various nodes.

[0008] The target parser that matches the original message to be sent is determined from the list of popular patterns, and a data packet is constructed based on the dynamic variable vector obtained by encoding the original message by the target parser and the target pattern number corresponding to the target parser.

[0009] The data packet is sent to the second node.

[0010] In one embodiment, determining a target parser that matches the original message to be sent from the list of popular patterns includes:

[0011] For each mode number in the list of popular modes, obtain its associated parser, and call the parser to encode the original message to be sent;

[0012] Determine whether the encoding was successful, and use the parser that successfully encoded the code as the target parser.

[0013] In one embodiment, the method further includes:

[0014] A pre-trained language model is invoked to perform inference on the local data stream to generate a parser, which includes an encoder and a decoder; wherein, the encoder is used to map the original message into a dynamic variable vector, and the decoder is used to map the dynamic variable vector back to the original message;

[0015] The identifier is calculated based on the code of the parser;

[0016] The current time is used as the reporting time, and the reporting time and the identifier of the parser are used as records to report to the central scheduling system.

[0017] Secondly, this application provides a data decompression method applied to a second node, the method comprising:

[0018] Receive data packets sent by the first node, and extract the target mode number and dynamic variable vector from the data packets;

[0019] Obtain the list of popular patterns cached locally, and find the identifier of the parser corresponding to the target pattern number from the list of popular patterns; wherein, the list of popular patterns is broadcast by the central scheduling system, which generates the list of popular patterns by tracking the parsers that appear relatively frequently in the recent reports of each node;

[0020] Based on the identifier, the parser is invoked to decode the dynamic variable vector to obtain the original message.

[0021] Thirdly, this application provides an information scheduling method applied to a central scheduling system, the method comprising:

[0022] Retrieve the records reported by each node, including the parser's identifier for each record;

[0023] Filter the recently reported records from the records, and construct a set of popular parsers based on the identifiers of parsers that appear relatively frequently;

[0024] Assign a pattern number to each identifier in the set of popular parsers, and generate a list of popular patterns based on the pattern number and the associated identifier;

[0025] The list of popular patterns is broadcast to each node in the distributed network system to instruct each node to compress the original message to be sent according to the list of popular patterns, or to decompress the received data packets according to the list of popular patterns.

[0026] In one embodiment, the parser is generated by the node through reasoning on the local data stream using a pre-trained language model, and the parser includes an encoder and a decoder;

[0027] The encoder is used to map the original message into a dynamic variable vector, and the decoder is used to map the dynamic variable vector back into the original message.

[0028] In one embodiment, recently reported records are filtered from the records, and a set of popular parsers is constructed based on the parsers that appear relatively frequently, including:

[0029] Based on the reporting time carried in each record, determine the historical records that are within the time window from the current time, and use the historical records as the most recently reported records;

[0030] The frequency of each identifier in the recently reported records is sorted in descending order, and the top K identifiers are selected to construct the popular parser set.

[0031] Fourthly, this application provides a data synchronization method applied to a distributed network system, the distributed network system including a central scheduling system and multiple nodes; the method includes:

[0032] The central scheduling system acquires records reported by each node, each record including the identifier of a parser; the central scheduling system filters recently reported records from the records, and constructs a set of popular parsers based on the identifiers of parsers with relatively high frequency of occurrence; the central scheduling system assigns a pattern number to each identifier in the set of popular parsers, and generates a list of popular patterns based on the pattern number and the associated identifier; the central scheduling system broadcasts the list of popular patterns to each node in the distributed network system;

[0033] The first node obtains the popular pattern list cached locally, determines the target parser that matches the original message to be sent from the popular pattern list, and constructs a data packet based on the dynamic variable vector obtained by encoding the original message by the target parser and the target pattern number corresponding to the target parser. The first node then sends the data packet to the second node.

[0034] After receiving the data packet, the second node extracts the target pattern number and the dynamic variable vector from the data packet. The second node obtains the locally cached list of popular patterns, searches for the identifier of the parser corresponding to the target pattern number in the list of popular patterns, and calls the parser to decode the dynamic variable vector according to the identifier to obtain the original message.

[0035] Fifthly, this application provides a distributed network system, including a central scheduling system and multiple nodes, wherein the central scheduling system and each of the nodes are communicatively connected, and at least two nodes are communicatively connected to each other;

[0036] When the first node needs to send a message to the second node, the first node executes the data compression and transmission method described in the first aspect above.

[0037] When the second node receives the data packet sent by the first node, the second node executes the data decompression method described in the second aspect above;

[0038] The central scheduling system is used to execute the information scheduling method described in the third aspect above.

[0039] Sixthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods described in the first, second, third, or fourth aspects above.

[0040] The aforementioned data compression rate information scheduling method, data compression transmission method, and data decompression method dynamically track the parsers that appear relatively frequently in recently reported records through a central scheduling system. These parsers represent the complex and non-deterministic hidden structures within the data, used for encoding or decoding. The central scheduling system assigns a very short pattern number to the selected high-frequency parsers. Based on the pattern number and associated parser identifier, a list of popular patterns is generated and broadcast to all nodes in the distributed network system. Nodes, by caching the list of popular patterns, can compress data using only a pattern number and a dynamic variable vector, effectively improving the compression rate for data with uncertain structures. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the structure of a distributed network system in one embodiment;

[0042] Figure 2 This is a flowchart illustrating an information scheduling method in one embodiment;

[0043] Figure 3This is a flowchart illustrating a data compression and transmission method in one embodiment;

[0044] Figure 4 This is a flowchart illustrating a data decompression method in one embodiment;

[0045] Figure 5 This is a flowchart illustrating a data synchronization method in one embodiment;

[0046] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0049] In scenarios such as log transmission and real-time data synchronization, the data stream to be transmitted (such as a large amount of log information) appears unstructured or semi-structured on the surface. However, its internal structure evolves over time or with events, revealing a hidden and complex internal structure that cannot be described using deterministic grammars (such as fixed schemas or regular expressions). Traditional data compression methods struggle to effectively compress this type of data. For example, logs generated within a short period of time, such as October 9, 2025, may contain a large amount of repetition in the year, month, day, and even hour portions of their timestamps; fields such as server identifiers and event types may also be highly concentrated within a certain range.

[0050] General lossless compression methods do not understand the semantics or structure of data at the byte level, and cannot identify and utilize higher-dimensional structural redundancy. For example, even if it is known that all logs within a certain period have the date "2025-10-09", such methods will still transmit the byte sequence of this string repeatedly. Therefore, their compression ratio is low for data with strong internal structure.

[0051] Compression methods based on predefined structures lack flexibility and cannot handle implicit structures. These methods require a fixed and explicit data structure. They are unsuitable for scenarios with log data that lacks a fixed format or whose structure dynamically evolves with business needs. They also fail to discover and utilize implicit relationships and patterns described by simple schemas. For example, they cannot express probabilistic structures such as "when field A has the value 'Error', field B is likely to have the value of one of several IP addresses."

[0052] Therefore, in order to compress uncertain data structures, the first problem that needs to be overcome is to automatically discover the complex and nondeterministic hidden structures in the data, and only then can the data compression be achieved by utilizing the hidden structures.

[0053] Based on the above analysis, in an implementation, Figure 1A schematic diagram of a distributed network system is provided. The system includes a central scheduling system 100 and multiple nodes 200. The central scheduling system 100 and each node 200 are communicatively connected, and at least two nodes 200 are interconnected. When a first node needs to send a message to a second node, the first node compresses the original message into a data packet using a data compression transmission method and sends it to the second node. When the second node receives the data packet sent by the first node, it performs a data decompression method to restore the data packet to the original message. The central scheduling system 100 generates a popular pattern list by executing an information scheduling method and broadcasts the popular pattern list to each node 200 in the distributed network system. This instructs each node 200 to compress the original message to be sent according to the popular pattern list, or to decompress the received data packets according to the popular pattern list.

[0054] The central scheduling system 100 can be implemented using a standalone server or a server cluster consisting of multiple servers. In some embodiments, the distributed network system further includes a data storage system 300, which can store the data that the central scheduling system 100 needs to process. The data storage system 300 can be integrated into the central scheduling system 100 or located in the cloud or on other network servers. Nodes 200 can be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc.

[0055] In one embodiment, Figure 2 A flowchart illustrating the information scheduling method, applied to a central scheduling system, is provided. The process includes the following steps:

[0056] Step S101: Obtain the records reported by each node, each record including the identifier of the parser.

[0057] The parser is generated by a node through reasoning on its local data stream using a pre-trained language model. The parser includes an encoder and a decoder; the encoder maps the original message to a vector of dynamic variables, and the decoder maps the vector of dynamic variables back to the original message. The parser's identifier can be a hash value.

[0058] Specifically, the parser code generated by the pre-trained language model through reasoning on the local data stream is executable program code that can parse specific data structures and perform mutual conversion between raw data and difference information.

[0059] Pre-trained language models can be LLMs (Large Language Models). LLMs refer to Transformer models that have been trained on massive amounts of data, are large in scale, and are powerful enough to perform complex tasks such as text understanding, generation, and code writing. The Transformer model is a deep learning model architecture based on a self-attention mechanism, suitable for processing sequential data.

[0060] Dynamic variable vectors (Diff, difference information) refer to compact information calculated based on the data structure understood by the parser code, containing only the changed parts of the data. In this embodiment, the dynamic variable vectors are not simple text comparisons, but rather structured and semantic differences.

[0061] Step S102: Filter the recently reported records from the records, and construct a set of popular parsers based on the identifiers of parsers that appear relatively frequently.

[0062] The central scheduling system determines the historical records that are within the time window from the current time based on the reporting time carried in each record, and regards the historical records as the recently reported records; it sorts the frequency of each identifier in the recently reported records in descending order, selects the top K identifiers, and constructs a set of popular parsers.

[0063] Step S103: Assign a pattern number to each identifier in the popular parser set, and generate a popular pattern list based on the pattern number and the associated identifier.

[0064] The Pattern ID is a globally unique and extremely short integer identifier assigned by the central scheduling system to the most popular parser recently.

[0065] Step S104: Broadcast the list of popular patterns to each node in the distributed network system to instruct each node to compress the original message to be sent according to the list of popular patterns, or to decompress the received data packets according to the list of popular patterns.

[0066] For example, the central scheduling system maintains a set containing all node-reported records:

[0067] R = {(h1,t1),(h2,t2),...};

[0068] The central scheduling system defines a time window ΔT (e.g., 300 seconds). At the current time t... now Filter out the set of recently reported records:

[0069] R recent ={(h,t)∈R∣t∈[t now-ΔT,t now ]};

[0070] For each recently occurring parser hash value h, calculate its frequency:

[0071] f(h)=|{(h^',t)∈R recent |h^'=h}|;

[0072] The central scheduling system sorts all recent resolver hash values ​​h in descending order by frequency f(h) and selects the top K hash values ​​to form a set of popular resolvers.

[0073] H topK ={h1,h2,...,h K};

[0074] The central scheduling system generates and maintains a list M of popular patterns to map a short integer (i.e., pattern number) p∈{1,2,...,K} to its corresponding hash value and complete parser code:

[0075] M:p→(h p ,P p );

[0076] Among them, h p ∈H topK The list of popular patterns M will be periodically broadcast to all nodes in the distributed network system.

[0077] In steps S101 to S104 above, the central scheduling system dynamically tracks parsers that appear relatively frequently in recently reported records. These parsers represent complex and non-deterministic hidden structures in the data, used for encoding or decoding. The central scheduling system assigns a very short pattern number to the selected high-frequency parsers. Based on the pattern number and the associated parser identifier, it generates a list of popular patterns and broadcasts it to all nodes in the distributed network system. When transmitting data, each node only needs to use the pattern number and dynamic variable vector to achieve effective compression. Through this core mechanism of recent high frequency, the entire distributed network system can automatically adapt to changes in data structure. When business changes lead to a new log or message format becoming mainstream, the central scheduling system automatically identifies it as a new popular pattern and broadcasts it, improving the data compression rate for data containing uncertain structures.

[0078] In an implementation, Figure 3 A flowchart of a data compression and transmission method is provided, applied to the first node. The process includes the following steps:

[0079] Step S201: Obtain the local cached list of popular modes. The list of popular modes includes interrelated mode numbers and parser identifiers. The list of popular modes is broadcast by the central scheduling system, which generates the list by tracking parsers that appear relatively frequently in the recent reports of each node.

[0080] Step S202: Determine the target parser that matches the original message to be sent from the list of popular patterns, and construct a data packet based on the dynamic variable vector obtained after the target parser encodes the original message and the target pattern number corresponding to the target parser.

[0081] For each pattern number in the popular pattern list, obtain its associated parser, call the parser to encode the original message to be sent; determine whether the encoding is successful, and use the parser that successfully encoded the message as the target parser.

[0082] Step S203: Send the data packet to the second node.

[0083] For example, when the first node needs to send a new message s new At that time, the first node will traverse the locally cached list of popular patterns M, as follows:

[0084] (1) For each pattern number p=1,...,K, obtain its corresponding parser. .

[0085] (2) Attempt to encode: .

[0086] (3) If encoding is successful, construct a data packet Packet=(p,v) and send it. The size of the data packet |p|+|v| is much smaller than the original message size |s. new The sending process has been terminated.

[0087] (4) If all popular parsers fail to match, then start the backup transmission mechanism, such as transmitting the complete parser.

[0088] In steps S201 to S203 above, the central scheduling system dynamically tracks parsers that appear relatively frequently in recently reported records. These parsers represent complex and non-deterministic hidden structures in the data, used for encoding or decoding. The central scheduling system assigns a very short pattern number to the selected high-frequency parsers. Based on the pattern number and the associated parser identifier, a list of popular patterns is generated and broadcast to all nodes in the distributed network system. The first node, by caching the list of popular patterns, can compress data using only a pattern number and a dynamic variable vector, effectively improving the compression rate for data with uncertain structures.

[0089] In some embodiments, the first node invokes a pre-trained language model to infer the local data stream and generate a parser. The parser includes an encoder and a decoder. The encoder maps the original message to a dynamic variable vector, and the decoder maps the dynamic variable vector back to the original message. The first node calculates an identifier based on the parser's code; it uses the current time as the reporting time, and reports the reporting time and the parser's identifier as records to the central scheduling system.

[0090] In this embodiment, it is assumed that the data stream generated by the first node is a message sequence S={S1,S2,...,S...} N}, where each message S i It is a sequence of tokens.

[0091] The parser P can be generated by a pre-trained language model LLM to process specific data structures; its essence is a functional pair. .

[0092] encoder This is used to map a raw message s that conforms to a specific structure to a compact dynamic variable vector v (i.e., the difference information Diff). If the message s does not conform to the structure, the mapping fails.

[0093] decoder , is used to map the dynamic variable vector v back to the original message s.

[0094] For any message s that conforms to the structure, it must satisfy... This ensures lossless compression.

[0095] The first node j corresponds to the local data stream S. j The LLM is called for processing, and the LLM is based on the training data. Generate a new parser: .

[0096] The first node j calculates a unique and definite identifier for the newly generated parser, which can be the hash value of its code content: .

[0097] The first node j is at the current time t report This event should be reported to the central dispatch system, with the reported content being a tuple (h). new ,t report ).

[0098] In this embodiment, the first node utilizes a pre-trained language model to automatically learn and discover complex, implicit internal structures and patterns in the data stream that cannot be described by deterministic grammars, achieving a paradigm shift from predefined structures to AI-discovered structures. The abstract structures learned by the pre-trained language model are concretized into an executable parser code, which becomes a dynamic carrier of the data structure, more flexible and powerful than static schemas or configuration files.

[0099] In one embodiment, Figure 4 A flowchart illustrating the data decompression method is provided, applied to the second node. The process includes the following steps:

[0100] Step S301: Receive the data packet sent by the first node and extract the target mode number and dynamic variable vector from the data packet.

[0101] Step S302: Obtain the list of popular modes in the local cache, and find the identifier of the parser corresponding to the target mode number from the list of popular modes; wherein, the list of popular modes is obtained by the central scheduling system broadcast, and the central scheduling system generates the list of popular modes by tracking the parsers that appear relatively frequently in the recent reports of each node.

[0102] Step S303: Based on the identifier, call the parser to decode the dynamic variable vector to obtain the original message.

[0103] For example, when the second node receives the data packet Packet=(p,v), the second node directly looks up the corresponding parser from the locally cached list of popular patterns M using the pattern number p. .

[0104] The decoder in the parser is called to restore the data: .

[0105] because The restored message With the original message s new Completely identical. This process involves only table lookups and function calls, with extremely low computational overhead.

[0106] In steps S301 to S303 above, the central scheduling system dynamically tracks parsers that appear relatively frequently in recently reported records. These parsers represent complex and non-deterministic hidden structures in the data, used for encoding or decoding. The central scheduling system assigns a very short pattern number to the selected high-frequency parsers. Based on the pattern number and the associated parser identifier, it generates a list of popular patterns and broadcasts it to all nodes in the distributed network system. Secondary nodes cache the list of popular patterns, simplifying the data restoration process to table lookups and function calls, avoiding expensive real-time code parsing, compilation, or semantic reasoning, and greatly saving the computing resources of the nodes.

[0107] In some embodiments, the second node invokes a pre-trained language model to infer the local data stream and generate a parser. The parser includes an encoder and a decoder. The encoder maps the original message to a dynamic variable vector, and the decoder maps the dynamic variable vector back to the original message. The second node calculates an identifier based on the parser's code; it uses the current time as the reporting time, and reports the reporting time and the parser's identifier as records to the central scheduling system. The specific process can be referred to the process of the first node reporting records in the above embodiments, and will not be repeated here.

[0108] In one embodiment, Figure 5 A flowchart illustrating a data synchronization method is provided, applicable to a distributed network system, which includes a central scheduling system and multiple nodes. The process includes the following steps:

[0109] In step S401, the central scheduling system obtains the records reported by each node, and each record includes the identifier of the parser; the central scheduling system filters the recently reported records from the records, and constructs a set of popular parsers based on the identifiers of parsers with relatively high frequency of occurrence; the central scheduling system assigns a pattern number to each identifier in the set of popular parsers, and generates a list of popular patterns based on the pattern number and the associated identifier; the central scheduling system broadcasts the list of popular patterns to each node in the distributed network system.

[0110] In step S402, the first node obtains the local cached list of popular patterns, determines the target parser that matches the original message to be sent from the list of popular patterns, and constructs a data packet based on the dynamic variable vector obtained after encoding the original message by the target parser and the target pattern number corresponding to the target parser. The first node then sends the data packet to the second node.

[0111] In step S403, after receiving the data packet, the second node extracts the target pattern number and dynamic variable vector from the data packet. The second node obtains the locally cached list of popular patterns, searches for the identifier of the parser corresponding to the target pattern number in the list of popular patterns, and calls the parser to decode the dynamic variable vector according to the identifier to obtain the original message.

[0112] A distributed network system can be a large microservice cluster containing multiple services such as authentication, order processing, and payment, with each service node generating logs. The internal operation of this large microservice cluster can include the following stages.

[0113] (1) Parser generation and reporting stage.

[0114] The LLM of the authentication service node analyzes its logs and reports its hash_auth to the central scheduling system.

[0115] The LLM of the order service node analyzes its logs, such as [ORDER] Svc-Order:User 123 created order 456 for product 789, generates OrderLogParser, and reports its hash_order to the central scheduling system.

[0116] In a short period of time, due to the high frequency of authentication and order operations, hash_auth and hash_order were reported in large numbers.

[0117] (2) Central scheduling and mode broadcasting stage.

[0118] The central dispatch system statistics show that in the past 5 minutes, hash_auth appeared 10,000 times and hash_order appeared 8,000 times, ranking first and second.

[0119] Central dispatch system allocation mode number:

[0120] {1:AuthLogParser_Code, 2:OrderLogParser_Code,...}.

[0121] The central scheduling system broadcasts the latest list of popular modes to the entire network, enabling all nodes (authentication, order, payment, and other services) to cache the code of the parser AuthLogParser and the parser OrderLogParser locally, and to know that their mode numbers are 1 and 2 respectively.

[0122] (3) Compression and transmission stage.

[0123] The authentication service node needs to send a new log: 2025-10-09 22:50:16,234 [INFO]{service-auth,tx_id=m3n4o5p6} User 'dave' logged in successfully from IP192.168.1.15.

[0124] The authentication service node found that this log is a perfect match for AuthLogParser with pattern number 1.

[0125] The authentication service node calls the parser to generate a Diff: [225016234, 0, "m3n4o5p6", 0, "dave","192.168.1.15"], which corresponds to the dynamic variable vector v in the formal expression.

[0126] The authentication service node ultimately transmits the data packet over the network: [1,[225016234,0,"m3n4o5p6", 0, "dave", "192.168.1.15"]], which corresponds to the compressed data packet (p,v) in the formal expression.

[0127] (4) Restoration and reception stage.

[0128] The log center (a receiving node) receives the data packet [1,[...]].

[0129] The log center extracts pattern number 1 from the data packet and retrieves the parser from the locally cached pattern list.

[0130] The log center sends the difference information to the parser to reconstruct the complete original message. The entire process does not require the use of an LLM or dynamic code compilation, resulting in extremely low computational overhead.

[0131] In steps S401 to S403 above, a central scheduling system is used to dynamically track and broadcast the data structure templates (i.e., parsers) that appear most frequently in recent events, and assign them a very short pattern number. This allows data transmission to reference large structure templates with just one number, greatly reducing transmission overhead. The subsequent data restoration process is simplified to table lookup and function calls, avoiding expensive real-time code parsing, compilation, or semantic reasoning, and greatly saving the computing resources of the nodes.

[0132] In one embodiment, a node device is provided, the node device comprising:

[0133] The cache acquisition module is used to obtain the list of popular patterns in the local cache. The list of popular patterns includes interrelated pattern numbers and parser identifiers. The list of popular patterns is broadcast by the central scheduling system, which generates the list of popular patterns by tracking parsers that appear relatively frequently in the recent reports of each node.

[0134] The data compression module is used to determine the target parser that matches the original message to be sent from the list of popular patterns, and to construct data packets based on the dynamic variable vector obtained after the target parser encodes the original message, and the target pattern number corresponding to the target parser.

[0135] The sending module is used to send data packets to the second node.

[0136] In one embodiment, the data compression module includes:

[0137] The parser call unit is used to obtain the parser associated with each pattern number in the popular pattern list, and call the parser to encode the original message to be sent;

[0138] The judgment unit is used to determine whether the encoding is successful and uses the parser that has successfully encoded the code as the target parser.

[0139] In one embodiment, the node device further includes:

[0140] The pre-trained language model invocation module is used to invoke the pre-trained language model to perform inference on the local data stream and generate a parser. The parser includes an encoder and a decoder. The encoder is used to map the original message into a dynamic variable vector, and the decoder is used to map the dynamic variable vector back to the original message.

[0141] The calculation module is used to calculate the identifier based on the parser's code;

[0142] The reporting module is used to report the current time as the reporting time, and the reporting time and the parser's identifier as records to the central scheduling system.

[0143] In one embodiment, another node device is provided, the node device comprising:

[0144] The receiving module is used to receive data packets sent by the first node and extract the target mode number and dynamic variable vector from the data packets;

[0145] The cache acquisition module is used to obtain the list of popular modes in the local cache and find the identifier of the parser corresponding to the target mode number from the list of popular modes. The list of popular modes is obtained by the central scheduling system broadcasting. The central scheduling system generates the list of popular modes by tracking the parsers that appear relatively frequently in the recent reports of each node.

[0146] The parser call module is used to invoke the parser to decode the dynamic variable vector based on the identifier, and obtain the original message.

[0147] In one embodiment, a server is provided, the server comprising:

[0148] The acquisition module is used to acquire the records reported by each node, and each record includes the parser's identifier;

[0149] The popular parser set construction module is used to filter recently reported records from the records and construct a popular parser set based on the identifiers of parsers that appear relatively frequently.

[0150] The popular pattern list generation module is used to assign pattern numbers to each identifier in the popular parser set and generate a popular pattern list based on the pattern number and the associated identifier.

[0151] The broadcast module is used to broadcast a list of popular patterns to each node in the distributed network system, so as to instruct each node to compress the raw message to be sent according to the list of popular patterns, or to instruct each node to decompress the received data packets according to the list of popular patterns.

[0152] In one embodiment, the parser is generated by a node through reasoning on a local data stream using a pre-trained language model. The parser includes an encoder and a decoder; wherein the encoder is used to map the original message into a dynamic variable vector, and the decoder is used to map the dynamic variable vector back into the original message.

[0153] In one embodiment, the popular parser set construction module includes:

[0154] The recent report filtering unit is used to determine the historical records that are within the time window from the current time based on the reporting time carried in each record, and to treat the historical records as the recent reports.

[0155] The identifier frequency statistics unit is used to sort the frequency of each identifier in the recently reported records in descending order, select the top K identifiers, and construct a set of popular parsers.

[0156] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0157] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores log data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a data compression and transmission method, a data decompression method, or an information scheduling method.

[0158] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0159] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0160] Retrieve the local cached list of popular patterns. The list of popular patterns includes the interrelated pattern numbers and the identifier of the parser. The list of popular patterns is broadcast by the central scheduling system, which generates the list by tracking the parsers that appear relatively frequently in the recent reports of each node.

[0161] The target parser that matches the original message to be sent is determined from the list of popular patterns. The data packet is constructed based on the dynamic variable vector obtained by encoding the original message by the target parser and the target pattern number corresponding to the target parser.

[0162] Send the data packet to the second node.

[0163] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0164] For each pattern number in the list of popular patterns, obtain its associated parser and call the parser to encode the original message to be sent;

[0165] Determine whether the encoding was successful, and use the parser that successfully encoded the code as the target parser.

[0166] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0167] The pre-trained language model is invoked to perform inference on the local data stream and generate a parser. The parser includes an encoder and a decoder. The encoder is used to map the original message into a dynamic variable vector, and the decoder is used to map the dynamic variable vector back to the original message.

[0168] The identifier is calculated based on the parser's code;

[0169] The current time is used as the reporting time, and the reporting time and the parser's identifier are used as records to report to the central scheduling system.

[0170] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0171] Receive data packets sent by the first node and extract the target pattern number and dynamic variable vector from the data packets;

[0172] Obtain the list of popular patterns cached locally, and find the identifier of the parser corresponding to the target pattern number from the list of popular patterns; the list of popular patterns is broadcast by the central scheduling system, which generates the list of popular patterns by tracking the parsers that appear relatively frequently in the recent reports of each node.

[0173] Based on the identifier, the parser is invoked to decode the dynamic variable vector to obtain the original message.

[0174] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0175] Retrieve the records reported by each node, including the parser's identifier for each record;

[0176] Filter the recently reported records from the records, and construct a set of popular parsers based on the identifiers of parsers that appear relatively frequently;

[0177] Assign pattern numbers to each identifier in the set of popular parsers, and generate a list of popular patterns based on the pattern numbers and associated identifiers;

[0178] The list of popular patterns is broadcast to each node in the distributed network system to instruct each node to compress the raw message to be sent according to the list of popular patterns, or to decompress the received data packets according to the list of popular patterns.

[0179] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0180] Based on the reporting time carried in each record, determine the historical records that are within the time window from the current time and treat them as the most recently reported records;

[0181] Sort the frequency of each identifier in the recently reported records in descending order, select the top K identifiers, and construct a set of popular parsers.

[0182] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0183] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0184] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0185] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data compression and transmission method, characterized in that, Applied to the first node, the method includes: Obtain a list of popular patterns cached locally. The list of popular patterns includes interrelated pattern numbers and parser identifiers. The list of popular patterns is broadcast by a central scheduling system, which generates the list by tracking the parsers that appear most frequently in recent reports from each node. The parser includes an encoder and a decoder, and the identifier is calculated based on the code of the parser. The target parser that matches the original message to be sent is determined from the list of popular patterns, and a data packet is constructed based on the dynamic variable vector obtained by encoding the original message by the target parser and the target pattern number corresponding to the target parser. The data packet is sent to the second node; The central scheduling system generates the popular pattern list by tracking the parsers that appear most frequently in the recent reports from each node. This includes: assigning pattern numbers to the identifiers of the parsers that appear most frequently in the recent reports from each node; and generating the popular pattern list based on the pattern numbers and the identifiers. Specifically, the frequency of each identifier in the recently reported records is sorted in descending order, and the top K identifiers are selected to construct a set of popular parsers. The pattern number is a globally unique and extremely short integer identifier.

2. The data compression and transmission method according to claim 1, characterized in that, Determining the target parser that matches the original message to be sent from the list of popular patterns includes: For each mode number in the list of popular modes, obtain its associated parser, and call the parser to encode the original message to be sent; Determine whether the encoding was successful, and use the parser that successfully encoded the code as the target parser.

3. The data compression and transmission method according to claim 1, characterized in that, The method further includes: A pre-trained language model is invoked to perform inference on the local data stream to generate a parser; wherein, the encoder is used to map the original message into a dynamic variable vector, and the decoder is used to map the dynamic variable vector back to the original message; The current time is used as the reporting time, and the reporting time and the identifier of the parser are used as records to report to the central scheduling system.

4. A data decompression method, characterized in that, Applied to the second node, the method includes: Receive data packets sent by the first node, and extract the target mode number and dynamic variable vector from the data packets; Obtain a list of popular patterns cached locally, and find the identifier of the parser corresponding to the target pattern number from the list of popular patterns; wherein, the list of popular patterns is broadcast by the central scheduling system, which generates the list of popular patterns by tracking the parsers that appear most frequently in the recent reports of each node; the parser includes an encoder and a decoder, and the identifier is calculated based on the code of the parser; Based on the identifier, the parser is invoked to decode the dynamic variable vector to obtain the original message; The central scheduling system generates the popular pattern list by tracking the parsers that appear most frequently in the recent reports from each node. This includes: assigning pattern numbers to the identifiers of the parsers that appear most frequently in the recent reports from each node; and generating the popular pattern list based on the pattern numbers and the identifiers. Specifically, the frequency of each identifier in the recently reported records is sorted in descending order, and the top K identifiers are selected to construct a set of popular parsers. The pattern number is a globally unique and extremely short integer identifier.

5. An information scheduling method, characterized in that, Applied to a central scheduling system, the method includes: Retrieve the records reported by each node, including the parser's identifier for each record; Filter the recently reported records from the records, and construct a set of popular parsers based on the identifiers of the parsers that appear most frequently; Assign a pattern number to each identifier in the set of popular parsers, and generate a list of popular patterns based on the pattern number and the associated identifier; The list of popular patterns is broadcast to each node in the distributed network system to instruct each node to compress the original message to be sent according to the data compression transmission method as described in claim 1, or to instruct each node to decompress the received data packets according to the data decompression method as described in claim 4.

6. The information scheduling method according to claim 5, characterized in that, The parser is generated by the node through reasoning on the local data stream using a pre-trained language model, and the parser includes an encoder and a decoder; The encoder is used to map the original message into a dynamic variable vector, and the decoder is used to map the dynamic variable vector back into the original message.

7. The information scheduling method according to claim 5, characterized in that, From the records, the most recently reported records are filtered out, and a set of popular parsers is constructed based on the parsers with the highest frequency of occurrence, including: Based on the reporting time carried in each record, determine the historical records that are within the time window from the current time, and use the historical records as the most recently reported records; The frequency of each identifier in the recently reported records is sorted in descending order, and the top K identifiers are selected to construct the popular parser set.

8. A data synchronization method, characterized in that, The method is applied to a distributed network system, which includes a central scheduling system and multiple nodes; the method includes: The central scheduling system acquires records reported by each node, each record including a parser identifier. The central scheduling system filters recently reported records from these records and constructs a popular parser set based on the identifiers of the most frequently occurring parsers. The central scheduling system assigns a pattern number to each identifier in the popular parser set and generates a popular pattern list based on the pattern number and its associated identifier. The central scheduling system broadcasts the popular pattern list to each node in the distributed network system. Specifically, the frequency of each identifier in the recently reported records is sorted in descending order, and the top K identifiers are selected to construct the popular parser set. Each parser includes an encoder and a decoder, and the identifier is calculated based on the parser's code. The pattern number is a globally unique and extremely short integer identifier. The first node obtains the popular pattern list cached locally, determines the target parser that matches the original message to be sent from the popular pattern list, and constructs a data packet based on the dynamic variable vector obtained by encoding the original message by the target parser and the target pattern number corresponding to the target parser. The first node then sends the data packet to the second node. After receiving the data packet, the second node extracts the target pattern number and the dynamic variable vector from the data packet. The second node obtains the locally cached list of popular patterns, searches for the identifier of the parser corresponding to the target pattern number in the list of popular patterns, and calls the parser to decode the dynamic variable vector according to the identifier to obtain the original message.

9. A distributed network system, characterized in that, It includes a central scheduling system and multiple nodes, wherein the central scheduling system and each of the nodes are communicatively connected, and at least two nodes are communicatively connected to each other; the distributed network system is used to execute the data synchronization method described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images