Information scheduling method, data compression transmission method and data decompression method

Abstract

The invention relates to an information scheduling method, a data compression transmission method and a data decompression method, and the method comprises the steps: obtaining a hot mode list of a local cache, the hot mode list comprising a mode number and an identifier of an analyzer, which are correlated with each other; wherein the hot mode list is broadcasted by the central scheduling system, and the central scheduling system generates the hot mode list by tracking the resolver with relatively high occurrence frequency in the report record of each node recently; determining a target parser matched with the to-be-sent original message from the hot mode list, and constructing a data packet according to a dynamic variable vector obtained after the target parser encodes the original message and a target mode number corresponding to the target parser; sending the data packet to a second node; and the compression rate of the data of the uncertain structure is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of distributed network data transmission, and in particular to an information scheduling method, a data compression transmission method and a data decompression method. BACKGROUND

[0002] Traditional data compression methods mainly include general lossless compression methods and compression methods based on predefined structures (Schema). Among them, general lossless compression methods, such as Gzip, Zstd, LZ4, etc., compress by finding repeated byte sequences in data streams. Compression methods based on predefined structures, such as JSON, Protobuf, Avro, etc., require data to strictly follow a predefined format or pattern.

[0003] In log transmission, real-time data synchronization and other scenarios, the data stream to be transmitted (such as a large amount of log information) is unstructured or semi-structured on the surface, and its internal structure is hidden and complex with the evolution of time or events, which cannot be described by a deterministic grammar (such as a fixed Schema or regular expression). Traditional data compression methods are difficult to effectively compress such data.

[0004] At present, there is no effective solution to the problem of effectively compressing data containing uncertain structures. SUMMARY

[0005] Therefore, it is necessary to provide an information scheduling method, a data compression transmission method and a data decompression method capable of improving data compression rate for data containing uncertain structures to solve the above technical problems.

[0006] In a first aspect, the present application provides a data compression transmission method applied to a first node, the method comprising:

[0007] obtaining a hot schema list cached locally, the hot schema list comprising a schema number and an identifier of a parser associated with each other; wherein the hot schema list is obtained by broadcasting from a central scheduling system, and the central scheduling system generates the hot schema list by tracking parsers with relatively high frequencies in recent node reporting records;

[0008] determining a target parser matching an original message to be sent from the hot schema list, and constructing a data packet according to a dynamic variable vector obtained by encoding the original message according to the target parser, and a target schema number corresponding to the target parser;

[0009] sending the data packet to a second node.

[0010] In one embodiment, determining a target parser that matches the original message to be sent from the list of popular patterns includes:

[0011] For each mode number in the list of popular modes, obtain its associated parser, and call the parser to encode the original message to be sent;

[0012] Determine whether the encoding was successful, and use the parser that successfully encoded the code as the target parser.

[0013] In one embodiment, the method further includes:

[0014] A pre-trained language model is invoked to perform inference on the local data stream to generate a parser, which includes an encoder and a decoder; wherein, the encoder is used to map the original message into a dynamic variable vector, and the decoder is used to map the dynamic variable vector back to the original message;

[0015] The identifier is calculated based on the code of the parser;

[0016] The current time is used as the reporting time, and the reporting time and the identifier of the parser are used as records to report to the central scheduling system.

[0017] Secondly, this application provides a data decompression method applied to a second node, the method comprising:

[0018] Receive data packets sent by the first node, and extract the target mode number and dynamic variable vector from the data packets;

[0019] Obtain the list of popular patterns cached locally, and find the identifier of the parser corresponding to the target pattern number from the list of popular patterns; wherein, the list of popular patterns is broadcast by the central scheduling system, which generates the list of popular patterns by tracking the parsers that appear relatively frequently in the recent reports of each node;

[0020] Based on the identifier, the parser is invoked to decode the dynamic variable vector to obtain the original message.

[0021] Thirdly, this application provides an information scheduling method applied to a central scheduling system, the method comprising:

[0022] Retrieve the records reported by each node, including the parser's identifier for each record;

[0023] Filter the recently reported records from the records, and construct a set of popular parsers based on the identifiers of parsers that appear relatively frequently;

[0024] assign a pattern number to each identifier in the hot pattern resolver set, and generate a hot pattern list according to the pattern number and the associated identifier;

[0025] broadcast the hot pattern list to each node in the distributed network system to instruct each node to compress the original message to be sent according to the hot pattern list or instruct each node to decompress the received data packet according to the hot pattern list.

[0026] In one embodiment, the resolver is generated by the node by inferring the local data stream through a pre-trained language model, and the resolver includes an encoder and a decoder;

[0027] The encoder is configured to map the original message to a dynamic variable vector, and the decoder is configured to map the dynamic variable vector back to the original message.

[0028] In one embodiment, the recent reported records are filtered from the records, and a hot pattern resolver set is constructed according to the resolvers with relatively high frequency, including:

[0029] According to the reporting time carried in each record, the historical records within a time window from the current time are determined as the recent reported records;

[0030] The frequencies of occurrence of each identifier in the recent reported records are arranged in descending order, and the top K identifiers are selected to construct the hot pattern resolver set.

[0031] In a fourth aspect, the present application provides a data synchronization method applied to a distributed network system, the distributed network system including a central scheduling system and multiple nodes; the method including:

[0032] The central scheduling system obtains the records reported by each node, and each record includes an identifier of a resolver; the central scheduling system filters recent reported records from the records, and constructs a hot pattern resolver set according to the identifiers of the resolvers with relatively high frequency; the central scheduling system assigns a pattern number to each identifier in the hot pattern resolver set, and generates a hot pattern list according to the pattern number and the associated identifier; and the central scheduling system broadcasts the hot pattern list to each node in the distributed network system;

[0033] The first node obtains the hot pattern list cached locally, determines a target resolver matching the original message to be sent from the hot pattern list, and constructs a data packet according to the dynamic variable vector obtained by encoding the original message according to the target resolver and the target pattern number corresponding to the target resolver, and the first node sends the data packet to the second node;

[0034] The second node receives the data packet, extracts the target mode number and the dynamic variable vector from the data packet, acquires a locally cached popular mode list, finds an identifier of a resolver corresponding to the target mode number from the popular mode list, and calls the resolver to decode the dynamic variable vector according to the identifier, to obtain the original message.

[0035] In a fifth aspect, the present application provides a distributed network system, comprising a central scheduling system and a plurality of nodes, the central scheduling system and each of the nodes being communicatively connected, and at least two of the nodes being communicatively connected to each other.

[0036] When the first node needs to send a message to the second node, the first node performs the data compression transmission method of the first aspect.

[0037] When the second node receives the data packet sent by the first node, the second node performs the data decompression method of the second aspect.

[0038] The central scheduling system is configured to perform the information scheduling method of the third aspect.

[0039] In a sixth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, the computer program being configured to implement the steps of the method of the first aspect, the second aspect, the third aspect or the fourth aspect when executed by a processor.

[0040] The information scheduling method, the data compression transmission method and the data decompression method have the following advantages. The central scheduling system dynamically tracks a resolver with a relatively high frequency of occurrence in recent reporting records, and the resolver is a complex and non-deterministic hidden structure in data, which is used for encoding or decoding data. The central scheduling system assigns a very short mode number to the high-frequency resolver, generates a popular mode list according to the mode number and an associated resolver identifier, and broadcasts the popular mode list to all nodes in the distributed network system. The nodes cache the popular mode list, and only need a mode number and a dynamic variable vector to compress data, thereby effectively improving the compression rate of data with uncertain structures. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 FIG. 1 is a structural schematic diagram of a distributed network system in an embodiment;

[0042] Figure 2 FIG. 4 is a flowchart of an information scheduling method in an embodiment;

[0043] Figure 3Fig. 1 is a flowchart of a data compression transmission method according to an embodiment of the present application;

[0044] Figure 4 Fig. 2 is a flowchart of a data decompression method according to an embodiment of the present application;

[0045] Figure 5 Fig. 3 is a flowchart of a data synchronization method according to an embodiment of the present application;

[0046] Figure 6 Fig. 4 is a block diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0048] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the general meaning understood by a person with ordinary skill in the art to which the present application belongs. In the present application, "one", "a", "an", "the", "these" and similar words do not represent a quantitative limitation, and they can be singular or plural. In the present application, the terms "include", "contain", "have" and any variants thereof are intended to cover non-exclusive inclusion; for example, a process, method and system, product or device containing a series of steps or modules (units) are not limited to the listed steps or modules (units), but can include steps or modules (units) not listed, or can include other steps or modules (units) inherent to the process, method, product or device. In the present application, the terms "connected", "connected", "coupled" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. In the present application, "multiple" means two or more. The association between the associated objects is described by the term "and / or", which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. In general, the character " / " represents an "or" relationship between the associated objects. In the present application, the terms "first", "second", "third" and the like are only used to distinguish similar objects, and do not represent a specific order of the objects.

[0049] In scenarios such as log transmission, real-time data synchronization, the data stream to be transmitted (such as a large amount of log information) is unstructured or semi-structured on the surface, and its internal structure is hidden and complex with the evolution of time or events, which cannot be described by a deterministic grammar (such as a fixed schema or regular expression). Traditional data compression methods are difficult to effectively compress such data. For example, on October 9, 2025, the current time, the logs generated in a short time have a large number of repeated years, months, days, and even hour parts in their timestamps; the server identifier and event type fields can also be highly concentrated within a certain range.

[0050] A general-purpose lossless compression method does not understand the semantics or structure of the data at the byte level and cannot identify and utilize higher-dimensional structural redundancy. For example, even if it is known that the date of all logs within a certain period of time is "2025-10-09", such a method will still transmit the byte sequence of this string repeatedly. Therefore, for data with strong internal structural properties, the compression rate is low.

[0051] Compression methods based on predefined structures have poor flexibility and cannot handle implicit structures. Such methods require data structures to be fixed and explicit. For log data without a fixed format or scenarios where the structure evolves dynamically with business needs, such methods are difficult to apply. It is impossible to discover and utilize implicit associations and patterns in the data that are described by simple schemas. For example, it is impossible to express the probabilistic structure "when the value of field A is 'Error', the value of field B is likely to be one of several IP addresses".

[0052] Therefore, to achieve compression of uncertain data structures, the first problem to be overcome is to automatically discover the complex and non-deterministic hidden structure in the data, and then it is possible to utilize the hidden structure to achieve data compression.

[0053] Based on the above analysis, in one embodiment, Figure 1A structural diagram of a distributed network system is provided, which comprises a central scheduling system 100 and a plurality of nodes 200, the central scheduling system 100 and the nodes 200 are communicatively connected, and at least two nodes 200 are communicatively connected with each other. When a first node needs to send a message to a second node, the first node compresses the original message into a data packet by executing a data compression transmission method, and sends the data packet to the second node. When the second node receives the data packet sent by the first node, the second node restores the data packet to the original message by executing a data decompression method. The central scheduling system 100 generates a popular pattern list by executing an information scheduling method, and broadcasts the popular pattern list to each node 200 in the distributed network system to instruct each node 200 to compress the original message to be sent according to the popular pattern list, or instruct each node 200 to decompress the received data packet according to the popular pattern list.

[0054] The central scheduling system 100 can be implemented by an independent server or a server cluster composed of a plurality of servers. In some embodiments, the distributed network system further comprises a data storage system 300, which can store data required to be processed by the central scheduling system 100. The data storage system 300 can be integrated on the central scheduling system 100, or placed on a cloud or other network server. The nodes 200 can be, but are not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc.

[0055] In one embodiment, Figure 2 A flowchart of an information scheduling method is provided, which is applied to a central scheduling system, and the flowchart comprises the following steps:

[0056] In step S101, records reported by each node are obtained, and each record comprises an identifier of a parser.

[0057] The parser is generated by the node by inferring a local data stream by using a pre-trained language model, and the parser comprises an encoder and a decoder. The encoder is used to map an original message to a dynamic variable vector, and the decoder is used to map the dynamic variable vector back to the original message. The identifier of the parser can be a hash value.

[0058] The specific parser code generated by inferring the local data stream by using the pre-trained language model is a Parser Code, which is executable program code capable of parsing a specific data structure and converting between original data and difference information.

[0059] The pre-trained language model can be a large language model (LLM). The LLM refers to a Transformer model with large scale and strong capability trained on massive data, which can perform complex tasks such as text understanding, generation, and code writing. The Transformer model is a deep learning model architecture based on a self-attention mechanism, which is suitable for processing sequence data.

[0060] The dynamic variable vector (Diff) refers to compact information containing only the changed part of the data, which is calculated based on the data structure understood by the parser code. The dynamic variable vector in this embodiment is not a simple text comparison difference, but a structured and semantic difference.

[0061] In step S102, the recent reported records are filtered from the records, and a popular parser set is constructed according to the identifiers of the parsers with relatively high frequencies.

[0062] The central scheduling system determines the historical records within a time window from the current time according to the reporting time carried in each record, and takes the historical records as recent reported records. The frequencies of occurrence of each identifier in the recent reported records are arranged in descending order, and the top K identifiers are selected to construct the popular parser set.

[0063] In step S103, a pattern number is assigned to each identifier in the popular parser set, and a popular pattern list is generated according to the pattern number and the associated identifier.

[0064] The pattern number (Pattern ID) is a globally unique and extremely short integer identifier assigned by the central scheduling system to the most popular parser.

[0065] In step S104, the popular pattern list is broadcast to each node in the distributed network system to instruct each node to compress the original message to be sent according to the popular pattern list, or to instruct each node to decompress the received data packet according to the popular pattern list.

[0066] For example, the central scheduling system maintains a set containing all the reported records of the nodes:

[0067] R={(h1,t1),(h2,t2),...};

[0068] The central scheduling system defines a time window ΔT (for example, 300 seconds). At the current time t now , the recent reported record set is filtered out:

[0069] R recent ={(h,t)∈R∣t∈[t now- AT, t now ]};

[0070] For each recent resolver hash value h, calculate its frequency:

[0071] f(h) = | {(h', t) e R recent | h' = h} |;

[0072] The central scheduling system sorts all recent resolver hash values h in descending order of frequency f(h), and selects the top K hash values to form a popular resolver set:

[0073] H topK = {h1, h2,..., h K};

[0074] The central scheduling system generates and maintains a popular pattern list M to map a short integer (i.e. pattern number) p e {1, 2,..., K} to its corresponding hash value and complete resolver code:

[0075] M: p→(h p , P p );

[0076] Where h p e H topK . The popular pattern list M will be periodically broadcast to all nodes in the distributed network system.

[0077] In the above steps S101 to S104, the central scheduling system dynamically tracks the relatively high frequency of the resolver appearing in the recent reporting record, which is the complex and non-deterministic hidden structure in the data for encoding or decoding data. The central scheduling system assigns a very short pattern number to the high-frequency resolver selected, generates a popular pattern list according to the pattern number and associated resolver identifier, and broadcasts it to all nodes in the distributed network system. Each node can achieve effective compression by using only the pattern number and dynamic variable vector when transmitting data. The entire distributed network system can automatically adapt to changes in data structure through the core mechanism of recent high frequency. When business changes cause new log or message formats to become mainstream, the central scheduling system will automatically identify them as new popular patterns and broadcast them, which can improve the data compression rate for data containing uncertain structures.

[0078] In one embodiment, Figure 3 A flowchart of a data compression transmission method is provided, which is applied to a first node and includes the following steps:

[0079] Step S201, obtaining a hot schema list cached locally, the hot schema list including schema numbers and identifiers of parsers associated with each other; wherein the hot schema list is obtained by broadcasting from a central scheduling system, and the central scheduling system generates the hot schema list by tracking parsers with relatively high frequencies in recent reporting records of each node.

[0080] Step S202, determining a target parser matching the original message to be sent from the hot schema list, and constructing a data packet according to a dynamic variable vector obtained by encoding the original message by the target parser, and a target schema number corresponding to the target parser.

[0081] For each schema number in the hot schema list, the associated parser is obtained, and the parser is called to encode the original message to be sent; it is judged whether the encoding is successful, and the parser with successful encoding is taken as the target parser.

[0082] Step S203, sending the data packet to the second node.

[0083] Exemplarily, when the first node needs to send a new message s new , the first node will traverse the hot schema list M cached locally, and the process is as follows:

[0084] (1) For each schema number p = 1,...,K, the corresponding parser is obtained.

[0085] (2) Try to encode: .

[0086] (3) If the encoding is successful, a data packet Packet = (p, v) is constructed and sent. The size of the data packet |p| + |v| is much smaller than the size of the original message |s new |. The sending process is terminated.

[0087] (4) If all hot parsers fail to match, a backup transmission mechanism is started, for example, a complete parser is transmitted.

[0088] In the above steps S201 to S203, the central scheduling system dynamically tracks parsers with relatively high frequencies in recent reporting records, and the parser is a hidden structure in data that is complex and uncertain, and is used for encoding or decoding data. The central scheduling system assigns a very short schema number to the high-frequency parser screened out, generates a hot schema list according to the schema number and the identifier of the associated parser, and broadcasts it to all nodes in the distributed network system. The first node compresses data by only one schema number and a dynamic variable vector by caching the hot schema list, which effectively improves the compression rate of data with uncertain structures.

[0089] In some embodiments, the first node will call the pre-trained language model to infer on the local data stream, generating a parser, which includes an encoder and a decoder; wherein the encoder is used to map the original message to a dynamic variable vector, and the decoder is used to map the dynamic variable vector back to the original message. The first node calculates the identifier according to the code of the parser; takes the current time as the reporting time, and reports the reporting time and the identifier of the parser to the central scheduling system as a record.

[0090] In this embodiment, it is assumed that the data stream generated by the first node is a message sequence S = {S1, S2,..., S N}, where each message S i is a Token sequence.

[0091] The parser P can be generated by a pre-trained language model LLM for processing a specific data structure, which is essentially a function pair .

[0092] The encoder is used to map an original message s conforming to a specific structure into a compact dynamic variable vector v (i.e. difference information Diff). If the message s does not conform to the structure, the mapping fails.

[0093] The decoder is used to map the dynamic variable vector v back to the original message s.

[0094] For any message s conforming to the structure, it must satisfy to ensure lossless compression.

[0095] The first node j calls the LLM to process the local data stream S j , and the LLM generates a new parser according to the training data : .

[0096] The first node j calculates a unique and deterministic identifier for the newly generated parser, which can be the hash value of its code content: .

[0097] The first node j reports this event to the central scheduling system at the current time t report , and the reporting content is the tuple (h new , t report ).

[0098] In this embodiment, the first node utilizes a pre-trained language model to automatically learn and discover complex and implicit internal structures and patterns in the data stream that cannot be described by a deterministic grammar, realizing a paradigm shift from pre-defined structures to AI-discovered structures. The abstract structures learned by the pre-trained language model are materialized into an executable parser code, which becomes a dynamic carrier of data structures, more flexible and powerful than static Schema or configuration files.

[0099] In one embodiment, Figure 4 A flowchart of a data decompression method is provided, which is applied to the second node, and the flow includes the following steps:

[0100] Step S301, receiving the data packet sent by the first node, and extracting the target mode number and the dynamic variable vector from the data packet.

[0101] Step S302, obtaining the hot mode list cached locally, and searching for the identifier of the parser corresponding to the target mode number from the hot mode list; wherein the hot mode list is obtained from the central scheduling system, and the central scheduling system generates the hot mode list by tracking the relatively high frequency of the parser appearing in the recent node reporting records.

[0102] Step S303, decoding the dynamic variable vector by calling the parser according to the identifier, to obtain the original message.

[0103] Exemplarily, when the second node receives the data packet Packet=(p, v), the second node directly searches for the corresponding parser from the hot mode list M cached locally using the mode number p. .

[0104] The decoder in the parser is called to restore the data: .

[0105] Since , the restored message is completely consistent with the original message s new . This process only involves table lookup and function call, and the computational overhead is extremely low.

[0106] In the steps S301-S303, the central scheduling system dynamically tracks the relatively high frequency of the analyzers in the recent reporting records, and the analyzer is the complex and non-deterministic hidden structure in the data for encoding or decoding the data. The central scheduling system assigns a very short mode number to the high-frequency analyzers screened out, generates a popular mode list according to the mode number and the associated analyzer identifier, and broadcasts the popular mode list to all nodes in the distributed network system. The second node simplifies the data restoration process to table lookup and function call by caching the popular mode list, avoids expensive real-time code analysis, compilation or semantic reasoning, and greatly saves the computing resources of the node.

[0107] In some embodiments, the second node will call the pre-trained language model to infer the local data stream and generate an analyzer including an encoder and a decoder; wherein the encoder is used to map the original message to a dynamic variable vector, and the decoder is used to map the dynamic variable vector back to the original message. The second node calculates the identifier according to the code of the analyzer, takes the current time as the reporting time, and reports the reporting time and the identifier of the analyzer to the central scheduling system as a record. The specific process can refer to the process of the first node reporting the record in the above-mentioned embodiments, which will not be described here.

[0108] In one embodiment, Figure 5 A flowchart of a data synchronization method is provided, which is applied to a distributed network system including a central scheduling system and a plurality of nodes. The flowchart includes the following steps:

[0109] In step S401, the central scheduling system obtains records reported by each node, and each record includes an identifier of an analyzer. The central scheduling system screens the recent reporting records from the records, constructs a set of popular analyzers according to the identifiers of the analyzers with relatively high frequency, assigns mode numbers to each identifier in the set of popular analyzers, generates a popular mode list according to the mode numbers and the associated identifiers, and broadcasts the popular mode list to each node in the distributed network system.

[0110] In step S402, the first node obtains the locally cached popular mode list, determines a target analyzer matching the original message to be sent from the popular mode list, and constructs a data packet according to the dynamic variable vector obtained by encoding the original message by the target analyzer and the target mode number corresponding to the target analyzer. The first node sends the data packet to the second node.

[0111] Step S403, after receiving the data packet, the second node extracts the target mode number and the dynamic variable vector from the data packet, the second node obtains the local cached popular mode list, finds the identifier of the parser corresponding to the target mode number from the popular mode list, and the second node calls the parser to decode the dynamic variable vector according to the identifier to obtain the original message.

[0112] The distributed network system can be a large micro-service cluster including authentication, order, payment and other services, and each service node will generate logs. The internal operation of the large micro-service cluster can include the following stages.

[0113] (1) Parser generation and reporting stage.

[0114] The LLM of the authentication service node analyzes its logs and hashes hash_auth and reports it to the central scheduling system.

[0115] The LLM of the order service node analyzes its logs, such as [ORDER] Svc-Order:User 123 created order 456 for product 789, generates OrderLogParser, and reports its hash hash_order to the central scheduling system.

[0116] In a short period of time, because the authentication and order operations are frequent, hash_auth and hash_order are reported in large quantities.

[0117] (2) Central scheduling and mode broadcast stage.

[0118] The central scheduling system finds that hash_auth has appeared 10,000 times and hash_order has appeared 8,000 times in the past 5 minutes, ranking the top two.

[0119] The central scheduling system assigns mode numbers:

[0120] {1:AuthLogParser_Code, 2:OrderLogParser_Code,...}.

[0121] The central scheduling system broadcasts the latest popular mode list to the entire network, so that all nodes (authentication, order, payment and other services) locally cache the codes of the parser AuthLogParser and the parser OrderLogParser, and know that their mode numbers are 1 and 2 respectively.

[0122] (3) Compression and transmission stage.

[0123] The authentication service node needs to send a new log: 2025-10-09 22:50:16,234 [INFO] {service-auth, tx_id=m3n4o5p6} User 'dave' logged in successfully from IP 192.168.1.15.

[0124] The authentication service node finds that this log exactly matches the AuthLogParser with pattern number 1.

[0125] The authentication service node invokes the parser to generate the Diff: [225016234, 0, "m3n4o5p6", 0, "dave", "192.168.1.15"], which corresponds to the dynamic variable vector v in the formal expression.

[0126] The authentication service node finally transmits the packet [1, [225016234, 0, "m3n4o5p6", 0, "dave", "192.168.1.15"]] over the network, which corresponds to the compressed packet (p, v) in the formal expression.

[0127] (4) The restore and receive phase.

[0128] The log center (a receiving node) receives the packet [1, [...]].

[0129] The log center extracts the pattern number 1 from the packet and invokes the parser Parser from the locally cached list of patterns.

[0130] The log center passes the Diff into the parser Parser and restores the complete original message. The entire process does not require LLM and does not need to dynamically compile code, with extremely low computational overhead.

[0131] In the above steps S401 to S403, a central scheduling system is used to dynamically track and broadcast the data structure template (i.e., the parser) that appears most frequently in recent events, and to assign it a very short pattern number, so that only a number needs to be transmitted to refer to the huge structure template when data is transmitted, greatly reducing the transmission overhead. The subsequent data restoration process is simplified to table lookup and function call, avoiding expensive real-time code parsing, compiling or semantic reasoning, greatly saving the computing resources of the nodes.

[0132] In one embodiment, a node device is provided, comprising:

[0133] The cache obtaining module is configured to obtain a hot schema list cached locally, the hot schema list comprising schema numbers and identifiers of the parsers associated with each other; wherein the hot schema list is obtained by broadcasting from a central scheduling system, and the central scheduling system generates the hot schema list by tracking the parsers with relatively high frequencies appearing in recent node reporting records;

[0134] The data compression module is configured to determine a target parser matching the original message to be sent from the hot schema list, and construct a data packet according to a dynamic variable vector obtained by encoding the original message by the target parser and a target schema number corresponding to the target parser.

[0135] The sending module is configured to send the data packet to the second node.

[0136] In an embodiment, the data compression module comprises:

[0137] The parser calling unit is configured to, for each schema number in the hot schema list, obtain the associated parser, and call the parser to encode the original message to be sent.

[0138] The judging unit is configured to judge whether the encoding is successful, and take the parser with successful encoding as the target parser.

[0139] In an embodiment, the node device further comprises:

[0140] The pre-trained language model calling module is configured to call a pre-trained language model to infer the local data stream and generate a parser, the parser comprising an encoder and a decoder; wherein the encoder is configured to map the original message to a dynamic variable vector, and the decoder is configured to map the dynamic variable vector back to the original message.

[0141] The computing module is configured to calculate the identifier according to the code of the parser.

[0142] The reporting module is configured to report the current time as a reporting time, and report the reporting time and the identifier of the parser to the central scheduling system as a record.

[0143] In an embodiment, another node device is provided, which comprises:

[0144] The receiving module is configured to receive the data packet sent by the first node, and extract the target schema number and the dynamic variable vector from the data packet.

[0145] The cache obtaining module is configured to obtain a hot schema list cached locally, and find the identifier of the parser corresponding to the target schema number from the hot schema list; wherein the hot schema list is obtained by broadcasting from a central scheduling system, and the central scheduling system generates the hot schema list by tracking the parsers with relatively high frequencies appearing in recent node reporting records.

[0146] The resolver calling module is configured to call the resolver to decode the dynamic variable vector according to the identifier, to obtain the original message.

[0147] In one embodiment, a server is provided, which comprises:

[0148] The acquisition module is configured to acquire records reported by the nodes, each record comprising an identifier of a resolver.

[0149] The popular resolver set construction module is configured to filter recent reported records from the records, and construct a popular resolver set according to identifiers of resolvers with relatively higher frequencies.

[0150] The popular pattern list generation module is configured to assign a pattern number to each identifier in the popular resolver set, and generate a popular pattern list according to the pattern number and the associated identifier.

[0151] The broadcasting module is configured to broadcast the popular pattern list to the nodes in the distributed network system, to instruct the nodes to compress the original messages to be sent according to the popular pattern list, or to instruct the nodes to decompress the received data packets according to the popular pattern list.

[0152] In one embodiment, the resolver is generated by a node through inferring a local data stream by a pre-trained language model, and the resolver comprises an encoder and a decoder; wherein the encoder is configured to map the original message into a dynamic variable vector, and the decoder is configured to map the dynamic variable vector back to the original message.

[0153] In one embodiment, the popular resolver set construction module comprises:

[0154] The recent reported record filtering unit is configured to determine historical records within a time window from a current time according to a reporting time carried in each record, and take the historical records as recent reported records.

[0155] The identifier frequency statistics unit is configured to arrange frequencies of each identifier in the recent reported records in descending order, select the first K identifiers, and construct the popular resolver set.

[0156] It should be noted that each of the above modules can be a functional module or a program module, and can be implemented by software or hardware. For the modules implemented by hardware, each of the above modules can be located in the same processor; or each of the above modules can also be located in different processors in any combination.

[0157] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown inFigure 6 The computer device shown in the figure includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is used to store log data. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a data compression transmission method, a data decompression method or an information scheduling method.

[0158] Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0159] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the following steps:

[0160] Obtain a hot pattern list cached locally, the hot pattern list including pattern numbers and identifiers of parsers associated with each other; wherein the hot pattern list is obtained by broadcasting from a central scheduling system, and the central scheduling system generates the hot pattern list by tracking the parsers with relatively high frequencies appearing in recent node reporting records;

[0161] Determine a target parser matching the original message to be sent from the hot pattern list, and construct a data packet according to the dynamic variable vector obtained by encoding the original message by the target parser, and the target pattern number corresponding to the target parser;

[0162] Send the data packet to the second node.

[0163] In one embodiment, the computer program is executed by the processor to further implement the following steps:

[0164] For each pattern number in the hot pattern list, obtain the associated parser, and call the parser to encode the original message to be sent;

[0165] Determine whether the encoding is successful, and take the parser with successful encoding as the target parser.

[0166] In one embodiment, the computer program is executed by the processor to further implement the following steps:

[0167] calling a pre-trained language model to perform inference on the local data stream, generating a parser, the parser including an encoder and a decoder; wherein the encoder is configured to map the original message to a dynamic variable vector, and the decoder is configured to map the dynamic variable vector back to the original message;

[0168] According to the code of the parser, the identifier is calculated;

[0169] The current time is taken as the reporting time, and the reporting time and the identifier of the parser are reported to the central scheduling system as records.

[0170] In one embodiment, the computer program is executed by the processor to further implement the following steps:

[0171] Receiving the data packet sent by the first node, extracting the target mode number and the dynamic variable vector from the data packet;

[0172] Obtaining a hot mode list cached locally, and finding the identifier of the parser corresponding to the target mode number from the hot mode list; wherein the hot mode list is obtained by broadcasting from the central scheduling system, and the central scheduling system generates the hot mode list by tracking the relatively high frequency of the appearance of the parser in the recent records reported by each node;

[0173] According to the identifier, calling the parser to decode the dynamic variable vector to obtain the original message.

[0174] In one embodiment, the computer program is executed by the processor to further implement the following steps:

[0175] Obtaining records reported by each node, each record including an identifier of a parser;

[0176] Filtering the records reported recently from the records, and constructing a hot parser set according to the identifiers of the parsers with relatively high frequency of appearance;

[0177] Assigning a mode number to each identifier in the hot parser set, and generating a hot mode list according to the mode number and the associated identifier;

[0178] Broadcasting the hot mode list to each node in the distributed network system to instruct each node to compress the original message to be sent according to the hot mode list, or to instruct each node to decompress the received data packet according to the hot mode list.

[0179] In one embodiment, the computer program is executed by the processor to further implement the following steps:

[0180] According to the reporting time carried in each record, determining the historical records within a time window from the current time as the records reported recently;

[0181] The frequencies of the identifiers in the recently reported records are arranged in descending order, and the top K identifiers are selected to construct a hot resolver set.

[0182] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0183] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without limitation. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without limitation.

[0184] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, as long as the combinations of technical features do not have contradictions, they shall be considered within the scope of the present disclosure.

[0185] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it shall not be understood as a limitation on the patent scope of the present application. It shall be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these shall be within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A data compression transmission method characterized by, The method applied to the first node comprises: Obtaining a hot schema list cached locally, the hot schema list comprising schema numbers and identifiers of parsers associated with each other; wherein the hot schema list is obtained by broadcasting from a central scheduling system, and the central scheduling system generates the hot schema list by tracking parsers with relatively high frequencies in recent node reporting records; Determining a target parser matching the original message to be sent from the hot schema list, and constructing a data packet according to a dynamic variable vector obtained by encoding the original message by the target parser and a target schema number corresponding to the target parser; Sending the data packet to the second node.

2. The data compression transmission method according to claim 1, characterized in that, Determining the target parser matching the original message to be sent from the hot schema list comprises: For each schema number in the hot schema list, obtaining the associated parser, and calling the parser to encode the original message to be sent; Determining whether the encoding is successful, and taking the parser with successful encoding as the target parser.

3. The data compression transmission method of claim 1, wherein, The method further comprises: Calling a pre-trained language model to infer a local data stream to generate a parser, the parser comprising an encoder and a decoder; wherein the encoder is used to map the original message to a dynamic variable vector, and the decoder is used to map the dynamic variable vector back to the original message; Calculating the identifier according to the code of the parser; Taking the current time as a reporting time, and reporting the reporting time and the identifier of the parser to the central scheduling system as a record.

4. A method of data decompression, characterized by, The method applied to the second node comprises: Receiving the data packet sent by the first node, and extracting a target schema number and a dynamic variable vector from the data packet; Obtaining a hot schema list cached locally, and finding the identifier of the parser corresponding to the target schema number from the hot schema list; wherein the hot schema list is obtained by broadcasting from a central scheduling system, and the central scheduling system generates the hot schema list by tracking parsers with relatively high frequencies in recent node reporting records; According to the identifier, calling the parser to decode the dynamic variable vector to obtain the original message.

5. An information scheduling method characterized by comprising: The method applied to the central scheduling system comprises: Obtaining records reported by nodes, each record comprising an identifier of a parser; From the records, screening recent records, and according to the identifiers of parsers with relatively high frequencies, constructing a hot parser set; Assigning schema numbers to each identifier in the hot parser set, and generating a hot schema list according to the schema numbers and the associated identifiers; Broadcasting the hot schema list to each node in the distributed network system to instruct each node to compress the original message to be sent according to the hot schema list, or to instruct each node to decompress the received data packet according to the hot schema list.

6. The information scheduling method of claim 5, wherein, The parser is generated by the node by inferring a local data stream through a pre-trained language model, and the parser comprises an encoder and a decoder. The encoder is configured to map an original message into a dynamic variable vector, and the decoder is configured to map the dynamic variable vector back to the original message.

7. The information scheduling method of claim 5, wherein, The method comprises the following steps: According to the reporting time carried in each record, the historical records within a time window from the current time are determined, and the historical records are taken as the recently reported records; The identifiers in the recently reported records are arranged in descending order of frequency, and the top K identifiers are selected to construct the popular parser set.

8. A data synchronization method, characterized by, The method is applied to a distributed network system, which comprises a central scheduling system and multiple nodes; the method comprises the following steps: The central scheduling system obtains records reported by each node, and each record comprises an identifier of a parser; the central scheduling system screens recently reported records from the records, and constructs a popular parser set according to identifiers of parsers with relatively high frequencies; the central scheduling system allocates mode numbers to each identifier in the popular parser set, generates a popular mode list according to the mode numbers and the associated identifiers, and broadcasts the popular mode list to each node in the distributed network system; A first node obtains the popular mode list cached locally, determines a target parser matching a to-be-sent original message from the popular mode list, and constructs a data packet according to a dynamic variable vector obtained by encoding the original message by the target parser and a target mode number corresponding to the target parser; the first node sends the data packet to a second node; After receiving the data packet, the second node extracts the target mode number and the dynamic variable vector from the data packet, obtains the popular mode list cached locally, and finds an identifier of a parser corresponding to the target mode number from the popular mode list; the second node calls the parser to decode the dynamic variable vector according to the identifier, and obtains the original message.

9. A distributed network system, characterized by The distributed network system comprises a central scheduling system and multiple nodes, the central scheduling system and each node are in communication connection, and at least two nodes are in communication connection; the distributed network system is configured to execute the data synchronization method in claim 8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the steps of the method in any one of claims 1 to 8. The computer program is executed by a processor to implement the steps of the method in any one of claims 1 to 8.

Images