Method and apparatus for atlas relationship compression based on context-aware huffman coding

Abstract

This invention discloses a graph relation compression method and apparatus based on context-aware Huffman coding, belonging to the field of graph compression technology. The method includes: traversing the original data, calculating the global occurrence frequency of all relation types, and the conditional occurrence frequency of each relation type in each context instance; constructing a global master Huffman coding table based on the global occurrence frequency of all relation types; constructing a subordinate Huffman coding table based on the conditional occurrence frequency of relation types; filtering the subordinate Huffman coding table according to the difference between the benefit and cost; performing a streaming encoding process, for each relation triple to be compressed, sequentially determining whether the relation triple matches the corresponding context instance, and performing corresponding encoding based on the matching result; and packaging the encoded binary bitstream with the metadata required for decompression. This invention improves compression performance by converting global probabilities into conditional probabilities and using the structural information of the graph as prior conditions.

Description

Technical Field

[0001] This invention belongs to the field of graph compression technology, and in particular relates to a graph relation compression method and apparatus based on context-aware Huffman coding. Background Technology

[0002] Knowledge graphs store massive amounts of entity relationships in the form of triples. Compressing storage space is crucial for reducing storage costs and improving transmission efficiency. Existing general-purpose compression algorithms (such as GZIP) use general dictionary encoding, which does not consider the topological characteristics of graph data, thus limiting their compression ratio. Most existing graph-specific compression methods focus on compressing node identifiers (such as adjacency list differential encoding and node reordering techniques), while failing to fully utilize the semantic features of relation types (i.e., the predicates in triples). Traditional Huffman coding encodes only based on the global frequency of relation types in the entire dataset, treating each relation as an independent and identically distributed event, ignoring the strong contextual dependencies of relations in the local structure of the graph. For example, when the source node type is "person" and the target node type is "company," the probability of the relation type "employed at" is much higher than other relation types, resulting in the encoding efficiency failing to reach the theoretical entropy limit. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a graph relation compression method and apparatus based on context-aware Huffman coding. By converting global probability into conditional probability and using the structural information of the graph as a priori conditions, the compression performance is improved.

[0004] This invention is achieved through the following technical solution:

[0005] The first aspect of this invention discloses a graph relation compression method based on context-aware Huffman coding, comprising:

[0006] Traverse the original graph data and count the global frequency of all relation types;

[0007] Multiple types of context features are extracted from the original graph data, and corresponding context instances are formed based on the context features. The frequency of occurrence of each relation type in each context instance is counted. The context instances include node type pair context instances, relation co-occurrence context instances, and frequent subgraph pattern context instances.

[0008] Construct a global master Huffman coding table based on the global occurrence frequency of all relation types;

[0009] For each context instance, a subordinate Huffman coding table is constructed based on the frequency of occurrence of conditions of the relation type corresponding to that context instance;

[0010] For each subordinate Huffman coding table, if the compression gain obtained by encoding with the subordinate Huffman coding table is greater than the metadata overhead required by the subordinate Huffman coding table and its triggering rules, then the subordinate Huffman coding table is retained and a unique context identifier is assigned to it. At the same time, the correspondence between the context identifier and the context instance is recorded as the triggering rule. Otherwise, the subordinate Huffman coding table is discarded.

[0011] The streaming encoding process is performed. During this process, for each relation triplet to be compressed, the context features of the relation triplet are extracted, and the triplet is sequentially judged according to a preset priority order to determine whether it matches a frequent subgraph pattern context instance, a node type pair context instance, or a relation co-occurrence context instance. If a frequent subgraph pattern context instance is matched, the joint encoding method of the frequent subgraph pattern context instance's identifier plus its internal offset is used for output. If a node type pair context instance or a relation co-occurrence context instance is matched, the context identifier corresponding to the context instance is written, and the subordinate Huffman coding table pointed to by the context identifier is used to encode the current relation type. If no context instance is matched, a preset escape identifier is written, and the global master Huffman coding table is used to encode the current relation type.

[0012] The encoded binary bitstream is packaged with the metadata required for decompression to form a self-describing compressed file format.

[0013] Furthermore, multiple types of contextual features are extracted, corresponding contextual instances are formed based on these features, and the frequency of occurrence of conditions for each relation type in each contextual instance is statistically analyzed, including:

[0014] Extract the type labels of the source node and the target node in each relation triple, treat each combination of source node type and target node type as a node type pair context instance, and count the number of times each relation type appears in the node type pair context instance;

[0015] Record the k relation type sequences preceding the current relation type using a sliding window, treat each relation type sequence of length k as a relation sequence context instance, and count the frequency of conditions for each relation type after that relation sequence context instance;

[0016] Extract the set of all outgoing edge relationship types of the source node, perform a hash operation on the set of outgoing edge relationship types to generate a structural fingerprint, take each structural fingerprint as an outgoing edge structural context instance, and count the frequency of occurrence of each relationship type in the outgoing edge structural context instance;

[0017] The frequent subgraph mining algorithm is used to identify connected subgraph structures in the graph that appear more than a preset threshold. Each connected subgraph structure is treated as a frequent subgraph pattern context instance, and a unique pattern identifier is assigned to each frequent subgraph pattern context instance. At the same time, the distribution of relation types at each position in the frequent subgraph pattern context instance is statistically analyzed.

[0018] Furthermore, the global master Huffman coding table assigns a unique binary codeword to each relation type, the length of which is inversely proportional to the global occurrence frequency of the corresponding relation type.

[0019] Furthermore, the formula for calculating the difference between the compression gain and the metadata overhead is as follows:

[0020] In the formula, This represents the number of times relation type p appears in this context instance. Let p be the code length of the relation type p in the global master Huffman coding table. The code length of relation type p in the subordinate Huffman-coded table. The metadata space required to store the subordinate Huffman-coded table and its triggering rules.

[0021] Furthermore, according to a preset priority order, it is determined whether the relation triple matches a frequent subgraph pattern context instance, a node type pair context instance, or a relation co-occurrence context instance, including:

[0022] Determine whether the current relation triple is part of the context instance of the constructed frequent subgraph pattern. If so, stop the subsequent priority determination.

[0023] If the current relation triple is not part of the already constructed frequent subgraph pattern context instance, then determine whether the combination of source node type and target node type in the current relation triple matches the already constructed node type pair context instance. If so, stop the subsequent priority judgment.

[0024] If the combination of source node type and target node type in the current relation triple does not match any constructed node type pair context instance, then determine whether the previous k relation type sequences of the current relation triple and the set of outgoing relation types of the source node match any constructed relation sequence context instance or outgoing edge structure context instance.

[0025] Furthermore, if a frequent subgraph pattern context instance is matched, the unique identifier of the frequent subgraph pattern context instance and the position number of the current relation type in the frequent subgraph pattern context instance are output.

[0026] If a node type is matched to a context instance or a relation co-occurrence context instance, the codeword pointing to the context identifier of the subordinate Huffman coding table is searched in the global master Huffman coding table and written to the output stream. Then, the local codeword corresponding to the current relation type is searched in the matched subordinate Huffman coding table and written to the output stream.

[0027] If no context instance is matched, the codeword of the escape identifier is searched in the global master Huffman coding table and written to the output stream. Then, the global codeword corresponding to the current relation type is searched in the global master Huffman coding table and written to the output stream.

[0028] Furthermore, the metadata required for decompression includes:

[0029] The symbol mapping dictionary is used to store the bijective mapping between relational type strings and internal integer identifiers. It adopts the format of the total number of records plus a variable-length string sequence.

[0030] The context rule table stores the correspondence between context identifiers and context instances. Each record includes a context identifier, rule type, and rule parameter tuple.

[0031] The Huffman forest topology information includes a global master Huffman coding table and arrays of normalized Huffman coding code lengths corresponding to each subordinate Huffman coding table.

[0032] Furthermore, the compressed data payload area in the compressed file format is composed of consecutive variable-length decoding units, and each decoding unit adopts a context switching instruction format or a default escape instruction format.

[0033] The context switching instruction format includes a context identifier codeword and a local relation type codeword, wherein the context identifier codeword is parsed by the global master Huffman coding table, and the local relation type codeword is parsed by the subordinate Huffman coding table corresponding to the context identifier.

[0034] The default escape instruction format includes an escape identifier codeword and a global relation type codeword, both of which are parsed by the global master Huffman coding table.

[0035] Furthermore, the graph relation compression method also includes a decompression step:

[0036] Parse the header of the compressed file, reconstruct the mapping table between relation type strings and internal integer identifiers from the symbol mapping dictionary, load the trigger rules from the context rule table and initialize the context state machine, obtain the code length array from the Huffman forest topology information and reconstruct the Huffman tree corresponding to the global master Huffman coding table and the Huffman trees corresponding to all subordinate Huffman coding tables.

[0037] Then, a cyclic decompression process is executed, which includes: calculating the current expected context state based on the decompressed historical data; reading the next decoding unit from the binary bit stream, decoding using the global master Huffman tree, locking the subordinate Huffman tree corresponding to the context identifier if the decoding result is a context identifier, and locking the global master Huffman tree if the decoding result is an escape identifier; using the locked Huffman tree to continue reading from the binary bit stream and decoding the internal integer identifier of the relation type, converting it into a relation type string through the mapping table; combining the restored relation type string with the decompressed subject and object to form a complete triple and storing it in the output buffer, and updating the historical state of the context state machine;

[0038] Repeat the cyclic decompression process until the binary bit stream is processed, and output a list of triples that are completely consistent with the original spectrum data.

[0039] A second aspect of the present invention discloses a graph relation compression apparatus based on context-aware Huffman coding, comprising:

[0040] The traversal statistics module is used to traverse the original graph data and count the global frequency of all relation types.

[0041] The context extraction module is used to extract multiple types of context features from the original graph data, form corresponding context instances based on the context features, and count the frequency of occurrence of each relation type in each context instance. The context instances include node type pair context instances, relation co-occurrence context instances, and frequent subgraph pattern context instances.

[0042] The main table construction module is used to construct a global main Huffman-coded table based on the global occurrence frequency of all relation types;

[0043] The subordinate table construction module is used to construct a subordinate Huffman coding table for each context instance based on the frequency of occurrence of conditions of the relation type corresponding to that context instance.

[0044] The filtering module is used to retain each subordinate Huffman coding table if the compression gain brought by encoding with the subordinate Huffman coding table is greater than the metadata overhead required by the subordinate Huffman coding table and its triggering rules, and assign a unique context identifier to it. At the same time, the correspondence between the context identifier and the context instance is recorded as the triggering rule. Otherwise, the subordinate Huffman coding table is discarded.

[0045] The streaming encoding module is used to perform the streaming encoding process. In this process, for each relation triplet to be compressed, the context features of the relation triplet are extracted, and the module is judged in a preset priority order to determine whether the relation triplet matches a frequent subgraph pattern context instance, a node type pair context instance, or a relation co-occurrence context instance. If a frequent subgraph pattern context instance is matched, the module outputs the relation using a joint encoding method of the identifier of the frequent subgraph pattern context instance plus the internal offset. If a node type pair context instance or a relation co-occurrence context instance is matched, the module writes the context identifier corresponding to the context instance and uses the subordinate Huffman coding table pointed to by the context identifier to encode the current relation type. If no context instance is matched, the module writes a preset escape identifier and uses the global master Huffman coding table to encode the current relation type.

[0046] The packaging module is used to package the encoded binary bitstream with the metadata required for decompression, forming a self-describing compressed file format.

[0047] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0048] (1) This invention transforms global probability into conditional probability and uses the structural information of the graph as a priori condition to make the average code length of the code shorter. According to information theory, the conditional entropy H(R|C) is less than or equal to the global entropy H(R). Therefore, context-aware Huffman coding can achieve better compression performance than traditional Huffman coding.

[0049] (2) This invention can deeply mine the multi-level semantic rules of the graph, including the context of node type, the context of co-occurrence of relations and the context of frequent subgraph patterns, and realize high-precision context modeling of relation types; by incorporating node type constraints, relation sequence dependencies and subgraph structure features into the encoding decision, it can capture graph structure information that traditional compression algorithms cannot utilize, thereby removing redundancy to the maximum extent while maintaining lossless compression.

[0050] (3) The present invention filters the subordinate Huffman coding table through an evaluation mechanism based on benefits and costs, and retains only the context instances that can bring net compression gains. This can effectively control metadata expansion while improving compression ratio, thereby balancing compression efficiency and storage overhead.

[0051] (4) This invention focuses on the compression of the relation layer, which is independent of existing node identifier compression technologies (such as node reordering, adjacency list differential encoding, etc.). The two can be used together to further improve the overall compression ratio.

[0052] (5) The present invention adopts a streaming coding architecture, the encoder maintains a sliding window and context state, and can output compressed bit streams in real time without full data caching. It can handle ultra-large-scale knowledge graphs that exceed memory capacity and can be applied to online compression scenarios of large-scale graph data.

[0053] (6) The present invention uses a self-describing compressed file format to package the symbol mapping dictionary, context rule table and Huffman forest topology information together with the compressed bit stream, ensuring that the decompression end can completely reconstruct the encoding environment; the decompression process uses a context state machine synchronization mechanism to accurately restore each relation triplet, achieving byte-level lossless restoration. Attached Figure Description

[0054] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0055] Figure 1 This is a flowchart of a graph relationship compression method in this invention;

[0056] Figure 2 This is a block diagram of one component of the graph relationship compression device in this invention. Detailed Implementation

[0057] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0058] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0059] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "inner," and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the accompanying drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0060] In the description of this invention, unless otherwise explicitly specified and limited, the term "connection" or similar designation indicating a connection between components should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0061] like Figures 1 to 2 As shown in the figure, this embodiment discloses a graph relation compression method and apparatus based on context-aware Huffman coding.

[0062] The first aspect of this embodiment discloses a graph relation compression method based on context-aware Huffman coding, such as... Figure 1 As shown, the map relationship compression method includes steps S100 to S700.

[0063] Step S100. Traverse the original graph data and count the global frequency of all relation types.

[0064] The original graph data is a set of triples, each triple including a subject, a predicate, and an object, where the predicate is also called the relation type.

[0065] Relationship type refers to the predicate category that defines the association between entities in a knowledge graph. It is the predicate part of a triple and is used to describe the semantic connection between two entities.

[0066] The global frequency of all relation types refers to the frequency value obtained by counting the number of times each relation type appears in the entire dataset across all triples of the original graph data. This frequency value is used to characterize the prior distribution of each relation type across the entire graph.

[0067] For example, in a knowledge graph containing entities such as people, companies, and locations, the relationship type "employed at" may appear 300 times, the relationship type "located at" may appear 200 times, and the relationship type "born at" may appear 100 times. These statistics form the basis of the global frequency of occurrence.

[0068] Step S200. Extract various types of context features from the original graph data, form corresponding context instances based on the context features, and count the frequency of occurrence of each relation type in each context instance. The context instances include node type pair context instances, relation co-occurrence context instances, and frequent subgraph pattern context instances.

[0069] Conditional frequency of occurrence (CFO) refers to the number or proportion of occurrences of each relation type within a subset of data defined by a specific context instance. This frequency reflects the local distribution characteristics of relation types under that contextual condition. According to information theory, the conditional entropy H(R|C) of encoding based on CFO is less than or equal to the global entropy H(R), thus resulting in a shorter average code length.

[0070] In some implementations of this embodiment, multiple types of context features are extracted, corresponding context instances are formed based on the context features, and the frequency of occurrence of each relation type of condition in each context instance is counted, including steps S210 to S240.

[0071] Step S210. Extract the type labels of the source node and the target node in each relation triplet, take each combination of source node type and target node type as a node type pair context instance, and count the number of times each relation type appears in the node type pair context instance.

[0072] In a knowledge graph, the subject and object of a triple correspond to nodes in a directed graph, and the predicate corresponds to the directed edge connecting the nodes. For ease of description, in this embodiment, the subject is referred to as the source node, and the object as the target node.

[0073] The type of a node greatly limits the range of possible relation types. For example, when the source node type is "building" and the target node type is "geographical location," the probability of the relation type "located at" is close to 99%, while the probability of the relation type "born at" is 0. This embodiment constructs a mapping table. Record the frequency of occurrence of each relation type under each node type combination. This represents a mapping table, which is a data structure (such as a dictionary or hash table) used to store the mapping relationship between node type combinations and relation frequencies; Indicates the type of the source node, such as "building" in the image and text; Indicates the type of the target node, such as "geographical location" in the text and image; Indicates the relation type, referring to a specific... The actual specific graph edges / relationships that exist between node pairs (such as "located in", "born in", etc.); Indicates frequency of occurrence, referring to the corresponding relation type. In the context of the current specific node type pair, the specific number of times it appears in the entire graph data.

[0074] For example, when the source node type is "person" and the target node type is "company", in the context instance of this node type pair, the relation type "employed in" appears 1000 times, the relation type "founded" appears 100 times, and the relation type "invested" appears 50 times. These statistics constitute the conditional frequency distribution of this context instance.

[0075] Step S220. Record the k relation type sequences before the current relation type using a sliding window, treat each relation type sequence of length k as a relation sequence context instance, and count the frequency of condition occurrence for each relation type after the relation sequence context instance.

[0076] For example, when k is 2, the historical relation type sequence is ["Serving at", "Located at"] as a relation sequence context instance. After this context instance, the relation type "Own" appears 200 times and the relation type "Established at" appears 50 times.

[0077] Step S230. Extract the set of all outgoing edge relationship types of the source node, perform a hash operation on the set of outgoing edge relationship types to generate a structural fingerprint, take each structural fingerprint as an outgoing edge structural context instance, and count the frequency of occurrence of each relationship type in the outgoing edge structural context instance.

[0078] For example, for all source node groups that simultaneously possess both "employed at" and "graduated from", the structural fingerprint of their outbound relation type set serves as an outbound structure context instance, in which the relation type "received an award" has a higher probability of occurrence.

[0079] Step S240. Use the frequent subgraph mining algorithm to identify connected subgraph structures in the graph that appear more than a preset threshold, treat each connected subgraph structure as a frequent subgraph pattern context instance, assign a unique pattern identifier to each frequent subgraph pattern context instance, and statistically analyze the distribution of relation types at each position in the frequent subgraph pattern context instance.

[0080] For example, the frequent subgraph pattern "person-employed-company-located-city" appears 300 times in the entire graph. This pattern is an instance of a frequent subgraph pattern context. The relation type of the first position in the pattern is fixed as "employed", and the relation type of the second position is fixed as "located". These statistics form the basis of the pattern context.

[0081] Step S300. Construct a global master Huffman coding table based on the global occurrence frequency of all relation types.

[0082] The global master Huffman coding table is used to process relations in contexts without specific context or in low-frequency contexts.

[0083] In this step, based on the global occurrence frequency of all relation types as counted in step S100, a global master Huffman coding table is constructed using the standard Huffman coding algorithm. This global master Huffman coding table assigns a unique binary codeword to each relation type. The length of this binary codeword is inversely proportional to the global occurrence frequency of the corresponding relation type; that is, the higher the occurrence frequency of a relation type, the shorter the assigned binary codeword.

[0084] For example, based on the aforementioned frequency data, the relation type "Serving" has the highest frequency and may be assigned a 2-bit codeword "01"; the relation type "Located" has the next highest frequency and may be assigned a 3-bit codeword "001"; while lower frequency relation types such as "Investment" may be assigned a longer codeword such as "00011". In this embodiment, the specific codeword allocation depends on the specific implementation of the Huffman algorithm and the global frequency distribution.

[0085] Step S400. For each context instance, construct a subordinate Huffman coding table based on the frequency of occurrence of conditions of the relation type corresponding to the context instance.

[0086] In this step, for each context instance formed in step S200, a subordinate Huffman coding table is constructed using the Huffman coding algorithm based on the frequency of occurrence of the relation types statistically obtained in that context instance. Each subordinate Huffman coding table contains only the relation types that appear in that context instance. Since the local probability distribution is usually more concentrated than the global distribution, shorter local codewords can be generated.

[0087] For example, in a node type pair context instance ("person", "company"), the relationship type may only be a few such as "employed", "founded", "invested", etc., and "employed" is absolutely dominant, so it may be assigned only 1 bit codeword "0".

[0088] Step S500. For each subordinate Huffman coding table, if the compression gain obtained by encoding with the subordinate Huffman coding table is greater than the metadata overhead required by the subordinate Huffman coding table and its triggering rules, then the subordinate Huffman coding table is retained, and a unique context identifier is assigned to it. At the same time, the correspondence between the context identifier and the context instance is recorded as the triggering rule; otherwise, the subordinate Huffman coding table is discarded.

[0089] In some embodiments of this example, the formula for calculating the difference between the compression gain and the metadata overhead is as follows:

[0090] In the formula, This represents the number of times relation type p appears in this context instance. Let p be the code length of the relation type p in the global master Huffman coding table. The code length of relation type p in the subordinate Huffman-coded table. The metadata space required to store the subordinate Huffman-coded table and its triggering rules.

[0091] This formula is used to calculate the difference between the compression gain (i.e., the total shortened code length) brought about by encoding with this subordinate Huffman coding table and the overhead required to store the table. If ΔL > 0, the subordinate Huffman coding table is retained and a unique context identifier (Context_ID) is assigned, while the correspondence between the context identifier and the context instance is recorded as a trigger rule; otherwise, the subordinate Huffman coding table is discarded. The context identifier is used to refer to a specific local context model and the unique index information of the corresponding subordinate Huffman coding table. The context identifier occupies a specific code position in the global master Huffman coding table, or exists as an independent prefix code. Its function is to indicate to the decoder to switch from the default global decoding state to a specific local context decoding state.

[0092] For example, in the Huffman coding table of a node type for a context instance ("person", "company"), if the calculated ΔL is much greater than zero, it is retained; while for some context instances that occur very infrequently, their compression gain may not be able to cover the storage overhead, so they are discarded. Through this filtering mechanism, metadata bloat can be effectively controlled while improving the compression ratio.

[0093] Step S600. Perform the streaming encoding process. In this process, for each relation triplet to be compressed, extract the context features of the relation triplet, and determine whether the relation triplet matches a frequent subgraph pattern context instance, a node type pair context instance, or a relation co-occurrence context instance in a preset priority order: If a frequent subgraph pattern context instance is matched, output the result using a joint encoding method of the identifier of the frequent subgraph pattern context instance plus the internal offset; if a node type pair context instance or a relation co-occurrence context instance is matched, write the context identifier corresponding to the context instance, and encode the current relation type using the subordinate Huffman coding table pointed to by the context identifier; if no context instance is matched, write the preset escape identifier, and encode the current relation type using the global master Huffman coding table.

[0094] In this embodiment, during streaming encoding, an encoder (such as a streaming context-aware encoder) is responsible for performing the real-time compression task. This encoder includes a context feature extraction unit, a cascaded matching logic unit, and a hybrid bitstream generation unit. The specific working process of the encoder is as follows: The encoder maintains a sliding window buffer and a metadata query interface for real-time extraction of context features. For each relation triple to be compressed, the encoder queries the type labels of the source node and the target node in real time through the node identifier index to generate the current semantic feature vector; it reads the k most recently encoded relation type sequences in the sliding window as temporal context features; and it monitors the state of the frequent subgraph pattern matching cursor. If the cursor is in an active state (i.e., the current triple belongs to a frequently processed frequent subgraph pattern context instance), the current context state is marked as Pattern_Mode.

[0095] In some implementations of this embodiment, the relationship triple is judged sequentially according to a preset priority order to determine whether it matches a frequent subgraph pattern context instance, a node type pair context instance, or a relationship co-occurrence context instance. This includes: determining whether the current relationship triple is a component of an already constructed frequent subgraph pattern context instance; if so, the subsequent priority judgment is stopped; if the current relationship triple is not a component of an already constructed frequent subgraph pattern context instance, the combination of the source node type and the target node type in the current relationship triple is judged to match an already constructed node type pair context instance; if so, the subsequent priority judgment is stopped; if the combination of the source node type and the target node type in the current relationship triple does not match any already constructed node type pair context instance, the k relationship type sequences preceding the current relationship triple and the set of outgoing relationship types of the source node are judged to match an already constructed relationship sequence context instance or outgoing edge structure context instance.

[0096] In this embodiment, a "short-circuit evaluation" strategy is adopted. According to the preset priority order, the matching item is searched in the context hash map table in memory. Once a match is found at a high priority level, the subsequent search is stopped immediately.

[0097] In some implementations of this embodiment, the output uses a joint encoding method of the identifier of the frequent subgraph pattern context instance plus the internal offset, including: when the current relation triple is a component of the identified frequent subgraph pattern context instance, outputting the unique identifier of the frequent subgraph pattern context instance and the position number of the current relation type in the frequent subgraph pattern context instance.

[0098] For example, for an instance of the frequent subgraph pattern "person-employed-company-located-city", if the second relation type "located" in the pattern is currently being processed, the encoder outputs "pattern identifier=0x03" plus "location number=2". In this joint encoding method, the entire pattern only needs one pattern identifier and a few offset bits to represent multiple relation types, thus improving the compression ratio.

[0099] In some implementations of this embodiment, writing the context identifier corresponding to the context instance and using the subordinate Huffman coding table pointed to by the context identifier to encode the current relation type includes: searching for the codeword pointing to the context identifier of the subordinate Huffman coding table in the global master Huffman coding table and writing the codeword to the output stream; searching for the local codeword corresponding to the current relation type in the matched subordinate Huffman coding table and writing the local codeword to the output stream.

[0100] For example, for a subordinate table corresponding to a node type context instance ("person", "company"), its context identifier might be encoded as "10" in the main table. After writing "10", the local codeword "0" for the current relation type "employed in" in the subordinate table is then written. Through this "switch-lookup" mechanism, context-aware short code encoding is achieved.

[0101] In some implementations of this embodiment, writing a preset escape identifier and encoding the current relation type using the global master Huffman coding table includes: searching for the codeword of the escape identifier in the global master Huffman coding table and writing the codeword into the output stream; searching for the global codeword corresponding to the current relation type in the global master Huffman coding table and writing the global codeword into the output stream.

[0102] For example, the escape identifier might be encoded as "111" in the main table. After writing "111", the global codeword "00011" for the current relation type "Investment" is then written. This fallback scheme ensures that all relation types can be encoded, guaranteeing the integrity of lossless compression.

[0103] In this embodiment, after encoding the current relation triple, the generated binary encoded fragment is appended to the binary bitstream buffer, the current relation type is pushed into the sliding window, the oldest relation type is removed, and the relevant states (including the set of outbound relation types of the source node or the outbound degree counter) are updated to prepare for processing the next triple.

[0104] S700. Packages the encoded binary bitstream with the metadata required for decompression to form a self-describing compressed file format.

[0105] In this embodiment, after all triples have been processed, the final generated binary bitstream and the metadata required for decompression are packaged into a self-describing compressed file format. The metadata header is located at the beginning of the file and is stored using uncompressed or lightweight compression (such as Varint encoding) to reconstruct the encoding environment at the decompression end.

[0106] In some implementations of this embodiment, the metadata required for decompression includes: a symbol mapping dictionary, used to store the bijective mapping relationship between relation type strings and internal integer identifiers, which adopts the format of total number of records plus variable-length string sequences; a context rule table, used to store the correspondence between context identifiers and context instances, each record including a context identifier, rule type, and rule parameter tuple; and Huffman forest topology information, which includes a global master Huffman coding table and a normalized Huffman coding code length array corresponding to each subordinate Huffman coding table.

[0107] The symbol mapping dictionary is stored in a compact format of [Total_Count] + [Length_1][String_1] + [Length_2][String_2]... After the decompressor reads this segment, it can reconstruct the ID_to_String array in memory, ensuring that the decoded numeric ID is losslessly restored to the original relational type string.

[0108] The context rule table uses the entry format [Context_ID] : [Rule_Type] : [Parameter_Tuple], which is used to configure the context state machine on the decompression end, so that the decompressor knows which decoding table corresponding to which Context_ID should be switched to when what data characteristics are detected.

[0109] For example: ID=0x05 : Type=NodePair : (Source=Person_ID, Target=City_ID)

[0110] ID=0x0A : Type=Sequence : (Prev=Has_Friend_ID)

[0111] The Huffman forest topology information includes a global master Huffman coding table and arrays of code lengths for each subordinate Huffman coding table corresponding to the normalized Huffman codes. To save space, the tree structure is not stored directly; instead, arrays of code lengths for the normalized Huffman codes are stored. The decompressor uses a standard algorithm based on the code length arrays to deterministically reconstruct a set of Huffman binary trees that is completely identical to the compressed end.

[0112] The storage format is: [Num_Tables]

[0113] [Table_ID_0 (Master)] + [Symbol_1: BitLen] + [Symbol_2: BitLen]...

[0114] [Table_ID_1 (Sub)] + [Symbol_x: BitLen]...

[0115] The compressed data payload immediately follows the metadata header and consists of consecutive variable-length decoding units. To address the delimiting problem of variable-length encoding, the bitstream is logically divided into atomic decoding units.

[0116] In some embodiments of this example, the compressed data payload area in the compressed file format is composed of consecutive variable-length decoding units. Each decoding unit adopts a context switching instruction format or a default escape instruction format. The context switching instruction format includes a context identifier codeword and a local relation type codeword, wherein the context identifier codeword is parsed by the global master Huffman coding table, and the local relation type codeword is parsed by the subordinate Huffman coding table corresponding to the context identifier. The default escape instruction format includes an escape identifier codeword and a global relation type codeword, wherein both the escape identifier codeword and the global relation type codeword are parsed by the global master Huffman coding table.

[0117] In this embodiment, the design of the decoding unit enables the decompression end to seamlessly switch decoding tables and accurately define the encoding boundaries of each relation type.

[0118] In some embodiments of this example, the graph relation compression method further includes a decompression step: parsing the compressed file header, reconstructing the mapping table between relation type strings and internal integer identifiers from the symbol mapping dictionary, loading trigger rules from the context rule table and initializing the context state machine, obtaining the code length array from the Huffman forest topology information and reconstructing the Huffman tree corresponding to the global master Huffman coding table and the Huffman trees corresponding to all subordinate Huffman coding tables. Then, a cyclic decompression process is executed, which includes: calculating the current desired context state based on the decompressed historical data; reading the next decoding unit from the binary bit stream, decoding using the global master Huffman tree, locking the subordinate Huffman tree corresponding to the context identifier if the decoding result is a context identifier, and locking the global master Huffman tree if the decoding result is an escape identifier; using the locked Huffman tree to continue reading from the binary bit stream and decoding the internal integer identifier of the relation type, converting it into a relation type string through the mapping table; combining the restored relation type string with the decompressed subject and object into a complete triple and storing it in the output buffer, and updating the historical state of the context state machine. Repeat the cyclic decompression process until the binary bit stream is processed, and output a list of triples that are completely consistent with the original spectrum data.

[0119] For example, during decompression, the file header is read first to reconstruct the mapping table, rule table, and Huffman forest; then the first decoding unit is read from the bit stream, and 0x05 is obtained by decoding with the global master tree. According to the node type context corresponding to this identifier in the rule table, the corresponding subordinate tree is switched; the bit stream is read again, and 0x00 is decoded with the subordinate tree. This value corresponds to "serving in" in the node type context, thus restoring the relation type string "serving in".

[0120] The second aspect of this embodiment discloses a graph relation compression device based on context-aware Huffman coding, such as... Figure 2 As shown, the graph relationship compression device includes a traversal statistics module, a context extraction module, a master table construction module, a subordinate table construction module, a filtering module, a streaming encoding module, and a packaging module.

[0121] The traversal statistics module is used to traverse the original graph data and count the global frequency of all relation types.

[0122] The context extraction module is used to extract multiple types of context features from the original graph data, form corresponding context instances based on the context features, and count the frequency of occurrence of each relation type in each context instance. The context instances include node type pair context instances, relation co-occurrence context instances, and frequent subgraph pattern context instances.

[0123] The main table construction module is used to construct a global main Huffman coding table based on the global occurrence frequency of all relation types.

[0124] The subordinate table construction module is used to construct a subordinate Huffman coding table for each context instance based on the frequency of occurrence of conditions of the relation type corresponding to that context instance.

[0125] The filtering module is used to retain each subordinate Huffman coding table if the compression gain obtained by encoding with the subordinate Huffman coding table is greater than the metadata overhead required by the subordinate Huffman coding table and its triggering rules, and assign a unique context identifier to it. At the same time, the correspondence between the context identifier and the context instance is recorded as the triggering rule. Otherwise, the subordinate Huffman coding table is discarded.

[0126] The streaming encoding module is used to perform the streaming encoding process. In this process, for each relation triple to be compressed, the context features of the relation triple are extracted, and the relation triple is judged in a preset priority order to determine whether it matches a frequent subgraph pattern context instance, a node type pair context instance, or a relation co-occurrence context instance. If a frequent subgraph pattern context instance is matched, the joint encoding method of the identifier of the frequent subgraph pattern context instance plus the internal offset is used for output. If a node type pair context instance or a relation co-occurrence context instance is matched, the context identifier corresponding to the context instance is written, and the subordinate Huffman coding table pointed to by the context identifier is used to encode the current relation type. If no context instance is matched, a preset escape identifier is written, and the global master Huffman coding table is used to encode the current relation type.

[0127] The packaging module is used to package the encoded binary bitstream with the metadata required for decompression to form a self-describing compressed file format.

[0128] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system or device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0129] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for atlas relationship compression based on context-aware Huffman coding, characterized in that, include: Traverse the original graph data and count the global frequency of all relation types; Multiple types of context features are extracted from the original graph data, and corresponding context instances are formed based on the context features. The frequency of occurrence of each relation type in each context instance is counted. The context instances include node type pair context instances, relation co-occurrence context instances, and frequent subgraph pattern context instances. Construct a global master Huffman coding table based on the global occurrence frequency of all relation types; For each context instance, a subordinate Huffman coding table is constructed based on the frequency of occurrence of conditions of the relation type corresponding to that context instance; For each subordinate Huffman coding table, if the compression gain obtained by encoding with the subordinate Huffman coding table is greater than the metadata overhead required by the subordinate Huffman coding table and its triggering rules, then the subordinate Huffman coding table is retained and a unique context identifier is assigned to it. At the same time, the correspondence between the context identifier and the context instance is recorded as the triggering rule. Otherwise, the subordinate Huffman coding table is discarded. The streaming encoding process is performed. During this process, for each relation triplet to be compressed, the context features of the relation triplet are extracted, and the triplet is sequentially judged according to a preset priority order to determine whether it matches a frequent subgraph pattern context instance, a node type pair context instance, or a relation co-occurrence context instance. If a frequent subgraph pattern context instance is matched, the joint encoding method of the frequent subgraph pattern context instance's identifier plus its internal offset is used for output. If a node type pair context instance or a relation co-occurrence context instance is matched, the context identifier corresponding to the context instance is written, and the subordinate Huffman coding table pointed to by the context identifier is used to encode the current relation type. If no context instance is matched, a preset escape identifier is written, and the global master Huffman coding table is used to encode the current relation type. The encoded binary bitstream is packaged with the metadata required for decompression to form a self-describing compressed file format. 2.The context-aware Hauffman coding based graph relation compression method according to claim 1, wherein, Extracting multiple types of context features, forming corresponding context instances based on these features, and statistically analyzing the frequency of occurrence of each relation type of condition in each context instance, including: Extract the type labels of the source node and the target node in each relation triple, treat each combination of source node type and target node type as a node type pair context instance, and count the number of times each relation type appears in the node type pair context instance; Record the k relation type sequences preceding the current relation type using a sliding window, treat each relation type sequence of length k as a relation sequence context instance, and count the frequency of conditions for each relation type after that relation sequence context instance; Extract the set of all outgoing edge relationship types of the source node, perform a hash operation on the set of outgoing edge relationship types to generate a structural fingerprint, take each structural fingerprint as an outgoing edge structural context instance, and count the frequency of occurrence of each relationship type in the outgoing edge structural context instance; The frequent subgraph mining algorithm is used to identify connected subgraph structures in the graph that appear more than a preset threshold. Each connected subgraph structure is treated as a frequent subgraph pattern context instance, and a unique pattern identifier is assigned to each frequent subgraph pattern context instance. At the same time, the distribution of relation types at each position in the frequent subgraph pattern context instance is statistically analyzed. 3.The context-aware Hauffman coding based graph relation compression method according to claim 1, wherein, The global master Huffman coding table assigns a unique binary codeword to each relation type, and the length of the binary codeword is inversely proportional to the global occurrence frequency of the corresponding relation type. 4.The context-aware Hauffman coding based graph relation compression method according to claim 1, wherein, The formula for calculating the difference between the compression gain and the metadata overhead is as follows: ； wherein is the number of occurrences of the relation type p in this context instance, is the code length of the relation type p in the global primary Huffman coding table, is the code length of the relation type p in the dependent Huffman coding table, is the metadata space required to store the dependent Huffman coding table and its trigger rule. 5.The context-aware Hauffman coding based graph relation compression method according to claim 2, wherein, The relation triple is evaluated sequentially according to a preset priority order to determine whether it matches a frequent subgraph pattern context instance, a node type pair context instance, or a relation co-occurrence context instance, including: Determine whether the current relation triple is part of the context instance of the constructed frequent subgraph pattern. If so, stop the subsequent priority determination. If the current relation triple is not part of the already constructed frequent subgraph pattern context instance, then determine whether the combination of source node type and target node type in the current relation triple matches the already constructed node type pair context instance. If so, stop the subsequent priority judgment. If the combination of source node type and target node type in the current relation triple does not match any constructed node type pair context instance, then determine whether the previous k relation type sequences of the current relation triple and the set of outgoing relation types of the source node match any constructed relation sequence context instance or outgoing edge structure context instance.

6. The graph relation compression method based on context-aware Huffman coding according to claim 5, characterized in that, If a frequent subgraph pattern context instance is matched, output the unique identifier of the frequent subgraph pattern context instance and the position number of the current relation type in the frequent subgraph pattern context instance; If a node type is matched to a context instance or a relation co-occurrence context instance, the codeword pointing to the context identifier of the subordinate Huffman coding table is searched in the global master Huffman coding table and written to the output stream. Then, the local codeword corresponding to the current relation type is searched in the matched subordinate Huffman coding table and written to the output stream. If no context instance is matched, the codeword of the escape identifier is searched in the global master Huffman coding table and written to the output stream. Then, the global codeword corresponding to the current relation type is searched in the global master Huffman coding table and written to the output stream. 7.The context-aware Hauffman coding based graph relation compression method according to claim 1, wherein, The metadata required for decompression includes: The symbol mapping dictionary is used to store the bijective mapping between relational type strings and internal integer identifiers. It adopts the format of the total number of records plus a variable-length string sequence. The context rule table stores the correspondence between context identifiers and context instances. Each record includes a context identifier, rule type, and rule parameter tuple. The Huffman forest topology information includes a global master Huffman coding table and arrays of normalized Huffman coding code lengths corresponding to each subordinate Huffman coding table. 8.The context-aware Hauffman coding based graph relation compression method of claim 1, wherein, The compressed data payload area in the compressed file format consists of continuous variable-length decoding units, each of which adopts a context switching instruction format or a default escape instruction format. The context switching instruction format includes a context identifier codeword and a local relation type codeword, wherein the context identifier codeword is parsed by the global master Huffman coding table, and the local relation type codeword is parsed by the subordinate Huffman coding table corresponding to the context identifier. The default escape instruction format includes an escape identifier codeword and a global relation type codeword, both of which are parsed by the global master Huffman coding table. 9.The context-aware Hauffman coding based graph relation compression method of claim 1, wherein, The graph relation compression method further includes a decompression step: Parse the header of the compressed file, reconstruct the mapping table between relation type strings and internal integer identifiers from the symbol mapping dictionary, load the trigger rules from the context rule table and initialize the context state machine, obtain the code length array from the Huffman forest topology information and reconstruct the Huffman tree corresponding to the global master Huffman coding table and the Huffman trees corresponding to all subordinate Huffman coding tables. Then, a cyclic decompression process is executed, which includes: calculating the current expected context state based on the decompressed historical data; reading the next decoding unit from the binary bit stream, decoding using the global master Huffman tree, locking the subordinate Huffman tree corresponding to the context identifier if the decoding result is a context identifier, and locking the global master Huffman tree if the decoding result is an escape identifier; using the locked Huffman tree to continue reading from the binary bit stream and decoding the internal integer identifier of the relation type, converting it into a relation type string through the mapping table; combining the restored relation type string with the decompressed subject and object to form a complete triple and storing it in the output buffer, and updating the historical state of the context state machine; Repeat the cyclic decompression process until the binary bit stream is processed, and output a list of triples that are completely consistent with the original spectrum data.

10. An atlas relationship compression apparatus based on context-aware Huffman coding, characterized by, include: The traversal statistics module is used to traverse the original graph data and count the global frequency of all relation types. The context extraction module is used to extract multiple types of context features from the original graph data, form corresponding context instances based on the context features, and count the frequency of occurrence of each relation type in each context instance. The context instances include node type pair context instances, relation co-occurrence context instances, and frequent subgraph pattern context instances. The main table construction module is used to construct a global main Huffman-coded table based on the global occurrence frequency of all relation types; The subordinate table construction module is used to construct a subordinate Huffman coding table for each context instance based on the frequency of occurrence of conditions of the relation type corresponding to that context instance. The filtering module is used to retain each subordinate Huffman coding table if the compression gain brought by encoding with the subordinate Huffman coding table is greater than the metadata overhead required by the subordinate Huffman coding table and its triggering rules, and assign a unique context identifier to it. At the same time, the correspondence between the context identifier and the context instance is recorded as the triggering rule. Otherwise, the subordinate Huffman coding table is discarded. The streaming encoding module is used to perform the streaming encoding process. In this process, for each relation triple to be compressed, the context features of the relation triple are extracted, and the module is judged in a preset priority order to determine whether the relation triple matches a frequent subgraph pattern context instance, a node type pair context instance, or a relation co-occurrence context instance. If a frequent subgraph pattern context instance is matched, the identifier of the frequent subgraph pattern context instance plus the internal offset is used for joint encoding. If a node type pair context instance or a relation co-occurrence context instance is matched, the context identifier corresponding to the context instance is written, and the subordinate Huffman coding table pointed to by the context identifier is used to encode the current relation type. If no context instance is matched, a preset escape identifier is written, and the current relation type is encoded using the global master Huffman coding table; The packaging module is used to package the encoded binary bitstream with the metadata required for decompression, forming a self-describing compressed file format.

Images