Method for compressing and processing large-scale graph data in resource-constrained environment

Abstract

The invention belongs to the technical field of graph data processing, and particularly relates to a method for compressing and processing large-scale graph data in a resource-constrained environment. The invention discloses a method for compressing and processing large-scale graph data in a resource-constrained environment. The method comprises the following steps: acquiring the large-scale graph data; dividing the large graph data to obtain a plurality of initial sub-graphs; performing a parallel compression algorithm on the plurality of initial sub-graphs to obtain a plurality of compressed target sub-graphs; and recovering the plurality of compressed target sub-images. According to the method, through the memory-perceived self-adaptive graph slicing module, the parallel rule-based sub-graph compression module and the graph expression recovery module, the storage space requirement of the graph data can be effectively reduced, and the processing efficiency of the graph data is optimized, especially in a resource-limited environment. According to the method, the development of a large-scale graph data processing technology is promoted, and a new solution is provided for graph data application in a resource-limited environment.

Description

Technical Field

[0001] The present application belongs to the field of graph data processing technology, and in particular, relates to a method for compressing and processing large-scale graph data in a resource-constrained environment. Background Art

[0002] As a data structure that expresses entities and their relationships, graph data plays an important role in many fields, such as social network analysis, bioinformatics, and transportation network planning. As the scale of data continues to increase, traditional graph data processing technology faces huge challenges, especially in a resource-constrained environment. How to effectively store, compress, and process large-scale graph data has become a research hotspot.

[0003] The main challenges of graph data lie in its complex structure and large-scale data volume. Graph data not only contains a large number of vertices (nodes) and edges (relationships connecting nodes), but also the relationships between these vertices and edges form a complex network structure. This structure leads to the storage and processing of graph data requiring a large amount of memory and computing resources. In addition, the dynamic nature of graph data, that is, the graph structure changes over time, increases the complexity of graph data processing.

[0004] At present, existing graph data processing technologies mainly focus on the compression and efficient calculation of graph data. Graph data compression aims to reduce the storage space requirements of graph data and identify and remove redundant information in graph data through various algorithms. However, many compression algorithms often sacrifice data access speed and processing efficiency, or cannot effectively support the processing of dynamic graph data. In addition, existing graph processing frameworks often require the entire graph data to be loaded into memory for processing, which is unrealistic for large-scale graph data, especially in resource-constrained environments. In resource-constrained environments, such as embedded systems, mobile devices, and cloud computing resources, higher requirements are placed on graph data processing technology. In these environments, not only limited storage space becomes a constraint, but computing power and energy consumption are also key points that need to be considered. Therefore, how to minimize resource consumption while ensuring graph data compression rate and processing efficiency has become an important direction for the development of graph data processing technology. Summary of the invention

[0005] Based on this, it is necessary to provide a method for compressing and processing large-scale graph data in a resource-constrained environment to address the above technical problems.

[0006] A first aspect of the present invention provides a method for compressing and processing large-scale graph data in a resource-constrained environment, the method comprising:

[0007] Get large graph data;

[0008] Dividing the large graph data into multiple initial subgraphs;

[0009] Performing a parallel compression algorithm on the multiple initial subgraphs to obtain multiple compressed target subgraphs;

[0010] The compressed multiple target sub-graphs are restored.

[0011] In some practicable manners, the step of dividing the large graph data to obtain a plurality of initial subgraphs includes:

[0012] Get memory parameters and utilization;

[0013] According to the parameters and utilization of the memory, by predicting the scale of the binary rules that may be generated, the size of the initial subgraph is adjusted so that the size of the initial subgraph and the intermediate result is within the range of available memory, wherein the intermediate result represents the size of the temporarily occupied memory.

[0014] In some practicable manners, the step of performing a parallel compression algorithm on the plurality of initial subgraphs to obtain a plurality of compressed target subgraphs includes:

[0015] Scan the input initial subgraph and assign serial numbers to all possible binary rules;

[0016] Assign serial numbers according to all possible binary rules, assign different threads, and obtain all binary rules;

[0017] The initial subgraph is segmented, and different initial subgraphs after segmentation are assigned to different threads, and rule replacement is performed to obtain a plurality of compressed target subgraphs.

[0018] In some practicable manners, the step of segmenting the initial subgraph, assigning different segmented initial subgraphs to different threads, performing rule replacement, and obtaining a plurality of compressed target subgraphs includes:

[0019] By merging the initial subgraphs compressed by the parallel rules, repeated binary rules between different initial subgraphs are eliminated to simplify the representation of the compressed multiple target subgraphs.

[0020] In some practicable manners, the step of restoring the compressed plurality of target sub-graphs further includes:

[0021] Merging the target subgraphs to obtain the large graph data;

[0022] The large graph data is reordered by breadth-first search, and the nodes are sorted according to the sequence numbers of neighbors, and the nodes represent vertices in the graph data.

[0023] A second aspect of the present invention provides a system for compressing and processing large-scale graph data in a resource-constrained environment, which is applied to the method for compressing and processing large-scale graph data in the aforementioned resource-constrained environment. The system for compressing and processing large-scale graph data in a resource-constrained environment comprises:

[0024] An acquisition unit, used to acquire large graph data;

[0025] A partitioning unit is used to partition large graph data to obtain multiple initial subgraphs;

[0026] A compression unit, used for performing a parallel compression algorithm on the plurality of initial subgraphs to obtain a plurality of compressed target subgraphs;

[0027] A restoration unit is used to restore the compressed target sub-graphs.

[0028] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the aforementioned method when executing the computer program.

[0029] A fourth aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0030] A fifth aspect of the present invention provides a computer program, which implements the steps of the aforementioned method when executed by a processor.

[0031] Beneficial Effects

[0032] The above-mentioned method for compressing and processing large-scale graph data in a resource-constrained environment first obtains large-scale graph data; divides the large-scale graph data to obtain multiple initial subgraphs; then, performs a parallel compression algorithm on the multiple initial subgraphs to obtain multiple compressed target subgraphs; next, restores the multiple compressed target subgraphs. The above-mentioned method uses memory-aware adaptive graph slicing, parallel rule-based subgraph compression, and graph expression recovery modules. The present invention can not only effectively reduce the storage space requirements of graph data, but also optimize the processing efficiency of graph data, especially in resource-constrained environments. The proposal of the present invention not only helps to promote the development of large-scale graph data processing technology, but also provides a new solution for graph data applications in resource-constrained environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 A flowchart of a method for compressing and processing large-scale graph data in a resource-constrained environment in one embodiment. DETAILED DESCRIPTION

[0035] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used herein includes any and all couplings of one or more related listed items.

[0037] It can be understood that the terms "first", "second", etc. used in the present application can be used in this article to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0038] Some terms involved in this application are explained below to facilitate understanding of this application:

[0039] Binary rules refer to rules involving the relationship between two variables. In data mining, machine learning, and logical reasoning, binary rules are used to describe the association, causal relationship, or conditional dependency between two variables.

[0040] Breadth-first search (BFS) is a graph algorithm used to traverse or search for nodes in a graph. It starts from a specified node in the graph, visits its neighbor nodes layer by layer, and then continues to visit the neighbor nodes of the next layer according to the breadth until all nodes have been visited.

[0041] PageRank is an algorithm used to evaluate the importance and ranking of web pages.

[0042] The present application provides a method for compressing and processing large-scale graph data in a resource-constrained environment. The method divides the large-scale graph data to be read into multiple sub-graphs based on the available memory, and each sub-graph is represented by a CSR. A parallel rule compression method is used for each sub-graph to accelerate the compression of the graph data. After all sub-graphs are compressed, the results of the rule compression are merged, and the merged graphs are reordered and encoded and compressed to achieve less memory usage. The present invention aims to provide an efficient graph compression method to effectively reduce the storage space occupied in the shortest possible time.

[0043] like Figure 1 As shown, a method for compressing and processing large-scale graph data in a resource-constrained environment includes:

[0044] S100, obtaining large-scale graph data.

[0045] Among them, large-scale graph data refers to large-scale graph data formed by a large amount of graph data as the scale of data increases. Graph data includes nodes and edges. Nodes refer to the vertices of graph data, and edges refer to the relationship between nodes. The relationship between vertices and edges constitutes a complex network structure. The complex network structure also requires a large amount of memory and computing resources for storage and processing.

[0046] S200, dividing the large graph data into multiple initial subgraphs.

[0047] Specifically, obtaining multiple initial subgraphs includes the following steps:

[0048] Get memory parameters and utilization;

[0049] According to the parameters and utilization of the memory, by predicting the scale of the binary rules that may be generated, the size of the initial subgraph is adjusted so that the size of the initial subgraph and the intermediate result is within the range of available memory, wherein the intermediate result represents the size of the temporarily occupied memory.

[0050] It should be noted that the parameters and utilization of the memory can be determined according to the system configuration, that is, known conditions. When the parameters and utilization of the memory are determined, the scale of the binary rules that may be generated is predicted, wherein the scale of the binary rules refers to the complexity of the binary rules. The size of the initial subgraph is adjusted using the binary rule so that the size of the initial subgraph and the intermediate result are within the available memory range. Exemplarily, the binary rule designs two variables, one of which is the size of the initial subgraph, and the other variable is the intermediate result formed by the temporary memory occupancy. In other words, if the size of the initial subgraph is A, then the intermediate result is B, which can be represented by symbols as A→B, that is, the size of the initial subgraph corresponds to the intermediate result of the temporary memory occupancy. In this way, the size of the initial subgraph can be adjusted using the binary rule to obtain the result of temporary memory occupancy, and the size of the initial subgraph can be adjusted multiple times so that the size of each subgraph can use the current memory capacity.

[0051] It should also be noted that the large graph data is divided into multiple initial sub-graphs, each of which contains a portion of data. A graph partitioning algorithm (such as METIS) can be used to divide the large graph data into several initial sub-graphs.

[0052] The present invention dynamically divides large graph data into multiple subgraphs through a memory-aware adaptive algorithm (binary rule). This process takes into account the limitations of available memory resources and aims to ensure that the size of each subgraph can adapt to the current memory capacity, thereby avoiding excessive memory requirements during subsequent compression and processing. In this way, the present invention can effectively balance memory usage and reduce performance bottlenecks caused by memory overflow.

[0053] S300, performing a parallel compression algorithm on the multiple initial subgraphs to obtain multiple compressed target subgraphs.

[0054] Specifically, obtaining the compressed multiple target subgraphs includes the following steps:

[0055] Scan the input initial subgraph and assign serial numbers to all possible binary rules;

[0056] Assign serial numbers according to all possible binary rules, assign different threads, and obtain all binary rules;

[0057] The initial subgraph is segmented, and different initial subgraphs after segmentation are assigned to different threads, and rule replacement is performed to obtain a plurality of compressed target subgraphs.

[0058] It should be noted that after the initial subgraph is obtained in the above steps, the initial subgraph is scanned and all possible binary rules are assigned unique serial numbers. Next, different binary rules are assigned to different threads according to the allocation requirements, and parallel processing is performed to generate all binary rules.

[0059] The initial subgraph is split into multiple subgraphs, and these subgraphs are assigned to different threads for parallel processing. By splitting the initial subgraph into multiple smaller subgraphs and processing these subgraphs in parallel, the resources of multi-core processors or computing clusters can be fully utilized, greatly improving computing efficiency. Each thread processes one subgraph, and multiple threads work simultaneously, which can significantly shorten the processing time. In addition, if the initial subgraph is very large, a single thread processing the entire initial subgraph may cause insufficient memory or inefficiency. By splitting the initial subgraph, each thread only needs to process smaller data blocks, thereby reducing the memory burden of a single thread and improving memory utilization. Splitting the initial subgraph can achieve better load balancing. By dynamically assigning subgraphs to different threads, it can ensure that the workload of each thread is roughly the same, avoiding overloading or idleness of some threads. This helps to make full use of computing resources and improve overall processing efficiency.

[0060] Furthermore, the step of dividing the initial subgraph, assigning different initial subgraphs after division to different threads, performing rule replacement, and obtaining a plurality of compressed target subgraphs also includes:

[0061] By merging the initial subgraphs compressed by the parallel rules, repeated binary rules between different initial subgraphs are eliminated to simplify the representation of the compressed multiple target subgraphs.

[0062] It should be noted that in each thread, the generated binary rules are used to replace the assigned subgraphs with rules, wherein the rule replacement can simplify the complexity of the graph. For example, if there are multiple edges that can be simplified into one edge by a certain rule, this replacement can reduce the complexity of the graph. As mentioned above, according to the binary rules, some nodes or edges can be merged. For example, if there are multiple edges between two nodes, these edges can be merged into one edge according to the rules, or the nodes themselves can be merged. Rule replacement can eliminate redundant information in the graph. For example, repeated edges or nodes can be deleted, and a unique rule representation is retained. For example, suppose there is a social network graph, in which the nodes in the graph represent users and the edges represent the relationship between users. Some binary rules are generated, such as "If user A and user B are both friends of user C, then it can be simplified to user A and user B are also friends". Based on this rule, a rule replacement operation can be performed. The core idea of ​​performing rule replacement is to use pre-generated binary rules to simplify and optimize graph data, reduce complexity and redundancy, and thus improve processing efficiency and storage efficiency. In parallel computing and large-scale data processing, rule replacement is an effective optimization strategy that helps to better manage and analyze data. Thus, each initial subgraph is processed and rule compressed to generate binary rules for each subgraph, wherein rule compression can be achieved by deleting redundant rules, merging similar rules, and the like.

[0063] Thanks to the subgraph slicing in the previous step, the present invention applies a parallel rule-based compression algorithm to each subgraph in this step. This algorithm reduces data redundancy by identifying and replacing repeated structures in the subgraph (such as frequently occurring vertex pairs or patterns). Parallel processing not only speeds up the compression process, but also ensures the efficiency and effectiveness of the compression process by accurately controlling the compression ratio and the quality of the compressed data. In addition, this step greatly reduces the size of the graph data by eliminating data redundancy, thereby reducing resource consumption in subsequent processing.

[0064] S400: Restoring the compressed target subgraphs.

[0065] Specifically, restoring the compressed target subgraphs comprises the following steps:

[0066] Merging the target subgraphs to obtain the large graph data;

[0067] The large graph data is reordered by breadth-first search, and the nodes are sorted according to the sequence numbers of neighbors, and the nodes represent vertices in the graph data.

[0068] It should be noted that merging the target subgraph means merging multiple compressed and processed subgraphs into an overall, larger graph structure. This process can be interpreted as bringing together all the nodes and edges in the graph to form a complete graph containing all the original subgraph information. Next, the large graph data is reordered, and the breadth-first search (BFS) algorithm can be used to traverse the entire large graph data. The BFS algorithm starts from a starting node, visits its neighbor nodes layer by layer, and then sorts them according to the access order. Within the node, sorting according to the sequence number of the neighbors means sorting the neighbor list of each node to ensure that the neighbor nodes are arranged according to a certain rule or index order. This sorting can be achieved according to the order of node connection, node attributes or other rules, so that the nodes and their relationships can be accessed and operated more efficiently in the subsequent analysis or processing process. The purpose of merging the target subgraphs to obtain large graph data is to integrate and consolidate all relevant information, while the breadth-first search reordering is to optimize the structure and access method of the data to ensure that the graph data can be efficiently processed and analyzed in large-scale data processing.

[0069] Although the compression process reduces the data size, it may also affect the structure and expression ability of the graph data, and thus affect subsequent graph analysis tasks. Therefore, in step S400 of the present invention, graph expression recovery is introduced to optimize and restore the structural expression of the compressed graph data. This includes readjusting the arrangement order of vertices to improve the locality of the data, thereby improving the speed and efficiency of graph data processing without significantly increasing the data size. Through this step, the present invention can ensure that even after compression processing, the graph data can still effectively support various graph analysis algorithms and tasks.

[0070] Example

[0071] For the uk-2007-05 dataset, steps S100 to S400 achieve a compression ratio of 8.23 ​​while ensuring the expressiveness of the graph, making the performance of the compressed graph exceed that of the original graph in executing graph algorithms such as PageRank.

[0072] The UK-2007-05 dataset refers to a Web Graph dataset, where "UK" stands for the United Kingdom and "2007-05" stands for the collection time of the dataset, i.e. May 2007.

[0073] The pseudo code of the algorithm provided by the embodiment of the present invention is:

[0074]

[0075] In summary, the method for compressing and processing large-scale graph data in a resource-constrained environment of the present invention has the following advantages:

[0076] 1. Significantly improve compression rate and data processing efficiency

[0077] The present invention uses memory-aware adaptive graph slicing and parallel rule-based subgraph compression technology to effectively identify and remove redundant information in graph data, achieve a high compression rate, and significantly reduce the storage space requirements of graph data. In addition, the graph expression recovery technology optimizes the compressed graph structure and improves the locality of the data, thereby accelerating the processing of graph data. This combination of compression and processing optimization enables the present invention to significantly improve the performance and efficiency of graph data processing while ensuring data integrity.

[0078] 2. Effectively alleviate the challenges of resource constraints

[0079] For mobile devices, edge computing devices, and cloud computing resource-constrained application scenarios, this invention effectively alleviates the challenge of resource constraints by reducing the storage and computing resource requirements of graph data. Memory-aware graph slicing technology ensures that memory usage during compression and processing is always within a controllable range, avoiding performance degradation or system crashes caused by resource overload, and improving the stability and reliability of graph data processing.

[0080] 3. Reduce energy consumption and extend device life

[0081] When processing large-scale graph data on mobile and edge computing devices, energy consumption is a factor that cannot be ignored. This invention significantly reduces energy consumption by improving the efficiency of data processing and reducing the amount of data transmission, which helps to extend the battery life of the device. This advantage is particularly important for energy-sensitive application scenarios, which can support longer data analysis and processing tasks and improve user experience.

[0082] 4. Enhance the flexibility and adaptability of graph data processing

[0083] The memory-aware adaptive graph slicing technology of the present invention can dynamically adjust the processing strategy of graph data according to available resources, thereby enhancing the flexibility and adaptability of graph data processing. This makes the present invention not only suitable for resource-constrained environments, but also can make full use of available resources when resources are sufficient to achieve more efficient graph data processing. In addition, the method of the present invention has good scalability, can adapt to graph data of different sizes and types, and supports a variety of graph analysis algorithms and application requirements.

[0084] A second aspect of the present invention provides a system for compressing and processing large-scale graph data in a resource-constrained environment, which is applied to the method for compressing and processing large-scale graph data in the aforementioned resource-constrained environment. The system for compressing and processing large-scale graph data in a resource-constrained environment comprises:

[0085] An acquisition unit, used to acquire large graph data;

[0086] A partitioning unit is used to partition large graph data to obtain multiple initial subgraphs;

[0087] A compression unit, used for performing a parallel compression algorithm on the plurality of initial subgraphs to obtain a plurality of compressed target subgraphs;

[0088] A restoration unit is used to restore the compressed target sub-graphs.

[0089] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the aforementioned method when executing the computer program.

[0090] A fourth aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0091] A fifth aspect of the present invention provides a computer program, which implements the steps of the aforementioned method when executed by a processor.

[0092] In the description of the embodiments of the present invention, those skilled in the art should know that the embodiments of the present invention can be implemented as methods, devices, electronic devices and computer-readable storage media. Therefore, the embodiments of the present invention can be specifically implemented in the following forms: complete hardware, complete software (including firmware, resident software, microcode, etc.), a combination of hardware and software. In addition, in some embodiments, the embodiments of the present invention can also be implemented in the form of a computer program product in one or more computer-readable storage media, and the computer-readable storage medium contains computer program code.

[0093] The above-mentioned computer-readable storage medium may adopt any combination of one or more computer-readable storage media. Computer-readable storage media include: electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM), flash memories, optical fibers, compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices or any combination of the above. In an embodiment of the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device, or device.

[0094] The computer program code contained in the above-mentioned computer-readable storage medium can be transmitted using any appropriate medium, including: wireless, wire, optical cable, radio frequency (RF) or any suitable combination thereof.

[0095] The computer program code for performing the operation of the embodiments of the present invention can be written in assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or in one or more programming languages ​​or a combination thereof, the programming language including an object-oriented programming language, such as Java, Smalltalk, C++, and also including a conventional procedural programming language, such as C language or a similar programming language. The computer program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, and completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer or to an external computer through any type of network, including a local area network (LAN) or a wide area network (WAN).

[0096] The embodiments of the present invention describe the provided methods, devices, and electronic devices through flowcharts and / or block diagrams.

[0097] It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, and these computer-readable program instructions are executed by a computer or other programmable data processing device to produce a device that implements the functions / operations specified by the boxes in the flowchart and / or block diagram.

[0098] These computer-readable program instructions may also be stored in a computer-readable storage medium that enables a computer or other programmable data processing device to work in a specific manner. In this way, the instructions stored in the computer-readable storage medium produce an instruction device product including functions / operations specified in the blocks in the flowchart and / or block diagram.

[0099] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable data processing apparatus to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0100] The terms "first" and "second" in the description and claims of the embodiments of the present invention are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0101] In the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0102] In the description of the embodiments of the present invention, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for compressing and processing large-scale graph data in a resource-constrained environment, characterized in that: The method comprises: Get large graph data; Dividing the large graph data into multiple initial subgraphs; Performing a parallel compression algorithm on the multiple initial subgraphs to obtain multiple compressed target subgraphs; The compressed multiple target sub-graphs are restored.

2. The method for compressing and processing large-scale graph data in a resource-constrained environment according to claim 1, characterized in that: The step of dividing the large graph data to obtain a plurality of initial subgraphs includes: Get memory parameters and utilization; According to the parameters and utilization of the memory, by predicting the scale of the binary rules that may be generated, the size of the initial subgraph is adjusted so that the size of the initial subgraph and the intermediate result is within the range of available memory, wherein the intermediate result represents the size of the temporarily occupied memory.

3. The method for compressing and processing large-scale graph data in a resource-constrained environment according to claim 1, characterized in that: The step of performing a parallel compression algorithm on the multiple initial subgraphs to obtain multiple compressed target subgraphs includes: Scan the input initial subgraph and assign serial numbers to all possible binary rules; Assign serial numbers according to all possible binary rules, assign different threads, and obtain all binary rules; The initial subgraph is segmented, and different initial subgraphs after segmentation are assigned to different threads, and rule replacement is performed to obtain a plurality of compressed target subgraphs.

4. The method for compressing and processing large-scale graph data in a resource-constrained environment according to claim 1, characterized in that: The step of dividing the initial subgraph, assigning different initial subgraphs after division to different threads, performing rule replacement, and obtaining a plurality of compressed target subgraphs includes: By merging the initial subgraphs compressed by the parallel rules, repeated binary rules between different initial subgraphs are eliminated to simplify the representation of the compressed multiple target subgraphs.

5. The method for compressing and processing large-scale graph data in a resource-constrained environment according to claim 1, characterized in that: The step of restoring the compressed plurality of target subgraphs further comprises: Merging the target subgraphs to obtain the large graph data; The large graph data is reordered by breadth-first search, and the nodes are sorted according to the sequence numbers of neighbors, and the nodes represent vertices in the graph data.

6. A system for compressing and processing large-scale graph data in a resource-constrained environment, characterized in that: The system for compressing and processing large-scale graph data in a resource-constrained environment includes: An acquisition unit, used to acquire large graph data; A partitioning unit is used to partition large graph data to obtain multiple initial subgraphs; A compression unit, used for performing a parallel compression algorithm on the plurality of initial subgraphs to obtain a plurality of compressed target subgraphs; A restoration unit is used to restore the compressed target sub-graphs.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images