Dynamic graph data storage method and device oriented to hybrid workloads

Abstract

The invention discloses a dynamic graph data storage method and device oriented to hybrid workloads. According to the method, aiming at the problem that various graph workloads are difficult to efficiently support in a single system in the prior art, a hash-based index construction, dynamic hierarchy edge storage design and a fine-grained concurrency control mechanism are provided. According to the method, efficient indexing of the vertexes is realized by adopting the hash table, and hash conflicts are solved in combination with a hash bucket strategy, so that the data query performance is improved. In the aspect of edge data storage, a compact array structure with a continuous memory is adopted, so that the traversal efficiency of graph data is optimized; meanwhile, aiming at the power law distribution characteristic of the graph data, the invention provides a dynamic hierarchical storage structure to improve the storage and updating efficiency, and an ordered storage strategy is adopted for the edge data of the vertex of the height number to efficiently support the graph pattern matching type workload. The invention provides a control mechanism, which improves the throughput of the system under a more novel working load while ensuring the data consistency.

Description

Technical Field

[0001] The present invention relates to the fields of database technology and data storage technology, and in particular to a dynamic graph data storage method, device, electronic device, computer-readable storage medium and computer program product for mixed workloads, so as to realize efficient storage, updating and analysis of large-scale dynamic graph data. Background Art

[0002] As a fundamental data structure for expressing inter-entity relationships, graph data, with its excellent expressive power, is widely used in scenarios such as financial risk control, social network analysis, and e-commerce recommendations. Currently, many applications require executing diverse workloads on graph datasets, such as graph updates, graph traversals, and graph mining. Therefore, building a dynamic graph data storage system that can effectively support diverse workloads is challenging, and the key lies in the design of the underlying storage structure. However, different workloads place varying demands on data structures: graph update workloads (such as vertex and edge insertion, deletion, and lookup) require efficient indexing capabilities for quickly locating the locations being manipulated; graph traversal workloads (such as breadth-first traversal, single-source shortest path, and web page ranking) require data structures that can quickly scan vertex neighbors and visit all adjacent edges to improve traversal performance; and graph mining workloads (such as triangle counting, K-core, and K-truss) prefer ordered data storage to efficiently perform subgraph mining and matching operations. Therefore, designing an efficient dynamic graph data storage solution for mixed workloads has become a critical issue that needs to be addressed.

[0003] The highly sparse nature of graph data means that the number of edges in the graph is relatively small, far less than the maximum possible number of edges between vertices. For example, for a graph with n vertices, since every two vertices can have an edge, the maximum number of edges is n(n-1) / 2. However, in reality, the number of edges is much smaller than this value. Storing sparse graphs using storage methods like adjacency matrices results in significant space waste. However, real-world graph data is often highly sparse. Therefore, to efficiently store graph data, the Compressed Sparse Row (CSR) structure is widely used in graph processing systems. CSR uses continuous edge storage, which offers good locality and effectively supports graph traversal workloads. However, when graph data involves updates such as vertex or edge insertions and deletions, CSR requires moving a large number of data elements, making it difficult to efficiently support graph update workloads. To address both the advantages and limitations of the CSR structure, some studies have proposed improved CSR-like storage structures that mitigate the inefficiency of CSR updates by reserving gaps. However, this type of structure has shortcomings in supporting graph mining workloads and faces high rebalancing overhead, which still limits overall update efficiency. Designing fast hash indexes to accelerate graph data lookups can improve the throughput of graph update workloads, but because edges to the same vertex are stored in discontinuous memory spaces, performance is poor when executing graph traversal and graph pattern matching workloads. Graph storage structures that support mixed workloads must maintain a large amount of version information, resulting in high storage overhead and low space utilization. In addition, some research has expanded support for graph mining workloads, but there are still limitations in data storage, updates, and computational efficiency, leaving room for further optimization.

[0004] While current research on graph storage structures has made progress in specific scenarios, it has exposed performance deficiencies when faced with diverse workloads. There is a lack of solutions to the difficult problem of coordinating the optimization of storage efficiency, update performance, and compute friendliness. Summary of the Invention

[0005] Based on the above research, the present invention starts from the aspects of workload execution mode, data layout and access characteristics, data concurrency control mechanism, etc., deeply analyzes the key factors affecting the dynamic graph storage performance for mixed workloads, and designs a dynamic graph data storage method for mixed workloads, aiming to improve the efficient support capabilities of a single dynamic graph storage system for multiple types of graph workloads.

[0006] In view of the shortcomings of the existing technology, the present invention proposes a dynamic graph data storage method for mixed workloads, which includes:

[0007] The initial step is to first obtain the edge information to be inserted, calculate the hash value of the source vertex in the edge information, obtain the location pointer of the hash value according to the hash table, and locate the hash bucket pointed to by the location pointer;

[0008] A judgment step, according to the bit vector in the hash bucket, accessing a non-empty vertex information storage unit in the bit vector and judging whether the source vertex already exists therein, if so, executing the insertion step, otherwise executing the development step;

[0009] An opening step, according to the bit vector, inserting the source vertex into an empty vertex information storage unit in the hash bucket, obtaining a vertex lock for the insertion position, opening an edge storage space for the vertex, and storing a pointer to the edge storage space in the hash bucket corresponding to the source vertex; the edge storage space includes a short array of a preset capacity and a bit vector for maintaining the short array; inserting the edge connected to the source vertex in the edge information into a specified position of the short array in the edge storage space, increasing the value of the short array by the number of inserted edges, releasing the vertex lock, and ending the insertion process;

[0010] The insertion step obtains the vertex lock corresponding to the source vertex and the pointer of the source vertex to the edge storage space to access the short array in the edge storage space and obtain the position of the free edge storage unit in the edge storage space through the bit vector. The edge connected to the source vertex in the edge information is inserted into the specified position of the short array in the edge storage space. The value of the short array increases by the number of inserted edges, the vertex lock is released, and the insertion process ends.

[0011] In the dynamic graph data storage method for mixed workloads, the inserting step includes:

[0012] According to the short array in the edge storage space, determining whether there is an idle edge storage unit position in the edge storage space, and if not, executing the expansion step;

[0013] The expansion step expands the capacity of the short array in the edge storage space, accesses the expanded short array in the edge storage space, and obtains the position of the free edge storage unit in the edge storage space through the bit vector. The edge connected to the source vertex in the edge information is inserted into the specified position of the short array in the edge storage space. The value of the short array increases by the number of inserted edges, the vertex lock is released, and the insertion process ends.

[0014] The dynamic graph data storage method for mixed workloads, wherein the expansion step includes:

[0015] Determine whether the capacity of the expanded short array is greater than the specified threshold. If so, sort the elements in the short array according to the destination vertex, open a sorted array, and the initial value of the stored elements at the head of the sorted array is 0, the number of elements marked for deletion is 0, and the number of storage units opened is the specified threshold. The sorted edges of the short array are stored in the sorted array, and then the short array is cleared. The position pointer of the opened space is saved in the short array, and the insertion step is performed again.

[0016] The dynamic graph data storage method for mixed workloads, wherein the edge information includes a source vertex and a destination vertex, the source vertex and the destination vertex are both user IDs in a social platform; or the source vertex and the destination vertex are respectively the seller ID and buyer ID of a trading platform.

[0017] like Figure 6 As shown, the present invention also proposes a dynamic graph data storage device for mixed workloads, which includes:

[0018] In the initial module, the storage device obtains the edge information to be inserted, calculates the hash value of the source vertex in the edge information, obtains the location pointer of the hash value according to the hash table, and locates the hash bucket pointed to by the location pointer;

[0019] A judgment module, according to the bit vector in the hash bucket, accesses the non-empty vertex information storage unit in the bit vector and determines whether the source vertex already exists therein, if so, executes the insertion module, otherwise executes the development module;

[0020] A development module inserts the source vertex into an empty vertex information storage unit in the hash bucket according to the bit vector, obtains a vertex lock for the insertion position, develops an edge storage space for the vertex, and stores a pointer to the edge storage space in the hash bucket corresponding to the source vertex; the edge storage space includes a short array of a preset capacity and a bit vector for maintaining the short array; inserts the edge in the edge information connected to the source vertex into a specified position of the short array in the edge storage space, increases the value of the short array by the number of inserted edges, releases the vertex lock, and ends the insertion process;

[0021] The insertion module obtains the vertex lock corresponding to the source vertex and the pointer of the source vertex to the edge storage space to access the short array in the edge storage space and obtain the position of the free edge storage unit in the edge storage space through the bit vector. The edge connected to the source vertex in the edge information is inserted into the specified position of the short array in the edge storage space. The value of the short array increases by the number of inserted edges, the vertex lock is released, and the insertion process ends.

[0022] The dynamic graph data storage device for mixed workloads, wherein the insertion module includes:

[0023] According to the short array in the edge storage space, determine whether there is an idle edge storage unit position in the edge storage space, and if not, execute the expansion module;

[0024] The expansion module expands the capacity of the short array in the edge storage space, accesses the expanded short array in the edge storage space and obtains the position of the free edge storage unit in the edge storage space through the bit vector, inserts the edge connected to the source vertex in the edge information into the specified position of the short array in the edge storage space, increases the value of the short array by the number of inserted edges, releases the vertex lock, and ends the insertion process.

[0025] The dynamic graph data storage device for mixed workloads, wherein the expansion module includes:

[0026] Determine whether the capacity of the expanded short array is greater than the specified threshold. If so, sort the elements in the short array according to the destination vertex, open a sorted array, and the initial value of the stored elements at the head of the sorted array is 0, the number of elements marked for deletion is 0, and the number of storage units opened is the specified threshold. The edges of the sorted short array are stored in the sorted array, and then the short array is cleared, and the position pointer of the opened space is saved in the short array, and then the insertion module is executed again.

[0027] The present invention also proposes an electronic device, which includes a dynamic graph data storage device for mixed workloads. The electronic device may be connected to an information display device, which is used to display the graph data storage results using display parameters and attributes set by the user or through an artificial intelligence model.

[0028] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the dynamic graph data storage method for mixed workloads.

[0029] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the dynamic graph data storage method for mixed workloads are implemented.

[0030] It can be seen from the above scheme that the advantages of the present invention are:

[0031] Compared with the prior art, the above-mentioned technical solution conceived by the present invention reduces vertex indexing time through hash indexing, and by designing the dynamic hierarchical edge storage as a variable double-layer edge array structure, it can store the edges in the same area, ensure memory sequential scanning and high spatial locality, and thus efficiently support traversal workloads. The short array in the double-layer structure can also act as a buffer structure, and the elements in the short array can be merged with the elements in the sorted array in batches, greatly improving the sorting effect. In general, the present invention reduces indexing time and speeds up traversal, so that the system can efficiently support mixed workloads. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The overall data structure diagram of the present invention

[0033] Figure 2 Insert edge flow chart for the present invention

[0034] Figure 3 Designed for hash bucket structure

[0035] Figure 4 Designed for dynamic hierarchical edge storage structure

[0036] Figure 5 Schematic diagram of the hardware device structure for implementing the embodiment of the present invention

[0037] Figure 6 This is a module diagram of the device of the present invention;

[0038] Figure 7 This is a schematic structural diagram of a first electronic device of the present invention;

[0039] Figure 8 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0040] Figure 9 This is a schematic structural diagram of a second electronic device according to the present invention.

[0041] Reference numerals:

[0042] A-First electronic device;

[0043] B-Dynamic graph data storage device for mixed workloads;

[0044] C-data acquisition equipment;

[0045] D-information display device;

[0046] 1000- second electronic device;

[0047] Ⅰ-computing unit;

[0048] II-ROM;

[0049] III-RAM;

[0050] IV-bus;

[0051] V-interface;

[0052] VI-input unit;

[0053] VII-output unit;

[0054] VIII-Storage medium;

[0055] IX-Communication unit. DETAILED DESCRIPTION

[0056] It should be noted that, in this application, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0057] Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0058] The processor described in the present invention is the control center of an electronic device and can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0059] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0060] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smartphones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0061] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0062] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0063] The above embodiments can be implemented in whole or in part through software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired method (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0064] It should also be understood that the term "and / or" in this document simply describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " in this document generally indicates an "or" relationship between the related objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0065] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0066] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0067] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0068] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0069] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0070] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0071] While researching storage structures for dynamic graph data, the inventors discovered that existing technologies struggle to efficiently support multiple workloads within a single graph storage system. Different workloads place varying demands on the graph data storage structure: Graph update workloads require a data structure with fast location capabilities, namely, an efficient indexing mechanism. Therefore, the present invention employs a hash table for fast vertex indexing and a hash bucket strategy to resolve hash collisions. A hash table is an array that uses a hash function to determine where a value should be stored. For example, hashing the source vertex 6 yields a value. Assuming that value is 3, the element 6 is stored in the third position of the hash table. However, different values can produce the same result after running the hash function. For example, hashing both 6 and 15 might yield 3, but both 6 and 15 cannot be stored in the array, resulting in a hash collision. Hash buckets are one such solution. By storing a pointer in the hash table to a fixed-size bucket array, all elements with the same hash value are placed in this bucket to resolve hash collisions. The fast indexing capabilities of hash tables provide strong support for vertex indexing in the present invention. Graph traversal workloads require a data structure capable of quickly traversing vertex neighbor information. To this end, the present invention adopts a compact and memory-contiguous array structure for edge storage to improve the traversal efficiency of graph data. In addition, in response to the problem of uneven vertex degree distribution in power-law graphs, the present invention proposes a dynamic hierarchical storage structure design to optimize the storage and update efficiency of vertices of different degrees. Graph mining workloads usually involve a large number of set intersection operations during execution. Ordered data storage can significantly reduce redundant calculations during the execution of this process. Therefore, in the edge storage design, the present invention adopts an ordered storage structure for the edges of high-degree vertices to efficiently support graph mining workloads. Finally, in order to improve the concurrent processing capability of the system, the present invention proposes a fine-grained optimistic locking mechanism to improve the concurrent reading and writing performance of data, thereby realizing high-throughput update workload processing. In summary, the present invention designs a dynamic graph data storage method for mixed workloads, which can realize efficient storage, updating and analysis of large-scale dynamic graph data.

[0072] To achieve the above goals, this paper proposes three key storage optimization strategies: hash-based index construction, dynamic hierarchical edge storage design, and fine-grained concurrent locking mechanism. The corresponding details are as follows:

[0073] Key point 1, hash-based index construction:

[0074] In order to improve the performance of vertex indexing, the present invention proposes a vertex index construction strategy based on hashing. In order to solve the hash collision problem that may occur during the hash indexing process, a hash bucket mechanism is used for efficient processing. The hash bucket is an improved chain address method that stores elements with the same hash value by pre-allocating a fixed-size array to solve the problem of low efficiency of linked list traversal in the chain address method. The maximum capacity of the hash bucket can be configured according to user needs, and the default size is set to 64. Each hash bucket is as follows: Figure 3 The diagram includes the following information: a 64-bit bit vector, a 64-bit lock array, and 64 vertex information storage units. The bit vector is used to mark the occupancy of each of the 64 storage units in the bucket. If a location is occupied, the corresponding location is set to 1; otherwise, it is set to 0. Bitwise operations allow for rapid location of available storage units. The lock array ensures consistent access to vertex data in a concurrent environment. The vertex information storage units store vertex information corresponding to the graph data. Vertex information includes the vertex ID, optional vertex attributes, and vertex neighbor pointers (pointers to the starting offsets of all neighboring edges for the vertex). During index construction, a hash value is first calculated based on the vertex value, and the corresponding hash bucket is determined. The vertices within the bucket are then traversed to check whether the target vertex already exists. If the vertex exists, the access operation is performed. If not, the bit vector is used to quickly locate an available storage location, store the vertex-related information, and complete the vertex index construction. This strategy effectively improves vertex index location efficiency and concurrent access capabilities through an efficient hash bucket design, combined with bit vector indexing and a locking mechanism.

[0075] Key Point 2: Dynamic Hierarchical Edge Storage Design:

[0076] In order to better adapt to different types of graph workload requirements, this paper proposes a dynamic hierarchical edge storage design, such as Figure 4As shown in the figure, this design combines the fast update characteristics of unordered storage with the efficient computational advantages of ordered storage, employing a dynamically variable short array and sorted array structure for hierarchical storage. During storage, the short array uses an unordered storage strategy to improve insertion performance. Furthermore, considering the power-law distribution of vertex degrees in graph data, the size of the short array is dynamically variable to reduce storage space waste. The corresponding short array size ranges from [4, 8, 16, 32, 64]. Initially, a short array of size 4 is used. When storage space is insufficient, the short array is doubled. When the number of neighboring edges of a vertex exceeds 64 (i.e., the maximum storage space of the short array cannot accommodate all neighboring edge information), the system dynamically allocates a sorted array and connects it to the short array via a pointer. The neighboring edge information in the short array is sorted in parallel and then asynchronously moved to the sorted array. At this point, the original short array data is cleared to continue storing the newly added neighboring edges. When the short array reaches its capacity again, it is sorted and then merged with the sorted array. The short array includes a header that stores array type information, an expiration flag, a timestamp, a 64-bit wide bit vector to assist with insertion and search, and a pointer to the sorted array. The sorted array header records the number of elements currently in the sorted array and the number marked for deletion to assist with merging. By designing a dynamic hierarchical edge storage structure, this invention not only improves the processing efficiency of graph update workloads, but also meets the requirements for efficient computation and processing for graph traversal and mining workloads.

[0077] Key point 3, fine-grained concurrency control mechanism:

[0078] To improve the throughput of graph storage systems when handling update workloads, this invention fully utilizes the multi-threaded resources of the hardware platform to achieve high-concurrency data reading and writing. To ensure consistency in concurrent data processing, this invention proposes a fine-grained concurrency control mechanism. This mechanism employs fine-grained locks for vertices in the graph data. Specifically, in the vertex storage of Key Point 1, an independent lock is designed for each vertex. During data reading and writing, different vertices acquire their own locks, thus avoiding data conflicts. Generally speaking, the finer the lock granularity, the stronger the concurrency capability, so this mechanism can significantly improve system throughput. To address read and write conflicts in graph data, this invention employs an optimistic locking mechanism. During read operations, the reader process does not need to acquire a lock but directly accesses the data. After the read is complete, the version number is checked to determine data consistency. If the version number is the same before and after the read, the data has not been modified and the read result is consistent. If the version number changes, the data has been modified during the read, and the data must be read again. For write operations, due to the potential for conflicts with other threads, the writer thread must acquire the corresponding vertex lock before executing the write operation to ensure that no other writer threads access the same vertex. After the write operation is completed, the version number is updated and the lock is released. This optimistic locking strategy effectively improves the efficiency of read and write operations. The fine-grained concurrency control mechanism designed in this invention ensures the correctness and high throughput of the system in a high-concurrency environment.

[0079] To illustrate the above-mentioned features and effects of the present invention more clearly and easily, the following embodiments are specifically described below with reference to the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are for illustrative purposes only. The scope of protection of the present invention is not limited to the disclosed embodiments; the present invention is defined by the appended claims.

[0080] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention will be further described in detail below in conjunction with specific embodiments. Obviously, the embodiments described are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0081] This example provides an edge insertion process using this method. The overall data structure of the invention is as follows Figure 1 As shown, based on this structure, the overall flow chart of the edge insertion scheme is as follows Figure 2 As shown, the specific steps include:

[0082] Step 1: First, obtain the information of the edge to be inserted, calculate the hash value based on the source vertex of the edge, access the corresponding hash value position in the hash table according to the hash value, obtain the information of the position, which is a pointer to the corresponding bucket, and then access the hash bucket;

[0083] An edge is a structure that represents a relationship. For example, in a social network graph, an edge between vertices A and B indicates a follow-subscribe relationship between A and B. In the purchase process, when a buyer pays a seller, an edge can be abstracted to represent a direct transaction between the buyer and seller. The most basic edge information only contains a source vertex and a destination vertex. For example, (v1, v2) indicates an edge from source vertex V1 to destination vertex V2, meaning there is a connection between the two vertices. An edge is actually represented as a vertex pair. Vertices are typically represented by a numeric ID.

[0084] Step 2: Traverse the hash bucket with the bit vector (the bit vector of the vertex) in the hash bucket. Because of the existence of the bit vector, only the vertex information storage unit with a non-empty bit vector can be accessed. According to the source vertex value (id), check whether the source vertex already exists in the hash bucket. If not, proceed to step 3. If so, proceed to the insertion step.

[0085] Step 3: If it does not exist, use the bit vector to quickly calculate the first free vertex information storage unit, insert the source vertex into this position, obtain the vertex lock for this position, open up the edge storage space for this vertex, and store the edge storage space pointer in the corresponding source vertex in the hash bucket. The initially opened edge storage space includes a short array of 4 edge storage units and a bit vector (edge bit vector) used to maintain the array, which is used for subsequent vertex statistics and to assist in fast insertion. Then insert the edge to be inserted into the first position of the short array in the edge storage space, and release the lock to end the insertion process.

[0086] The vertex bit vector is used to assist vertex operations, and the edge bit vector is used to assist edge operations. The main function of both is to speed up the operation. Figure 3 and Figure 4 As you can see, the two are stored in different locations and act on different objects.

[0087] Insertion step: If it exists, get the pointer of the source vertex to the edge storage space, and get the vertex lock corresponding to the source vertex. According to the pointer, access the short array in the edge storage space to determine whether the short array meets the expansion condition. That is, the space opened up during access is 4, and now 4 edges are stored in the short array. This situation meets the expansion condition. If it has not been reached, calculate the first free edge storage unit position through the bit vector, insert the edge to be inserted into this position in the short array, and then release the lock to end the insertion process. If the maximum capacity has been reached, proceed to step 5.

[0088] Step 5: If the expansion condition is met, double the capacity of the short array based on the current maximum capacity. For example, if the current number of edge storage units is 4, expand the number of storage units to 8; if it is currently 8, expand it to 16, and so on. Then, calculate the first free storage unit, insert the edge to be inserted into it, and release the lock to end the insertion process. If the current storage space is 64 and all 64 storage units are occupied, proceed to step 6.

[0089] Capacity expansion refers to allocating more storage space within the computer's memory space. As long as the computer's memory space is sufficient, expansion can be carried out.

[0090] Step 6. Through the above steps, the present invention stores all edges of the same source vertex in the short array and sorted array pointed to by the source vertex. Therefore, you only need to store the destination vertex to know who the edge comes from and where it needs to go. Sort the elements in the short array according to the serial number of the destination vertex, and then open up the sorted array. Because it is the first time to open up, the initial value of the stored elements at the header of the sorted array is 0, the number marked for deletion is 0, and the number of storage units opened is 64. The edges after the short array is sorted are stored in the sorted array, and then the short array is cleared to continue storing the newly added adjacent edges, and the position pointer of the opened space is saved in the short array, corresponding to Figure 1 The edge to be inserted is inserted into the first position of the cleared short array, and the lock is released to end the insertion process. If this is not the first time the sorted array is allocated, a quick merge sort is performed on the sorted short array contents and the elements in the sorted array, and the contents are stored in the sorted array; the edge to be inserted is then inserted into the first position of the cleared short array, and the lock is released to end the insertion process.

[0091] After completing the above steps and storing the graph data, the stored graph data can be queried, retrieved, scanned, and other operations. For example, when performing a query operation: when querying edge (3, 1), the first step is to determine whether the source vertex exists in the graph data. If the source vertex does not exist, then the edge certainly does not exist. The hash value of source vertex 3 is calculated. Based on the hash value, the corresponding hash bucket array is found. Then, using bit vectors to assist in scanning the bucket array, the bucket array is scanned to determine whether the source vertex exists in the bucket. If not, it indicates that edge (3, 1) does not exist. If so, the edge storage space (i.e., the short array and the sorted array) for the source vertex is accessed. First, using bit vectors to assist in traversing the short array, the short array is checked to see if destination vertex 1 exists. If so, the relevant information is returned. If not, the sorted array is checked to see if it exists. If it does, the edge does not exist. If the sorted array exists, the sorted array is accessed. Because sorted arrays are stored in order, a binary search can be used to determine whether destination vertex 1 exists. If so, the relevant information is returned. If not, it indicates that edge (3, 1) does not exist, and the query process ends.

[0092] Figure 5 Provided is a schematic diagram of the structure of a hardware device for implementing an embodiment of the present invention. The device may include: at least one processor, such as a CPU (Central Processing Unit), at least one memory, at least one communication bus, at least one input device, and at least one output device.

[0093] The CPU consists of an arithmetic logic unit (ALU) and a control unit (CU). The ALU is responsible for program operations and logical judgment, while the CU coordinates the work between peripheral components and units. The CPU performs functions or processes data by executing non-volatile programs, instructions, or modules stored in memory.

[0094] The memory is a memory component used to store programs and various data information. It can store program instructions and data corresponding to the parallel single-source shortest path graph processing optimization method in the example of the present invention. The memory can include volatile memory, such as high-speed random access memory (Random Access Memory, referred to as RAM), and can also include non-volatile memory, such as a hard disk, flash memory or other non-volatile solid-state storage device.

[0095] The communication bus is used to realize the connection and data communication between these components, including the data bus, address bus and control bus, which are used to transmit data, data address and control signals respectively.

[0096] Input devices can receive numbers, characters, etc. input by the user, and can also obtain input control signals or function setting information, etc., and can include a mouse, keyboard, etc.

[0097] The output device can display the output information of the device and may include a display device such as a display screen.

[0098] The program instructions corresponding to the described method are stored in the memory. When they are executed by one or more processors through an input device, the parallel single-source shortest path graph processing optimization method described in the above embodiment can be implemented, and the final result can be displayed on an output device.

[0099] The above-mentioned specific implementation methods are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementation methods. Any appropriate changes or substitutions made by any ordinary technician in the technical field that conform to the claims of the present invention for a hardware resource allocation method applied to a data center container cloud platform shall fall within the patent protection scope of the present invention.

[0100] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0101] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0102] like Figure 6 As shown, the present invention also proposes a dynamic graph data storage device for mixed workloads, which includes:

[0103] In the initial module, the storage device obtains the edge information to be inserted, calculates the hash value of the source vertex in the edge information, obtains the location pointer of the hash value according to the hash table, and locates the hash bucket pointed to by the location pointer;

[0104] A judgment module, according to the bit vector in the hash bucket, accesses the non-empty vertex information storage unit in the bit vector and determines whether the source vertex already exists therein, if so, executes the insertion module, otherwise executes the development module;

[0105] A development module inserts the source vertex into an empty vertex information storage unit in the hash bucket according to the bit vector, obtains a vertex lock for the insertion position, develops an edge storage space for the vertex, and stores a pointer to the edge storage space in the hash bucket corresponding to the source vertex; the edge storage space includes a short array of a preset capacity and a bit vector for maintaining the short array; inserts the edge in the edge information connected to the source vertex into a specified position of the short array in the edge storage space, increases the value of the short array by the number of inserted edges, releases the vertex lock, and ends the insertion process;

[0106] The insertion module obtains the vertex lock corresponding to the source vertex and the pointer of the source vertex to the edge storage space to access the short array in the edge storage space and obtain the position of the free edge storage unit in the edge storage space through the bit vector. The edge connected to the source vertex in the edge information is inserted into the specified position of the short array in the edge storage space. The value of the short array increases by the number of inserted edges, the vertex lock is released, and the insertion process ends.

[0107] The dynamic graph data storage device for mixed workloads, wherein the insertion module includes:

[0108] According to the short array in the edge storage space, determine whether there is an idle edge storage unit position in the edge storage space, and if not, execute the expansion module;

[0109] The expansion module expands the capacity of the short array in the edge storage space, accesses the expanded short array in the edge storage space and obtains the position of the free edge storage unit in the edge storage space through the bit vector, inserts the edge connected to the source vertex in the edge information into the specified position of the short array in the edge storage space, increases the value of the short array by the number of inserted edges, releases the vertex lock, and ends the insertion process.

[0110] The dynamic graph data storage device for mixed workloads, wherein the expansion module includes:

[0111] Determine whether the capacity of the expanded short array is greater than the specified threshold. If so, sort the elements in the short array according to the destination vertex, open a sorted array, and the initial value of the stored elements at the head of the sorted array is 0, the number of elements marked for deletion is 0, and the number of storage units opened is the specified threshold. The edges of the sorted short array are stored in the sorted array, and then the short array is cleared, and the position pointer of the opened space is saved in the short array, and then the insertion module is executed again.

[0112] like Figure 7 As shown, the present invention further proposes a first electronic device A in another embodiment, which includes the dynamic graph data storage device for mixed workloads.

[0113] like Figure 8 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to collect the image data to be stored, and the information display device D is used to display the storage results, status and query results.

[0114] The information display device D can organize and process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll, etc. The user is presented with the key information specified by the user, such as news updates, system information, etc. The user can understand this information more promptly without having to visit the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present the user with rich and necessary key information.

[0115] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a readable storage medium. When the computer program is executed by a processor, the computer can execute the dynamic graph data storage method for mixed workloads provided by the above methods.

[0116] In another embodiment, the present invention further proposes a storage medium VIII for storing a computer program for executing the dynamic graph data storage method for hybrid workloads. It should be understood that the storage medium in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM).

[0117] Figure 9 A schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention is shown. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0118] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. Various programs and data required for the operation of the device 1000 can also be stored in the RAM III. The computing unit I, ROM II, and RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0119] Multiple components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard and mouse; an output unit VII, such as various types of displays and speakers; a storage medium VIII, such as a magnetic disk and optical disk; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0120] Computing unit I can be various general and / or special processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. Computing unit I performs the various methods and processes described above, such as method steps S1-S4. For example, in some embodiments, the method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the method described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0121] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A dynamic graph data storage method for mixed workloads, characterized in that: include: The initial step is to first obtain the edge information to be inserted, calculate the hash value of the source vertex in the edge information, obtain the location pointer of the hash value according to the hash table, and locate the hash bucket pointed to by the location pointer; A judgment step, according to the bit vector in the hash bucket, accessing a non-empty vertex information storage unit in the bit vector and judging whether the source vertex already exists therein, if so, executing the insertion step, otherwise executing the development step; An opening step, according to the bit vector, inserting the source vertex into an empty vertex information storage unit in the hash bucket, obtaining a vertex lock for the insertion position, opening an edge storage space for the vertex, and storing a pointer to the edge storage space in the hash bucket corresponding to the source vertex; the edge storage space includes a short array of a preset capacity and a bit vector for maintaining the short array; inserting the edge connected to the source vertex in the edge information into a specified position of the short array in the edge storage space, increasing the value of the short array by the number of inserted edges, releasing the vertex lock, and ending the insertion process; The insertion step obtains the vertex lock corresponding to the source vertex and the pointer of the source vertex to the edge storage space to access the short array in the edge storage space and obtain the position of the free edge storage unit in the edge storage space through the bit vector. The edge connected to the source vertex in the edge information is inserted into the specified position of the short array in the edge storage space. The value of the short array increases by the number of inserted edges, the vertex lock is released, and the insertion process ends.

2. The method for storing dynamic graph data for mixed workloads according to claim 1, wherein: The insertion step includes: According to the short array in the edge storage space, determining whether there is an idle edge storage unit position in the edge storage space, and if not, executing the expansion step; The expansion step expands the capacity of the short array in the edge storage space, accesses the expanded short array in the edge storage space, and obtains the position of the free edge storage unit in the edge storage space through the bit vector. The edge connected to the source vertex in the edge information is inserted into the specified position of the short array in the edge storage space. The value of the short array increases by the number of inserted edges, the vertex lock is released, and the insertion process ends.

3. The dynamic graph data storage method for mixed workloads according to claim 2, characterized in that: The expansion steps include: Determine whether the capacity of the expanded short array is greater than the specified threshold. If so, sort the elements in the short array according to the destination vertex, open a sorted array, and the initial value of the stored elements at the head of the sorted array is 0, the number of elements marked for deletion is 0, and the number of storage units opened is the specified threshold. The sorted edges of the short array are stored in the sorted array, and then the short array is cleared. The position pointer of the opened space is saved in the short array, and the insertion step is performed again.

4. The method for storing dynamic graph data for mixed workloads according to claim 2, wherein: The edge information includes a source vertex and a destination vertex, where the source vertex and the destination vertex are both user IDs in a social platform; or the source vertex and the destination vertex are respectively a seller ID and a buyer ID in a trading platform.

5. A dynamic graph data storage device for mixed workloads, characterized in that: include: In the initial module, the storage device obtains the edge information to be inserted, calculates the hash value of the source vertex in the edge information, obtains the location pointer of the hash value according to the hash table, and locates the hash bucket pointed to by the location pointer; A judgment module, according to the bit vector in the hash bucket, accesses the non-empty vertex information storage unit in the bit vector and determines whether the source vertex already exists therein, if so, executes the insertion module, otherwise executes the development module; A development module inserts the source vertex into an empty vertex information storage unit in the hash bucket according to the bit vector, obtains a vertex lock for the insertion position, develops an edge storage space for the vertex, and stores a pointer to the edge storage space in the hash bucket corresponding to the source vertex; the edge storage space includes a short array of a preset capacity and a bit vector for maintaining the short array; inserts the edge in the edge information connected to the source vertex into a specified position of the short array in the edge storage space, increases the value of the short array by the number of inserted edges, releases the vertex lock, and ends the insertion process; The insertion module obtains the vertex lock corresponding to the source vertex and the pointer of the source vertex to the edge storage space to access the short array in the edge storage space and obtain the position of the free edge storage unit in the edge storage space through the bit vector. The edge connected to the source vertex in the edge information is inserted into the specified position of the short array in the edge storage space. The value of the short array increases by the number of inserted edges, the vertex lock is released, and the insertion process ends.

6. The dynamic graph data storage device for mixed workloads according to claim 5, characterized in that: The plug-in module includes: According to the short array in the edge storage space, determine whether there is an idle edge storage unit position in the edge storage space, and if not, execute the expansion module; The expansion module expands the capacity of the short array in the edge storage space, accesses the expanded short array in the edge storage space and obtains the position of the free edge storage unit in the edge storage space through the bit vector, inserts the edge connected to the source vertex in the edge information into the specified position of the short array in the edge storage space, increases the value of the short array by the number of inserted edges, releases the vertex lock, and ends the insertion process.

7. The dynamic graph data storage device for mixed workloads according to claim 6, wherein: The extension module includes: Determine whether the capacity of the expanded short array is greater than the specified threshold. If so, sort the elements in the short array according to the destination vertex, open a sorted array, and the initial value of the stored elements at the head of the sorted array is 0, the number of elements marked for deletion is 0, and the number of storage units opened is the specified threshold. The edges of the sorted short array are stored in the sorted array, and then the short array is cleared, and the position pointer of the opened space is saved in the short array, and then the insertion module is executed again.

8. An electronic device, characterized in that: It includes a dynamic graph data storage device for mixed workloads as described in claims 5-7, and the electronic device is connected to an information display device, which is used to display the graph data storage results according to the display parameters and attributes set by the user or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the dynamic graph data storage method for mixed workloads as recited in any one of claims 1 to 4.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the dynamic graph data storage method for mixed workloads described in any one of claims 1 to 4 are implemented.

Images