Bitmap-based map data storage and query method and system, equipment and medium

Abstract

The invention relates to the field of knowledge graphs, and discloses a bitmap-based graph data storage and query method, system, device and medium, the method adopts a unified bitmap model to store graph data, converts graph structure data into a bitmap suitable for parallel computing, maps a graph query request into a bitmap operation expression tree, and stores the graph data in the bitmap operation expression tree; therefore, graph query is converted into bitmap operation, and the query performance of high-concurrency and complex graph query is remarkably improved; moreover, a reasonable computing resource scheduling decision is dynamically set according to an input graph query request, and for high-concurrency and complex graph query, thousands of cores of a large-scale parallel computing engine are used for parallel processing, so that the random access bottleneck and the CPU computing power bottleneck of traditional graph traversal are thoroughly avoided, the query speed of order of magnitude can be increased, and the query efficiency is improved. A plurality of complex atlas analysis queries can be processed at the same time, and the real-time requirement of complex atlas query can be met.

Description

Bitmap-based methods, systems, devices, and media for storing and querying graph data. Technical Field

[0001] This invention relates to the field of knowledge graph technology, and specifically discloses a bitmap-based graph data storage and query method and system, electronic device, and computer-readable storage medium. Background Technology

[0002] Knowledge graphs (or graph data), with their powerful ability to express relationships, are widely used in search engines, recommendation systems, social network analysis, and other fields. Currently, the storage and retrieval of graph data mainly employs the following two methods:

[0003] 1) Adjacency List Storage and Traversal Computation Based on Graph Databases: Taking native graph databases like Neo4j as an example, they use adjacency lists or property graph models to store data. When performing complex queries (such as multi-hop queries, common neighbor calculations, etc.), the system needs to start from the initial node and traverse adjacent nodes hop by hop using pointers until the query conditions are met. The advantage of this method is its intuitiveness and conformity to the abstraction of graph theory. However, when performing complex queries, graph databases require a large amount of random memory access to follow the pointers. Random access overhead is high, CPU cache hit rate is low, resulting in high query latency. Especially when dealing with deep jumps or high fan-out nodes, performance drops sharply and cannot meet the real-time requirements of queries.

[0004] 2) Bitmap Index-Assisted Queries in Relational Databases: In analytical databases such as ClickHouse and Druid, bitmap indexes are created for low-cardinality columns (e.g., enumerated fields like gender and city) to accelerate filtering queries on massive datasets. When performing multi-condition queries (e.g., city = Beijing AND occupation = programmer), the system can quickly retrieve the bitmap indexes for each condition and perform bitwise operations such as AND and OR on the CPU to obtain the final set of row IDs that meet the conditions. This method is extremely efficient for multi-dimensional filtering queries. However, relational databases with bitmap indexes are not specifically designed for complex graph structure queries; they only support simple graph queries and are difficult to implement complex ones. Furthermore, although they can efficiently handle multi-dimensional filtering through bitmap indexes, when the data volume is extremely large, the size of the bitmap itself may reach GB or even TB levels. Performing intersection, union, and difference operations on these ultra-large bitmaps on the CPU, even with compression technologies such as Roaring Bitmap and SIMD instruction set optimizations, still cannot meet the requirements of low-latency, high-concurrency queries due to their inherent serial and limited parallel processing modes.

[0005] Therefore, existing map data query methods cannot meet the requirements of real-time performance, complexity, and efficiency in map queries. Summary of the Invention

[0006] This invention provides a bitmap-based method and system for storing and querying graph data, as well as an electronic device and a computer-readable storage medium. It can significantly improve the query performance of high-concurrency and complex graph queries, meet the real-time requirements of complex graph queries, and improve versatility.

[0007] According to one aspect of the present invention, a method for storing and querying map data based on bitmaps is provided, comprising the following:

[0008] Receive map query requests;

[0009] A bitmap operation expression tree is generated based on the graph query request, mapping the abstract query conditions to the basic operation sequence of the bitmap;

[0010] Dynamic computation scheduling decisions are made based on bitmap computation expression trees, and the bitmap computation expression trees are allocated to CPU computing engines or massively parallel computing engines according to the computation scheduling decisions.

[0011] The CPU computing engine or massively parallel computing engine performs query tasks in the graph database based on the bitmap operation expression tree and returns the query results; wherein, the graph database uses a unified bitmap model to store all graph data.

[0012] Furthermore, the process of making dynamic computation scheduling decisions based on the bitmap operation expression tree, and allocating the bitmap operation expression tree to the CPU computing engine or the massively parallel computing engine according to the computation scheduling decisions, includes the following:

[0013] The total amount of data and parallelism of the bitmap to be processed are estimated based on the bitmap operation expression tree. If the total amount of data is less than or equal to the preset data amount threshold or the parallelism is less than or equal to the preset parallelism threshold, the bitmap operation expression tree is assigned to the CPU computing engine. If the total amount of data is greater than the preset data amount threshold or the parallelism is greater than the preset parallelism threshold, the bitmap operation expression is assigned to the large-scale parallel computing engine.

[0014] Furthermore, the process of making dynamic computation scheduling decisions based on the bitmap operation expression tree and allocating the bitmap operation expression tree to the CPU computing engine or the massively parallel computing engine according to the computation scheduling decisions also includes the following:

[0015] The current task queues of the CPU computing engine and the massively parallel computing engine are monitored in real time. If the current task queue of one engine is full, the bitmap operation expression tree is assigned to another engine.

[0016] Furthermore, the unified bitmap model includes an attribute and entity bitmap area, a relation bitmap matrix area, and a metadata area. The attribute and entity bitmap area configures a bitmap for each entity type. The relation bitmap matrix area maintains a logical two-dimensional bitmap matrix for each relation. The rows and columns of the matrix correspond to the global IDs of the entities. If a relation exists between two entities, the corresponding element in the matrix is ​​set to 1; otherwise, it is set to 0. The metadata area maintains a data dictionary, which includes global ID allocation, the mapping relationship between bitmaps and entities, and the mapping relationship between bitmaps and relations.

[0017] Furthermore, the attribute and entity bitmap area configures a bitmap for each discrete value or value range of each attribute.

[0018] Furthermore, the two-dimensional bitmap matrix is ​​stored in blocks, and each block is compressed for storage.

[0019] Furthermore, when compressing and storing each block, the proportion of zero elements in each block is calculated. If the proportion of zero elements in a block is greater than a preset proportion threshold, coordinate storage is used; otherwise, bitmap storage is used.

[0020] In addition, the present invention also provides a bitmap-based spectral data storage and query system, comprising:

[0021] The map query input module is used to receive map query requests;

[0022] The graph query request mapping module is used to generate a bitmap operation expression tree based on the graph query request, mapping the abstract query conditions to the basic operation sequence of the bitmap;

[0023] The computation scheduling decision-making module is used to make dynamic computation scheduling decisions based on the bitmap operation expression tree, and allocate the bitmap operation expression tree to the CPU computation engine or the massively parallel computing engine according to the computation scheduling decision.

[0024] The graph data query module is used by the CPU computing engine or the massively parallel computing engine to perform query tasks in the graph database based on the bitmap operation expression tree and return the query results; wherein, the graph database uses a unified bitmap model to store all graph data.

[0025] In addition, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the steps of the method described above by calling the computer program stored in the memory.

[0026] In addition, the present invention provides a computer-readable storage medium for storing a computer program for storing and querying spectral data based on bitmaps, wherein the computer program executes the steps of the method described above when running on a computer.

[0027] This invention presents a bitmap-based graph data storage and query method. By employing a unified bitmap model to store graph data, it transforms graph structure data into bitmaps suitable for parallel computing. Furthermore, by mapping graph query requests to bitmap operation expression trees, it converts graph queries into bitmap operations, significantly improving the query performance of high-concurrency, complex graph queries. Moreover, it dynamically sets reasonable computing resource scheduling decisions based on the input graph query requests, significantly improving bit operation efficiency and further enhancing query performance. Especially for high-concurrency, complex graph queries (such as multi-hop queries and large-scale common neighbor calculations), it utilizes thousands of cores of a massively parallel computing engine for parallel processing, completely avoiding the random access bottleneck and CPU computing power bottleneck of traditional graph traversal, achieving orders-of-magnitude improvements in query speed. The high throughput of the massively parallel computing engine enables the system to handle multiple complex graph analysis queries simultaneously and meets the real-time requirements of complex graph queries. In addition, it provides a unified storage and computing architecture that can efficiently handle both simple attribute filtering queries and complex topology queries, improving versatility.

[0028] In addition, the bitmap-based map data storage and query system of the present invention also has the above-mentioned advantages.

[0029] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the figures. Attached Figure Description

[0030] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0031] Figure 1 is a flowchart illustrating a bitmap-based spectral data storage and query method according to a preferred embodiment of this application.

[0032] Figure 2 is a schematic diagram of the module structure of a bitmap-based map data storage and query system according to another embodiment of this application. Detailed Implementation

[0033] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] This application is primarily used in a next-generation data application architecture, which specifically includes three nodes: data source and ingestion nodes, a next-generation data platform / middleware, and business application layer nodes. The data source and ingestion nodes are the sources of graph data, including but not limited to business relational databases (such as MySQL), real-time log streams (such as Kafka), and existing structured knowledge graph files (such as RDF format), representing the raw data input. This application mainly obtains data from these data sources through data integration tools or ETL processes, then transforms and loads it into the storage layer (i.e., the graph database). The business application layer nodes are systems that directly face end users or handle core business processes, such as recommendation microservices, risk control engines, and visualization management platforms. These application nodes submit complex graph query requests to this application through standard API interfaces (such as GraphQL, gRPC, or RESTful API). The next-generation data platform / middleware, as a unified data infrastructure layer, integrates various data storage and computing capabilities, mainly providing consistent data services for upper-layer businesses. It includes three components: data lake / warehouse, graph computing engine, and real-time computing engine. This application embeds the graph computing engine, which works in collaboration with components such as data lake / warehouse and real-time computing engine. The data lake / warehouse is responsible for offline batch analysis of massive historical data, while the real-time computing engine is responsible for streaming data processing. The graph computing engine focuses on solving the problem of real-time deep query of relational data, thereby empowering various upper-layer business applications to achieve advanced functions such as real-time recommendation and instant risk identification, which were previously difficult to achieve. The real-time computing engine is a dedicated accelerator in the entire architecture. Its core responsibility is to receive query requests from the business application layer involving complex relational relationships (multi-hop queries, common neighbors, dense subgraph discovery, etc.) and return query results at extremely high speed using this application. The real-time computing engine relies on other components of the data platform to obtain raw data and provides dedicated, high-performance graph query computing services for various business applications. It fills the gaps in traditional data warehouses' inability to handle complex graph queries and traditional graph databases' inability to cope with real-time analysis of ultra-large-scale data.

[0035] Referring to Figure 1, a preferred embodiment of this application provides a bitmap-based spectral data storage and query method, including the following:

[0036] Step S1: Receive a map query request.

[0037] In step S1, the user submits a graph query request through an interactive query terminal (real-time query interface). The interactive query terminal is a visual graph exploration tool or a business intelligence dashboard. The user submits ad-hoc queries through the terminal and quickly obtains visual results. Alternatively, the user can provide high-performance batch analysis tasks to upstream business systems (such as recommendation systems, anti-fraud engines, etc.) through a data service interface. The upstream business systems can execute complex graph query logic in batches to support their core business by calling API interfaces.

[0038] Specifically, the system receives map query requests through a real-time query interface or a data service interface.

[0039] Step S2: Generate a bitmap operation expression tree based on the graph query request, mapping the abstract query conditions to the basic operation sequence of the bitmap.

[0040] In step S2, since the input graph query request is usually a standard graph query statement, such as an extension of Cypher or Gremlin, or a custom SQL, the system will convert the input high-level query language (standard graph query statement) into an efficient low-level execution plan after receiving the graph query request. For example, a graph query request of "find friends in Beijing who are older than 30" can be converted into: (entity type bitmap [User]) AND (attribute bitmap [city=Beijing]) AND (attribute bitmap [age>30]); a graph query request of "find user A's two-hop friends" will be converted into a "multiplication" operation on the relation bitmap matrix (simulated through bitwise operations).

[0041] Specifically, the input standard graph query statement is first subjected to lexical and syntactic analysis to understand the query semantics. Then, the query is rewritten and optimized by an optimizer, adjusting the operation order to minimize the amount of intermediate data. Abstract query conditions (such as attribute filtering, N-degree relation traversal, etc.) are mapped to basic bitmap operation sequences, generating a bitmap operation expression tree. The lexical analysis, syntactic analysis, and optimizer mentioned above are all existing technologies and will not be elaborated on here.

[0042] Step S3: Make dynamic computation scheduling decisions based on the bitmap operation expression tree, and allocate the bitmap operation expression tree to the CPU computing engine or the massively parallel computing engine according to the computation scheduling decisions.

[0043] In step S3, after mapping the abstract query conditions to the basic operation sequence of the bitmap, computational resource scheduling is required to maximize system throughput and reduce average query latency. Specifically, this involves evaluating the bitmap operation expression tree using multi-dimensional metrics, making dynamic computational scheduling decisions based on the evaluation results, and allocating the bitmap operation expression tree to either the CPU computing engine or the massively parallel computing engine based on these decisions. The massively parallel computing engine can utilize GPUs, many-core CPUs, FPGAs, dedicated AI chips, etc., and has a stronger ability to process bitwise operations in parallel compared to the CPU computing engine.

[0044] The process of making dynamic computation scheduling decisions based on bitmap operation expression trees and allocating the bitmap operation expression trees to CPU computing engines or massively parallel computing engines according to the computation scheduling decisions includes the following:

[0045] The total amount of data and parallelism of the bitmap to be processed are estimated based on the bitmap operation expression tree. If the total amount of data is less than or equal to the preset data amount threshold or the parallelism is less than or equal to the preset parallelism threshold, the bitmap operation expression tree is assigned to the CPU computing engine. If the total amount of data is greater than the preset data amount threshold or the parallelism is greater than the preset parallelism threshold, the bitmap operation expression is assigned to the large-scale parallel computing engine.

[0046] Specifically, the total amount of data and parallelism of the bitmap to be processed are first estimated based on the bitmap operation expression tree, i.e., multi-dimensional indicator evaluation. The specific estimation process is existing technology and will not be elaborated here. If the total amount of data is less than or equal to a preset data volume threshold, the query task is determined to be a small-scale operation, and the bitmap operation expression tree is assigned to the CPU computing engine to perform bit operations. If the parallelism is less than or equal to a preset parallelism threshold, the query task is determined to be a simple operation, such as a control-intensive operation or a sequential operation, and the bitmap operation expression tree is assigned to the CPU computing engine to perform bit operations. If the total amount of data is greater than the preset data volume threshold, the query task is determined to be a large-scale operation, and the computing power of the CPU computing engine is limited. In this case, the bitmap operation expression is assigned to a more powerful large-scale parallel computing engine to achieve efficient operation. If the parallelism is greater than the preset parallelism threshold, the query task is determined to be a complex operation, such as a highly parallelized computationally intensive operation (such as AND, OR), and the bitmap operation expression is assigned to a large-scale parallel computing engine to achieve more efficient parallel operation. Therefore, for small-scale or simple computational tasks, they are assigned to the CPU computing engine for execution, while for large-scale or complex computational tasks, they are assigned to the massively parallel computing engine. The reasonable computing resource scheduling decision can be dynamically set according to the input graph query request, which can significantly improve bit operation efficiency and thus improve query performance.

[0047] In other embodiments of the present invention, only a massively parallel computing engine may be configured, with all computational tasks executed by the massively parallel computing engine. Of course, a computing architecture that combines a CPU computing engine and a massively parallel computing engine has greater advantages. It makes the system more robust, capable of handling fragmented, control-intensive tasks unsuitable for massively parallel computing engines. Through dynamically formulated computing resource scheduling decisions, the system can more intelligently utilize heterogeneous resources, reducing latency and resource consumption in small query scenarios, improving overall system efficiency, and forming the advantage of collaborative computing.

[0048] Optionally, the process of making dynamic computation scheduling decisions based on the bitmap operation expression tree and allocating the bitmap operation expression tree to the CPU computing engine or the massively parallel computing engine according to the computation scheduling decisions also includes the following:

[0049] The current task queues of the CPU computing engine and the massively parallel computing engine are monitored in real time. If the current task queue of one engine is full, the bitmap operation expression tree is assigned to another engine.

[0050] Understandably, when making dynamic computation scheduling decisions, the current task queues of the CPU computing engine and the massively parallel computing engine can be monitored in real time. When the CPU computing engine's current task queue is full, but the massively parallel computing engine's current task queue is not full, the bitmap operation expression tree is assigned to the massively parallel computing engine for execution; conversely, when the CPU computing engine's current task queue is not full, but the massively parallel computing engine's current task queue is full, the bitmap operation expression tree is assigned to the CPU computing engine for execution, thus achieving load balancing. For example, for a large-scale computation task, if the massively parallel computing engine's current task queue is full, it can be rolled back to the CPU computing engine for execution. Alternatively, the large-scale computation task can be broken down into multiple smaller computational subtasks before rolling back to the CPU computing engine. This invention considers the load balancing of the two heterogeneous computing engines when making computation scheduling decisions, which is beneficial for further improving the execution efficiency of bit operations.

[0051] Step S4: The CPU computing engine or massively parallel computing engine performs a query task in the graph database based on the bitmap operation expression tree and returns the query results; wherein, the graph database uses a unified bitmap model to store all graph data.

[0052] In step S4, the CPU computing engine's bitmap operation library contains a highly optimized bitmap algorithm that fully utilizes the SIMD instruction set of modern CPUs (such as AVX-512) to accelerate bit operations, enabling efficient execution of small-scale or simple computational tasks. Meanwhile, a massively parallel computing engine, such as a GPU, performs parallel computation through thousands of cores, efficiently executing computationally intensive large-scale bitmap parallel computations. Furthermore, to achieve high-speed data transfer between the CPU and GPU, the GPU uses paged memory and manages multiple asynchronous data transfer streams. The CPU computing engine or massively parallel computing engine performs bitmap operations based on a bitmap operation expression tree to perform query tasks in the graph database. The graph database decodes the resulting bitmap into a list of entity IDs, which is then returned as the query result to the user or upstream business system.

[0053] Furthermore, the graph database uses a unified bitmap model to store all graph data. For large-scale or complex computing tasks, the graph database transforms graph structure data into bitmaps suitable for parallel computing, thereby transforming graph queries into bitmap operations, providing a data foundation for improving the query performance of high-concurrency and complex graph queries. The unified bitmap model specifically includes an attribute and entity bitmap area, a relationship bitmap matrix area, and a metadata area. The attribute and entity bitmap area configures a bitmap for each entity type (e.g., User, Product, etc.). A bit in the bitmap being 1 indicates that the entity with the corresponding global ID belongs to that entity type. In the relationship bitmap matrix area, a logical two-dimensional bitmap matrix is ​​maintained for each relationship (e.g., Follows). The rows and columns of the matrix correspond to the global IDs of the entities. If a relationship exists between two entities, the corresponding element in the matrix is ​​set to 1; otherwise, it is set to 0. For example, if there is a Follows relationship between entity i and entity j, the element in the i-th row and j-th column of the matrix is ​​set to 1; otherwise, it is set to 0. The metadata area maintains the data dictionary for the entire system. This data dictionary includes global ID allocation, bitmap-entity mapping relationships, and bitmap-relationship mapping relationships, ensuring accurate encoding and decoding.

[0054] Understandably, this invention not only assigns a global ID to each entity, but also configures a bitmap for each entity type and a two-dimensional bitmap matrix for each relationship. This allows graph structure data to be accurately transformed into bitmaps suitable for parallel computing, enabling graph queries to be transformed into bitmap operations suitable for parallel computing, thus greatly improving the query performance of high-concurrency and complex graph queries.

[0055] Optionally, the attribute and entity bitmap area can also be configured with a bitmap for each discrete value or value range of each attribute (e.g., city = Beijing, age > 30) to achieve a more efficient inverted index, which is beneficial to further improve query performance.

[0056] Furthermore, since each relationship corresponds to a huge two-dimensional bitmap matrix, when the graph undergoes a small change (such as adding an edge), the entire two-dimensional bitmap matrix needs to be updated and reconstructed. Optionally, in this invention, the two-dimensional bitmap matrix adopts a block-based storage method, and each block is compressed for storage. When the graph undergoes a small change, it is not necessary to reconstruct the entire bitmap; only the affected blocks need to be updated, thus improving the efficiency of incremental updates. When compressing and storing each block, the sparsity of each block is evaluated based on the proportion of zero elements. Specifically, the proportion of zero elements in each block is first calculated. If the proportion of zero elements in a block is greater than a preset proportion threshold, the block is determined to be a sparse block, and coordinate storage is used directly for sparse blocks. If the proportion of zero elements in a block is less than or equal to the preset proportion threshold, the block is determined to be a dense block, and bitmap storage is used. This is beneficial for further optimizing storage and computation efficiency, enabling efficient processing of sparse matrices, and greatly saving storage space when storing sparse graphs. When compressing and storing each block, existing compression encoding techniques such as Roaring Bitmap are used. Compared with traditional adjacency list storage, this can greatly save storage space when storing sparse graphs.

[0057] Understandably, the bitmap-based graph data storage and query method in this embodiment uses a unified bitmap model to store graph data, transforming graph structure data into bitmaps suitable for parallel computing. Furthermore, by mapping graph query requests to bitmap operation expression trees, graph queries are transformed into bitmap operations, significantly improving the query performance of high-concurrency, complex graph queries. Moreover, it dynamically sets reasonable computing resource scheduling decisions based on the input graph query requests, significantly improving bit operation efficiency and further enhancing query performance. Especially for high-concurrency, complex graph queries (such as multi-hop queries and large-scale common neighbor calculations), it utilizes thousands of cores of a large-scale parallel computing engine for parallel processing, completely avoiding the random access bottleneck and CPU computing power bottleneck of traditional graph traversal, achieving orders-of-magnitude improvements in query speed. The high throughput of the large-scale parallel computing engine enables the system to handle multiple complex graph analysis queries simultaneously and also meets the real-time requirements of complex graph queries. In addition, it provides a unified storage and computing architecture that can efficiently handle both simple attribute filtering queries and complex topology queries, improving versatility.

[0058] In addition, as shown in Figure 2, another embodiment of the present invention also provides a bitmap-based spectral data storage and query system, preferably employing the bitmap-based spectral data storage and query method described above, including:

[0059] The map query input module is used to receive map query requests;

[0060] The graph query request mapping module is used to generate a bitmap operation expression tree based on the graph query request, mapping the abstract query conditions to the basic operation sequence of the bitmap;

[0061] The computation scheduling decision-making module is used to make dynamic computation scheduling decisions based on the bitmap operation expression tree, and allocate the bitmap operation expression tree to the CPU computation engine or the massively parallel computing engine according to the computation scheduling decision.

[0062] The graph data query module is used by the CPU computing engine or the massively parallel computing engine to perform query tasks in the graph database based on the bitmap operation expression tree and return the query results; wherein, the graph database uses a unified bitmap model to store all graph data.

[0063] It is understood that the bitmap-based graph data storage and query system of this embodiment uses a unified bitmap model to store graph data, transforming graph structure data into bitmaps suitable for parallel computing. Furthermore, by mapping graph query requests to bitmap operation expression trees, it transforms graph queries into bitmap operations, significantly improving the query performance of high-concurrency, complex graph queries. Moreover, it dynamically sets reasonable computing resource scheduling decisions based on the input graph query requests, significantly improving bit operation efficiency and further enhancing query performance. Especially for high-concurrency, complex graph queries (such as multi-hop queries and large-scale common neighbor calculations), it utilizes thousands of cores of a large-scale parallel computing engine for parallel processing, completely avoiding the random access bottleneck and CPU computing power bottleneck of traditional graph traversal, achieving orders-of-magnitude improvements in query speed. The high throughput of the large-scale parallel computing engine enables the system to handle multiple complex graph analysis queries simultaneously and also meets the real-time requirements of complex graph queries. In addition, it provides a unified storage and computing architecture that can efficiently handle both simple attribute filtering queries and complex topology queries, improving versatility.

[0064] In addition, another embodiment of the present invention provides an electronic device including a processor and a memory, wherein the memory stores a computer program, and the processor executes the steps of the method described above by calling the computer program stored in the memory.

[0065] In addition, another embodiment of the present invention provides a computer-readable storage medium for storing a computer program for storing and querying map data based on bitmaps, wherein the computer program executes the steps of the method described above when running on a computer.

[0066] Common computer-readable storage media include: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tape, any other physical media with perforated patterns, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), flash erasable programmable read-only memory (FLASH-EPROM), any other memory chips or cartridges, or any other media readable by a computer. Instructions may further be transmitted or received by a transmission medium. The term transmission medium can include any tangible or intangible medium used to store, encode, or carry instructions for execution by a machine, and includes digital or analog communication signals or intangible media that facilitate communication of such instructions. Transmission media include coaxial cables, copper wires, and optical fibers, which contain conductors for transmitting a bus of computer data signals.

[0067] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0068] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0069] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0070] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0071] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0072] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0073] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for storing and querying spectral data based on bitmaps, characterized in that, Includes the following: receiving map query requests; A bitmap operation expression tree is generated based on the graph query request, mapping the abstract query conditions to the basic operation sequence of the bitmap; Dynamic computation scheduling decisions are made based on bitmap operation expression trees, and the bitmap operation expression trees are allocated to CPU computing engines or massively parallel computing engines according to the computation scheduling decisions; the CPU computing engine or massively parallel computing engine performs query tasks in the graph database based on the bitmap operation expression trees and returns the query results; wherein, the graph database uses a unified bitmap model to store all graph data.

2. The bitmap-based graph data storage and query method as described in claim 1, characterized in that, The process of making dynamic computation scheduling decisions based on bitmap operation expression trees and allocating the bitmap operation expression trees to CPU computing engines or massively parallel computing engines according to the computation scheduling decisions includes the following: The total amount of data and parallelism of the bitmap to be processed are estimated based on the bitmap operation expression tree. If the total amount of data is less than or equal to the preset data amount threshold or the parallelism is less than or equal to the preset parallelism threshold, the bitmap operation expression tree is assigned to the CPU computing engine. If the total amount of data is greater than the preset data amount threshold or the parallelism is greater than the preset parallelism threshold, the bitmap operation expression is assigned to the large-scale parallel computing engine.

3. The bitmap-based spectral data storage and query method as described in claim 2, characterized in that, The process of making dynamic computation scheduling decisions based on bitmap operation expression trees and allocating bitmap operation expression trees to CPU computing engines or massively parallel computing engines according to computation scheduling decisions also includes the following: The current task queues of the CPU computing engine and the massively parallel computing engine are monitored in real time. If the current task queue of one engine is full, the bitmap operation expression tree is assigned to another engine.

4. The bitmap-based graph data storage and query method as described in claim 1, characterized in that, The unified bitmap model includes an attribute and entity bitmap area, a relation bitmap matrix area, and a metadata area. The attribute and entity bitmap area configures a bitmap for each entity type. The relation bitmap matrix area maintains a logical two-dimensional bitmap matrix for each relation. The rows and columns of the matrix correspond to the global IDs of the entities. If a relation exists between two entities, the corresponding element in the matrix is ​​set to 1; otherwise, it is set to 0. The metadata area maintains a data dictionary, which includes global ID allocation, the mapping relationship between bitmaps and entities, and the mapping relationship between bitmaps and relations.

5. The bitmap-based spectral data storage and query method as described in claim 4, characterized in that, The attribute and entity bitmap area configures a bitmap for each discrete value or value range of each attribute.

6. The bitmap-based spectral data storage and query method as described in claim 4, characterized in that, The two-dimensional bitmap matrix is ​​stored in blocks, and each block is compressed.

7. The bitmap-based spectral data storage and query method as described in claim 6, characterized in that, When compressing and storing each block, the proportion of zero elements in each block is calculated. If the proportion of zero elements in a block is greater than a preset proportion threshold, coordinate storage is used; otherwise, bitmap storage is used.

8. A bitmap-based spectral data storage and query system, characterized in that, include: The map query input module is used to receive map query requests; The graph query request mapping module is used to generate a bitmap operation expression tree based on the graph query request, mapping the abstract query conditions to the basic operation sequence of the bitmap; The computation scheduling decision-making module is used to make dynamic computation scheduling decisions based on the bitmap operation expression tree, and allocate the bitmap operation expression tree to the CPU computing engine or the massively parallel computing engine according to the computation scheduling decision; the graph data query module is used to enable the CPU computing engine or the massively parallel computing engine to perform query tasks in the graph database based on the bitmap operation expression tree, and return the query results; wherein, the graph database uses a unified bitmap model to store all graph data.

9. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the steps of the method as described in any one of claims 1 to 7 by calling the computer program stored in the memory.

10. A computer-readable storage medium for storing a computer program for storing and querying map data based on bitmaps, characterized in that, The computer program, when run on a computer, performs the steps of the method as described in any one of claims 1 to 7.

Images