Query performance evaluation method and system for multi-model database

Abstract

The invention belongs to the technical field of database performance evaluation, and discloses a query performance evaluation method and system for a multi-model database. The method comprises the following steps of: constructing a test data set coexisting with four models, namely a relationship, a document, a graph and a vector, by an open-source data source and a native model, and performing global entity ID semantic association; four types of cross-model query workloads are defined, point query, aggregation, vector ANN and graph matching scenes are covered, and the method is used for systematically evaluating the comprehensive capacity of the multi-model database in the aspects of connection sequence selection, model data conversion cost estimation, aggregation operation optimization and the like. During testing, under the conditions of fixed hardware, a memory buffer area and concurrent parameters, the workload is automatically executed, the median execution time is collected, then the performance score of each load is calculated based on a logarithm normalization speed-up ratio model, and fair quantitative comparison among different multi-model databases is achieved. The method is reproduced in a common multi-model database system to verify the feasibility of the evaluation benchmark.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database performance evaluation, and particularly relates to a query performance evaluation method and system for a multi-model database. BACKGROUND

[0002] With the deepening of big data and artificial intelligence application scenarios, data forms are evolving from a single model to a multi-model. Heterogeneous models such as relations, documents, graphs, and vectors often exist in the same business system. To meet the organization and management needs of multi-model, the traditional "polyglot persistence" strategy needs to select, deploy, optimize and maintain a special database for each model, which in turn causes problems such as data silos, high cross-model query complexity, performance degradation, and a sharp rise in operation and maintenance costs. Multi-model databases (MMDB) support multiple data models in a single kernel or unified query engine, significantly reducing system complexity, and have become the focus of the academic and industrial communities as the next generation of data management platforms.

[0003] However, MMDB brings new challenges to the performance evaluation framework while bringing architectural convenience. The current multi-model database evaluation benchmark is scarce and has obvious defects: at the data set construction level, a single model benchmark generator is often used first (for example, the E-Commerce scenario of M2Bench reuses the relational data of TPC-DS, and UniBench reuses the graph data of LDBC-SNB), and then the data is derived or converted into other models through rules, rather than starting directly from the multi-model perspective of the same entity in real business. Therefore, it is difficult to restore the multi-model characteristics that the same object naturally has in the real world. At the same time, the existing benchmark does not cover the key vector model in the era of large models, making it unable to effectively evaluate the performance of mixed queries involving vector similarity search combined with graph / structured data, forming an obvious evaluation blind area. In addition, part of the data set is not completely open source and needs to be registered or paid for use (M2Bench's DrugBank data set); at the workload level, the query lacks scientific classification based on core features, the semantics is simple, and key model operations such as relational window aggregation, graph high-order algorithm, document field existence and inclusion relationship detection, and vector similarity nearest neighbor search are missing, resulting in limited scene applicability; at the evaluation index level, there is a lack of unified quantitative standard. The existing evaluation mainly relies on the absolute response time of each query, but due to the time difference of different types of queries, which can reach several orders of magnitude, it is difficult to directly list the time data to intuitively reflect the comprehensive performance of the system, which makes it difficult to make a comprehensive comparison between different database systems. Therefore, there is an urgent need for a query performance evaluation method for MMDB, which can construct a multi-model evaluation benchmark with clear semantics, complete models, diverse queries, and scalability based on real scenario data sets, filling the technical gap in the current multi-model database evaluation field. SUMMARY

[0004] The present application proposes a query performance evaluation method and system for multi-model databases, aiming to solve the systematic defects of existing evaluation benchmarks in terms of query scene reality, data model coverage integrity, key operation completeness, and evaluation index uniformity. A multi-model data set with native cross-model capability and semantic association is constructed, and a complex workload is designed based on the data set to evaluate the key technical capabilities of multi-model databases in cross-model query optimization, including connection order selection, model data conversion cost estimation, and aggregation operation optimization. Under the conditions of fixed hardware configuration, data set size, query statement version, and concurrency parameters, the workload is repeatedly executed multiple times to collect performance data, obtaining statistically significant evaluation results, providing quantifiable and reproducible benchmark basis for performance evaluation and optimization of multi-model database systems.

[0005] The technical scheme of the present application is as follows: a query performance evaluation method for multi-model databases, comprising the following steps:

[0006] S1. Construct a multi-model test dataset based on the OpenAlex open-source academic dataset; the multi-model test dataset contains model data of relationships, documents, graphs, and vectors in the same semantic space, and the four types of model data maintain cross-model semantic association through globally unique entity IDs;

[0007] S2. Import the multi-model test dataset into the multi-model database to be tested, and uniformly configure the environment of the database to be tested, including cache size, parallelism, log level, and index strategy;

[0008] S3. Construct four types of cross-model query workloads; each type of cross-model query workload contains multiple query templates, each of which spans at least two types of model data and sequentially forms a semantically coherent query task chain, used to evaluate the comprehensive ability of the multi-model database to be tested in connection order selection, model data conversion cost estimation, and aggregation operation optimization;

[0009] S4. Under the conditions of fixed hardware, dataset size, query statement version, and concurrency parameters, repeatedly execute the cross-model query workload and collect the median execution time of each query statement, then calculate the performance score of each type of cross-model query workload based on the logarithmic normalized speedup ratio model, and obtain the horizontal performance comparison between different multi-model databases according to the performance score.

[0010] The step S1 is specifically:

[0011] S1.1 Obtain the OpenAlex data snapshot with CC0 license;

[0012] S1.2 Split the four types of core entities of papers, authors, topics, and institutions into relationship model data, document model data, graph model data, and vector model data;

[0013] S1.3 Assign globally unique IDs in large integer format to papers, authors, topics, and institutions, and keep the global IDs as cross-model connection keys in the four types of model data to ensure that the data has native cross-model query capability.

[0014] The relationship model data stores entity basic fields; the document model data stores nested or dynamic schema fields; the graph model data stores entity relationships, converting cooperative relationships, reference relationships, and ownership relationships into isomorphic or heterogeneous edge tables; and the vector model data uses BERT model to embed topic descriptions and article abstracts to form a vector table.

[0015] The step S2 is specifically:

[0016] S2.1 Install the multi-model database system to be tested; install the corresponding database distribution. If it depends on plugins to extend the multi-model query capability, install the corresponding plugins simultaneously so that the interfaces of each model are available within the same instance.

[0017] S2.2 Unify the settings for memory buffer size, query parallelism, and log level parameters;

[0018] S2.3 Based on the import requirements of the multi-model database under test, a data preprocessing script can be optionally written to parse the nested JSON fields in the CSV and complete the source-target node pointing fields required for the edge table data, and output them in JSONL format, thereby enabling the multi-model database under test to complete the batch import.

[0019] S2.4 Create consistent indexes on primary key, foreign key, and vector fields respectively to ensure that acceleration conditions are equivalent across systems.

[0020] In step S3, the four types of cross-model query workloads are:

[0021] Hybrid-IO class: Cross-model detail extraction query with point lookup as the main function, used to evaluate hybrid IO throughput;

[0022] Aggregation-Group class: Statistical analysis queries involving grouping, window functions, and sorting, used to evaluate CPU and memory bandwidth bottlenecks in aggregation and sorting scenarios;

[0023] Vector-ANN class: It adds relational, graph or document predicate filtering to vector nearest neighbor search to evaluate the recall-throughput tradeoff;

[0024] The Graph-Matching class matches specific subgraphs using graph patterns and associates them with other models to supplement semantics. It is used to evaluate cache invalidation and traversal pruning efficiency.

[0025] The log-normalized speedup model is as follows:

[0026]

[0027] in, The shortest median execution time for query q across all tested systems. This represents the median execution time of system s on query q. The number of queries is for load category C; the performance score is used to quantify the average logarithmic speedup of different databases relative to their extreme performance, in order to eliminate the impact of differences in query volume on the scoring results, reduce the weight of extreme long-tail values, and achieve horizontal comparison against outlier interference.

[0028] A query performance evaluation system for multi-model databases, implementing a method for evaluating query performance for multi-model databases, including:

[0029] The dataset building module is used to generate native multi-model test datasets;

[0030] The data import module is used to import data into the system under test and configure a unified environment.

[0031] The workload module is used to define and instantiate specific multi-model query workloads;

[0032] The performance acquisition and scoring module is used to collect median execution time and output log-normalized scores, enabling horizontal comparison and optimization verification of different multi-model databases.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] Firstly, this invention, based on the OpenAlex open-source academic dataset, constructs a native multi-model dataset covering four mainstream model types: relational, document, graph, and vector. Unlike the traditional benchmark's construction path of "single model generation + rule derivation," this invention directly starts from the multi-perspective description of core entities in real-world business scenarios, naturally mapping different dimensions of the same object to heterogeneous model data. This accurately restores the multi-model features and complete semantic relationships inherent in real-world objects. This method fundamentally avoids the semantic fragmentation and data distortion caused by artificial synthesis, realistically simulating the multi-model collaboration needs in actual business scenarios, and significantly improving the representativeness and credibility of the evaluation results.

[0035] Secondly, this invention proposes a cross-model query workload system based on core computational feature classification. This system systematically covers commonly used operators across various models and abstracts the workload into four core categories: Hybrid-IO, Aggregation-Group, Vector-ANN, and Graph-Matching. This system not only fills the gap in vector evaluation for multi-model hybrid query scenarios but also incorporates key operators previously overlooked in multi-model benchmarking, such as relation window aggregation, higher-order graph algorithms, and document field probing, into the same test set. This makes the evaluation results more technically in-depth and comprehensive, enabling a complete assessment of the processing capabilities of multi-model databases in complex query scenarios and promoting technological advancements in multi-model collaborative optimization.

[0036] Third, this invention pre-defines query statement templates, data import scripts, and index creation schemes for mainstream multi-model databases such as ArangoDB and AgensGraph, supporting users to expand as needed. Under fixed hardware, dataset size, query statement version, and concurrency parameters, it can generate statistically significant evaluation results by repeatedly and automatically executing workloads and collecting performance data, thereby lowering the evaluation threshold and achieving standardization, automation, and reproducibility of the evaluation process. This significantly improves the scientific rigor and fairness of performance comparisons between different systems, providing a reliable basis for the selection and optimization of multi-model databases.

[0037] Fourth, this invention introduces a cross-model performance scoring model based on log normalization in terms of evaluation indicators. It unifies the time consumption of four types of cross-model query paradigms into the average log speedup ratio relative to the limit performance, forming a horizontally comparable quantitative scoring system that is resistant to outlier interference. This enables fair comparison between different systems and provides a unified and interpretable indicator basis for the performance evaluation of multi-model databases. Attached Figure Description

[0038] Figure 1 A detailed pattern information graph for a multi-model dataset;

[0039] Figure 2 A working example diagram of the task chain load for querying V1;

[0040] Figure 3 This is a flowchart illustrating the specific workflow of the present invention. Detailed Implementation

[0041] To achieve the above objectives, the present invention adopts the following technical measures in terms of dataset construction, model selection, and workload design:

[0042] Firstly, in terms of dataset construction, this invention selects datasets from the public domain that are clearly licensed and have a realistic structure, ensuring that the data distribution, skew characteristics, and noise levels can reflect the actual application scenario, thereby effectively exposing the potential bottlenecks of the database system in terms of cache management, indexing mechanisms, and query optimization.

[0043] Secondly, regarding the selection of data models, to accurately measure the query performance of mainstream multi-model databases in real-world business scenarios, this invention comprehensively considers the representativeness and technical differences of various model data in actual systems, selecting relational model data, document model data, graph model data, and vector model data as evaluation models. Relational model data is suitable for structured data management scenarios and serves as the de facto standard for strongly consistent scenarios such as finance and accounting. Document model data supports semi-structured data storage and flexible expansion, meeting the needs of rapid iteration in e-commerce and mobile applications. Graph model data is suitable for modeling highly correlated data networks and is widely used in fields such as anti-fraud and knowledge graphs. Vector model data, as a new type of infrastructure in the era of large models, is used to support approximate retrieval tasks for high-dimensional semantic data.

[0044] Thirdly, regarding workload design, this invention constructs a composite workload covering core single-model operations and cross-model joint queries, focusing on examining the execution efficiency and optimization capabilities of multi-model databases in cross-model semantic association queries. The workload design adheres to the principle of semantic consistency, avoiding simply piecing together single-model operations. Instead, it systematically evaluates the system's comprehensive capabilities in data conversion between models, query plan generation, and execution cost control by constructing cross-model query task chains with business logic connections.

[0045] Fourth, regarding evaluation metrics, considering that previous multi-model benchmark tests only recorded the execution time of each query and lacked a quantitative measurement of the overall capabilities of the multi-model database, this invention abstracts all workloads into four representative cross-model query workloads and provides specific quantitative scoring metrics based on the query time of each type, enabling horizontal comparisons between different database systems.

[0046] Through the above methods, the present invention can realize the performance comparison and optimization verification of multi-model database systems under a unified evaluation framework, make up for the shortcomings of existing evaluation methods in evaluating the collaborative processing capabilities of multi-model systems, and improve the technical integrity and applicability of benchmark tests.

[0047] After comprehensively considering the above factors, this invention proposes a query performance evaluation method for multi-model databases, comprising the following steps:

[0048] S1: Construct a multi-model query test dataset based on the open-source OpenAlex Academic dataset;

[0049] S2: Set up the experimental environment for the system to be tested, complete the unified parameter configuration, and load the test dataset constructed in step S1;

[0050] S3: Define four types of cross-model query workloads, and write corresponding multi-model query test statements on the system to be tested built in S2;

[0051] S4: Run the multi-model query test statements defined in S3 on the system to be tested built in S2, collect performance data and perform statistical analysis.

[0052] The specific process of step S1 includes:

[0053] S1.1: Download the OpenAlex open-source academic dataset. The multi-model dataset construction method described in this invention is based on the OpenAlex open-source academic dataset. OpenAlex is an open academic literature dataset maintained by a non-profit organization, integrating multiple open data sources including Crossref, ORCID, DOAJ, ROR, and Wikidata. It covers five core entities: papers, authors, institutions, journals, and research fields, forming a heterogeneous graph data with complete structure and clear semantic relationships. This dataset possesses authenticity and completeness, and its data size can be expanded from GB to TB, making it suitable for constructing large-scale evaluation datasets with multi-model features. The OpenAlex dataset adopts the CC0 (Creative Commons Zero v1.0 Universal) open license agreement, supporting the free acquisition, transformation, trimming, and redistribution of data, meeting the requirements of data legality, reproducibility, and scalability in the evaluation benchmark construction process.

[0054] S1.2: Data Re-splitting to Construct a Multi-Model Dataset Including Relationships, Documents, Graphs, and Vectors. The original OpenAlex dataset was released in JSONL format. This invention re-abstracted and split it into four types of model data: basic fields such as papers, authors, topics, institutions, and their geographical information form a relationship table; nested or dynamically modal supplementary information (e.g., lists of authors' former names, ORCIDs, author / topic lists, abstract and keyword inverted indexes, etc.) are converted into documents; collaboration relationships between authors and citation relationships between papers are converted into isomorphic graphs; authorship relationships between papers and authors and attribution relationships between papers and topics are converted into heteromorphic graphs; and topic descriptions and article titles and abstracts are converted into vector models after processing by a BERT vector embedding model. All data is finally processed into a CSV format containing nested JSON structures for easy import into the test system. The dataset processed in the above way retains complete semantics on the same dataset and has cross-model query capabilities.

[0055] S1.3: Data modeling, determining the data tables and main fields for each model. Based on the multi-model data obtained in S1.2, globally unique BIGINT type IDs are assigned to the four types of entities: papers, authors, topics, and institutions. These IDs are retained in all models as cross-model join keys. The table structures for each model are as follows (see...). Figure 1).

[0056] (1) Relational Model. Used to store basic information of various entities, including: author (author basic information table, main fields include name, total number of papers, total number of citations, etc.), work (paper basic information table, main fields include title, publication time, type, DOI number, etc.), topic (topic table, main fields include topic name, description information, field name, etc.), institution (institution table, main fields include institution name, institution type, etc.), institution_geo (institution geographical information table, main fields include city, latitude and longitude, etc.).

[0057] (2) Document Model. Used to store optional or multi-valued attributes, including: author_doc (author supplementary information table, with fields including a list of authors' former names and optional ORCID, used to simulate the shallow nested structure of documents) and work_doc (paper supplementary information table, with fields including a deeply nested structure of "author list" and "topic list", where the author list subfields include author ID, author order, and author name, and the topic list subfields include topic ID, topic name, and relevance score. An inverted index of the abstract text is also stored to support field existence detection).

[0058] (3) Graph model. Used to express the relationships between entities, it includes: work_v (article graph vertex table), author_v (author graph vertex table), topic_v (topic graph vertex table), author_author_e (author collaboration relationship edge table), work_referenced_work_e (paper citation relationship edge table), work_topic_e (paper topic relationship edge table), and work_author_e (paper author relationship edge table). Each table retains only a few high-frequency filtering fields to support graph traversal pushdown conditions. The remaining attributes are obtained by associating them with the relational model through ID.

[0059] (4) Vector model. Used for performance evaluation of vector nearest neighbor search, it includes: work_vec (paper vector table) and topic_vec (topic vector table), which store the semantic vectors of papers and topics.

[0060] The specific process of step S2 includes:

[0061] S2.1: Install the multi-model database to be tested. If the target system itself is a native multi-model kernel, directly install its distribution; if it depends on plugins to extend multi-model query capabilities, install the corresponding plugins simultaneously to make relational, document, graph, and vector model interfaces available within the same instance, ensuring that subsequent workloads can be executed across models within a unified query.

[0062] S2.2: Configure environment-related parameters. In instances where S2.1 has been deployed, the configuration files are uniformly modified to set the same values ​​for parameters affecting the execution plan, such as memory buffer and query parallelism, for each system under test, in order to eliminate horizontal comparison biases caused by configuration differences.

[0063] S2.3: Write the data preprocessing script. In the dataset generated in step S1.2, nested fields in the document model and graph model are stored as JSON text in CSV columns. However, some systems under test cannot fully parse this mixed format of "CSV with embedded JSON"; in addition, edge tables in the graph model also need to be converted to a structure conforming to a specific pattern (for example, in some multi-model databases, edge table data must contain from and to fields for pointing to graph nodes) in order to be correctly imported by the system. To solve the above compatibility issues, this step converts the CSV data containing nested JSON structures into a standard JSONL format file conforming to specific pattern requirements through field mapping configuration, thereby supporting batch import of data by the systems under test.

[0064] S2.4: Importing Data and Creating Indexes. Given the lack of a unified syntax for data import and multi-model index definition in existing multi-model databases, this invention provides script templates for importing data and creating indexes in common multi-model databases such as ArangoDB and AgensGraph. If the system under test exceeds this scope, it can be extended according to the template. In addition to creating regular indexes on primary keys and foreign keys, the script also creates vector indexes on vector models, ensuring consistency in index type and granularity across databases, providing equal acceleration conditions for subsequent query performance comparisons.

[0065] The specific process of step S3 includes:

[0066] S3.1: Define the query workload. Based on the multi-model dataset obtained in step S1, this invention abstracts four different categories of cross-model query workloads according to the core characteristics and usage scenarios of the query workload. Each type of workload contains 4-6 query task templates. Each task template is built based on a real business scenario (see Table 1) and spans at least two data models, while ensuring coverage of the core operators of the models involved (see Table 2). Each template consists of several interconnected subqueries. The output of the previous subquery can be used as the input of the next subquery. The subqueries are interconnected, forming a complete cross-model query task chain, which is used to systematically evaluate the coherence and optimization capabilities of the tested system in multi-model collaborative execution. The four different types of multi-model query workloads are listed below:

[0067] Table 1. Query Business Scenarios and Corresponding Query Numbers

[0068]

[0069] (1) Hybrid-IO workload (simple point queries across models): This type of query mainly uses point queries as the entry point, connecting multiple data models to extract information (which may involve simple calculations). The results are mostly lightweight detailed records, without complex data analysis processing. It is mainly used to provide detailed data views. The main focus is on the hybrid IO overhead of heterogeneous data sources—relational block queries, JSONB large field lookups, graph traversal, etc. It tests the throughput of multi-model hybrid reads of storage engines under different cache localities. Typical application scenarios include real-time rendering of web data homepages. Specific queries in this category are listed below:

[0070] H1: List all authors who have used the given name, along with a list of papers by each author.

[0071] H2: Lists other authors who have directly collaborated with the author with the specified ID, returning the author's name, number of citations, and institution name.

[0072] H3: Lists the top 10 papers with the highest topic similarity scores published under the specified topic and displays their titles, citation counts, and average similarity scores.

[0073] H4: Lists the 10 most recent papers cited by the specified paper, and displays the title, publication year, and author information for each cited paper.

[0074] H5: Lists papers along the shortest citation path to the specified paper, displaying their titles, authors, topics, and citation counts, sorted by similarity to the target topic.

[0075] Table 2. Models and key operators involved in each query

[0076]

[0077]

[0078] (2) Aggregation-Group type workload (complex statistical analysis queries across models). This type of query involves grouping and statistical analysis from multiple data models to identify the most active fields, authors, or papers. It is usually used to identify industry trends, leading researchers, or high-impact research. The main focus is on the CPU and memory bandwidth bottlenecks of aggregation and sorting—large tables need to be sorted in full memory after grouping, and window functions are often involved, testing the connectivity and memory management strategies of the multi-model execution engine. Specific queries in this category are listed below:

[0079] A1: List the top three research institutions with the most publications in the past five years, and show their top three most popular research areas during this period and their publication frequency.

[0080] A2: Analyze the evolution of the author collaboration network: List the top three collaborators with the most collaborations for the target author each year, and display the collaborator ID list for each year.

[0081] A3: List the top 10 most popular research topics in a certain institution, sorted by the number of papers published by that institution.

[0082] A4: List the top 3 research institutions with the most publications on the topic in the past five years, and show the number of papers they have published on that topic, sorted by the number of papers.

[0083] A5: Compile statistics on authors who published papers containing specified keywords in a particular field between 2022 and 2025, list the number of their corresponding papers, and the percentage of each paper in the total number of papers published by that author.

[0084] A6: List the top 5 most contributing (i.e., most cited) papers spanning two disciplines since 2020, displaying the contribution of each paper and sorting them from highest to lowest contribution.

[0085] (3) Vector-ANN type load (cross-model similar nearest neighbor query). This type of query mainly focuses on vector similarity nearest neighbor query as the main task, and then superimposed predicate filtering of relation, graph or document models to return recommendation results that simultaneously satisfy semantic similarity and multi-model constraints. The main points of examination are the recall-throughput trade-off of vector index in mixed scenarios, and the gain or loss of ANN performance after multi-model predicate pushdown. The typical application scenario is recommendation system. The specific queries in this category are listed below:

[0086] V1: Based on the "seed paper" with the highest number of citations in a certain field, expand its 1-2 layer citation network outward, and recommend the Top-10 documents that are semantically closest to the seed paper.

[0087] V2: Lists the top 10 papers since 2020 whose abstracts contain the specified keywords and are most relevant to the given topic, presenting their titles and abstracts in order of topic similarity.

[0088] V3: Find papers that mention a certain keyword in their abstracts, vectorize them, compare them with known papers using vector nearest neighbor, and return the IDs of the top 10 most similar papers.

[0089] V4: Find the top 20 papers from 2018 to 2023 whose abstracts contain the keyword "multi-model database" and are most similar to the specified paper in terms of topic. Display detailed information such as authors, titles, publication dates, types, citation information, and DOIs of these papers. The output should be in nested JSON format.

[0090] (4) Graph-Matching workload (cross-model complex graph pattern matching query). The main workload of this type of query is concentrated on the graph model, used to match specific complex graph structures (such as shortest path, multi-hop subgraph), and associate the hit results with other data to supplement the semantics. The main points of examination are cache invalidation caused by multi-hop random memory access, recursive traversal overhead, and multi-hop pruning efficiency of complex path patterns. The typical application scenario is graph retrieval augmentation generation (Graph-RAG). The specific queries in this category are listed below:

[0091] G1: Search the global collaboration network for researchers who are 2–4 hops away from the specified author and have no direct connection to them. Calculate the average vector distance between the published paper and the target topic. Select the three researchers with the smallest distance as potential collaborators and return their IDs.

[0092] G2: Lists the shortest citation path between two specified papers, calculates the influence score (the sum of citations and average author influence) of each paper on the path, and returns the paper IDs and titles in descending order.

[0093] G3: Perform BFS on a given paper in the citation network to collect information on its 1-hop direct citations and 2-hop indirect citations.

[0094] G4: Run PageRank on the author collaboration network and the paper citation network respectively, and select the top 50 authors and papers in each network; finally, select scholars who have works in both the "high-impact author list" and the "high-impact list" and return them.

[0095] G5: Executes the WCC algorithm on the author collaboration graph to count metadata such as the number of authors and collaborations for each weakly connected component, and returns detailed author ID information for the top N components.

[0096] G6: Detect and identify community structures in research collaboration networks. This query analyzes the author collaboration graph, uses the CDLP algorithm to find the three largest communities, and returns the main research areas (top 5) and core authors (top 10) for each community.

[0097] S3.2: Write the query statement for the system to be tested. Since there is no unified standard for multi-model query statements in existing multi-model databases, each database designs its own query syntax, resulting in significant differences. This invention provides cross-model query task chain statement examples for two common multi-model databases, ArangoDB and AgensGraph, in the embodiment section of the specification. If the system to be tested is not covered, it can be extended by referring to the same task chain order and returned field structure to achieve equivalent queries.

[0098] The specific process of step S4 includes:

[0099] S4.1: Write the query script. Configure the connection information (username, database name, password, IP address, port) and workload parameters (number of threads, cache switch, load path, exclusions, output directory, number of test rounds) of the database under test uniformly in the query script. The main function of the script is to automatically read the query statements under the load path, while excluding some queries that cannot be executed (if the multi-model database under test lacks relevant functions), and repeatedly execute the complete multi-model workload according to the predetermined rounds. For each test, automatically pull execution statistics from the database, obtain detailed database execution time, write it to the specified output file in real time, and dynamically update the median query time for each load, achieving automated load testing with one-time configuration and multiple reproductions.

[0100] S4.2: Start the query script and calculate the score. Start the query script, summarize the data in the corresponding output directory, and calculate the average score for each type of load according to the following formula:

[0101] For any load category C and the tested system s, the performance score is defined as:

[0102]

[0103] in, , representing the shortest median execution time to complete query q across all tested systems, is used as a benchmark for extreme performance. This is the median execution time of the current system s obtained from multiple actual measurements on query q. This represents the actual number of queries contained in load category C. The formula first log-normalizes the median time for each query, then takes the arithmetic mean of all queries within the category, thus obtaining the average log speedup ratio of that category relative to the limit performance as the final score.

[0104] Figure 3 The specific workflow of this invention has been given, and the workflow of this invention will now be described in detail with reference to specific implementation examples.

[0105] S1: Construct a multi-model test dataset.

[0106] Use the AWS CLI to download the official OpenAlex data snapshot from Amazon S3 Object Storage Service, and convert it into four data models according to the following rules:

[0107] Relationship: Extract the basic fields of author, paper, institution, and topic into a CSV file, and extract the last numeric part of the REST API address as the primary key as a BIGINT type ID.

[0108] Document: Convert nested fields (author aliases, ORCID, paper abstracts, keywords, reference list, etc.) into a nested JSON structure and write them into a CSV file along with the primary key extracted from the relational table.

[0109] The graph extracts some hotspot fields from the relational data to form the vertex table of the graph. It generates author collaboration edge tables and paper author edge tables based on the paper's author list, paper citation edge tables based on the paper's citation list, and paper topic edge tables based on the paper's topic list. All data is written into a CSV file.

[0110] Vectors: The paper abstracts and topic descriptions are processed using the open-source BERT-BASE model. After dimensionality reduction by PCA, 384-dimensional vectors are obtained and stored in CSV format, with the primary key consistent with the relational table.

[0111] S2: Setting up the environment for the system to be tested

[0112] Taking the mainstream open-source multi-model databases ArangoDB and AgensGraph as examples, on a physical machine with an Intel Xeon E7-8880 v4 CPU, 512GB DRAM, and 2TB HDD, ArangoDB 3.12.5, AgensGraph 2.16.0, and pgvector 0.6.2 are installed. ArangoDB, as a native multi-model database, does not require the installation of plugins, while AgensGraph requires the installation of the pgvector extension to support vector queries.

[0113] In the ArangoDB configuration file arangod.conf, set block-cache-size to 64GB, and add the startup parameter --experimental-vector-index to ArangoDB in / etc / systemd / system / arangodb3.service to enable experimental vector search functionality. Similarly, set work_mem to 64GB in postgresql.conf for AgensGraph. Adjust the log level of both to ERROR to eliminate differences in log flushing to disk.

[0114] ArangoDB cannot directly read CSV data containing nested JSON format. A script needs to be written to parse the nested JSON fields in the CSV into memory, while also adding the "_from" and "_to" pointer fields required by ArangoDB when importing graph data, ultimately outputting a standard JSONL file. Subsequently, a data table is created, and the data file is imported into the system using the arangoimport tool, creating a persistent index on the primary foreign key and an IVF-FLAT index on the vector fields to complete the data loading.

[0115] AgensGraph can directly import CSV data without preprocessing: first, create the corresponding data table, then load the data using the standard SQL LOAD statement, then create a B-tree index for the primary and foreign keys, and an IVF-FLAT index for the vector fields to complete the data loading.

[0116] S3: Define and instantiate multi-model query workloads

[0117] Write the multi-model database query statements corresponding to the workload, and set the query parameters. Taking cross-model similar nearest neighbor query V1 as an example, such as... Figure 2 As shown, this query is a typical cross-model query task chain: based on the "seed paper" with the highest number of citations in a certain field (relational document cross-model connection + document inclusion predicate filtering), it expands outward to its 1-2 layer citation network (multi-hop pattern matching of graph), and recommends the Top-10 documents that are semantically closest to the seed paper (vector ANN search).

[0118] Set the query parameter in this query (in this example, the research field) to "Impact of Climate Change on Human Health" (corresponding topic_id is 11244) to obtain the complete query statements for ArangoDB and AgensGraph, as shown below:

[0119] ArangoDB V1 query load:

[0120] LET TopCitedPaper = (

[0121] For w IN work

[0122] FILTER w.publication_year >= 2020 - 5

[0123] FOR wc in work_doc

[0124] FILTER w.id == wc.id and 'Impact of Climate Change onHuman Health' in wc.doc.topics[*].display_name

[0125] SORT w.cited_by_count DESC, w.id ASC

[0126] LIMIT 1

[0127] RETURN {work_id: w.id} )

[0129] LET PaperCitationNetwork = (

[0130] FOR tcp IN TopCitedPaper

[0131] FOR v IN 1..2 OUTBOUND CONCAT("work_v / ", to_string(tcp.work_id)) work_referenced_work_e

[0132] RETURN distinct {cited_work_id: v.id} )

[0134] LET PaperVector = (

[0135] UNION (

[0136] (FOR pcn IN PaperCitationNetwork

[0137] FOR wv IN work_vec

[0138] FILTER wv.id == pcn.cited_work_id

[0139] RETURN {id: wv.id, vec: wv.vec}

[0140] ),

[0141] (FOR tcp IN TopCitedPaper

[0142] FOR wv IN work_vec

[0143] FILTER wv.id == tcp.work_id

[0144] RETURN {id: wv.id, vec: wv.vec}) ) )

[0147] LET topVec = (FOR pv IN PaperVector FILTER pv.id == TopCitedPaper[0].work_id RETURN pv.vec)[0]

[0148] LET SimilarityScore = (

[0149] FOR pcn IN PaperCitationNetwork

[0150] FOR pv1 IN PaperVector

[0151] FILTER pv1.id == pcn.cited_work_id

[0152] LET similarity_score = L2_DISTANCE(topVec, pv1.vec)

[0153] SORT similarity_score ASC , pcn.cited_work_id ASC

[0154] RETURN {cited_work_id: pcn.cited_work_id, similarity_score} )

[0156] FOR ss IN SimilarityScore

[0157] LIMIT 10

[0158] FOR w IN work

[0159] FILTER w.id == ss.cited_work_id

[0160] RETURN {

[0161] title: w.title,

[0162] cited_by_count: w.cited_by_count,

[0163] similarity_score: ss.similarity_score

[0164] }

[0165] AgensGraph V1 query load:

[0166] WITH TopCitedPaper AS (

[0167] SELECT w.id AS work_id

[0168] FROM work w, work_doc wc

[0169] WHERE w.publication_year >= 2020 - 5

[0170] and w.id = wc.id

[0171] and wc.doc->'topics' @> '[{"display_name":"Impact of ClimateChange on Human Health"}]'

[0172] ORDER BY w.cited_by_count DESC,w.id asc

[0173] LIMIT 1

[0174] ),

[0175] PaperCitationNetwork AS (

[0176] MATCH (p1: work_v)-[: work_referenced_work_e*1..2]->(p2: work_v)

[0177] where p1.id = (select to_jsonb(tcp.work_id) from TopCitedPaper tcp )

[0178] return DISTINCT p2.id AS cited_work_id

[0179] ),

[0180] PaperVector AS (

[0181] SELECT wv.id, wv.vec

[0182] FROM work_vec wv

[0183] WHERE wv.id IN (SELECT cited_work_id::bigint FROMPaperCitationNetwork)

[0184] UNION

[0185] SELECT wv.id, wv.vec

[0186] FROM work_vec wv

[0187] WHERE wv.id = (SELECT work_id FROM TopCitedPaper)

[0188] ),

[0189] SimilarityScore AS (

[0190] SELECT pcn.cited_work_id::bigint, (wv1.vec <-> wv2.vec) ASsimilarity_score

[0191] FROM PaperCitationNetwork pcn

[0192] JOIN PaperVector wv1 ON wv1.id = pcn.cited_work_id::bigint

[0193] JOIN PaperVector wv2 ON wv2.id = (SELECT work_id FROM TopCitedPaper)

[0194] ORDER BY (wv1.vec <-> wv2.vec) ASC, pcn.cited_work_id::bigint ASC

[0195] LIMIT 10 )

[0197] SELECT w.title, w.cited_by_count, ss.similarity_score

[0198] FROM SimilarityScore ss

[0199] JOIN work w ON w.id = ss.cited_work_id

[0200] ORDER BY ss.similarity_score ASC,ss.cited_work_id ASC;

[0201] S4: Run workloads and collect performance data

[0202] Write an execution script and disable the execution plan cache uniformly within the script: set `cache = False` on the ArangoDB side and `plan_cache_mode = force_custom_plan` on the AgensGraph side; simultaneously force single-threaded execution—ArangoDB is single-threaded by default and requires no additional configuration, while on the AgensGraph side, set `max_parallel_workers_per_gather = 0`. Specify the output directory in the script and run the predetermined number of workloads simultaneously, collecting the runtime and writing the median runtime of each workload to the output file in real time.

[0203] After all loads have been executed, the average score for each cross-model load category of each system is calculated using formula (1). ArangoDB: H(0.839) A(0.913) V(1.000) G(0.860), AgensGraph: H(0.868) A(0.967) V(0.953) G(0.691). The score results show that the query performance of the two is not much different in H-class queries and is at the same level. AgensGraph has a slight advantage in A-class queries, ArangoDB has a slight advantage in V-class queries, and ArangoDB has a significant advantage in G-class queries. After investigation, it was found that AgensGraph's performance was significantly lower than that of ArangoDB in graph VLE (Variable Length Expand) operation (i.e., multi-hop traversal operator), which proves the effectiveness of the performance evaluation method.

Claims

1. A query performance evaluation method for multi-model databases, characterized in that, Includes the following steps: S1. Based on the OpenAlex open-source academic dataset, a multi-model test dataset is constructed; the multi-model test dataset contains model data of relations, documents, graphs and vectors in the same semantic space, and the four types of model data maintain cross-model semantic association through globally unique entity IDs; S2. Import the multi-model test dataset into the multi-model database to be tested, and unify the environment configuration of the database to be tested, including cache size, parallelism, log level and indexing strategy; S3. Construct four types of cross-model query workloads; each type of cross-model query workload contains multiple query templates, each query template spans at least two types of model data and forms a semantically coherent query task chain in sequence, which is used to evaluate the comprehensive capabilities of the multi-model database under test in terms of join order selection, model data transformation cost estimation and aggregation operation optimization. S4. Under fixed hardware, dataset size, query statement version, and concurrency parameters, repeatedly execute the cross-model query workload and collect the median execution time of each query statement. Then, calculate the performance score of each type of cross-model query workload based on the log-normalized speedup model, and obtain a horizontal performance comparison between different multi-model databases based on the performance score.

2. The query performance evaluation method for multi-model databases according to claim 1, characterized in that, Step S1 specifically involves: S1.1 Obtain a snapshot of OpenAlex data under the CC0 license; S1.2 breaks down the four core entities—papers, authors, topics, and institutions—into: relational model data, document model data, graph model data, and vector model data. S1.3 A unified global ID in large integer format is assigned to papers, authors, topics, and institutions, and this global ID is retained as a cross-model connection key in the four types of model data to ensure that the data has native cross-model query capabilities.

3. The query performance evaluation method for multi-model databases according to claim 2, characterized in that, The relational model data stores entity basic fields; the document model data stores nested or dynamic pattern fields; the graph model data stores relationships between entities, transforming cooperation, referencing, and attribution relationships into isomorphic or heterogeneous edge tables; and the vector model data uses the BERT model to embed topic descriptions and article summaries to form vector tables.

4. The query performance evaluation method for multi-model databases according to claim 1, characterized in that, Step S2 specifically involves: S2.1 Install the multi-model database system to be tested; install the corresponding database distribution. If it depends on plugins to extend the multi-model query capability, install the corresponding plugins simultaneously so that the interfaces of each model are available within the same instance. S2.2 Unify the settings for memory buffer size, query parallelism, and log level parameters; S2.3 Based on the import requirements of the multi-model database under test, a data preprocessing script can be optionally written to parse the nested JSON fields in the CSV and complete the source-target node pointing fields required for the edge table data, and output them in JSONL format, thereby enabling the multi-model database under test to complete the batch import. S2.4 Create consistent indexes on primary key, foreign key, and vector fields respectively to ensure that acceleration conditions are equivalent across systems.

5. The query performance evaluation method for multi-model databases according to claim 1, characterized in that, In step S3, the four types of cross-model query workloads are: Hybrid-IO class: Cross-model detail extraction query with point lookup as the main function, used to evaluate hybrid IO throughput; Aggregation-Group class: Statistical analysis queries involving grouping, window functions, and sorting, used to evaluate CPU and memory bandwidth bottlenecks in aggregation and sorting scenarios; Vector-ANN class: It adds relational, graph or document predicate filtering to vector nearest neighbor search to evaluate the recall-throughput tradeoff; The Graph-Matching class matches specific subgraphs using graph patterns and associates them with other models to supplement semantics. It is used to evaluate cache invalidation and traversal pruning efficiency.

6. The query performance evaluation method for multi-model databases according to claim 1, characterized in that, The log-normalized speedup model is as follows: ； in, The shortest median execution time for query q across all tested systems. This represents the median execution time of system s on query q. The number of queries is for load category C; the performance score is used to quantify the average logarithmic speedup of different databases relative to their extreme performance, in order to eliminate the impact of differences in query volume on the scoring results, reduce the weight of extreme long-tail values, and achieve horizontal comparison against outlier interference.

7. A query performance evaluation system for multi-model databases, characterized in that, Implementing the query performance evaluation method for multi-model databases as described in any one of claims 1-6, comprising: The dataset building module is used to generate native multi-model test datasets; The data import module is used to import data into the system under test and configure a unified environment. The workload module is used to define and instantiate specific multi-model query workloads; The performance acquisition and scoring module is used to collect median execution time and output log-normalized scores, enabling horizontal comparison and optimization verification of different multi-model databases.

Images