A method for optimizing a list expansion operation of a graph database

Abstract

This invention discloses an optimized method for expanding a list in a graph database, relating to the field of graph databases. The method includes: obtaining a batch of vectors containing a list to be expanded; evaluating the batch of vectors to obtain the length information of the list to be expanded; determining whether the length information of the list to be expanded is less than a preset minimum batch threshold; if so, expanding the batch of vectors using a row-by-row copying method; otherwise, expanding using a constant vector optimization method; and returning the generated expanded batch of vectors as output. Beneficial effects: By introducing constant vector technology, memory usage is reduced, execution efficiency is improved, and the risk of memory overflow during large-scale data expansion is avoided.

Description

Technical Field

[0001] This invention relates to the field of graph databases, and more particularly to an optimized method for expanding a list in a graph database. Background Technology

[0002] With the rapid development of applications such as social networks, knowledge graphs, and recommendation systems, graph databases, as data management systems based on nodes (vertices) and edges (edges), have been widely adopted due to their advantages in handling complex relational data. Graph Query Language (GQL), as the main interaction interface for graph databases, typically needs to support the parsing and manipulation of list types, among which the "unwind" operation is a common and crucial data transformation operation.

[0003] During graph query execution, the list expansion operation is used to transform nested list structures into multiple rows of data. For example, if a column in the query result contains the list value [1, 2, 3], then after expansion, it should result in three rows: {1}, {2}, and {3}. This operation is frequently called in path queries, aggregation result expansion, and multi-level nested queries, and is one of the important operators in graph database execution engines.

[0004] However, existing graph database systems generally suffer from high memory consumption and low execution efficiency when performing list expansion operations. Traditional implementations typically employ either "row-by-row copying" or "pre-allocated space":

[0005] Row-by-row copying: Creates a new record for each element in the list and copies the complete data for all non-expanded columns. When the list length is N and the number of non-expanded columns is M, M×N values ​​need to be stored repeatedly. This method is simple to implement, but it has high memory overhead, high CPU copying cost, poor cache locality, and is prone to memory overflow during large-scale queries.

[0006] Pre-allocated space method: This method reduces the number of dynamic allocations by pre-calculating the total number of rows after expansion and allocating memory at once. However, it still requires N data copies of non-expanded columns, which cannot fundamentally reduce memory usage and copying overhead, and it does not support streaming output, thus lacking flexibility.

[0007] In typical graph query benchmarks, traditional list expansion operators perform poorly. For example, in scenarios involving expanding lists with thousands of nodes, memory usage can exceed 600MB, and single execution time can exceed 1500ms, severely impacting query performance and system throughput.

[0008] Furthermore, as graph databases evolve towards vectorized execution models, data is typically passed between operators in the form of "vector batches." Traditional list unrolling implementations cannot fully utilize this execution mechanism, still primarily using row-based data, leading to complex memory management, low vector cache hit rates, and an inability to leverage hardware parallelism. Summary of the Invention

[0009] Purpose of the invention: The purpose of this invention is to solve the technical problems in the prior art and provide an optimized method for expanding a list in a graph database.

[0010] Technical solution: An optimized method for expanding list operations in a graph database, comprising:

[0011] Retrieve vector batches containing lists to be expanded;

[0012] The vector batches are evaluated to obtain the length information of the list to be expanded;

[0013] Determine whether the length of the list to be expanded is less than the preset minimum batch threshold. If so, expand the vector batch by copying it line by line. If not, expand it by optimizing the constant vector.

[0014] The generated expanded vector batches are returned as output.

[0015] Preferably, obtaining a vector batch containing a list to be expanded further includes extracting the non-expanded column vectors from the vector batch.

[0016] Preferably, the vector batch is evaluated to obtain the length information of the list to be expanded, including:

[0017] For each row of data in the batch of input vectors, calculate the value of the expression to be expanded to obtain a list or a binding table reference;

[0018] Record the list length information corresponding to each of the aforementioned lists or bound table references.

[0019] Preferably, the line-by-line copying method includes:

[0020] Iterate through the elements in the list to be expanded one by one;

[0021] For each element, create a new row and copy the values ​​of all non-expanded columns into that new row;

[0022] Append the new row to the output table.

[0023] Preferred methods for optimizing the expansion of constant vectors include:

[0024] For each unexpanded column in the input row, a constant vector representation is created, wherein the constant vector stores only a single constant value and vector length for that column, without storing multiple repeated values;

[0025] Convert the list to be expanded into a regular vector containing all elements of the list;

[0026] The constant vector and the ordinary vector are assembled into a batch of output vectors.

[0027] Preferably, the constant vector representation includes:

[0028] A constant value used to store the common value of this column across all expanded rows;

[0029] A length parameter represents the number of rows represented by the constant vector; wherein the memory usage of the constant vector is O(1) and is independent of the vector length.

[0030] Preferably, assembling the constant vector and the ordinary vector into a batch of output vectors includes:

[0031] Create a new vector batch containing:

[0032] The constant vectors corresponding to all non-expanded columns, wherein each constant vector shares the same constant value;

[0033] The ordinary vector corresponding to the expanded column contains all elements in the list to be expanded;

[0034] Set the number of rows in the new vector batch to the length N of the list to be expanded.

[0035] Preferably, the list to be expanded includes one of the following types:

[0036] A list type containing several elements of the same type;

[0037] Bind a table reference type to reference a complete data table;

[0038] For bound table reference types, the referenced table is converted into a vector representation and then the same constant vector optimization is performed.

[0039] Preferably, returning the generated expanded vector batches as output also includes streaming processing when the expanded results exceed the capacity limit of a single vector batch:

[0040] The expanded result is divided into multiple vector batches;

[0041] Results are returned in batches according to the flow pipeline model;

[0042] Downstream operators can begin consuming data while upstream operations are still processing it.

[0043] Preferably, returning the generated expanded vector batches as output also includes using a zero-copy method, including:

[0044] The constant vectors share the values ​​of the unexpanded columns in the original input rows by reference;

[0045] The ordinary vector is directly used from the vector converted from the list to be expanded;

[0046] The output vector batches are assembled by including references to the constant vector and the ordinary vector.

[0047] This application also proposes an optimized apparatus for expanding a list in a graph database, comprising:

[0048] The acquisition unit is used to acquire a batch of vectors containing a list to be expanded;

[0049] An evaluation unit is used to evaluate the vector batch and obtain the length information of the list to be expanded;

[0050] The execution unit is used to determine whether the length information of the list to be expanded is less than the preset minimum batch threshold. If so, the vector batch is expanded by copying line by line. If not, the constant vector optimization expansion method is used.

[0051] The output unit is used to return the generated expanded vector batches as output.

[0052] Beneficial effects:

[0053] 1. By introducing constant vector technology, for duplicate values ​​in non-expanded columns, only a single constant value and vector length are stored, instead of N complete copies. Memory usage is reduced from O(M×N) in the traditional method to O(M+N), where M is the number of non-expanded columns and N is the list length. In the LDBC benchmark test (1528 nodes, 10 attribute columns), memory usage decreased from 635,969.770 KiB to 4,727.301 KiB, achieving a memory compression ratio of up to 134.5 times. This effectively avoids the risk of memory overflow during large-scale data expansion.

[0054] 2. Eliminating redundant copying: Constant vector technology fundamentally eliminates M×(N-1) data copying operations for non-expanded columns. This significantly reduces the number of dynamic memory allocations and the creation of temporary objects, lowering garbage collection (GC) pressure. The data representation is more compact, improving CPU cache locality and increasing cache hit rate. In the same test scenario, the execution time of the list expansion operator itself decreased from 1,552.562ms to 10.070ms, a performance improvement of 154.2 times, and end-to-end query performance also achieved a 76-fold improvement.

[0055] 3. The optimization effect of this invention becomes increasingly significant with the growth of data scale. The larger the list length N and the more non-expanded columns M, the more obvious the memory savings and performance improvements. This makes this invention particularly suitable for large-scale graph analysis scenarios handling millions or even tens of millions of nodes, providing effective technical support for graph databases to cope with ever-increasing data volumes.

[0056] 4. By introducing a configurable minimum batch threshold parameter, an adaptive optimization strategy is implemented: for small lists (length below the threshold), a simple row-by-row copying method is used to avoid the additional overhead of the optimization mechanism itself. For large lists (length reaches or exceeds the threshold), constant vector optimization is enabled to achieve the ultimate performance gain. This design ensures that the method remains efficient under various data distribution scenarios, combining simplicity and efficiency.

[0057] 5. This invention is designed based on a vectorized execution engine model, achieving zero-copy batch assembly through the combination of constant vectors and ordinary vectors. This method does not disrupt existing pipelined execution patterns, fully supports streaming processing, and allows downstream operators to begin processing immediately after the previous batch of data is ready, further reducing query latency. Attached Figure Description

[0058] Figure 1 A schematic diagram of the method framework for this invention is provided;

[0059] Figure 2 This is a block diagram of a device structure provided in one embodiment of this application;

[0060] Figure 3 This is a block diagram of an electronic device structure provided in one embodiment of this application. Detailed Implementation

[0061] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Example 1

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art. The terms "comprising" and similar expressions used herein mean that the element or object preceding the word covers the element or object listed following the word and its equivalents, but do not exclude other elements or objects.

[0064] The terms used in this application are as follows:

[0065] Unwind operation: An operation in Graph Query Language (GQL) used to expand a list of columns into multiple rows of data. For example, expanding a single row containing [1, 2, 3] into three rows, each containing 1, 2, and 3 respectively.

[0066] Vector: A data structure used in database execution engines to store data of the same type in batches; it is the basic unit of columnar storage.

[0067] A constant vector is a special type of vector representation used to store recurring identical values. It only requires storing a constant value and the vector length, eliminating the need to store complete data for each position.

[0068] Vector Batch: A data batch consisting of multiple vectors, representing several rows of data in a table, and is the basic unit of data flow in the database execution engine.

[0069] Binding Table: A table structure used to store intermediate results during graph query processing, consisting of row types and actual data.

[0070] RowType: Defines the schema information of the table, including column names and the data type of each column.

[0071] Minimum batch threshold (minUnwindBatch): A configuration parameter that controls when constant vector optimization is enabled. Constant vector optimization is only used when the list length exceeds this threshold.

[0072] In response to the problems existing in the current technology, such as Figure 1As shown, an optimization method for list expansion operations in graph databases is proposed, including:

[0073] Step S101: Obtain a vector batch containing the list to be expanded; the system receives a vector batch containing the list to be expanded. This vector batch contains vectors of one or more columns, including vectors of expanded columns and vectors of non-expanded columns (vectors of non-expanded columns are optional). This vector batch is the basic unit of data flow in the graph database execution engine, storing the data to be processed.

[0074] Step S102: Evaluate the vector batch to obtain the length information of the list to be expanded; evaluate the data in the received vector batch and calculate the length of the list to be expanded. The system obtains the number of elements in the list to be expanded by traversing each row of data in the vector batch and records the length information of the list. This evaluation process is the basis for subsequent optimization strategy selection.

[0075] Step S103: Determine whether the length of the list to be expanded is less than the preset minimum batch threshold. If yes, expand the vector batch by copying it line by line. If no, expand it by optimizing the constant vector.

[0076] Based on the obtained list length information, the system determines whether the length of the list to be expanded is less than a preset minimum batch threshold. If the length of the list to be expanded is less than the threshold, the system will use the traditional line-by-line copying method to expand it; if the length of the list to be expanded is greater than or equal to the threshold, the system will use a constant vector optimized expansion method to significantly improve memory efficiency and execution speed.

[0077] Row-by-row copying method: For lists smaller than the minimum batch threshold, the system still uses the row-by-row copying method, expanding each element one by one, creating new rows and copying the data of the non-expanded columns.

[0078] Constant vector optimization expansion: For lists with values ​​greater than or equal to a threshold, the system optimizes the unexpanded columns by constructing a constant vector, storing only a constant value and the vector length, instead of storing complete data for each position. This method significantly reduces memory usage.

[0079] Step S104: Return the generated expanded vector batches as output. The system generates expanded vector batches according to the selected expansion method (row-by-row copying or constant vector optimization expansion) and returns them as output. These expanded vector batches contain all the expanded data and can be further processed by downstream graph query operations.

[0080] This invention achieves significant performance improvements and memory optimization by optimizing list expansion operations in graph databases for large-scale data scenarios. Regarding memory usage, traditional row-by-row copying methods have a memory footprint of O(M×N), while this invention reduces memory usage to O(M+N) using constant vector technology, significantly reducing memory consumption and meeting the high-efficiency processing requirements of large-scale graph data. Simultaneously, execution time has been significantly shortened in multiple real-world tests, improving the overall system processing capabilities.

[0081] In some specific embodiments, obtaining a batch of vectors containing a list to be expanded further includes extracting the non-expanded column vectors from the batch of vectors.

[0082] In some specific embodiments, the vector batch is evaluated to obtain the length information of the list to be expanded, including:

[0083] For each row of data in the input vector batch, the value of the expression to be expanded is calculated, resulting in either a list or a bound table reference. The system evaluates each row of data in the input vector batch and calculates the value of the field to be expanded. These fields may be lists (e.g., a list of adjacent nodes of a node) or bound table references (e.g., a table reference of query results).

[0084] Record the list length information corresponding to each of the aforementioned lists or bound table references.

[0085] For each list or bound table reference to be expanded, the system records its length. This step helps determine later which lists can be expanded using constant vector optimization.

[0086] List length information: For list data, this records the number of elements it contains.

[0087] Number of rows in the bound table: For bound table references, this records the number of rows in the table.

[0088] In some specific embodiments, the line-by-line copying method includes:

[0089] Iterate through the elements in the list to be expanded one by one;

[0090] For each element, create a new row and copy the values ​​of all non-expanded columns into that new row;

[0091] Append the new row to the output table.

[0092] In some specific embodiments, the constant vector optimization expansion method includes:

[0093] For each unexpanded column in the input row, a constant vector representation is created, wherein the constant vector stores only a single constant value and vector length for that column, without storing multiple repeated values; in the constant vector optimization expansion, the system first creates a constant vector representation for each unexpanded column in the input row.

[0094] The characteristic of constant vectors: Constant vectors store only a single constant value for the column and the vector length, instead of storing multiple duplicate values ​​for each data location. This means that if all values ​​in a column are the same, only one constant value and the length of the column need to be stored, rather than storing multiple identical values ​​for each row.

[0095] Convert the list to be expanded into a regular vector containing all elements of the list;

[0096] The system converts the list to be expanded into a regular vector containing all the elements in the list.

[0097] Ordinary vectors: Unlike constant vectors, ordinary vectors directly store all the actual elements in the list.

[0098] The constant vector and the ordinary vector are assembled into a batch of output vectors.

[0099] Finally, the system combines the constant vector and the ordinary vector into a batch of output vectors. This batch contains all the expanded data, with each row consisting of elements from both the constant vector and the ordinary vector.

[0100] Constant vector: Used to store repeated values ​​of non-expanded columns, such as constant values.

[0101] Ordinary vector: Used to store elements in an expanded list.

[0102] The core of this step is to combine a constant vector (which stores only constant values) with a regular vector (which contains all elements of the list), reducing redundancy in memory storage.

[0103] In some specific embodiments, the constant vector representation includes:

[0104] A constant value used to store the common value of this column across all expanded rows;

[0105] Each constant vector contains a constant value that is the same across all expanded rows. In other words, if all values ​​in a column are the same during the expansion process, the system only needs to store this constant value once, instead of storing the same value for every row.

[0106] One length parameter indicates the number of rows represented by the constant vector; the constant vector also contains a length parameter indicating the number of rows represented by the constant vector, that is, the number of times the constant value is repeated in the expanded result.

[0107] The memory usage of the constant vector is O(1), and it is independent of the vector length.

[0108] The memory usage of a constant vector is O(1), meaning that a constant vector only needs to store constant values ​​and a length parameter. Regardless of how many rows the data is expanded, the memory usage of a constant vector remains fixed and is independent of the vector's length N.

[0109] In some specific embodiments, assembling the constant vector and the ordinary vector into a batch of output vectors includes:

[0110] Create a new vector batch containing:

[0111] The constant vectors corresponding to all non-expanded columns, wherein each constant vector shares the same constant value;

[0112] The system will create a new vector batch that contains all the expanded data. The new vector batch consists of two types of vectors: constant vectors and ordinary vectors.

[0113] The ordinary vector corresponding to the expanded column contains all elements in the list to be expanded;

[0114] The newly created vector batch first contains the constant vectors corresponding to all non-expanded columns.

[0115] Each non-expanded column of constant vectors shares the same constant value, and only stores that constant value and the corresponding vector length.

[0116] A regular vector containing expanded columns:

[0117] The vector batch also contains regular vectors corresponding to the expanded columns. These regular vectors will contain all elements in the list to be expanded.

[0118] This process ensures the complete transformation of the data to be expanded.

[0119] Set the number of rows in the new vector batch to the length N of the list to be expanded.

[0120] After assembly, the system sets the number of rows in the new vector batch to the length N of the list to be expanded. If there are N elements in the list to be expanded, then the output vector batch will have N rows of data, each row including the constant value of the non-expanded column and the actual element of the expanded column.

[0121] In some specific embodiments, the list to be expanded includes one of the following types:

[0122] A list type that contains several elements of the same type; a list type represents a list containing several elements of the same type.

[0123] The bound table reference type refers to a complete data table; the bound table reference type indicates that the data to be expanded comes from a reference to a complete data table.

[0124] For bound table reference types, the referenced table is converted into a vector representation and then the same constant vector optimization is performed.

[0125] In some specific embodiments, returning the generated expanded vector batches as output also includes streaming processing when the expanded results exceed the capacity limit of a single vector batch:

[0126] The expanded result is divided into multiple vector batches. When the size of the expanded result exceeds the capacity limit of a single vector batch, the system divides the expanded result into multiple smaller vector batches. Each vector batch contains a portion of the expanded data, ensuring that each batch meets the system's capacity limit. This is to prevent a single vector batch from becoming too large, leading to excessive memory consumption. By processing in batches, the system can avoid memory overflow or performance bottlenecks caused by excessive data.

[0127] Results are returned in batches according to the streaming pipeline model. The streaming pipeline model allows data to be transmitted in batches without waiting for all data to be generated. This means that once the first batch of data is ready, the system immediately returns it to the downstream operator, instead of waiting for all expansion operations to complete. This allows downstream operators to begin consuming the returned data while upstream operators continue processing other data. This enables parallel data processing, thereby speeding up the overall query process.

[0128] Downstream operators can begin consuming data while upstream operations are still processing it. Through streaming processing, downstream operators can begin consuming already returned vector batches while upstream operations are still processing data. This parallel processing approach significantly improves system throughput and response speed. This method avoids data backlog, reduces waiting time, and improves processing efficiency. When processing large amounts of data, the streaming pipeline model can significantly improve performance, especially in scenarios involving large-scale graph database queries.

[0129] In some specific embodiments, returning the generated expanded vector batch as output also includes employing a zero-copy method, including:

[0130] The constant vector shares the values ​​of the unexpanded columns in the original input row by reference; in the zero-copy method, the constant vector does not create a new copy of the data, but rather shares the values ​​of the unexpanded columns in the original input row by reference. This means that for columns that do not need to be expanded, the constant vector only stores references to these columns, without copying the entire data. The constant vector only stores references to the data, instead of storing it multiple times. This reduces memory usage and avoids unnecessary data copying.

[0131] The ordinary vector directly uses the vector converted from the list to be expanded; that is, it directly uses the vector converted from the list to be expanded. That is, no new data copies are created; instead, the already generated vectors are used directly. This avoids storing the same data multiple times in memory, further reducing memory usage.

[0132] The output vector batches are assembled by including references to both the constant vectors and the ordinary vectors. When generating output vector batches, these batches are assembled by including references to both constant and ordinary vectors. This means that output vector batches do not create copies of the constant and ordinary vectors; instead, they directly contain references to them. The system avoids the overhead of copying data and memory allocation, thus improving efficiency. All vectors (constant and ordinary vectors) are only referenced to data in shared memory, without copying the data itself.

[0133] For example:

[0134] Example 1: Simple List Expansion

[0135] 1. Enter your query:

[0136] This query defines a list a = [1, 2, 3] and sets b and c to constants 100 and 200, respectively. Then, iterates through list a using a FOR loop, returning element i along with the constant values ​​b and c at each iteration.

[0137] 2. Input vector batch. Number of rows: 1, number of columns: [a, b, c], value: {[1, 2, 3], 100, 200};

[0138] 3. The execution process assumes that the configured minimum batch threshold minUnwindBatch=2, meaning that constant vector optimization is enabled when the length of the list to be expanded is greater than or equal to 2. The detailed execution steps are as follows:

[0139] 3.1 Evaluate the list to be expanded: The list to be expanded is a=[1,2,3], with a length N=3.

[0140] 3.2 Determine whether to enable constant vector optimization: Since N=3 is greater than or equal to minUnwindBatch=2, constant vector optimization is enabled.

[0141] 3.3 Constructing a constant vector:

[0142] For the unexpanded columns b and c, construct constant vectors respectively:

[0143] b_vec = ConstantVector(100, 3), which is a constant vector representing the constant value 100 repeated 3 times.

[0144] c_vec=ConstantVector(200, 3), this constant vector represents the constant value 200, repeated 3 times.

[0145] Memory usage: The memory usage of each constant vector is O(1), which is independent of the vector length.

[0146] 3.4. Convert the list to a vector:

[0147] Convert the list to be expanded, a = [1, 2, 3], into a regular vector: i_vec = FlatVector([1, 2, 3]).

[0148] Memory usage: The memory usage of this ordinary vector is O(3) because the list has 3 elements.

[0149] 3.5 Assemble the output vector batch:

[0150] Create a new vector batch containing constant vectors b_vec, c_vec, and the expanded vector i_vec.

[0151] 3.6 Set the number of output rows to N=3, which is 3 rows of data after flattening.

[0152] 4. Output Results

[0153] After optimization with constant vectors, the expanded output is as follows: Output:

[0154] {i: 1, b: 100, c: 200}

[0155] {i: 2, b: 100, c: 200}

[0156] {i: 3, b: 100, c: 200}

[0157] 5. Memory Comparison:

[0158] Traditional method: 3×3=9 values ​​are stored;

[0159] This invention stores 3+2=5 values ​​(44% optimization).

[0160] Example 2: Large-scale graph node expansion:

[0161] 1. Input query. This query first uses USEgraph to specify the graph data source for the query, then uses MATCH to match all Person nodes and collects these nodes into a list persons. Then, it uses FOR loop to traverse the list persons and returns the name and age attributes of each node.

[0162] 2. Execution process

[0163] 2.1 First-stage aggregation: The system first performs an aggregation operation, collecting all Person nodes into a list named persons. At this point, the persons list contains references to 10,000 Person nodes.

[0164] 2.2 The second phase unfolds:

[0165] In the second stage, the system will expand the list of persons:

[0166] List length: The length of the persons list is 10,000.

[0167] Non-expanded column: In this example, all data comes from inside the list, so the non-expanded column is 0.

[0168] Number of rows after expansion: After expansion, there will be 10,000 rows of data.

[0169] Traditionally, the system needs to copy the attributes of each node row by row and perform an expand operation on each node. This method is inefficient when processing large-scale data, typically taking about 1500 milliseconds to complete the entire query. By employing vectorization and constant vector optimization, the solution of this invention can significantly improve data processing efficiency. When processing the list of persons, the system only needs to convert the list into a vector representation and directly return the expanded vector batch, with an execution time of approximately 10 milliseconds. Compared to the traditional row-by-row copying method, the optimized method of this invention achieves a 150-fold performance improvement, significantly accelerating the query speed for expanding large-scale graph nodes.

[0170] Example 3: Nested List Expansion. This query defines a list `data` containing two objects. Each object contains an `id` field and a `tags` field, where the `tags` field is a list containing multiple tags. The query first iterates through each object in the `data` list, then expands the `tags` list within each object twice, returning each tag and its corresponding `id`.

[0171] 1. First expansion (data):

[0172] Input: 1 line of data, containing the fields id and tags.

[0173] Output: 2 lines of data, each line containing an id and a list of tags.

[0174] Column: [id, tags];

[0175] 2. Second expansion (tags):

[0176] For the first line tags=['A', 'B', 'C']: length: 3

[0177] deal with:

[0178] Create a constant vector ConstantVector(1, 3) to represent the id column, with the id value of 1 for all rows.

[0179] Create a regular vector FlatVector(['A', 'B', 'C']) to represent the tag column, which contains all elements from the tags list.

[0180] For the second line, tags=['D', 'E']:

[0181] Length: 2

[0182] Processing: Create a constant vector `ConstantVector(2, 2)` to represent the `id` column, with all rows having an `id` value of 2. Create a regular vector `FlatVector(['D', 'E'])` to represent the `tag` column, containing all elements from the `tags` list.

[0183] 3. Final output:

[0184] The expanded data is shown below:

[0185] {id: 1, tag: 'A'}

[0186] {id: 1, tag: 'B'}

[0187] {id: 1, tag: 'C'}

[0188] {id: 2, tag: 'D'}

[0189] {id: 2, tag: 'E'};

[0190] Traditional row-by-row copying method: In the final five-row output, the id column needs to store five specific values: 1, 1, 1, 2, 2.

[0191] The optimization method of this invention only needs to store two constant values ​​(1 and 2) for the id column, and map them to the final five rows of data through two constant vectors (with lengths of 3 and 2 respectively).

[0192] In this example, for the id column alone, the present invention achieves a 60% memory saving (reducing the number of value stores from 5 to 2). This demonstrates that the optimization strategy of the present invention can effectively handle complex nested query structures, recursively applying constant vector techniques in each level of expansion operation, thereby achieving significant overall memory resource savings.

[0193] In other embodiments of the invention, combined with Figure 2 This invention discloses an optimization device for expanding a list in a graph database, comprising:

[0194] Acquisition unit 201 is used to acquire a batch of vectors containing a list to be expanded;

[0195] Evaluation unit 202 is used to evaluate the vector batch and obtain the length information of the list to be expanded;

[0196] The execution unit 203 is used to determine whether the length information of the list to be expanded is less than the preset minimum batch threshold. If so, the vector batch is expanded by copying line by line. If not, the constant vector optimization expansion method is used.

[0197] Output unit 204 is used to return the generated expanded vector batch as output.

[0198] In other embodiments of the present invention, an electronic device 400 is disclosed, such as... Figure 3 As shown, the electronic device may include: one or more processors 401; a memory 402; a display 403; one or more application programs (not shown); and one or more computer programs 404. These devices can be connected via one or more communication buses 405. The one or more computer programs 404 are stored in the memory 402 and configured to be executed by the one or more processors 401. The one or more computer programs 404 include instructions that can be used to perform actions such as... Figure 1 And the various steps in the corresponding embodiments.

[0199] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0200] In the various embodiments of this invention, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0201] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0202] The above description is merely a specific implementation of the embodiments of the present invention, but the protection scope of the embodiments of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of the present invention should be covered within the protection scope of the embodiments of the present invention. Therefore, the protection scope of the embodiments of the present invention should be determined by the protection scope of the claims.

Claims

1. An optimized method for expanding a list in a graph database, characterized in that, include: Retrieve vector batches containing lists to be expanded; The vector batches are evaluated to obtain the length information of the list to be expanded; Determine whether the length of the list to be expanded is less than the preset minimum batch threshold. If so, expand the vector batch by copying it line by line. If not, expand it by optimizing the constant vector. Return the generated expanded vector batches as output; Optimization methods for constant vector expansion include: For each unexpanded column in the input row, a constant vector representation is created, wherein the constant vector is a vector that stores only a single constant value and the vector length of that column, without storing multiple repeated values; A regular vector is a vector that converts a list to be expanded into a vector containing all the elements of the list; The constant vector and the ordinary vector are assembled into a batch of output vectors.

2. The optimized method for expanding a list in a graph database according to claim 1, characterized in that, Obtaining a batch of vectors containing a list to be expanded also includes extracting the non-expanded column vectors from the batch of vectors.

3. The optimized method for expanding a list in a graph database according to claim 1, characterized in that, The vector batch is evaluated to obtain the length information of the list to be expanded, including: For each row of data in the batch of input vectors, calculate the value of the expression to be expanded to obtain a list or a binding table reference; Record the list length information corresponding to each of the aforementioned lists or bound table references.

4. The optimized method for expanding a list in a graph database according to claim 1, characterized in that, The line-by-line copying method includes: Iterate through the elements in the list to be expanded one by one; For each element, create a new row and copy the values ​​of all non-expanded columns into that new row; Append the new row to the output table.

5. The optimized method for expanding a list in a graph database according to claim 2, characterized in that, The constant vector representation includes: A constant value used to store the common value of this column across all expanded rows; A length parameter, representing the number of rows represented by the constant vector; The memory usage of the constant vector is O(1), and it is independent of the vector length.

6. The optimized method for expanding a list in a graph database according to claim 2, characterized in that, Assembling the constant vector and the ordinary vector into a batch of output vectors includes: Create a new vector batch containing: The constant vectors corresponding to all non-expanded columns, wherein each constant vector shares the same constant value; The ordinary vector corresponding to the expanded column contains all elements in the list to be expanded; Set the number of rows in the new vector batch to the length of the list to be expanded.

7. The optimized method for expanding a list in a graph database according to claim 2, characterized in that, The list to be expanded includes one of the following types: A list type containing several elements of the same type; Bind a table reference type to reference a complete data table; For bound table reference types, the referenced table is converted into a vector representation and then the same constant vector optimization is performed.

8. The optimized method for expanding a list in a graph database according to claim 1, characterized in that, Returning the generated expanded vector batches as output also includes streaming processing when the expanded results exceed the capacity limit of a single vector batch: The expanded result is divided into multiple vector batches; Results are returned in batches according to the flow pipeline model; Downstream operators can begin consuming data while upstream operations are still processing it.

9. The optimized method for expanding a list in a graph database according to claim 6, characterized in that, Returning the generated expanded vector batches as output also includes using a zero-copy method, including: The constant vectors share the values ​​of the unexpanded columns in the original input rows by reference; The ordinary vector is directly used from the vector converted from the list to be expanded; The output vector batches are assembled by including references to the constant vector and the ordinary vector.

Images