Unified rdbms framework for mixed vector search of different data types via SQL and nosql

Abstract

Techniques are provided for a unified relational database framework for hybrid vector search. In one technique, a plurality of documents are accessed and a vector table and a text table are generated. For each accessed document, data within the document is converted into plain text, a plurality of chunks are generated based on the plain text, the embedding model generates a vector for each of the chunks, the vectors are stored in a vector table together with a document identifier identifying the accessed document, a segmented word is generated based on the plain text, and the segmented word is stored in the vector table together with the document identifier identifying the accessed document. The word segments are stored in a text table together with a document identifier. Such processing may be performed in a database system in response to a single database statement to create a hybrid index. In response to receiving the hybrid query, a vector query and a textual query are generated and executed, and the corresponding results may be combined.

Description

A unified RDBMS framework for performing mixed vector searches of different data types via SQL and NoSQL. Technical Field

[0001] This disclosure generally relates to database systems, and more specifically to presenting a single search interface to a user to perform different types of searches. Background Technology

[0002] Traditionally, users of database systems are provided with search tools to initiate queries on data stored in relational tables. These queries involve keywords and / or other search criteria that data items must contain or satisfy in order for those data items to be returned as results. This keyword approach is traditional information retrieval.

[0003] Using vector databases, users can perform semantic searches on documents for which embeddings have been generated. To run a semantic search, the user with the document (such as a PDF / Word document) must first transform the document into a vector using a series of steps, typically performed outside the index and database system. These transformations include converting the document to plain text, chunking it into fragments, and converting them into vectors using a vector embedding model. For JSON (JavaScript Object Notation) documents where one or more fields need to be vectorized, the user must provide explicit instructions to compute the vectors for those fields and include those vectors back into the JSON to be indexed. During a query, the user either has access to a simple search API or access to SQL, but not both.

[0004] The methods described in this section are feasible methods, but not necessarily methods that have been previously conceived or implemented. Therefore, unless otherwise indicated, no method described in this section should be assumed to be prior art solely because it is included in this section. Attached Figure Description

[0005] In the picture:

[0006] Figure 1 is a block diagram depicting an example database system in an embodiment;

[0007] Figure 2 is a block diagram depicting an example vector table generated by the vectorizer in an embodiment;

[0008] Figure 3 illustrates examples of different final result sets given the result set from performing a text search on the hybrid index and the result set from performing a vector search on the hybrid index in the embodiments;

[0009] Figure 4 illustrates examples of different final result sets given a result set from a text search performed on a hybrid index and a result set from a vector search performed on a hybrid index;

[0010] Figure 5 is a flowchart depicting an example process for processing mixed queries in an embodiment;

[0011] Figure 6 is a block diagram illustrating a computer system on which embodiments of the present invention can be implemented;

[0012] Figure 7 is a block diagram of a basic software system that can be used to control the operation of a computer system. Detailed Implementation

[0013] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the invention. However, it will be clear that the invention can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the invention.

[0014] General Overview

[0015] A system and method for performing mixed searches on different data types are provided. In one technique, the database system constructs a single index that supports information retrieval searches as well as vector similarity searches, and optionally also supports relational predicates. In related techniques, the database system supports block-level and document-level queries using one or more result fusion and scoring methods. This single index can be queried through a single search API and accessed through views.

[0016] The embodiments improve upon computer-related technologies associated with searching in databases. Users can submit a single query, which is automatically transformed into multiple queries, one for performing a semantic search and another for performing a text-based search. Furthermore, the embodiments reduce the consumption of computer resources (such as CPU and memory) by performing a single document ingestion and processing for both vector database construction and text database construction. Further, with the embodiments, users do not need to use complex and expensive solutions that involve having RDBMS assets for relational search, separate assets for unstructured search, and separate dedicated vector database assets for vector similarity search. The embodiments reduce this complexity and cost by indexing all data items into a single hybrid index that can be searched using a single user query.

[0017] System Overview

[0018] Figure 1 is a block diagram depicting an example database system 100 in an embodiment. Database system 100 may be a relational database management system (RDBMS) for storing and managing relational tables. Database system 100 includes a data source 110, a filter 120, a chunker 130, a vectorizer 140, a tokenizer 150, a hybrid index 160, and a query engine 170. These components of database system 100 (except for query engine 170) comprise a single index pipeline that includes "forks" or splits in the data flow, as described herein. Filter 120, chunker 130, vectorizer 140, tokenizer 150, and query engine 170 may be implemented in software, hardware, or any combination of software and hardware.

[0019] Data source 110 includes one or more types of data items, such as columns, files, and documents. Data items can be JSON documents, XML documents, text data (e.g., VARCHAR2), and / or BLOBs (Binary Large Objects). Examples of other document types include PDFs, HTML, word processing documents, spreadsheet documents, and presentation / slideshow documents, each of which is encoded text data.

[0020] Data source 110 can be a database, file system, network location, directory, or a combination thereof. Data source 110 can be local to database system 100 or remote. If remote, data items in data source 110 may have originated from another source (such as a remote source), and therefore, the results of data retrieval operations may already be stored in data source 110. For example, database system 100 receives a request to index data items, where the data items are stored remotely relative to database system 100; therefore, the request includes one or more references (e.g., Uniform Resource Locators (URLs)) that database system 100 uses to request those data items. This request can be an HTTP request. In response, database system 100 receives the requested data items and stores those data items in data source 110.

[0021] In relevant embodiments, data items include references to data to be indexed. For example, a JSON document might include references, such as URLs, in one of the JSON objects within the JSON document. These references point to a remote data source. The references could point to text data, image data, video data, or audio data. Therefore, when reading a data item, database system 100 (or a component thereof) detects the references, uses them to generate and send a request to another entity, receives a response from that entity, and performs zero or more operations on the data within the response. Such operations can be performed by filter 120. Example operations include performing speech-to-text recognition on audio data (which may be part of video data) and performing optical character recognition (OCR) on image data, which may also be part of video data.

[0022] In this embodiment, the data items in data source 110 are heterogeneous, spanning different database columns (e.g., JSON, XML, text) and / or different types of files (such as PDF, Word, HTML, RAR, ZIP, etc.). Furthermore, some of the data items input to filter 120 may be read from one location (e.g., a database, local directory, or network), while others may be read from another location.

[0023] Index creation instructions

[0024] Users of database system 100 provide instructions to database system 100 on how to construct a hybrid index given a set of data items to be ingested into the index pipeline. These instructions may be in the form of Data Definition Language (DDL) statements, such as the following:

[0025] CREATE HYBRID VECTOR INDEX index_name ON table_name(column_name)PARAMETERS(...);

[0026] In this example, the name of the hybrid index is "index_name", the name of the table storing the data items to be indexed is "table_name", and the name of the column in which the data items are stored is "column_name". Also in this example, the DDL statement includes a list of one or more parameters. Some parameters can be semantically related, while others can be text-related.

[0027] Examples of semantically relevant parameters include the name (or location) of the embedding model used to generate embeddings (or vectors) for a given chunk of text, the chunking method (e.g., by word or by sentence), the type of vector index (e.g., inverted vector file (IVF) index or hierarchical navigable small world (HNSW)), the distance function (e.g., cosine distance, dot product distance, Euclidean distance, and Manhattan distance), the vector index search precision (e.g., a scale between 0 and 100), whether to vectorize the entire JSON object / record, and one or more specific JSON fields (within the JSON object) to be vectorized (if any). For each possible parameter, a default value can be used if no value is specified in the DDL statement.

[0028] Examples of text-related parameters include data storage, filter type, lexer, sectioner, and stoplist. Data storage refers to how the document / text is obtained. The document / text can be obtained via file references or URLs. If the data storage parameter has no value, the text can be a column. Filter type refers to extracting text from rich content files such as PDFs. Lexer refers to how the text is tokenized. Several techniques for tokenization may be supported. Sectioner refers to the process of recording sections of text, for example, through tags such as... <title>The stop word list is a list of words excluded from the index, such as "the".< / title>

[0029] Filter

[0030] Filter 120 ingests data items and outputs plain text, which is ultimately input to chunker 130 and tokenizer 150. In other words, the same plain text output by filter 120 is sent to both chunker 130 and tokenizer 150. If the data item is already plain text, filter 120 passes that plain text to chunker 130 and tokenizer 150. If the data item is not plain text, filter 120 performs one or more operations on the data item to convert the data within the data item into plain text. For example, if the data item is a PDF document, filter 120 extracts plain text from the PDF document. As another example, if the data item is a compressed file, filter 120 performs a decompression operation on the compressed file to output data that may be plain text or may need to be converted to plain text.

[0031] In this embodiment, filter 120 processes different types of data items to generate plain text. For example, filter 120 performs a different set of one or more operations depending on the type of data item. As a specific example, filter 120 performs a first set of operations on a JSON document and a second set of operations on a BLOB document. The first set of operations may involve detecting and extracting plain text from the JSON document. The second set of operations may involve decoding the encoded data within the BLOB document to obtain plain text.

[0032] In an embodiment, filter 120 processes the JSON document in one of several ways. For example, in a first way, filter 120 accepts a JSON document that already contains vectors. Filter 120 may be informed where the vectors are located or may detect vectors while parsing the JSON document. For example, metadata of JSON objects within the JSON document may indicate that a particular field includes vector data. Therefore, filter 120 reads this metadata and identifies each instance of that particular field within the JSON document in order to extract the vectors from each instance.

[0033] In the second approach, filter 120 identifies which fields in the JSON document contain text data to be converted into vectors, and passes the contents of those fields to chunker 130 or directly to vectorizer 140. The identification of such fields can be based on the values ​​of specific parameters in a database statement created using a hybrid index.

[0034] In the third approach, filter 120 identifies which JSON fields contain references to remote data sources, uses these references to retrieve data from those remote data sources, performs any necessary transformations on the retrieved data (i.e., to obtain plain text), and causes vectorizer 140 to generate one or more embeddings based on the plain text.

[0035] In the fourth approach, filter 120 causes the entire JSON document to be passed to chunker 130, and chunks are sent to vectorizer 140 so that embeddings are generated based on those chunks.

[0036] Block divider

[0037] Chunker 130 generates text blocks for each of the multiple parts of a set of plain text received from filter 120. Such "chunking" is necessary because many sets of plain text have a text size larger than the context window of the embedding model (which generates embeddings for the input text). Therefore, a set of plain text can be divided into blocks and each block is "vectorized".

[0038] Chunking can be performed by selecting a threshold number of characters or bytes that fits the context window of the embedding model. To prevent splitting words or sentences, the chunker 130 can consider words or sentences when performing the chunking operation. For example, if a threshold number of characters would cause the last sentence to be split, resulting in a chunk that does not end with a period, the chunker 130 identifies the end of the previous sentence and selects only the text up to that point when generating chunks. Thus, each chunk and each subsequent chunk can begin with a complete sentence. Such chunking can help produce vectors with less noisy data.

[0039] If different embedding models that change over time are used to vectorize blocks, and these different embedding models have different context windows, then the chunker 130 can take those different context windows into account when generating blocks based on text data items. Therefore, the chunker 130 generates relatively small blocks for embedding models with relatively small context windows and larger blocks for embedding models with larger context windows.

[0040] As described herein, the database statement instructing database system 100 to create a hybrid index can specify the embedding model to use. If the embedding model is not supported by database system 100, an error may be presented. Alternatively, database system 100 can send blocks to a remote vectorization service, which generates vectors based on the blocks and returns those vectors to database system 100. If the database statement instructing database system 100 to create a hybrid index does not specify the embedding model to use, the vectorizer can choose the default embedding model.

[0041] Vectorizer

[0042] Vectorizer 140 generates embeddings for each block input to it. For example, if a document is divided into one hundred blocks, and those blocks are passed as input to vectorizer 140, vectorizer 140 will generate one hundred embeddings for that document, one embedding per block. Embeddings are also referred to as vectors. Vectors can be stored differently in one of several ways. For example, a dense vector (where the number of dimensions with zero values ​​is relatively small, if any) will contain the entire embedding, while a sparse vector will exclude dimension values ​​with zero values. Furthermore, vectors can be stored in vector objects that contain not only the vector but also metadata about the vector, such as the number of dimensions, indicators of the dimension format, and one or more next-version values ​​referencing other versions of the vector (if other versions exist). Other versions of the vector can be stored in the same vector object or can be stored separately.

[0043] In this embodiment, the embedded model is loaded directly into the memory of the database system 100 as a first-class object, meaning it is a native object. An example format for the embedded model is ONNX (OpenNeural Network Exchange). As a native object, the embedded model can be named, invoked, and deleted. Furthermore, permissions can be granted to users, user groups, and / or organizations regarding the embedded model. Therefore, the vectorizer 140 can call or invoke the embedded model, passing a block of text as input and receiving the embedded model as the output of the call.

[0044] In an alternative embodiment, vectorizer 140 makes third-party calls to remote providers implementing the embedded model, such as OpenAI and OCIGeNIA. An example of a third-party call is a REST API call to one of these remote providers. Alternatively, vectorizer 140 makes local API calls (e.g., REST API calls) to locally installed providers implementing the embedded model, such as Ollama.

[0045] Vectorizer 140 (or another component of database system 100) stores vectors in vector table 162, which is a data structure that is part of hybrid index 160. Although hybrid index 160 comprises multiple data structures, it is presented to its users as a single point of contact, providing a single interface for efficient access to data stored in both vector and text formats. This interface is described in more detail herein.

[0046] Vector table 162 comprises multiple rows, one row for each vector. Figure 2 is a block diagram depicting an example vector table 210 generated by vectorizer 140 (or another component of database system 100) in an embodiment. Vector table 210 includes vector columns for storing vectors. In this example, other columns of vector table 210 include columns for document identifiers (IDs), columns for row IDs, columns for offsets, columns for lengths, and columns for blocks. The document ID uniquely identifies the document from which the corresponding vector originates. The row ID uniquely identifies the row in the table storing the corresponding document. The offset value is the offset in the corresponding document (at which the start of the text block corresponding to the vector is found), and the length value is the length of the text block. Different blocks can have different lengths. Each block stored in the block column is the input to the embedding model that generates the corresponding vector.

[0047] Hybrid index 160 may also include vector index 164 constructed by database system 100 based on vector table 162. Such construction can be performed in response to receiving a database statement that generates hybrid index 160. Vector index 164 allows vector searches to be performed more quickly. Without vector index 164, semantic search would have to examine every vector in vector table 162 and perform a distance operation for each vector, where the query vector (in semantic search) is compared to each vector and the distance between them is calculated. Example distance operations include cosine distance, dot product distance, Euclidean distance, and Manhattan distance. Examples of vector index 164 include IVF indexes and HNSW indexes. Using vector index 164 avoids many distance operations because only a small subset of distance operations is performed to identify the vector "closest" to the query vector. However, there is no guarantee that the vector "closest" to the query vector is indeed the closest; this can only be guaranteed through an exhaustive search of vector table 162.

[0048] Word segmenter

[0049] Tokenizer 150 generates tokens for each text data item received from filter 120. Examples of tokenization include identifying keywords and storing them in a text table 166, which is a data structure that is another part of a hybrid index 160. Before storing the keywords in text table 166, tokenizer 150 may perform one or more operations on the words and / or phrases it detects in the text data items, such as stemming, lemmatization, singularizing plural words, etc.

[0050] The tokenizer 150 can also identify sections within plain text. Examples of sections include any text that is tagged, such as author, title, and custom tags.

[0051] Figure 2 also depicts text table 220 corresponding to text table 166. Text table 220 is an inverted index. Text table 220 includes five columns, but may include more or fewer columns. These five columns are: a text column storing tokens or words, a first document column storing document IDs, a last document column also storing document IDs, a document count storing numbers, and an inverted list column storing a list of document IDs. The document ID identifier in the first document column includes the first document containing the corresponding token. The document ID identifier in the last document column includes the last document containing the corresponding token. The document count is the number of documents containing the corresponding token. The document ID list contains the document IDs of the documents containing the corresponding token.

[0052] In an embodiment, tokenizer 150 (or another component of database system 100) generates a text index 168 that indexes words or phrases in text table 166. An example of text index 168 is a B-tree index. Such generation can be performed in response to receiving a database statement that constructs hybrid index 160.

[0053] Querying mixed indexes

[0054] Queries targeting hybrid index 160 are called "hybrid queries". From the perspective of a user requesting the results of a hybrid query, hybrid index 160 appears as a single data structure. Users may not be aware that hybrid index 160 comprises multiple data structures or that different subqueries are generated based on the hybrid query to access those data structures.

[0055] In this embodiment, a user can specify different types of mixed queries to query the mixed index 160. The query engine 170 is capable of handling different types of mixed queries that may be received from different users or the same user. One type of mixed query that can be submitted to the query engine 170 is an SQL query. Another type of mixed query that can be submitted to the query engine 170 is a NoSQL query (e.g., JSON or XML) or a "low SQL" query that includes both SQL and non-SQL elements. An example of a low SQL mixed query is as follows:

[0056] select DBMS_HYBRID_VECTOR.SEARCH (

[0057] json (

[0058] '{ "hybrid_index_name": "idx",

[0059] "search_text": "C, Python, Database"

[0060] }'

[0061] )

[0062] from dual?

[0063] The first field in the JSON object "json" is "hybrid_index_name" and its value is "idx". Therefore, the query is calling a hybrid index, such as hybrid index 160. The second field in the JSON object is "search_text" and its value is "C, Python, Database". Therefore, the query requests documents containing these three words: C, Python, and Database.

[0064] Search API Options

[0065] In one embodiment, query engine 170 provides one or more search API options that can be specified for hybrid queries. At least some of the search API options can be default options. In a related embodiment, the DDL statement used to create the hybrid index specifies one or more search API options.

[0066] Example search API options include a scorer function, a fusion function, a return format, top N counts, search text for semantic / vector search, search patterns for semantic / vector search, score weights, rank penalties, and an aggregator function for semantic / vector search.

[0067] The example returns data in JSON, XML, and plain text formats. The first N counts can be any positive integer. If a value greater than 1000 is specified, a prompt can be displayed asking the user if they want that value.

[0068] Example search patterns used for vector searches include documents and blocks. For example, a user submits a request to return a document ID (or the document itself) as a result of a mixed query. Therefore, if the vector search produces five blocks, and they are all part of the same document, only a single document (or document ID) is returned as part of that vector search. As another example, a user submits a request to return a block ID (or the block itself) as a result of a mixed query. Again, the value of the search pattern can be specified at query time or when the mixed index is created, or it can be a default value.

[0069] In this embodiment, a hybrid query can specify a text string for a vector search and a list of one or more words or phrases for a text search. Therefore, different search terms can be used to search both vector table 162 and text table 166.

[0070] Aggregator functions are used to identify which document(s) to return given multiple blocks identified as results of a vector search. The input to an aggregator function is a list of block scores. Examples of aggregator functions include maximum, median, and average. Other aggregator functions are described in this article. If no aggregator function is specified in the mixed query, a default aggregator function (such as maximum or another aggregator function described in this article) can be used. Additional aggregator functions are described elsewhere in this article.

[0071] In this embodiment, search results from vector search and search results from text search are weighted differently. These weights can be specified via weight options. For example, the weight for vector search can be 2 and the weight for text search can be 1, meaning that results from vector search will be weighted twice as much as results from text search.

[0072] Ranking penalties are used in some ranking formulas, such as the Reciprocal Rank Fusion (RRF) formula, which combines search results from multiple sources, such as text and vector sources. In RRF, "reciprocal" means "1 / rank". Therefore, the highest rank results in 1 / 1, the second highest rank results in 1 / 2, and so on. Without penalties, a document's score could be 1 / text_rank + 1 / vector_rank. However, if one set of results is better than another, penalties can be added to the denominator. For example, the final score = 1 / (text_rank + text_penalty) + 1 / (vector_rank + vector_penalty). If text_penalty is set to 5 and vector_penalty is set to 1, this means that the contribution of the text ranking is penalized.

[0073] Another example of a text search API option is a value that indicates whether to return only documents containing all specified words from the text search, or whether to return only a strict subset of documents containing only the specified words from the text search.

[0074] Example Search API

[0075] Below is an example of the complete search API. Other implementations may have more or fewer parameters.

[0076] dbms_hybrid_vector.search(

[0077] json(

[0078] '{ "hybrid_index_name" : "my_index",

[0079] "search_text" : "query for text and vector sear",

[0080] "search_scorer" : "rrf",

[0081] "search_fusion" : "INTERSECT / ...",

[0082] "vector":

[0083] {

[0084] "search_text" : "something",

[0085] "search_vector" : " [...] ",

[0086] "search_mode" : "DOCUMENT / CHUNK",

[0087] "aggregator" : "MAX / AVG / ..",

[0088] "score_weight" : 10 ,

[0089] "rank_penalty" : 1

[0090] },

[0091] "text":

[0092] {

[0093] "contains" : "about(cats)",

[0094] "scorer" : 'definescore, mergescore' / / - contains syntaxincludes operators to control scoring

[0095] "paths" : ["mypath.a.b.c", "p.q.r"], / / - JSON paths

[0096] "score_weight" : 1,

[0097] "rank_penalty" : 5

[0098] }

[0099] "filter_by": / /

[0100] {

[0101] "user_name" : ["eq", 5,]

[0102] }

[0103] "return":

[0104] {

[0105] "topN" : 10,

[0106] "values" : ["rowid", "chunk_text",..],

[0107] "format" : "JSON"

[0108] }

[0109] }'

[0110] )

[0111] )

[0112] { "title": "dbms_hybrid_vector.search",

[0113] "description": "hybrid search parameters",

[0114] "type" : "object",

[0115] "properties" : {

[0116] "hybrid_index_name" : {"type" : "string"},

[0117] "partition_name" : {"type" : "string"},

[0118] "search_text" : {"type" : "string"},

[0119] "search_scorer" : {"type" : "string", "enum" : [ "RSF", "RRF" ]},

[0120] "search_fusion" : {"type" : "string", "enum" : [ "UNION", "INTERSECT",

[0121] "TEXT_ONLY", "VECTOR_ONLY", "MINUS_TEXT", "MINUS_VECTOR" ]},

[0122] "vector" : {"type" : "object",

[0123] "properties" : {

[0124] "search_text" : {"type" : "string"},

[0125] "search_vector" : {"type" : "string"},

[0126] "search_mode" : {"type" : "string",enum : ["DOCUMENT",

[0127] "CHUNK" ]},

[0128] "score_weight" : {"type" : "number", "minimum" : 1},

[0129] "rank_penalty" : {"type" : "number", "minimum" : 0},

[0130] "aggregator" : {"type" : "string",enum : [ "COUNT", "SUM", "MIN",

[0131] "MAX", "AVG", "MEDIAN", "BONUSMAX", "MAXAVGMED", "WINAVG", "ADJBOOST"]}

[0132] },

[0133] "additionalProperties": false

[0134] },

[0135] "text" : {"type" : "object",

[0136] "properties" : {

[0137] "contains" : {"type" : "string"},

[0138] "score_weight" : {"type" : "number", "minimum" : 1},

[0139] "rank_penalty" : {"type" : "number", "minimum" : 0}

[0140] },

[0141] "additionalProperties": false

[0142] },

[0143] "return" : {"type" : "object",

[0144] "properties" : {

[0145] "topN" : {"type" : "integer", "minimum" : 1},

[0146] "format" : {"type" : "string", enum :[ "JSON", "XML" ]},

[0147] "values" : { "type": "array", "items":{ "type": "string", "enum":

[0148] ["rowid", "score", "vector_score", "text_score", "vector_rank", "text_rank", "chunk_text", "chunk_id"]}}

[0149] },

[0150] "additionalProperties": false

[0151] }

[0152] },

[0153] "additionalProperties": false,

[0154] "required": [ "hybrid_index_name" ]

[0155] }

[0156] In this embodiment, the database system 100 can accept at least four broad types of queries via the search API:

[0157] a. (1) Queries with one or more keywords that will be used to issue both vector queries and text queries (e.g., a search for "generative AI" in both vector and text searches simultaneously), where the results of the two queries are combined (e.g., union or join) and the results are re-ranked.

[0158] b. (2) The query includes (i) a first input for a vector query (provided as a keyword or a pre-computed vector) and (ii) a second input for a text query, wherein the results of the two queries are combined (e.g., union, join) and the results are re-ranked (e.g., a search for "stock fraud" in a vector search and a search for "ABC corporation" in a text search).

[0159] c. (3) Text-only queries; and

[0160] d. (4) Vector query only.

[0161] Subquery generation and processing

[0162] In this embodiment, given a query, query engine 170 generates multiple subqueries, each targeting a different set of one or more data structures. For example, one subquery targets vector table 162 and vector index 164, another subquery targets text table 166 and text index 168, and yet another subquery targets a column in the table that may already exist before the generation of hybrid index 160.

[0163] For example, the subquery targets text index 168 and text table 166. Given the words "ai" and "technology" as search terms in the mixed query, query engine 170 uses text index 168 to identify entries in text table 166 that have "ai" in a text column of text table 166, and then retrieves a list of document IDs from the inverted list columns of text table 166. Similarly, query engine 170 uses text index 168 to identify entries in text table 166 that have "technology" in a text column of text table 166, and then retrieves a list of document IDs from the inverted list columns of text table 166. Therefore, query engine 170 has two inverted lists and performs an intersection to identify documents that contain both "ai" and "technology". If the inverted lists are relatively long (e.g., hundreds of document IDs), query engine 170 can utilize the first and last document columns of text table 166 (which can also be reflected in text index 168). Based on the corresponding document ID ranges of the two text search terms, query engine 170 can determine that the document ID range of one text search term does not overlap with the document ID range of the other text search term. Therefore, query engine 170 can determine that the intersection of the two inverted lists is zero, meaning that no document contains both text search terms. Thus, query engine 170 does not need to access the inverted list of either text search term, let alone perform a match between the two inverted lists. This process using the first and last document columns saves significant time and computing resources.

[0164] Query Engine 170 can also use document count columns when processing subqueries. For example, if the document count associated with a word is relatively high, the result associated with that word may rank relatively low. Conversely, if the document count associated with a word is relatively low, the result associated with that word may rank relatively high.

[0165] Query engine 170 also enables one or more query vectors to be generated based on one or more search terms indicated in the hybrid query. These one or more search terms can be the same search terms used in the text subquery (or "text search query" or "text search"). Alternatively, the one or more search terms used for the vector subquery (or "vector search query" or "vector search") can be different from the one or more search terms used in the text subquery. Therefore, the hybrid query can specify different search terms for each of two or more subqueries. The one or more search terms of the vector subquery are fed into an embedding model, i.e., the same embedding model used by vectorizer 140 to generate the vectors. The embedding model outputs one or more query vectors that become part of the vector subquery.

[0166] Query engine 170 processes subqueries (e.g., by executing an execution plan generated from the subqueries), producing a result set for each subquery. The results in the result sets are document identifiers and / or block identifiers. Query engine 170 produces result data for each result (e.g., a block) in the result set. Examples of result data include the row ID of the corresponding document, the overall (mixed) score of the result (whether it's a document or a block within a document), the vector score of the result, the text score of the result, the vector rank of the result, the text rank of the result, and the text of the block (if the result is a block).

[0167] Query engine 170 can combine results from all result sets into a single final result set and present that final result set. Combining these results may involve consulting a mapping from document IDs to row IDs to obtain a set of row IDs for a given document ID in the final result set. Each row ID uniquely identifies a row in the database table where the document is stored. Query engine 170 can then retrieve the document stored in the row corresponding to that row ID.

[0168] Execution plan processing

[0169] In this embodiment, query engine 170 generates multiple execution plans, each including multiple operations, to produce the final result of a mixed query. Each execution plan for the mixed query may include a different set of operations. Two or more execution plans may include the same set of operations, but in a different order. For example, one execution plan includes a set of subqueries executed in one order, and another execution plan includes the same set of subqueries executed in a different order. For each execution plan, query engine 170 generates an estimated cost for executing that execution plan. The estimated cost may be time-related, computer resource-related (e.g., CPU, memory, disk I / O, network I / O, etc.), or a combination of both. Query engine 170 selects the execution plan associated with the lowest estimated cost. If the selected execution plan results in an error during its execution, query engine 170 may select another execution plan associated with the next lowest estimated cost.

[0170] In relevant embodiments involving the intersection of search results from different searches (e.g., vector search and text search), query engine 170 determines the selectivity of each subquery among a plurality of subqueries. The selectivity of each subquery can determine which subquery is executed first. Therefore, instead of executing multiple subqueries of a mixed query in parallel, at least two subqueries can be executed sequentially, where the result set of the first subquery is used to execute the second subquery.

[0171] For example, if a mixed query includes one or more very unique search terms (meaning a text search query would result in very few results), query engine 170 may determine to execute the text search query before the vector search query (and optionally, before the relational search query, if it exists). In cases where the text search query returns relatively few document IDs as a result, those document IDs are used when executing the vector search query. For example, before performing a distance calculation between the query vector and candidate vectors in vector index 168, query engine 170 identifies the document IDs of the candidate vectors and determines whether those document IDs are found in the result set of the text search query. If not, the distance calculation is avoided, and another candidate vector is considered.

[0172] As another example, if the mixed query is associated with the first N parameter values ​​less than a specific value (e.g., five), query engine 170 determines to first execute a vector search query to identify, for example, five block IDs and determine the corresponding document IDs, which may be fewer than five. This determination can also be based on the assumption that the text search query is not very selective (meaning that the text search query is estimated to return a relatively large number of document IDs). After identifying five or fewer document IDs from the vector search query, those document IDs are used to reduce the number of document IDs that will be considered when executing the text search query.

[0173] Example scorer function

[0174] As described herein, another search API option is a scorer function. Example scorer functions that can be specified as search API options include Relative Score Fusion (RSF) and Reverse Rank Fusion (RRF). In this embodiment, query engine 170 (or a component thereof) implements one or both of RSF and RRF.

[0175] RSF is calculated as follows: the sum of (i) the product of vector score and vector weight and (ii) the product of text score and text weight is divided by (b) the sum of vector weight and text weight. The formula for RSF is as follows:

[0176] RSF = ((text_score * text_weight)+(vector_score * vector_weight)) / (text_weight+ vector_weight)

[0177] RRF is calculated as follows: calculate the RRF value (RRFVAL) and the RRF maximum value (RRFMAX), take the ratio, and then multiply by one hundred. The formula for RRF is as follows:

[0178] RRFVAL = 1 / (rank+penalty) + 1 / (rank+penalty)

[0179] RRFMAX = 1 / (1+5) + 1 / (1+1)

[0180] RRF = 100 * (RRFVAL / RRFMAX)

[0181] The "5" and the final "1" are values ​​used for ranking penalties. These values ​​can be default values ​​or values ​​that have been set in the search API call to the query. The other values ​​are predefined.

[0182] Figure 3 illustrates an example of two distinct final result sets given a result set 312 from a text search performed on the hybrid index 160 and a result set 314 from a vector search performed on the hybrid index 160. Each "intermediate" result set is based on the same single query received by the query engine 170.

[0183] Result set 312 and result set 314 can be combined using RRF to generate final result set 322. Alternatively, result set 312 and result set 314 can be combined using RSF to generate final result set 324.

[0184] Example fusion function

[0185] Example fusion functions include intersection, union, and difference. The intersection function returns documents that appear in both vector and text search results. In contrast, the union function returns documents that appear in either search result. The difference function returns all results from the vector search results that do not appear in the text search results, or all results from the text search results that do not appear in the vector search results. If difference is the selected fusion function, the user can specify which search results to subtract from in the fusion query.

[0186] Figure 4 illustrates an example of two distinct final result sets given a result set 412 from a text search performed on the hybrid index 160 and a result set 414 from a vector search performed on the hybrid index 160. Each "intermediate" result set is based on the same single query received by the query engine 170.

[0187] The intersection function can be used to combine result set 412 and result set 414 to generate a final result set 422. Alternatively, the union function can be used to combine result set 412 and result set 414 to generate a final result set 424, which includes all results from result set 412 and result set 414.

[0188] In this example, the re-ranking score is the vector score, while the combined score is a weighted sum of the vector and text scores. In this example, the vector and text scores are weighted equally in the combined score. However, in other implementations, the weighting can be different.

[0189] Aggregate block fraction

[0190] When performing a semantic search that attempts to return documents rather than blocks, it's possible to identify documents associated with multiple blocks for which scores have been generated. Examples of aggregation operations that can be performed on multiple block scores to generate a single document score include maximum, average, and median. However, each aggregation operation has its drawbacks. For example, identifying the maximum block score and using that maximum block score might result in a large document with only a single, relevant block, while all other blocks in the large document are irrelevant. In this example, it would be preferable to select different documents associated with multiple blocks that have scores slightly lower than that maximum block score.

[0191] In this embodiment, a document score is generated based on the scores of multiple physically close blocks within the document. There are two main methods: window averaging and adjustment boosting.

[0192] In the window averaging method, the maximum average score for a scrolling window is determined. First, the top N (e.g., ten) block scores of the document are identified. For each "window" of the identified blocks, the average block score of the blocks within that window is calculated. Depending on the number of blocks and the window size, one or more average block scores are generated. If multiple average block scores exist, the highest average block score is selected as the aggregate score for the document.

[0193] For example, if the document blocks identified by the execution vector query are 2, 8, 10, and 19, and the window size is 3, then the window sizes for the blocks are {2, 8, 10} and {8, 10, 19}. From these two sets of blocks, the average block score in each set is calculated, and the maximum average is selected as the aggregate score for that document.

[0194] In the ADJBoost method, an average boost block score is calculated for the first N (e.g., five) blocks. The maximum boost for each block is M minus the block score. An example value for M is one hundred. Each of the N block scores is increased by the following value: the maximum boost for that block score multiplied by the average score of the surrounding blocks. The formula for calculating the ADJBoost is as follows:

[0195] BOOSTSCORE i = CHUNKSCORE i + (M - CHUNKSCORE i ) * ((PRIORSCOREi +NEXTSCORE i ) / 2M) / / This is calculated for each (i) block in the first N blocks.

[0196] ADJBOOST = SUM(boostscore 1, ...,boostscore N ) / N

[0197] Therefore, if a particular block is surrounded by two unscored blocks, then the boost score for that particular block is its block score. Thus, if none of the N blocks returned as part of a vector search have neighboring blocks that are also part of those N blocks, then ADJBOOST is a simple average of the N block scores.

[0198] Example process

[0199] Figure 5 is a flowchart depicting an example process 500 for processing mixed queries in an embodiment.

[0200] At box 510, multiple documents are accessed. These documents may be stored in data source 110. Some or all of these documents may originate from one or more remote data sources. The accessed documents can be of different types, such as PDF and Word. Furthermore, some accessed documents may come from one column data type (e.g., JSON), while others may come from another column data type (e.g., text). Box 510 can be initiated by a database statement instructing database system 100 to create a mixed index.

[0201] At box 520, a vector table and a text table are generated. Box 520 can be executed some time before or after box 530. Initially, the vector table and the text table are empty.

[0202] At box 530, a document is selected. Box 530 may involve random selection from the accessed documents, or it may involve selecting the accessed documents in a specific order. The document selected in one iteration of box 530 may be different from the document selected in another iteration of box 530. Box 530 may be performed by filter 120.

[0203] At box 540, the data of the selected document is converted into plain text. Box 540 can be performed by filter 120. Box 540 can involve determining the type of the selected document and determining one or more conversion operations associated with that type. Thus, conversion operations can include decoding data, transforming data, and / or decompressing compressed data, etc. Therefore, depending on the type of document, different sets of one or more conversion operations can be performed to generate plain text. Box 540 can also involve filter 120 passing the plain text to chunker 130.

[0204] At box 550, multiple blocks are generated based on plain text. Box 550 can be executed by block divider 130. The specific type of the blocks can be a default type, or it can be specified in the database statement that initiates the creation of the hybrid index.

[0205] At box 560, the embedding model generates multiple vectors based on the generated blocks. In other words, the embedding model generates a vector for each block. Box 560 can be initiated by calling the embedding model through vectorizer 140.

[0206] At box 570, vectors are stored in a vector table along with document identifiers (identifying the selected document from which the corresponding block originates). Each vector is stored in a different row of the vector table. Box 570 can be performed by vectorizer 140 or another component of database system 100.

[0207] At box 580, multiple word segments are generated based on the plain text. Box 580 can be performed by word segmenter 150.

[0208] At box 590, the tokens are stored in a text table along with the document identifier, associated with each token. Each token is stored in a different row of the text table. Box 590 can also be performed by tokenizer 150.

[0209] Process 500 may also involve generating vector indexes on vector tables and text indexes on text tables.

[0210] Hardware Overview

[0211] According to one embodiment, the techniques described herein are implemented by one or more dedicated computing devices. The dedicated computing device may be hardwired to execute the techniques, or may include digital electronic devices (such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) that are persistently programmed to execute the techniques, or may include one or more general-purpose hardware processors programmed to execute the techniques according to program instructions in firmware, memory, other storage devices, or a combination thereof. Such dedicated computing devices may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to implement the techniques. The dedicated computing device may be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hardwired and / or program logic to implement the techniques.

[0212] For example, Figure 6 is a block diagram illustrating a computer system 600 on which embodiments of the present invention may be implemented. The computer system 600 includes a bus 602 or other communication mechanism for conveying information, and a hardware processor 604 coupled to the bus 602 for processing information. The hardware processor 604 may be, for example, a general-purpose microprocessor.

[0213] Computer system 600 also includes main memory 606, such as random access memory (RAM) or other dynamic storage device, coupled to bus 602, for storing information and instructions to be executed by processor 604. Main memory 606 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 604. When such instructions are stored in a non-transitory storage medium accessible to processor 604, computer system 600 presents itself as a dedicated machine tailored to perform the operations specified in the instructions.

[0214] The computer system 600 further includes a read-only memory (ROM) 608 or other static storage device coupled to a bus 602 for storing static information and instructions for the processor 604. A storage device 610, such as a disk, optical disk, or solid-state drive, is provided and coupled to the bus 602 for storing information and instructions.

[0215] Computer system 600 can be coupled via bus 602 to a display 612, such as a cathode ray tube (CRT), for displaying information to a computer user. Input devices 614, including alphanumeric keys and other keys, are coupled to bus 602 to transmit information and command selections to processor 604. Another type of user input device is a cursor control 616, such as a mouse, trackball, or cursor arrow keys, used to transmit directional information and command selections to processor 604 and to control cursor movement on display 612. This input device typically has two degrees of freedom in two axes (a first axis (e.g., x) and a second axis (e.g., y)), allowing the device to specify a position in a plane.

[0216] Computer system 600 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware and / or program logic (which, when combined with the computer system, make computer system 600 a special-purpose machine or program it as such)

[0217] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such storage media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs, magnetic disks, or solid-state drives, such as storage device 610. Volatile media include dynamic memory, such as main memory 606. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips, or magnetic tape cartridges.

[0218] Storage media differ from transmission media but can be used in conjunction with them. Transmission media participate in transferring information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including lines containing bus 602. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0219] Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 604 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit them over a telephone line using a modem. A modem local to computer system 600 may receive data over a telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on bus 602. Bus 602 carries the data to main memory 606, from which processor 604 retrieves and executes the instructions. The instructions received in main memory 606 may optionally be stored on storage device 610 before or after execution by processor 604.

[0220] Computer system 600 also includes a communication interface 618 coupled to bus 602. Communication interface 618 provides bidirectional data communication coupled to network link 620 connected to local network 622. For example, communication interface 618 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem providing a data communication connection to a corresponding type of telephone line. As another example, communication interface 618 may be a local area network (LAN) card providing a data communication connection to a LAN-compatible network. A wireless link may also be implemented. In any such implementation, communication interface 618 transmits and receives electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information.

[0221] Network link 620 typically provides data communication to other data devices via one or more networks. For example, network link 620 can provide a connection via local network 622 to host 624 or to data equipment operated by Internet Service Provider (ISP) 626. ISP 626 then provides data communication services via a global packet data communication network now commonly referred to as the "Internet" 628. Both local network 622 and Internet 628 use electrical, electromagnetic, or optical signals that carry digital data streams. Signals through various networks, as well as signals on network link 620 and through communication interface 618 (which carry digital data to and from computer system 600), are example forms of transmission media.

[0222] Computer system 600 can send messages and receive data, including program code, through one or more networks, network links 620, and communication interfaces 618. In the Internet example, server 630 can transmit application request codes through the Internet 628, ISP 626, local network 622, and communication interface 618.

[0223] The received code may be executed by processor 604 upon receipt and / or stored in storage device 610 or other non-volatile storage device for later execution.

[0224] In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary with implementation. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. The unique and exclusive indications of the scope of this invention, and what the applicant intends to be within the scope of this invention, are the literal and equivalent scope of the set of claims claimed in this application, in the specific form claimed by those claims, including any subsequent corrections.

[0225] Software Overview

[0226] Figure 7 is a block diagram of a basic software system 700 that can be used to control the operation of computer system 600. The software system 700 and its components, including their connections, relationships, and functions, are intended to be exemplary only and are not intended to limit the implementation of one or more example embodiments. Other software systems suitable for implementing one or more example embodiments may have different components, including components with different connections, relationships, and functions.

[0227] Software system 700 is provided to guide the operation of computer system 600. Software system 700, which may be stored in system memory (RAM) 606 and on fixed storage devices (e.g., hard disk or flash memory) 610, includes a kernel or operating system (OS) 710.

[0228] OS 710 manages low-level aspects of computer operations, including managing process execution, memory allocation, file input and output (I / O), and device I / O. One or more applications, designated 702A, 702B, 702C...702N, can be "loaded" (e.g., transferred from fixed storage device 610 to memory 606) for execution by system 700. Applications or other software intended for use on computer system 600 can also be stored as a set of downloadable computer-executable instructions, for example, for downloading and installing from an internet location (e.g., a web server, application store, or other online service).

[0229] Software system 700 includes a graphical user interface (GUI) 715 for receiving user commands and data graphically (e.g., "clicks" or "touch gestures"). System 700 can then act on these inputs according to instructions from operating system 710 and / or (one or more) applications 702. GUI 715 is also used to display the results of operations from OS 710 and (one or more) applications 702, whereby the user can provide additional input or terminate the session (e.g., log off).

[0230] OS 710 can be executed directly on the bare hardware 720 of computer system 600 (e.g., one or more processors 604). Alternatively, a hypervisor or virtual machine monitor (VMM) 730 can be inserted between the bare hardware 720 and OS 710. In this configuration, VMM 730 acts as a software "buffer" or virtualization layer between OS 710 and the bare hardware 720 of computer system 600.

[0231] VMM 730 instantiates and runs one or more virtual machine instances ("guest machines"). Each guest machine includes a "guest" operating system, such as OS 710, and one or more applications designed to run on the guest operating system, such as application(s)702. VMM 730 presents a virtual operating platform for the guest operating system and manages the execution of the guest operating system.

[0232] In some cases, VMM 730 can allow a guest operating system to run as if it were running directly on the bare hardware 720 of computer system 600. In these cases, the same version of the guest operating system configured to run directly on the bare hardware 720 can also run on VMM 730 without modification or reconfiguration. In other words, in some situations, VMM 730 can provide complete hardware and CPU virtualization for a guest operating system.

[0233] In other cases, the guest operating system can be specifically designed or configured to run on the VMM 730 for improved efficiency. In these cases, the guest operating system "knows" that it is running on the virtual machine monitor. In other words, in some situations, the VMM 730 can provide paravirtualization for the guest operating system.

[0234] Computer system processes include hardware processor time allocation and (physical and / or virtual) memory allocation. Memory allocation is used to store instructions executed by the hardware processor, to store data generated by the execution of instructions by the hardware processor, and / or to store hardware processor state (such as the contents of registers) between hardware processor time allocations when the computer system process is not running. Computer system processes run under the control of the operating system and can also run under the control of other programs executing on the computer system.

[0235] The basic computer hardware and software described above are presented to illustrate the basic underlying computer components that can be used to implement one or more example embodiments. However, the one or more example embodiments are not necessarily limited to any particular computing environment or computing device configuration. Rather, according to this disclosure, the one or more example embodiments can be implemented in any type of system architecture or processing environment that will be understood by those skilled in the art to be capable of supporting the features and functionality of the one or more example embodiments shown herein.

[0236] cloud computing

[0237] The term "cloud computing" is generally used in this article to describe a computing model that enables on-demand access to a shared pool of computing resources, such as computer networks, servers, software applications, and services, and allows for the rapid provisioning and release of resources with minimal management effort or service provider interaction.

[0238] Cloud computing environments (sometimes referred to as cloud environments or the cloud itself) can be implemented in a wide variety of ways to best meet different needs. For example, in a public cloud environment, the underlying computing infrastructure is owned by the organization that makes its cloud services available to other organizations or the public. In contrast, private cloud environments are typically designed for use by a single organization or within a single organization. Community clouds are designed to be shared by several organizations within a community; while hybrid clouds include two or more types of clouds (e.g., private, community, or public) that are bound together by data and application portability.

[0239] Typically, cloud computing models enable some of the responsibilities that might have previously been provided by an organization's own IT departments to be delivered as service layers within a cloud environment for consumers (whether internal or external, depending on the public / private nature of the cloud). Depending on the specific implementation, the precise definition of the components or features provided or internal to each cloud service layer can vary, but common examples include: Software as a Service (SaaS), where consumers use software applications running on cloud infrastructure, while the SaaS provider manages or controls the underlying cloud infrastructure and applications; Platform as a Service (PaaS), where consumers can use software programming languages ​​and development tools supported by the PaaS provider to develop, deploy, and otherwise control their own applications, while the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything in the runtime execution environment); and Infrastructure as a Service (IaaS), where consumers can deploy and run arbitrary software applications and / or provision processing, storage, networking, and other basic computing resources, while the IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything below the operating system layer). Database as a Service (DBaaS) is a service in which consumers use database servers or database management systems running on cloud infrastructure, while DBaaS providers manage or control the underlying cloud infrastructure, applications, and servers, including one or more database servers.

[0240] In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary with implementation. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. The unique and exclusive indications of the scope of this invention, and what the applicant intends to be within the scope of this invention, are the literal and equivalent scope of the set of claims claimed in this application, in the specific form claimed by those claims, including any subsequent corrections.

Claims

1. A method for execution within a database system, comprising: Access multiple documents; Generate a vector table and a text table; for each of the multiple documents: convert the data within each document into plain text; Multiple blocks are generated based on the plain text; Multiple vectors are generated based on the multiple blocks using an embedding model; the multiple vectors are stored in the vector table; a document identifier is stored in the vector table associated with each of the multiple vectors, the document identifier identifying each document; multiple word segments are generated based on the plain text; the multiple word segments are stored in the text table; the document identifier is stored in the text table associated with each of the multiple word segments; wherein the method is performed by one or more computing devices.

2. The method according to claim 1, wherein: The plurality of documents includes a first document of a first type and a second document of a second type different from the first type; converting the data in each document includes: for the first document among the plurality of documents, performing a first conversion operation on the first document based on the first document being of the first type; and for the second document among the plurality of documents, performing a second conversion operation on the second document based on the second document being of the second type.

3. The method according to claim 1, further comprising: Receives a database statement instructing the database system to create a hybrid index; The generation of the vector table and the text table is performed in response to receiving the database statement.

4. The method according to claim 3, further comprising: In response to receiving the database statement, a vector index is generated on the vector table; In response to receiving the database statement, a text index is generated on the text table.

5. The method of claim 3, wherein the database statement indicates the embedded model.

6. The method of claim 3, wherein the database statement indicates the type of the vector index, the method further comprising: A vector index of the type is generated in response to receiving the database statement.

7. The method of claim 3, wherein the database statement indicates a technique for segmenting the plain text, wherein generating the plurality of blocks is performed using the technique.

8. The method of claim 3, wherein the database statement indicates a distance operation for calculating the distance between two vectors.

9. The method according to claim 1, further comprising: Receive mixed queries indicating one or more search terms; Generate one or more query vectors based on the one or more search terms; Based on the one or more query vectors, identify one or more document identifiers from the vector table.

10. The method of claim 9, wherein the one or more search terms are one or more first search terms, the method further comprising: In response to receiving the mixed query, a text query including one or more second search terms is generated, wherein the one or more second search terms are (i) one or more of the one or more first search terms, or (ii) different from the one or more first search terms; Executing the text query includes: identifying one or more second document identifiers from the text table based on the one or more second search terms.

11. The method of claim 9, wherein the hybrid query is a first hybrid query, wherein the first hybrid query is an SQL query, and the method further comprises: Receive a second hybrid query that indicates the hybrid index, wherein the second hybrid query is a NoSQL query or a low SQL query.

12. The method according to claim 9, further comprising: Use the one or more query vectors to identify multiple blocks in the vector table; Determine that two or more of the plurality of blocks belong to a specific document among the plurality of documents; for a first block among the two or more blocks: identify a first plurality of block scores of the blocks surrounding the first block; generate a first adjusted score for the first block based on the first plurality of block scores and the first block score of the first block; for a second block among the two or more blocks: identify a second plurality of block scores of the blocks surrounding the second block; generate a second adjusted score for the second block based on the second plurality of block scores and the second block score of the second block; determine the score of the specific document based on the first adjusted score and the second adjusted score.

13. The method of claim 9, further comprising: Use the one or more query vectors to identify a set of blocks in the vector table; Determine that multiple blocks in the set of blocks belong to a specific document in the multiple documents; identify a window size smaller than the number of blocks in the multiple blocks; identify multiple subsets of the multiple blocks; for each subset of the multiple subsets: generate an average score of the block scores of the blocks in each subset; Add the average score to a set of average scores; identify the maximum average score in the set of average scores; associate the maximum average score with the specific document.

14. The method according to claim 1, further comprising: Receive a mixed query indicating one or more search terms; in response to receiving the mixed query, generate (1) a text subquery for the text table and (2) a vector subquery for the vector table; execute the text subquery to generate a first set of results; Execute the vector subquery to generate a second set of results; combine the first set of results with the second set of results to generate the final result set.

15. The method of claim 14, wherein the first set of results is a first set of document identifiers, wherein the second set of results is a second set of document identifiers, wherein the final result set is (i) the union of the first set of document identifiers and the second set of document identifiers, or (ii) the intersection of the first set of document identifiers and the second set of document identifiers.

16. A method executed by a database system, comprising: Access multiple documents, including documents of different types; for each of the multiple documents: identify the type of each document; Select a transformation operation corresponding to the type; use the transformation operation to convert the data in each document into plain text; generate a vector index based on the plain text; generate a text index based on the plain text; wherein the method is performed by one or more computing devices.

17. One or more non-transitory storage media storing instructions that, when executed by one or more computing devices of a database system, cause the method according to any one of claims 1-16 to be performed.

Images