Tensor engine driven vector database method and system

Abstract

The application provides a tensor engine driven vector database method and system, including the following steps: S1, adaptive data recognition and vector conversion; S2, obtaining intelligent data, dynamically selecting a state query strategy, and finally outputting adaptive results; S3, collecting user satisfaction and continuously optimizing a model. The application is different from the traditional CPU-oriented database architecture 10-20 years ago, discards these old computing architectures, starts from a tensor computing engine naturally supporting GPU acceleration, and constructs a new analytical vector database oriented to GPU computing and surrounding tensor data computing. This makes it have a significant advantage in the vector analysis and search scene, and can utilize the characteristics (such as GPUDirect RDMA) in the CUDA code library supported by the tensor computing framework to accelerate network transmission.

Description

Technical Field

[0001] This invention relates to the field of databases, and more specifically to a vector database method and system driven by a tensor engine. Background Technology

[0002] With the rapid development of modern internet, sensor technology, and applications, we are observing an increasing amount of data taking the form of high-dimensional vectors, such as images, text, and audio. The prevalence and complexity of this data have exceeded the processing capabilities of traditional database systems.

[0003] Against this backdrop, the development of deep learning has made data embedding a mainstream data representation method. Through deep learning models, we can map complex data structures such as text and images into a high-dimensional vector space. These vectors not only capture the surface features of the data but also reveal its semantic information in depth. This vector representation is particularly important when it comes to tasks such as similarity calculation and classification.

[0004] Furthermore, an increasing number of applications require fast similarity searches on large-scale datasets, such as image search, recommendation systems, and natural language processing. However, traditional database indexing methods are inefficient when dealing with high-dimensional data, which is precisely the problem that vector databases attempt to solve.

[0005] From 2022 to 2023, although the GPT large model showed a strong interest in vector databases, research on vector databases remained relatively limited at important academic conferences such as VLDB and SIGMOD, indicating that this is an emerging and yet-to-be-explored technological field. From a market perspective, DBRank statistics show that major vector databases such as Milvus, pinicone, and weakate have all emerged within the last five years.

[0006] Currently, the main application of vector databases focuses on vector similarity search, which relies on pre-built vector indexes to achieve millisecond-level similarity queries. This query method is well-suited for similarity searches across various formats, including text, audio / video, and biological data.

[0007] However, just as indexing is not suitable for all scenarios in OLTP databases, vector databases face similar challenges. Beyond similarity search, there are a range of analytical needs, such as clustering, anomaly detection, dimensionality reduction, and association analysis, which require a more in-depth analysis of the entire vector dataset. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, this invention proposes a vector database method and system based on a tensor engine. Unlike traditional CPU-oriented database architectures from 10-20 years ago, this invention abandons these outdated computing architectures and builds a completely new analytical vector database oriented towards GPU computing and focused on tensor data computation, starting from a tensor computing engine that naturally supports GPU acceleration. This gives it significant advantages in vector analysis and search scenarios. The CUDA codebase supported by the tensor computing framework can support network acceleration similar to GPUDirect RDMA, offering greater potential for distributed data computation.

[0009] To achieve the above objectives, this invention provides a vector database method driven by a tensor engine, comprising:

[0010] Step S1: Adaptive data recognition and vector transformation;

[0011] Step S2: Acquire intelligent data, dynamically select a query strategy, and finally output adaptive results;

[0012] Step S3: Collect user satisfaction data and continuously optimize the model.

[0013] Furthermore, step S1 is detailed as follows:

[0014] Step S11: Use a decision tree model to predict and determine the data type, and select the most appropriate method for subsequent transformation;

[0015] Step S12: Use a collaborative filtering recommendation system to automatically select the best vector transformation method;

[0016] Step S13: Based on the data content, apply deep autoencoder technology to automatically generate vector representations in order to capture deeper levels of information;

[0017] Step S14: Store the vector data and the original data in KineVectorStore;

[0018] Step S15: Dynamically update the data using the B-tree index based on the data update status and query frequency.

[0019] Furthermore, step S2 is detailed as follows:

[0020] Step S21: Use KineVectorCompute in conjunction with the LRU (Least Recently Used) intelligent caching system to quickly retrieve frequently queried data;

[0021] Step S22: Use Support Vector Machine (SVM) to predict the user's possible query needs and dynamically select a query strategy;

[0022] Step S23: Based on query analysis and user historical feedback, optimize using K-means clustering and return the most relevant result data.

[0023] Furthermore, step S3 is as follows:

[0024] Step S31: Provide a React-based interface or RESTful API for users to rate the query results or leave feedback;

[0025] Step S32: Based on user feedback and query logs, continuously adjust and train the vector transformation and query strategy models using online learning strategies.

[0026] Furthermore, step S1 also includes data augmentation strategies, specifically using generative adversarial networks (GANs) to generate augmented data, increasing the diversity and robustness of vector data, and improving the availability of rare or uncommon data samples through random noise injection.

[0027] Furthermore, step S2 also includes an adaptive visualization interface, as detailed below:

[0028] Step S24: Based on the user's query history and preferences, use the D3.js library to provide an automatically adjusted data visualization interface.

[0029] Step S25: Provide an interactive exploration tool based on Three.js for the returned results to enhance the user experience.

[0030] A vector database system driven by a tensor engine, applicable to the aforementioned vector database method driven by a tensor engine, is characterized by including a client layer and a computation management layer (KineVectorCompute Layer).

[0031] The client layer is used to provide users with multiple access and interaction methods to meet different application scenarios and user habits;

[0032] The computing management layer is used for core computing and query management, and processes all data requests sent by clients.

[0033] Furthermore, the client layer includes a native Client API, an SQL-like syntax interface, and a RESTful API;

[0034] The Client native API provides developers with a dedicated set of APIs that allow them to interact directly with the database;

[0035] The SQL-like syntax interface provides a familiar query method for users familiar with traditional database operations;

[0036] The RESTful API allows users to access the database remotely over the network from anywhere, on any device.

[0037] Furthermore, the computational management layer includes a Kine Coordinator component, a MetaService component, and a Native Tensor Worker component;

[0038] The Kine Coordinator component serves as the main entry point of the system and is responsible for parsing and routing client requests.

[0039] The MetaService component maintains the database's metadata, such as table structure, permissions, and node information.

[0040] The Native Tensor Worker component is the computational core of KineVectorDB, utilizing a tensor computation engine, particularly GPUs, to perform efficient vector computation and analysis.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] 1. This invention provides a vector database method and system based on a tensor engine. Unlike traditional CPU-oriented database architectures from 10-20 years ago, it abandons these outdated computing architectures and builds a completely new analytical vector database oriented towards GPU computing and centered around tensor data computation, starting from a tensor computing engine that naturally supports GPU acceleration. This gives it significant advantages in vector analysis and search scenarios.

[0043] 2. This invention provides a vector database method and system based on a tensor engine, which supports external custom vector computation algorithms, such as AI models, custom vector extraction and computation algorithms, and third-party plugins, and allows users to use the API of a native tensor engine (such as PyTorch) to perform custom vector processing.

[0044] 3. This invention provides a vector database method and system based on a tensor engine, which can efficiently handle large-scale tensor calculations because it is based on a tensor calculation engine that naturally supports such calculations and is accelerated by GPUs.

[0045] 4. This invention provides a vector database method and system based on a tensor engine. It utilizes a tensor computing engine like PyTorch and has CUDA / ROCm GPU codebase support, enabling efficient execution on Nvidia / AMD graphics cards. It also supports parallel and distributed computing acceleration for single-machine multi-GPU and multi-machine multi-GPU setups.

[0046] 5. This invention provides a vector database method and system based on tensor engine. Compared with traditional databases and some modern vector databases, this invention can utilize the features of the CUDA codebase supported by the tensor computing framework (such as GPUDirect RDMA) to accelerate network transmission, especially in distributed data computing. Attached Figure Description

[0047] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the process of this invention;

[0049] Figure 2 This is a data flow diagram of the present invention;

[0050] Figure 3 This is a diagram of the overall architecture of the present invention;

[0051] Figure 4 This is a schematic diagram of the client layer;

[0052] Figure 5 This is a schematic diagram of the KineCoordinator component;

[0053] Figure 6 This is a schematic diagram of the task coordinator's processing flow;

[0054] Figure 7 This is a schematic diagram of the MetaService component;

[0055] Figure 8 This is a schematic diagram of the Native Tensor Worker component;

[0056] Figure 9 This is a schematic diagram of KineVectorStore. Detailed Implementation

[0057] The technical solution of the present invention will be more clearly and completely explained below with reference to the accompanying drawings and through the description of preferred embodiments of the present invention.

[0058] Explanation of technical terms:

[0059] TensorFlow: An open-source deep learning framework developed by Google, featuring powerful tensor computation capabilities and supporting CPU and GPU acceleration.

[0060] PyTorch: An open-source deep learning framework developed by Facebook, it also boasts powerful tensor computation capabilities and supports CPU and GPU acceleration. PyTorch is characterized by its dynamic computation graph, making model building and debugging more flexible.

[0061] MXNet: An open-source deep learning framework, originally developed by Amazon and later becoming an Apache Foundation project. MXNet supports multiple programming languages ​​and boasts excellent computational performance.

[0062] Caffe: A deep learning framework developed by the Berkeley Center for Vision and Learning, primarily used for image recognition tasks, featuring efficient forward propagation computation.

[0063] Theano: A deep learning framework developed by the University of Montreal, primarily focused on automatic differentiation of mathematical expressions, and is no longer under active development.

[0064] Faiss: A vector similarity search library open-sourced by Facebook.

[0065] Proxima: An open-source vector similarity calculation library from DAMO Academy, similar to Faiss.

[0066] Vector: a one-dimensional array

[0067] Matrix: a 2D array

[0068] Tensor: Any multidimensional array is called a tensor. Vectors, matrices, and scalars can all be special tensors;

[0069] Tensor computing engines: Computation engines that use tensors as the basic data structure, such as PyTorch, TensorFlow, MXNet, Caffe, Theano, and other common deep learning computing frameworks in the AI ​​field, are all collectively referred to as tensor computing engines.

[0070] Source Data: This refers to the data originating from a source. It encompasses various types of data, including structured, semi-structured, and unstructured data.

[0071] Structured data refers to data organized in a tabular or table-like format, with clearly defined rows and columns, and each cell containing a specific value. This data is typically stored in traditional relational databases.

[0072] Unstructured data: refers to unstructured data, such as images, audio, video, and text.

[0073] Semi-structured data: refers to semi-structured data, such as XML / HTML / JSON content.

[0074] transform: represents the process of converting data of various types into another format.

[0075] Vectors: Represents vector data.

[0076] Latent-sematic Data: Data that has certain business significance.

[0077] Store: Represents the storage layer.

[0078] KineVectorStore: Indicates the vector storage engine of this patent.

[0079] KineVectorCompute: Represents the vector computation component of this patent.

[0080] Client Layer: This refers to the client logic layer in the architecture of this patented solution.

[0081] Meta Service: refers to the metadata service in this patent.

[0082] Native Tensor Worker: This refers to the native tensor computation node in this patented solution, which encapsulates the AI-native tensor computation engine as a processor for complex vector analysis computation.

[0083] KineCoordinator: This refers to the task coordinator in this patented solution, which identifies user requests and coordinates the corresponding components for processing.

[0084] RESTful API: It is essentially an HTTPS / HTTP network request interface.

[0085] SQL-Like: This is essentially the SQL-like operation interface provided by this patent.

[0086] Client API: This refers to the API direct call interface provided in this patent solution.

[0087] WebService: In this patent, a web server for the component is specifically designed to receive external requests for interface calls to the component.

[0088] Parser: This refers to the user request parser in this patent, which is used to identify the task intent of the user request content and construct the corresponding executable task.

[0089] Scheduler: This refers to the task scheduler in this patent, which schedules the execution of user-requested tasks.

[0090] Processor: This refers to the task processing thread of the coordinator in this patented solution, which is mainly responsible for executing user tasks.

[0091] Meta API: Represents the externally exposed interfaces of the metadata service.

[0092] Access Manager: This refers to the access management controller, which is responsible for verifying access permissions / identity.

[0093] MetaManager: Represents the metadata manager, which manages all the metadata information required by the system, such as Database (database definition metadata), Table (table definition metadata), Partitions (partition definition metadata), Columns (column definition metadata), Function (function definition metadata), Privileges (permission definition metadata), and so on.

[0094] Persistence: This encapsulates the interface for the actual persistence of this metadata, ensuring that the corresponding metadata is truly stored in memory; Save / Load represents data storage or retrieval operations.

[0095] Distributed Vector Storage: Represents the data storage for this vector database.

[0096] Worker API: Represents the external operation interface of this compute node; Task Parser: Represents the task parser constructor; Task Runner: Represents the task runner, which will execute each task in four steps: config (step), DataLoad (data loading step), Prepare (optional data preprocessing step), and Runner (actual task execution stage).

[0097] Tensor Compute Engine: This refers to the native tensor computation engine, which has various functions, including Tensor (representation of tensor data structures), Functions (various tensor computation functions), Neural Network (definition of neural network structures), DP (Data Parallel) (data parallel processing capabilities), GPU (graphics card acceleration capabilities), and Distributed (distributed computing capabilities); KineVectorStorage (vector storage); and Load Vectordata (loading vector data).

[0098] Vector / Scalar Unified API: This represents the unified operation interface for this memory.

[0099] Pluggable NoSQL System: This indicates a pluggable NoSQL system that is encapsulated as the actual data storage system.

[0100] Vector databases are a special type of database designed to store and manage high-dimensional vector data and provide efficient vector similarity search functionality. Unlike traditional relational databases or document databases, vector databases focus on processing data represented by vectors, such as image features, text embeddings, and audio features. These vectors can be generated in deep learning models or obtained through other feature extraction methods.

[0101] like Figure 1 As shown, the present invention includes:

[0102] Step S1: Adaptive data recognition and vector transformation;

[0103] Step S2: Acquire intelligent data, dynamically select a query strategy, and finally output adaptive results;

[0104] Step S3: Collect user satisfaction data and continuously optimize the model.

[0105] Step S1 is as follows:

[0106] Step S11: Use machine learning models to predict and determine the data type, and select the most suitable method for subsequent transformation;

[0107] Step S12: Use a recommender system or machine learning model to automatically select the best vector transformation method;

[0108] Step S13: Based on the data content, apply deep learning technology to automatically generate vector representations in order to capture deeper information;

[0109] Step S14: Store the vector data and the original data in KineVectorStore;

[0110] Step S15: Dynamically update the data index based on data update status and query frequency.

[0111] Step S2 is as follows:

[0112] Step S21: Use KineVectorCompute in conjunction with an intelligent caching system to quickly retrieve frequently queried data;

[0113] Step S22: Use machine learning models to predict users' possible query needs and dynamically select query strategies;

[0114] Step S23: Based on query analysis and user historical feedback, optimize and return the most relevant result data.

[0115] Step S3 is as follows:

[0116] Step S31: Provide an interface or API for users to rate the query results or leave feedback;

[0117] Step S32: Based on user feedback and query logs, continuously adjust and train the model for vector transformation and query strategies.

[0118] Step S1 also includes data augmentation strategies, specifically generating augmented data through machine learning models to increase the diversity and robustness of vector data, and then improving the availability of rare or uncommon data samples through synthesis techniques.

[0119] Step S2 also includes an adaptive visualization interface, as detailed below:

[0120] Step S24: Provide an automatically adjusted data visualization interface based on the user's query history and preferences.

[0121] Step S25: Provide interactive exploration tools for the returned results to enhance the user experience.

[0122] like Figure 2 As shown, the main data flow in the insert stage is as follows: the source data is transformed into vector data and then stored in KineVectorStore; the main data flow in the query stage is as follows: KineVectorCompute traverses the vector data in KineVectorStore and uses the tensor calculation engine to accelerate the analysis and calculation to obtain the corresponding results, and then outputs the result data.

[0123] Source Data represents various types of external data, including structured, semi-structured, and unstructured data.

[0124] Structured data refers to data organized in tabular or table-like formats, with clearly defined rows and columns, and each cell containing a specific value. This data is typically stored in traditional relational databases. To convert this structured data into vector data, various AI algorithms can be employed. For example, a column of numeric data can be normalized to obtain vector data; a column of text data can be converted into vectors using AI models such as word2vec / text2vec; and boolean types can be directly represented by simple 0 / 1 vectors.

[0125] For semi-structured data, such as XML / HTML / JSON, third-party libraries can typically be used to extract the required content according to business needs. Then, the numerical and textual content can be vectorized using methods similar to those described above. The conversion methods can still be categorized into two types: algorithmic methods and AI model-based methods.

[0126] For unstructured data, such as images, audio, video, and text, AI methods can typically be used to extract feature vectors. For example, convolutional neural networks can be used to extract local features to represent the image, while NLP techniques can be used to extract feature vectors from text data. Audio data can be converted into vector representations such as Mel-frequency spectra using audio technology. These are well-established methods in the AI ​​field and will not be elaborated upon here.

[0127] During the insert process, in addition to being converted into specific feature vectors, the original data is also retained as payload and stored together in the KineVectorStore storage system. Here's an example to illustrate why it's necessary to store both vector data and original data, and their logical storage format.

[0128] For example, given a 256*256 RGB image, we extract features from it using convolutional neural networks like ResNet / VGG to obtain a high-dimensional vector (collectively called a tensor). This high-dimensional vector doesn't store complete image information; it only contains feature values ​​and cannot directly reconstruct the original image. This high-dimensional vector data is only used as a comparison condition for matching during real-time similarity searches. After a match is found, to obtain the original image that meets the criteria, a method is needed to retrieve it. Therefore, the stored data here is...<vectors,payload> It is stored in a way that calculates the matching degree using vectors; if the conditions are met, the payload contains the data it needs.

[0129] Finally, in the query analysis phase, KineVectorCompute represents the query engine component of our vector database. Based on the user's query task, it selects an appropriate query strategy to perform vector matching search or analysis and returns the final results. There are two types of queries here. The first is the traditional direct similarity search method using vector indexes, similar to current vector databases on the market, where the search is performed by first anticipating the results.<vectors,payload> The inverted vector index uses the similarity matching degree of vectors during searches to return payload metadata. Another type of business, which is the large-scale vector analysis business we need to solve, involves a full table scan to obtain vector data or raw payload data, performing comprehensive re-vector calculation, analysis, and statistics to obtain the final result. We will elaborate on these two query types here.

[0130] The first method: Fast search using vector similarity index

[0131] This is the main approach used by vector databases currently on the market. The core idea is to first calculate and transform the data into a numerical value, creating an index from the vector value to the original data. When searching for the closest match to a given piece of data, we only need to convert the query data into the corresponding vector value in the same way, then query this index to obtain the original payload data that meets the criteria. Once the original data is obtained, it can be returned. Vector indexes for various scenarios include PQ and HNSW, which will not be elaborated upon here.

[0132] The second method: Large-scale vector data analysis and computation

[0133] This type of analytical query is used for business needs that require large-scale vector data analysis and computation to obtain an overall picture. Examples include clustering, low-frequency anomalies, and computationally intensive specific analyses, as mentioned above. This type of analysis is suitable for...<vectors,payload> For data in general, a full table scan is required to retrieve all payload data, followed by re-performing the established business logic, including vector feature extraction and analysis, to finally obtain the results and return the source data that meets the requirements. Of course, if pre-built vectors are available, appropriate filtering can be performed. However, this is not the case in most situations. Just like ad-hoc OLAP online analysis, some analytical scenarios are non-fixed, flexible, and involve infrequent repetitive queries, making it impossible to determine in advance what kind of vector indexes to build.

[0134] like Figure 3The diagram shows the overall architecture of the vector database solution of this invention, which can be roughly divided into three core layers. The first layer, the Client Layer, is a high-level client layer. Its core is to encapsulate the user-side API interface of the vector database, including native client APIs, SQL-like syntax, and RESTful APIs. The second layer is the KineVectorCompute layer, which is responsible for handling user vector analysis, query requests, and routine database source data management. It consists of three core components: the Kine Coordinator component, the MetaService component, and the NativeTensor Worker component. The MetaService stores the metadata information of the vector database, including database table column definitions, access control, and node information for each worker. The Native Tensor Worker is a GPU-oriented tensor computation node encapsulated based on a tensor computation engine, responsible for various actual tensor computation tasks. The Kine Coordinator is the entry point for coordinating computation, responsible for receiving and parsing user requests, and then performing different operations based on the query type. For DDL operations, the metadata management in the metaservice will be directly manipulated; for DQL operations, it will be determined whether the query is analytical or real-time search. If it is a real-time search, the vector similarity index of KineVectorStore will be directly queried to obtain the search results; if it is an analytical operation, Native Tensor Workers will be coordinated to perform distributed tensor analysis calculations manually, and their calculation results will be coordinated and finally returned to the client.

[0135] Below, we will provide further detailed explanations for each level.

[0136] like Figure 4 As shown, for the Client Layer, as Figure 4 The darker section primarily encapsulates three API types for client use: Client API, SQL-Like, and RESTful API. All three types of APIs send user requests to the Kine Coordinator's WebService for subsequent request parsing and execution, ultimately returning the result. These API types differ only in their usage; their actual execution is parsed into a unified request task by KineCoordinator, facilitating subsequent unified scheduling and execution of the corresponding tasks.

[0137] The KineVectorCompute computation layer comprises three main core components: Kine Coordinator, MetaService, and Native Tensor Worker. The core design and functional roles of each component are described below.

[0138] like Figure 5 As shown, the KineCoordinator component of KineVectorCompute plays a crucial role. Its core function is to receive user requests from the WebService, parse them according to the request type using a Parser, transform them into a Task, and submit them to the Scheduler for processing. The Scheduler is responsible for scheduling the tasks that process user requests to the corresponding idle Processor threads. Each Task executes using the same Run method in the Processor, but different Task types have different specific Run logic. For example, DDL Tasks, Analytical Tasks, and Search Tasks are relatively common task types in databases, and their traditional technical details will not be elaborated here. Instead, the core interaction flow of these tasks will be described below.

[0139] like Figure 6 As shown, when the Processor thread runs an Analytical Task, its core function is to act as a coordinator for distributed tensor computation analysis tasks. It initiates and distributes tasks to native tensor workers on each node, allowing them to perform tensor computations. The results are then returned to the Analytical Task for aggregation, further merging the results before returning them to the upper-layer client. When the Processor thread runs a Search Task (a simple vector similarity search), it directly searches the Distributed Vector Storage. This storage system uses a plugin approach, allowing integration with any NoSQL / RDBMS storage system, simply by adding the current vector indexing technology plugin. This part essentially represents a common system approach combining a traditional database with a vector indexing plugin. When the Processor thread runs a DDL Task, it directly operates on our MetaService, modifying, updating, or querying metadata such as table columns and permissions in the vector database, and directly returning the results to the client.

[0140] like Figure 7As shown, the MetaService component of KineVectorCompute provides metadata services for a typical vector database. For example, the Meta Manager manages metadata information such as databases, tables, columns, partitions, and permissions, and provides a Meta API for external calls, primarily used by DDL Tasks. The Access Manager then performs permission checks for each API call. The final CRUD operations on the metadata are persisted to a unified Distributed Vector storage using a Persistence abstraction interface. This Distributed Vector Storage is specifically adapted for Vector storage but can also store regular non-vector data.

[0141] like Figure 8 As shown, the KineVectorCompute Native Tensor Worker component is responsible for the analysis and computation of large-scale tensor data. It can consist of one or more workers, each responsible for parsing the analytical task initiated by the coordinator and launching a Task Runner to wrap and execute the task. The execution logic of each analytical task typically includes configuration preparation, DataLoad loading the data required for the task's computation, Prepare performing necessary preprocessing, and the Runner actually executing the user's business tensor computation and analysis. The Runner leverages the Tensor Compute Engine to perform complex tensor computations. The Tensor Compute Engine encapsulates GPU-accelerated, complex tensor computation frameworks such as PyTorch / TensorFlow, supporting complex tensor computations, complex deep learning neural network computations, and acceleration for single-machine multi-GPU and multi-machine multi-GPU computing. Our Tensor Compute Engine's role is to transform our business computations into tensor computations, fully utilizing these naturally GPU-accelerated tensor computation engines to accelerate our vector data analysis.

[0142] like Figure 9As shown in the diagram, KineVectorStore consists of two core parts. The first part provides a unified Vector query interface and a Scalar query interface, both of which are standard database CRUD operation interfaces. The second part encapsulates a pluggable NoSQL system as a storage layer, simplifying the storage of general-purpose data. This allows for the rapid construction of a complete, ACID-compliant storage system for storing vector data, scalar data, and inverted vector index data.

[0143] For example, the Meta Service uses common queries to retrieve data from the key-value storage system just like querying traditional scalar data. When the query is a simple vector similarity search task, the vector index is used to match the results, and then the original data is retrieved by scalar. When the query is an analytical task, all vector data and original data are retrieved directly for upper-layer tasks to perform full-scale analysis, computation, aggregation, and statistics.

[0144] As seen above, our storage system utilizes general-purpose NoSQL storage as a backend wrapper, without being limited to any specific system. Because of its pluggable design, it can connect to HBase, Cassandra, GraphDB, and more, depending on the complex computational analysis required at the upper layer. For example, if the upper layer involves complex graph neural network computations, our underlying storage data can also be connected to a graph database to facilitate these computations. If it's other types of image feature data, a standard key-value system can also be used. In short, the system prioritizes maximizing memory utilization and minimizing disk operations.

[0145] Based on the above descriptive information, we obtain the following qualitative effect comparison table:

[0146]

[0147] As shown in the table above, in the field of vector databases, we divide them into three types:

[0148] The first type of vector database is a traditional database that has evolved into a vector database with vector indexing plugins. Common products include: Alibaba Cloud Elasticsearch + Proxima, Alibaba Cloud ADB PosgreSQL + PGVector, Ant Financial ZSearch, Alibaba Cloud PolarDB + PASE vector index, etc. These databases typically use third-party extension plugins to obtain vector indexes, and then the engine modifies the search support for vector indexes, thus adding vector index support to the original database functionality.

[0149] The second category consists of commonly available vector databases, such as Zilliz / Milvus, Chroma, Pinecone, and KDB. These are mostly vector databases developed from the perspective of vector database requirements, with their architectures tailored to specific positioning. Although they are entirely self-developed around vectors, their overall architecture still belongs to traditional database computing architectures, designed and developed around the CPU. For example, Chroma uses ClickHouse + vector plugins, leveraging ClickHouse's efficient CPU processing of data, such as SIMD optimizations, and combining it with vector indexing plugins to quickly build a vector database with OLAP + vector similarity search capabilities. Milvus, on the other hand, uses a Pulsar / Kafka + distributed computing + object storage architecture, positioned as a cloud-native, vector-centric, and eventually consistent vector database. Its architecture is relatively complex, involving many external components integrated into the product.

[0150] The third category is the tensor-based analytical vector database of this invention. The core of this solution lies in abandoning the computational architectures of all existing databases. These architectures are 10 or even 20 years old and CPU-oriented; completely transforming them to support complex tensor computations on GPUs is extremely difficult, requiring extensive reconstruction—essentially a redesign. Therefore, we start with tensor computation engines that naturally support GPU acceleration and complex tensor analysis, and on this basis, build a completely new analytical vector database oriented towards GPU computation and focused on tensor data computation. This architecture is relatively new; our target scenario is not to replace traditional databases, but rather to address the initial intent of searching and analyzing vector databases. The tensor engine can be PyTorch / TensorFlow, etc.

[0151] The above explains the three types of vector databases compared in our table. Now, we will focus on explaining the basis for the comparison items in the table above.

[0152] Firstly, regarding "vector similarity search," this function is essentially a vector index search function. This vector index typically has similarity values ​​calculated using publicly available indexing algorithms at the time of insertion, forming an inverted index. Index-based searches are largely similar and generally excellent. While many traditional databases are written in high-level languages ​​like Java / Scala that run on virtual machines, the core of vector similarity search lies in the use of the entire indexing system. In most cases, it fully utilizes engineering optimizations such as caching, resulting in minimal language differences and almost no algorithmic differences, as these are all open-source, such as Proxima from DAMO Academy, Faiss from Facebook, and the Anny library, etc.

[0153] Regarding "custom vector analysis and computation algorithms," the traditional database + vector plugin approach is inferior because the plugins typically only extend the database's internal vector indexing functionality. Some databases are even commercially closed-source, offering limited support for custom vector computation capabilities. In contrast, common vector databases excel because they primarily focus on storing and computing vector data, generally reserving space for custom vector indexing algorithm implementations. For example, Milvus reserves space for external ResNet / VGG, custom algorithms, and the third-party plugin Towheeembedding framework to compute custom vector extraction methods. Like ordinary vector databases, the vector database of this invention supports external custom vector computation algorithms, including AI models, custom vector extraction and computation algorithms, and third-party plugins. It also supports using native tensor engine APIs to customize vector computation methods; for example, it allows the use of PyTorch-like functions for custom vector processing. This advantage stems from the fact that the tensor computation engine of this invention is a native PyTorch tensor computation engine.

[0154] In terms of "large-scale vector complex analysis," traditional vector indexing methods are insufficient. Supporting this requires full table scans and CPU computation on each vector data point. While some might incorporate GPU libraries, these are not yet mature. This is one of the core advantages of this invention. This invention leverages a tensor computation engine that natively supports large-scale tensor computation and GPU acceleration to perform various computational analyses on large-scale vectors. This computational architecture also has a natural advantage in "GPU acceleration." For example, based on the PyTorch tensor computation engine, it has CUDA / ROCm GPU codebase support, allowing some computations to be executed efficiently on Nvidia / AMD graphics cards. Furthermore, it supports parallel execution acceleration on a single machine with multiple GPUs through the DATA PARALLEL function, and distributed computing acceleration on multiple machines with multiple GPUs through the DISTRIBUTED DATA PARALLEL feature.

[0155] Regarding "network transmission acceleration," traditional databases typically rely on standard network programming or combinations with Thrift / Netty / Protobuf / gRPC, with limited optimizations for data transmission acceleration. Few databases can be modified to support RDMA-optimized network transmission. Vector databases are similar, but their newer architecture allows for more customization options, hence their good rating. This invention excels in this area because the CUDA codebase supported by the tensor computing framework supports network acceleration similar to GPUDirect RDMA, offering greater potential for distributed data computation.

[0156] The above-described specific embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Various modifications, substitutions, and improvements made by those skilled in the art to the technical solutions of the present invention based on the provided textual description and drawings, without departing from the design concept and spirit of the present invention, should all fall within the scope of protection of the present invention. The scope of protection of the present invention is determined by the claims.

Claims

1. A vector database method based on a tensor engine, characterized in that, include: Step S1: Adaptive data recognition and vector transformation, as detailed below: Step S11: Use machine learning models to predict and determine the data type, and select a method for subsequent transformation; Step S12: Automatically select a vector transformation method using a recommender system or machine learning model; Step S13: Based on the data content, apply deep learning technology to automatically generate vector representations in order to capture deeper information; Step S14: Store the vector data and the original data in KineVectorStore; Step S15: Dynamically update the data index based on data update status and query frequency; Step S2: Acquire intelligent data, dynamically select query strategies, and finally output adaptive results, as follows: Step S21: Use KineVectorCompute in conjunction with an intelligent caching system to quickly retrieve frequently queried data; Step S22: Use machine learning models to predict users' possible query needs and dynamically select query strategies; Step S23: Based on query analysis and user historical feedback, optimize and return the most relevant result data; Step S3: Collect user satisfaction data and continuously optimize the model, as detailed below: Step S31: Provide an interface or API for users to rate the query results or leave feedback; Step S32: Based on user feedback and query logs, continuously adjust and train the model for vector transformation and query strategies.

2. The vector database method based on tensor engine driving according to claim 1, characterized in that, Step S1 also includes data augmentation strategies, specifically generating augmented data through machine learning models to increase the diversity and robustness of vector data, and then improving the availability of rare or uncommon data samples through synthesis techniques.

3. The vector database method based on tensor engine driving according to claim 1, characterized in that, Step S2 also includes an adaptive visualization interface, as detailed below: Step S24: Provide an automatically adjusted data visualization interface based on the user's query history and preferences; Step S25: Provide interactive exploration tools for the returned results to enhance the user experience.

4. A vector database system driven by a tensor engine, applicable to the vector database method driven by a tensor engine as described in any one of claims 1-3, characterized in that, Includes the client layer and the computing management layer; The client layer is used to provide users with multiple access and interaction methods to meet different application scenarios and user habits; The computing management layer is used for core computing and query management, and processes all data requests sent by clients.

5. The vector database system based on tensor engine driving according to claim 4, characterized in that, The client layer includes a native Client API, an SQL-like syntax interface, and a RESTful API; The Client native API provides developers with a dedicated set of APIs that allow them to interact directly with the database; The SQL-like syntax interface provides a familiar query method for users familiar with traditional database operations; The RESTful API allows users to access the database remotely over the network from anywhere, on any device.

6. The vector database system based on tensor engine driving according to claim 4, characterized in that, The computational management layer includes the Kine Coordinator component, the MetaService component, and the Native Tensor Worker component; The Kine Coordinator component serves as the main entry point of the system, responsible for parsing and routing client requests; The MetaService component maintains the database's metadata, such as table structure, permissions, and node information; The Native Tensor Worker component is the computational core of KineVectorDB, utilizing a tensor computation engine to perform efficient vector computation and analysis.

Images