Intelligent endogenous database system and data processing method thereof

Abstract

The invention relates to the technical field of databases, and particularly discloses an intelligent endogenous database system and a data processing method. According to the system, an artificial intelligence calculation engine is determined to be a native component which is at the same level as a relational query engine and is deeply integrated into a database kernel, and a vectorization scheduler and a unified query optimization engine are created. The basic principle of the method is as follows: in response to an unstructured data write-in transaction, a vectorization scheduler automatically triggers an artificial intelligence calculation engine in a database transaction boundary, an AI model is scheduled to perform in-situ calculation and vectorization on data in a database, and a result and original data are stored in a unified storage management layer in an atomized association manner. According to the method, a safe closed loop of a data processing flow in a single database system process is realized, intrinsically safe data without ex-warehouse, cross-domain zero-copy in-situ high-performance calculation and intelligent optimization of mixed query are achieved, and the systematic defects of a traditional externally-hung architecture in safety, performance and complexity are overcome.

Description

Technical Field

[0001] This invention relates to the field of databases, and more specifically, to an intelligent, endogenous database system and its data processing method. Background Technology

[0002] With the current trend of big data and artificial intelligence converging, database systems have evolved from simple structured data management platforms into core infrastructures capable of processing and analyzing massive amounts of unstructured data such as text, images, audio, and video. To enable databases to understand unstructured data content, the industry is widely exploring combining them with artificial intelligence models to achieve intelligent vectorization and semantic retrieval of data—this is the core development direction of "intelligent databases."

[0003] Currently, the mainstream technical approaches to achieving this goal can be summarized as plug-in or add-on integration solutions. Specifically, there are two main types: First, application-layer calls, where the application retrieves data from the database, calls an external AI service for vectorization, and then stores the results in a separate vector database. This process requires writing a significant amount of code for workflow coordination and data synchronization. Second, database plugins, which add extension plugins to traditional databases to enable the database to store and retrieve vectors. However, the vector computation itself still needs to be delegated to external processes or services through external function calls.

[0004] However, these existing technical solutions suffer from a series of inherent and interconnected serious flaws. First, regarding data security, unstructured data and its vectors must leave the controlled memory space of the database process and flow between multiple independent systems, fundamentally undermining the original security boundaries of the database and introducing significant risks of sensitive data leakage. Second, in terms of performance efficiency, the multiple serialization, deserialization, network transmission, and cross-process copying of data across multiple systems introduce huge additional overhead, resulting in high processing latency and low throughput, making it difficult to meet real-time business requirements. Third, in terms of system architecture and operation and maintenance, the introduction of multiple heterogeneous components such as databases, AI services, and vector databases leads to an exponential increase in system complexity, deployment costs, and operational burden. Finally, and crucially, regarding query capabilities, because the AI ​​computation process is an invisible "black box" to the database optimizer, the optimizer cannot estimate its computational cost and therefore cannot perform global cost optimization for mixed queries that simultaneously include structured condition filtering and unstructured semantic retrieval, resulting in poor query performance.

[0005] In summary, existing architectures suffer from inherent contradictions in terms of security, performance, complexity, and intelligence, necessitating a novel systemic solution that natively integrates AI capabilities with the database kernel. Summary of the Invention

[0006] The purpose of this invention is to provide a database system solution that natively integrates AI capabilities with the database kernel.

[0007] According to a first aspect of the present invention, an intelligent endogenous database system is proposed, comprising: The intelligent processing layer includes: Relational query engines are used to process structured query requests; The artificial intelligence computing engine, as a native engine at the same level as the relational query engine, is used to load and run at least one AI model within the database system process; A unified storage management layer is used to manage the persistent storage of raw data and its derived vectorized data in a unified manner. The vectorized scheduler is used to respond to write operations on unstructured data. Within the database transaction boundary, it automatically triggers the artificial intelligence computing engine, schedules at least one AI model to perform calculations on the unstructured data to generate corresponding vectorized data, and submits the corresponding vectorized data to the unified storage management layer for atomic associative storage.

[0008] According to some embodiments, in the intelligent endogenous database system of the first aspect of the present invention, the artificial intelligence computing engine includes: The model runtime framework provides standardized interfaces for model loading, inference execution, and resource release. The plug-in management module is used to perform dynamic lifecycle management of at least one AI model that is packaged as a pluggable computing unit.

[0009] According to some embodiments, in the intelligent endogenous database system of the first aspect of the present invention, the model runtime framework is configured as follows: During the model loading phase, the corresponding driver plugin is called according to the model format to parse the model file into an internally unified intermediate representation; During the execution phase, computing tasks are dispatched to CPUs or hardware accelerators through a device abstraction layer.

[0010] According to some embodiments, the intelligent endogenous database system of the first aspect of the present invention also includes a unified query optimization engine for globally optimizing mixed query requests that simultaneously involve structured data relational operations and unstructured data intelligent processing.

[0011] According to some embodiments, in the intelligent endogenous database system of the first aspect of the present invention, the unified query optimization engine performs optimization through its built-in cost model, which includes an AI operator cost sub-model for estimating the computational cost of at least one AI model.

[0012] According to some embodiments, in the intelligent endogenous database system of the first aspect of the present invention, the unified query optimization engine is configured to: dynamically decide the execution order of relation filtering operation and vector similarity retrieval operation in hybrid queries based on the estimation results of AI operator cost sub-model, so as to generate a globally optimal execution plan.

[0013] According to some embodiments, in the intelligent endogenous database system of the first aspect of the present invention, the system further includes a resource arbitrator; The resource arbitrator is used to allocate independent resource quotas to the relational query engine and the artificial intelligence computing engine, and to arbitrate and schedule computing tasks according to preset priorities when system resources are scarce.

[0014] According to some embodiments, in the intelligent endogenous database system of the first aspect of the present invention, the resource arbitrator creates an independent running sandbox for the artificial intelligence computing engine through operating system-level process isolation or containerization technology.

[0015] According to some embodiments, in the intelligent endogenous database system of the first aspect of the present invention, the vectorization scheduler provides a declarative interface that enables users to bind specific AI models to database tables or columns through an extended data definition language to trigger automatic vectorization. The system further includes a unified memory management component for maintaining a unified virtual address space across different physical memory domains; When vectorization is performed in response to a binding declaration, the unified memory management component enables unstructured data to be processed to be directly transferred between the database buffer pool and the AI ​​model input buffer in a zero-copy manner.

[0016] According to a second aspect of the present invention, a data processing method for an intelligent endogenous database system according to the first aspect of the present invention is provided, comprising: Native loading steps: Load and reside at least one AI model within the database system process; Triggering and Vectorization Steps: In response to the write transaction of unstructured data, the AI ​​model is automatically triggered. Within the context of the write transaction, the unstructured data is directly computed to generate the corresponding vectorized data. Atomic associative storage steps: Establish an association between unstructured data and its corresponding vectorized data within the same database transaction and persist them together.

[0017] Based on the core technical features of the solution, the technical effects of this invention can be summarized as follows: Firstly, regarding data security and processing workflows, a closed-loop data processing environment is constructed by integrating the artificial intelligence computing engine as a native component and relying on the triggering mechanism of the vectorized scheduler within database transactions. Unstructured data is processed entirely within the database process from writing to vectorization, effectively preventing data from flowing between multiple independent systems. This reduces the risk of sensitive data leakage at the architectural level and simplifies the complexity of security management.

[0018] Secondly, in terms of system performance and query capabilities, the solution brings significant improvements from both data processing and optimization perspectives. On the one hand, the zero-copy data transfer path supported by the unified memory management component, combined with the in-situ vectorization process, greatly reduces redundant copying and serialization overhead during data processing, improving throughput and reducing latency. On the other hand, the unified query optimization engine with an embedded AI cost model enables the database to generate more efficient overall execution plans for queries that combine structured filtering and unstructured semantic retrieval, improving the response performance of complex queries.

[0019] Finally, regarding the reliability and availability of the system engineering, the solution ensures resource isolation and stability between high-load AI computing tasks and core database transaction services through resource arbitrators and isolation mechanisms. Meanwhile, the declarative interface of "model as index" abstracts the complex underlying capabilities at a high level, providing users with a simple and intuitive operation method and lowering the technical threshold for development and maintenance. Together, these two aspects enable the system to maintain its reliability and ease of use as foundational software while introducing powerful AI capabilities. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without exceeding the scope of protection claimed by the present invention.

[0021] Figure 1 This is a schematic diagram of the structure of an embodiment 1000 of an intelligent endogenous database system according to the present invention; Figure 2 for Figure 1 A schematic diagram of the structure of the artificial intelligence computing engine 1012 in Example 1000; Figure 3 This is a schematic diagram of an embodiment 2000 of an intelligent endogenous database system according to the present invention; Figure 4 This is a schematic diagram of the structure of an embodiment 3000 of an intelligent endogenous database system according to the present invention; Figure 5As one of the present invention Figure 1-4 The flowchart of embodiment 4000 of the data processing method of the database system shown is illustrated. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Figure 1 This is a schematic diagram of an embodiment 1000 of an intelligent endogenous database system according to the present invention. Figure 1 As shown, embodiment 1000 includes an intelligent processing layer 101, a unified storage management layer 102, and a vectorized scheduler 103.

[0024] Optionally, the intelligent processing layer 101 includes a relational query engine 1011 and an artificial intelligence computing engine 1012.

[0025] In some specific embodiments, the relational query engine 1011 is used to process structured query requests. Optionally, the relational query engine 1011 is the traditional and core path for this system to handle all structured query requests. In this invention, its role has evolved from single relational data processing to one of the coordinators and executors of mixed queries: it is not only responsible for pure structured queries, but more importantly, when faced with mixed queries that simultaneously involve conditional filtering (such as WHERE department='Sales') and semantic search (such as ORDER BY vector <=> '…'), it collaborates with the artificial intelligence computing engine 1012 to form a complete joint execution unit.

[0026] Optionally, the relational query engine 1011 internally adopts the classic relational database architecture, with enhancements to key components, including: (1) Parser: Receives SQL statements. When it recognizes extended vector types, similarity operators (such as <=>), or AI functions, it marks them as special operators and passes them to the optimizer.

[0027] (2) Unified Query Optimizer: This is the key intersection of the relational query engine 1011 and the overall system intelligence. This optimizer embeds an AI operator cost sub-model. When generating execution plans for hybrid queries, AI vectorization or vector retrieval is no longer treated as a black box, but its cost can be estimated. Subsequently, global optimization decisions are made, such as: should the relational engine be used to quickly filter out "all documents of the sales department" first, and then send the subset of results to the AI ​​engine for vectorization and sorting; or should all documents be vectorized first, and then the vectors of "sales department" be selected and sorted. The unified query optimizer will choose the path with lower estimated cost and generate a unified execution plan that integrates relational operators and AI computing nodes.

[0028] (3) Executor: Responsible for scheduling and executing the plans generated by the optimizer. For AI computing nodes in the plan, the executor will submit subtasks to the artificial intelligence computing engine 1012 through the efficient channel inside the system, obtain the results, and then perform subsequent processing such as concatenation and sorting of the results with the results of relational operations, and finally return them to the user.

[0029] In some specific embodiments, the artificial intelligence computing engine 1012, as a native engine at the same level as the relational query engine 1011, is used to load and run at least one AI model within the database system process.

[0030] Optionally, the artificial intelligence computing engine 1012 includes a model runtime framework 2A and a plug-in management module 2B.

[0031] The Model Runtime Framework 2A provides standardized interfaces for model loading, inference execution, and resource release. In some specific embodiments, the Model Runtime Framework 2A is configured to: during the model loading phase, call the corresponding driver plugin according to the model format to parse the model file into an internally unified intermediate representation; and during the execution phase, dispatch computational tasks to the CPU or hardware accelerator through a device abstraction layer.

[0032] Optionally, the plug-in management module 2B is used to perform dynamic lifecycle management on at least one AI model encapsulated as a pluggable computing unit. Optionally, the plug-in management module 2B is the manager of the model runtime framework 2A, responsible for the dynamic management and control of the AI ​​model computing units existing in the form of plug-ins throughout their entire lifecycle, ensuring that they can be used safely and efficiently.

[0033] Optionally, the unified storage management layer 102 is used for the integrated management of persistent storage of raw data and its derived vectorized data. Specifically, the core function of the unified storage management layer 102 is to achieve integrated, transactional, and associated storage of raw unstructured data and its derived vectorized data. The unified storage management layer 102 of this invention not only enables the physical coexistence of the two types of data, but also treats the two types of data as an inseparable intelligent data unit at both the logical and physical levels, ensuring consistency in its storage, retrieval, and transaction management.

[0034] In some specific embodiments, the unified storage management layer 102 designs and implements a new physical storage structure. For example, for a table storing documents, in addition to the traditional row data area, a vector data extension area is introduced. Each data row corresponds to a vector slot, which is strongly associated through pointers within the row or fixed offsets. The system catalog precisely records which columns are bound to the AI ​​model, as well as the metadata of the corresponding vector data.

[0035] When writing a new row containing bound columns, the unified storage management layer 102 reserves storage space for the vector data. After the vectorization scheduler 103 submits the computed vector, the unified storage management layer 102 writes it to the reserved location and establishes or strengthens the association index between the two. This design allows the system to obtain both the original content and the vector representation of a row of data with one or a minimal number of I / O operations.

[0036] In some specific embodiments, the unified storage management layer 102 provides a storage API that supports atomic commits to the vectorized scheduler 103, ensuring that the "raw data-vector data" pair committed in a single call will be acknowledged or rolled back as a whole in the storage system.

[0037] Within the database transaction, the vectorized scheduler 103 calls this storage API after obtaining the vector returned by the AI ​​engine. The unified storage management layer 102 utilizes the underlying transaction log mechanism to record the write operations of these two data items in the same log entry, ensuring that they maintain a consistent state even during system failure recovery, thereby achieving transaction atomicity at the storage layer.

[0038] Specifically, in some embodiments, the vectorization scheduler 103, in response to write operations on unstructured data, automatically triggers the artificial intelligence computing engine 1012 within the database transaction boundary, schedules at least one AI model to perform calculations on the unstructured data to generate corresponding vectorized data, and submits the corresponding vectorized data to the unified storage management layer 102 for atomic associative storage. The vectorization scheduler 103 is the core coordinator driving the data-intrinsic intelligence process, responsible for monitoring data changes, automatically triggering AI calculations within the transaction context, and bridging the AI ​​engine and the storage layer to ensure the automation and consistency of intelligent processing.

[0039] Optionally, the vectorization scheduler 103 provides a declarative interface that allows users to declare the binding of specific AI models to database tables or columns using an extended data definition language to trigger automatic vectorization. Optionally, the declarative interface provided by the vectorization scheduler 103 is a user-facing control plane that simplifies complex processes into configuration operations.

[0040] Specifically, the vectorized scheduler 103 provides a declarative interface and configuration management. This declarative interface extends the database's data definition language, supporting syntax such as `ALTER TABLE t1 ADD AI_COLUMN vec AS ('content') USINGMODEL 'model_name'`. Internally, the vectorized scheduler 103 maintains a binding relationship metadata table, recording all user-defined model-data column bindings. After a user executes a declarative statement, the vectorized scheduler 103 parses and persists the binding relationship. Subsequently, any write or update operations to the target column will be automatically included in its monitoring scope.

[0041] Optionally, the vectorized scheduler 103 also performs event listening and task triggering within a transaction, specifically including: The vectorized scheduler 103 is deeply integrated into the database's transaction management module. It can function as a transaction callback hook or by listening to the transaction log stream (Change Data Capture). It can accurately detect write operations on bound columns and the transaction ID to which the operation belongs.

[0042] Once a write event is detected, the vectorized scheduler 103 immediately submits a vectorized task asynchronously to the artificial intelligence computing engine 1012 within the context of the current transaction, passing the logical identifier of the data to be processed and the transaction context. This task, as a subtask of the transaction, is directly related to the final commit status of the parent transaction.

[0043] Optionally, in some specific embodiments, the system further includes a unified memory management component for maintaining a unified virtual address space across different physical memory domains. When vectorization is performed in response to a binding declaration, the unified memory management component enables the unstructured data to be processed to be directly transferred between the database buffer pool and the AI ​​model input buffer in a zero-copy manner.

[0044] Optionally, the vectorized scheduler 103 also works closely with a unified memory management component. When computation needs to be triggered, data is not copied; instead, the component retrieves a stable memory address or handle for the target data in the database buffer pool. Specifically, the vectorized scheduler 103 submits this memory handle along with the computation task to the AI ​​engine. The unified memory management component ensures that the AI ​​model can directly read the contents of this memory region during execution using a zero-copy method, thereby avoiding the expensive copying overhead between user-space memory and the AI ​​computing device, achieving extremely low-latency vectorized triggering.

[0045] According to such Figure 1 The embodiment shown in this invention constructs an intelligent endogenous database system that uses a relational query engine and an artificial intelligence computing engine as parallel native components to form an intelligent processing layer, which is coordinated by an independent vectorized scheduler. Finally, it relies on a unified storage management layer to achieve integrated persistence, thereby realizing unified, efficient and secure processing of structured and unstructured data within a single database system.

[0046] The system architecture proposed in this invention enables deep integration and automatic execution of traditional SQL queries and advanced AI vectorized computation within transaction boundaries: the relational engine processes structured queries and collaborates with an optimizer embedded with an AI cost model to plan hybrid queries; the AI ​​engine provides native computing power; and the vectorized scheduler automatically triggers the entire process from data writing, in-situ vectorization to atomic storage based on declarative binding. This not only eliminates the security risks and performance losses associated with data transfer between multiple heterogeneous systems, but also achieves significant improvements in data security, processing efficiency, and query intelligence through zero-copy, transaction consistency guarantees, and unified optimization, while reducing development and maintenance complexity with a minimalist interface.

[0047] Figure 2 for Figure 1 A schematic diagram of the structure of the artificial intelligence computing engine 1012 in Embodiment 1000. (See attached diagram.) Figure 2 As shown, the artificial intelligence computing engine 1012 includes a model runtime framework 2A and a plugin management module 2B.

[0048] Optionally, the model runtime framework 2A is used to provide standardized interfaces for model loading, inference execution, and resource release. In some specific embodiments, the model runtime framework 2A is configured to: during the model loading phase, call the corresponding driver plugin according to the model format to parse the model file into an internally unified intermediate representation; during the execution phase, dispatch computational tasks to the CPU or hardware accelerator through a device abstraction layer.

[0049] Optionally, the workflow of the model runtime framework 2A includes: (1) Model loading stage: When the system needs to load a new model, the model runtime framework 2A will search for and call the corresponding format parser in the registered driver plugin library according to the file extension or file header information of the model file.

[0050] Each driver plugin is an independent dynamic library dedicated to handling a specific format. For example, the GGUF driver plugin is responsible for reading the header information of GGUF files, parsing the model architecture, weight tensor data, and hyperparameters. The plugin's job is to convert all this heterogeneous format data and populate it into a unified intermediate representation data structure defined within the model runtime framework 2A. This intermediate representation contains all the runtime-necessary information, such as the model computation graph, tensor metadata, and weight data pointers, but no longer depends on the original file format.

[0051] Optionally, the model runtime framework 2A employs a driver plugin mechanism to address the fragmentation of AI model formats (such as PyTorch's PT, TensorFlow's SavedModel, ONNX, GGUF, etc.). Without abstraction, the database kernel would need to write hard-coded parsing logic for each format, resulting in a bloated and difficult-to-maintain system. The plugin mechanism externalizes this complexity, allowing support for new model formats to be achieved simply by developing and registering a new plugin, without modifying the database kernel code.

[0052] (2) Inference execution phase: When the vectorized scheduler 103 submits a computation task containing data pointers, the scheduler of the model runtime framework 2A will select the target model from the loaded model pool and prepare for execution.

[0053] Specifically, the model runtime framework 2A internally includes a device abstraction layer. This abstraction layer provides a unified API at the top and interfaces with the native computing libraries of different hardware (such as the ONNX Runtime on the CPU, cuBLAS / cuDNN on GPU CUDA, and the NPU's dedicated SDK) at the bottom. The model runtime framework 2A passes the model's unified intermediate representation and input data pointers to the device abstraction layer, which is responsible for: 1. Dispatch model weights and data to the target hardware as needed.

[0054] 2. Call the optimal computing library for the corresponding hardware to perform the actual neural network forward propagation (inference) computation.

[0055] 3. Manage temporary memory and other resources during the computing process.

[0056] The device abstraction layer is designed to shield the heterogeneity of the underlying hardware. Different CPU instruction sets, GPU architectures, and AI accelerator cards differ significantly in their programming interfaces and memory management. Through this abstraction layer, the model runtime framework 2A and the upper-level engine do not need to write specialized code for each type of hardware, achieving "write once, run everywhere."

[0057] (3) Resource release phase: When the model is no longer needed, such as when it is uninstalled or the system is shut down, the model runtime framework 2A notifies the device abstraction layer and driver plugin through the standard interface to perform the reverse operation.

[0058] The device abstraction layer is responsible for releasing all video memory / memory occupied by the model on the hardware. The driver plugin may clean up intermediate states related to the format. During model runtime, the framework 2A itself releases the memory occupied by maintaining the unified intermediate representation.

[0059] Optionally, the plug-in management module 2B is used to perform dynamic lifecycle management on at least one AI model encapsulated as a pluggable computing unit. Optionally, the plug-in management module 2B is the manager of the model runtime framework 2A, responsible for the dynamic management and control of the AI ​​model computing units existing in the form of plug-ins throughout their entire lifecycle, ensuring that they can be used safely and efficiently.

[0060] Specifically, the implementation and core responsibilities of the plug-in management module 2B include: In some specific embodiments, the plugin management module 2B performs plugin lifecycle management, managing driver plugins and model instances, including: During system startup or runtime, the plugin management module 2B is responsible for loading the specified driver plugin dynamic libraries into the database process space. When a user declares binding to a model, the plugin management module 2B coordinates the model runtime framework 2A to complete the model initialization. During model runtime, the plugin management module 2B monitors its health status. When the model needs to be updated or system resources need to be reclaimed, the plugin management module 2B is responsible for safely unloading the model instance and plugins.

[0061] Optionally, the plugin management module 2B maintains a plugin registry, recording the metadata, status, and corresponding model instance handle for each plugin. Optionally, the metadata includes supported formats, version, and compatibility; the status includes loaded / running / abnormal.

[0062] In some specific embodiments, the plug-in management module 2B provides dynamic hot-plugging and hot-updating support, allowing the addition of new format driver plug-ins or the replacement of existing model versions without restarting the database service, including: When a new plugin is added, the plugin management module 2B loads it and registers it in the plugin library of the model runtime framework 2A, making it immediately available. For hot model updates, the new version of the model is loaded and validated first, then new inference requests are gradually switched to the new version, and finally the old version is unloaded and its resources are reclaimed when there are no tasks.

[0063] Dynamic lifecycle management addresses the conflict between the frequent iterations and updates required for AI models and the 24 / 7 continuous and stable operation of databases. Traditional methods necessitate downtime for updates, while this design supports online operation. This solution significantly improves the system's maintainability, availability, and flexibility. Businesses can seamlessly upgrade AI model capabilities, meeting the stringent service continuity requirements of production environments.

[0064] According to such Figure 2 In the illustrated implementation, the artificial intelligence computing engine 1012 of this invention works closely with the plug-in management module 2B through the model runtime framework 2A to transform the external, complex, and ever-changing AI model ecosystem into a stable, unified, and predictable computing resource pool within the database kernel. Together, they solve the three major engineering challenges of compatibility, heterogeneity, and maintainability faced in deeply integrating AI into basic software, enabling the innovative concept of "AI models as native database operators" to be firmly implemented, providing high-performance and highly reliable intelligent computing services to the upper layers.

[0065] Figure 3 This is a schematic diagram of an embodiment 2000 of an intelligent endogenous database system according to the present invention. Figure 3 As shown, embodiment 2000 includes an intelligent processing layer 201, a unified storage management layer 202, a vectorized scheduler 203, and a unified query optimization engine 204. Among them, the intelligent processing layer 201, the unified storage management layer 202, the vectorized scheduler 203, and... Figure 1 The intelligent processing layer 101, unified storage management layer 102, and vectorized scheduler 103 in embodiment 1000 are the same and will not be described again here.

[0066] Optionally, the unified query optimization engine 204 is used to perform global optimization on mixed query requests that involve both structured data relational operations and unstructured data intelligent processing.

[0067] In some specific embodiments, the unified query optimization engine 204 first completes query parsing and feature extraction, specifically including: It receives raw query statements from the SQL interface. Its parser can recognize extended syntax elements, such as the VECTOR type, the similarity operator <=>, and user-defined AI functions, and mark them as special "AI calculation operators". At the same time, the Unified Query Optimization Engine 204 also extracts all relational operators (such as selection, projection, and join) and related statistics (such as table size and column selectivity).

[0068] For example, for the query SELECT * FROM documents WHERE dept='Sales' AND category='Report' ORDER BY content_vector <=> '[?]' LIMIT 10, the parser will identify the relational filtering condition (dept='Sales' AND category='Report') and the AI ​​calculation request (based on the similarity ranking of content_vector).

[0069] Optionally, the unified query optimization engine 204 implements optimization through its built-in cost model, which includes an AI operator cost sub-model for estimating the computational cost of at least one AI model. In some specific embodiments, the unified query optimization engine 204 is configured to dynamically decide the execution order of relation filtering operations and vector similarity retrieval operations in a hybrid query based on the estimation results of the AI ​​operator cost sub-model, in order to generate a globally optimal execution plan.

[0070] Specifically, the core innovation of the Unified Query Optimization Engine 204 lies in its cost estimation based on an AI operator cost sub-model. Its built-in cost model, in addition to the traditional I / O and CPU cost calculation modules, introduces a dedicated AI operator cost sub-model. This sub-model is not a fixed value, but rather an estimation function based on multiple dynamic parameters, whose input parameters typically include: (1) Model computational complexity: The inherent attributes of the AI ​​model to which the current query is bound, such as the number of parameters and the number of computation layers, can be obtained through the model metadata.

[0071] (2) Expected input data size: that is, the amount of data to be vectorized or to be compared for similarity.

[0072] (3) Current system load status: especially the utilization rate of hardware accelerators (GPU / NPU) and memory bandwidth usage.

[0073] For the above query, the unified query optimization engine 204 will plan multiple candidate execution paths. For example: Path A: First, use the relationship engine to perform filtering (dept='Sales' AND category='Report') to get a small result set (say 100 rows). Then, call the AI ​​engine to sort these 100 rows of data by vector similarity.

[0074] Path B: First, use the AI ​​engine to sort all rows in the documents table (assuming 10,000 rows) by vector similarity to obtain a global sorted list. Then, use the relationship engine to filter out the top 10 rows that satisfy dept='Sales' AND category='Report'.

[0075] The Unified Query Optimization Engine 204 uses the AI ​​operator cost sub-model to estimate the cost of "sorting 100 rows of data into vectors" in path A and the cost of "sorting 10,000 rows of data into vectors" in path B, and then compares them with the cost of relation filtering itself.

[0076] In some specific embodiments, the unified query optimization engine 204 evaluates and compares the total cost of all candidate execution paths based on the cost estimation results, using dynamic programming or heuristic algorithms. For example, assuming that the estimated cost of vector sorting of 10,000 rows of data is extremely high, while the relation filtering conditions are highly selective (able to filter out most data), then the total cost of path A will be much lower than that of path B. The engine will then decide to adopt the execution order of path A and generate a specific, executable fusion execution plan. This plan will explicitly indicate that: in transaction T1, the relation engine executes the filtering subtask; the intermediate dataset M1 of the filtering results is passed to the AI ​​engine via zero-copy; the AI ​​engine performs vector similarity calculation and sorting on M1; and the final result is returned.

[0077] In some specific embodiments, the unified query optimization engine 204 distributes the generated fusion execution plan to the relational query engine 2011 and the artificial intelligence computing engine 2012 of the intelligent processing layer 201, and the vectorized scheduler 203 coordinates the transfer of intermediate data when necessary. The unified query optimization engine 204 itself does not execute the plan, but the generated plan contains detailed cross-engine collaboration instructions. The specific implementation process includes: according to the fusion execution plan, the relational query engine 2011 is first invoked to complete filtering and generate intermediate result M1. Subsequently, through an internal system channel (usually involving a unified memory management component), a reference to M1, rather than a copy of the data, is submitted to the vectorized scheduler 203 or the artificial intelligence computing engine 2012 is directly triggered to perform computation.

[0078] According to such Figure 3In the implementation shown, the solution proposed in this invention achieves the following core technical effects by introducing an AI operator cost sub-model and performing global cost evaluation and dynamic path decision-making for hybrid queries: It solves the optimization blind spot problem of mixed queries: enabling the database optimizer to understand the overhead of AI computation, thereby transforming the originally independent and unoptimizable external AI calls into evaluable and optimizable internal operators, filling the gap in the field of intelligent query optimization in traditional databases.

[0079] This achieves a qualitative leap in query performance: by intelligently selecting the optimal or near-optimal execution order, it avoids significant performance waste caused by improper execution order. This transforms the performance of complex queries involving semantic retrieval from unpredictable to near-optimal, greatly improving the system's efficiency and stability in processing intelligent queries.

[0080] This improves system usability and resource utilization: users no longer need to manually write complex programs or provide hints to optimize query order; the system can automatically make the best decisions. Simultaneously, by being aware of hardware load, the optimizer can make more resource-efficient scheduling decisions, improving the overall system resource utilization.

[0081] Figure 4 This is a schematic diagram of an embodiment 3000 of an intelligent endogenous database system according to the present invention. Figure 4 As shown, embodiment 3000 includes an intelligent processing layer 301, a unified storage management layer 302, a vectorized scheduler 303, and a resource arbitrator 304. Among them, the intelligent processing layer 301, the unified storage management layer 302, the vectorized scheduler 303, and... Figure 1 The intelligent processing layer 101, unified storage management layer 102, and vectorized scheduler 103 in embodiment 1000 are the same and will not be described again here.

[0082] The resource arbitrator 304 is used to allocate independent resource quotas to the relational query engine 3011 and the artificial intelligence computing engine 3012 respectively, and to arbitrate and schedule computing tasks according to preset priorities when system resources are scarce. Optionally, the resource arbitrator 304 can create an independent runtime sandbox for the artificial intelligence computing engine 3012 through operating system-level process isolation or containerization technology.

[0083] Optionally, the resource arbitrator 304 is a core component of this system that ensures stability and service quality. Its core function is to act as a supervisor and scheduler of global resources, performing unified resource management and conflict arbitration for concurrent relational query tasks and artificial intelligence computing tasks. Its fundamental goal is to ensure that, in scenarios with limited or overloaded hardware resources, the response time and stability of core database services, especially OLTP transactions, are not affected by high-load AI computing tasks through fine-grained isolation and priority scheduling, thus ensuring that the system as a whole always provides predictable high-performance services.

[0084] In some specific embodiments, the resource arbitrator 304 employs a strategy combining static configuration and dynamic monitoring to allocate independent resource quotas to the two types of engines: (1) Static quota allocation: During the database initialization or configuration phase, resource limits are preset for the artificial intelligence computing engine sandbox and relational query engine based on physical hardware specifications and workload characteristics.

[0085] Artificial intelligence computing engine sandbox: Create an isolated runtime sandbox using cgroups (control groups) or containerization technologies such as Docker / Kubernetes runtime. Explicitly assign the following to this sandbox: a. CPU Resources: List of bound CPU cores and CPU time slice shares (cpu.shares).

[0086] b. Memory resources: memory usage limit (memory.limit_in_bytes) and memory overflow control strategy.

[0087] c. GPU Resources: Specifies the index of the GPU device that can be used, and can limit its memory usage limit and computing unit utilization.

[0088] Relational query engine: runs on the host machine or in the default cgroup, and its resource quota is usually all the remaining resources of the system, or it is also set with a high guarantee quota.

[0089] (2) Dynamic quota adjustment: Resource arbitrator 304 continuously monitors system-level resource indicators (such as global CPU load, memory pressure, and GPU utilization). When the overall system resources are detected to be strained, the resource limit of the AI ​​sandbox can be dynamically tightened according to the preset strategy to free up more resource space for core relational tasks.

[0090] Optionally, the resource arbitrator 304 maintains a multi-level task priority model for decision-making during resource contention.

[0091] Optionally, specific embodiments of priority definition include: Level 1, Highest Priority: Core Support Mission a. Real-time write transaction requests: These include data modification operations such as INSERT, UPDATE, and DELETE, and are currently active transactions. These tasks directly impact data consistency and durability, and their resources must be prioritized for rapid commit to avoid long-running transactions causing blockages.

[0092] b. Critical internal system maintenance tasks: such as synchronous flushing of transaction logs to disk, and creation and persistence of checkpoints. These are the cornerstones for ensuring database crash recovery capabilities and data integrity, and cannot be delayed.

[0093] Second-level high priority: Critical query and control tasks a. Real-time read-only transactions and interactive point queries: Simple SELECT queries initiated by users or front-end applications, expecting millisecond-level responses. These tasks directly determine the end-user experience.

[0094] b. Database control commands: DBA operations such as connection management, session termination, and online configuration changes (in some cases) require deterministic responses.

[0095] Level 3 (High Priority): Analytical and Background Processing Tasks a. Complex analytical queries: Long-running SELECT queries, such as reporting tasks involving multiple table joins and large-scale aggregations. These require significant resources but can tolerate some queuing delays.

[0096] b. Online Incremental Vectorization Tasks: Vectorized computation tasks triggered in real-time by user data writing and bound to write transactions. Their priority is related to the priority of their respective write transactions and must be completed before the transaction is committed.

[0097] Level 4 (Low to Medium Priority): Planned Background Operations a. Batch data import / export tasks: such as batch data movement triggered by the COPY command or backup tools.

[0098] b. Batch historical data vectorization task: Non-real-time vectorization supplementary calculation or recalculation task after model upgrade of existing historical data.

[0099] c. Automatic collection of statistical information and index rebuilding tasks: updating statistical information that the database optimizer depends on, and online index maintenance operations.

[0100] Level 5 Lowest Priority: Preemptible Resource-Intensive Tasks a. AI model training and fine-tuning tasks: Large-scale, long-term model training or parameter tuning operations performed within a database using data.

[0101] b. Resource-intensive management operations: such as non-urgent maintenance tasks like full database consistency verification and physical storage defragmentation.

[0102] Optionally, based on the above priority definition rules, the arbitration rules of resource arbitrator 304 include: (1) Preemptive scheduling: When a high-priority task is waiting due to insufficient resources, the resource arbitrator 304 can command the resource arbitrator to suspend or forcibly reduce the resource quota of low-priority AI tasks, or even put their tasks into the suspension queue to prioritize the resource needs of high-priority tasks.

[0103] (2) Queuing and Degradation: For newly submitted AI computing tasks, if the current AI sandbox resource utilization rate has exceeded the safety threshold, the arbitrator will place them in the waiting queue according to their priority. For low-priority AI tasks that are being executed, their available CPU share or GPU computing frequency can be dynamically reduced to decrease their resource consumption.

[0104] Optionally, in the priority mechanism, the final priority of a task is not entirely fixed. For example, the priority of an online incremental vectorization task inherits the priority of its triggering source—the real-time write transaction. If the write transaction is aborted, all vectorization tasks derived from it will also be cascaded and canceled.

[0105] Optionally, in some specific embodiments, the priority definition in the resource arbitrator 304 also includes a dynamic upgrade / downgrade mechanism: Upgrade: A medium-priority task that has been queued for a long time can have its priority dynamically increased as its waiting time exceeds a threshold, in order to prevent it from "starving".

[0106] Degradation: If a low-priority task is detected to have a resource usage pattern that severely impacts a high-priority task, its resource quota can be dynamically reduced or it can be suspended.

[0107] Optionally, in some specific embodiments, the coordination of resource quotas and priorities in the resource arbitrator 304 includes: high-priority tasks not only enjoy scheduling priority, but also have stronger resource quota guarantees. For example, a portion of non-preemptible CPU cores and memory are reserved for core guarantee tasks. Meanwhile, the lowest-priority tasks can only use elastic quotas when resources are idle, and are prioritized for compression or even evicted when resources are scarce.

[0108] In some specific embodiments, the resource arbitrator 304 runs continuously as a background daemon, and its typical workflow is as follows: (1) Monitoring and collection: Periodically collect resource metrics from the operating system, container runtime and database, and receive task submission events and priority tags from the query optimization engine and vectorized scheduler.

[0109] (2) Analysis and Decision-Making: Real-time analysis of resource usage. If the resources (such as average CPU wait time) of the environment where the relational engine is located exceed the threshold, or the resource usage of the AI ​​sandbox reaches the upper limit, the arbitration decision logic is triggered. The decision engine applies the above arbitration rules according to the priority of the currently running tasks and the tasks to be executed.

[0110] (3) Execution and Control: Transforming arbitration decisions into specific control instructions: The cgroup file system interface allows for dynamic adjustment of the AI ​​sandbox's cpu.cfs_quota_us (CPU time limit) or memory.limit_in_bytes (memory limit).

[0111] Limit GPU utilization for specific containers using commands from GPU management libraries such as NVIDIA's nvidia-smi or NVML.

[0112] Send instructions to the vectorized scheduler to suspend or resume task queues within a specific priority range.

[0113] (4) Feedback and recording: The arbitration action and system status are recorded in the system log, and real-time resource arbitration indicators can be provided to the database monitoring system for operation and maintenance insights.

[0114] According to such Figure 4 As illustrated in the embodiments, the resource arbitrator of this invention provides intelligent resource scheduling and conflict arbitration for core transaction processing and AI computing tasks through operating system-level resource isolation technology and a multi-level, dynamic task priority model. Its technical advantages lie in ensuring that the resource requirements of high-priority core database operations are always met first, thereby guaranteeing the overall stability and service quality of the system are not affected by high-load AI computing tasks. Simultaneously, through dynamic quota adjustment and task queuing mechanisms, it achieves fine-grained resource management of heterogeneous computing loads, enabling the system to gracefully degrade when resources are limited, and significantly improving the overall utilization of hardware resources and maintainability under mixed loads.

[0115] Figure 5 As one of the present invention Figure 1-4 The flowchart of embodiment 4000 of the data processing method of the database system is shown. Figure 5 As shown, Example 4000 includes a native loading step S1, a triggering and vectorization step S2, and an atomic associative storage step S3.

[0116] In some specific embodiments, in the native loading step S1, at least one AI model is loaded and resided within the database system process.

[0117] Optionally, the native loading step S1 is executed by the AI ​​computing engine: During database system initialization or runtime according to user instructions, the AI ​​computing engine, within its independent resource sandbox, loads at least one user-specified AI model directly from persistent storage into the database process's memory space through its integrated model runtime framework, and completes initialization. The model is managed in a plug-in manner, and its weight data can be loaded into CPU memory or GPU memory according to hardware configuration, achieving native model residency and laying the foundation for subsequent zero-latency calls.

[0118] In some specific embodiments, in the triggering and vectorization step S2, in response to the write transaction of unstructured data, the AI ​​model is automatically triggered, and within the context of the write transaction, the unstructured data is directly calculated to generate the corresponding vectorized data.

[0119] Optionally, the triggering and vectorization step S2 embodies the core functionality of the vectorization scheduler. When a user or application submits a transaction to the database to write unstructured data, the vectorization scheduler will listen for this event in real time; then, within the context and lifecycle of the write transaction, it will automatically initiate a computation request to the resident AI model.

[0120] The key innovation of triggering and vectorization step S2 lies in the fact that the data to be processed is transferred directly from the database transaction buffer to the AI ​​model's input buffer in a zero-copy manner through a unified memory management component. Subsequently, the artificial intelligence computing engine schedules the model to complete inference calculations and generate high-dimensional vectors. Step S2 ensures the atomic correlation and extremely low latency between vectorization processing and data writing.

[0121] In some specific embodiments, in the atomic association storage step S3, unstructured data and corresponding vectorized data are associated in the same database transaction and persistently stored together.

[0122] Optionally, the atomic associative storage step S3 is completed collaboratively by the vectorization scheduler and the unified storage management layer. After the vectorized data generated by the AI ​​engine is returned, the vectorization scheduler immediately pairs it with the corresponding original unstructured data and submits it to the unified storage management layer as an indivisible data unit. The unified storage management layer uses its converged storage engine and transaction mechanism to ensure that these two parts of data are persistently stored in the same database transaction and a strongly correlated index is established. This means that the entire operation of "writing original data - generating vectors - storing both" can be completed as an atomic transaction, either succeeding or rolling back as a whole, fundamentally guaranteeing the consistency of the data.

[0123] According to such Figure 5As shown in the implementation, the present invention fully realizes intelligent readiness upon data entry through the above three-step closed-loop process, simplifying the complex cross-system AI processing pipeline into a native, automated atomic operation within the database. Thus, within a single system boundary, it simultaneously achieves three core goals: data security without leaving the database, efficient zero-copy processing, and consistent atomic storage, providing users with an extremely simplified intelligent data management experience.

[0124] The embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of the present invention, its specific implementation methods, and its application scope, are all within the scope of protection of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An intelligent, endogenous database system, characterized in that, include: The intelligent processing layer includes: Relational query engines are used to process structured query requests; An artificial intelligence computing engine, as a native engine at the same level as the relational query engine, is used to load and run at least one AI model within the database system process; A unified storage management layer is used to manage the persistent storage of raw data and its derived vectorized data in a unified manner. A vectorized scheduler is used to respond to write operations on unstructured data, automatically trigger the artificial intelligence computing engine within the database transaction boundary, schedule at least one AI model to perform calculations on the unstructured data to generate corresponding vectorized data, and submit the corresponding vectorized data to the unified storage management layer for atomic associative storage.

2. The intelligent endogenous database system according to claim 1, characterized in that, The artificial intelligence computing engine includes: The model runtime framework provides standardized interfaces for model loading, inference execution, and resource release. The plug-in management module is used to perform dynamic lifecycle management on the at least one AI model that is packaged as a pluggable computing unit.

3. The intelligent endogenous database system according to claim 2, characterized in that, The model runtime framework is configured as follows: During the model loading phase, the corresponding driver plugin is called according to the model format to parse the model file into an internally unified intermediate representation; During the execution phase, computing tasks are dispatched to CPUs or hardware accelerators through a device abstraction layer.

4. The intelligent endogenous database system according to claim 1, characterized in that, The system also includes a unified query optimization engine, which is used to globally optimize mixed query requests that involve both structured data relational operations and unstructured data intelligent processing.

5. The intelligent endogenous database system according to claim 4, characterized in that, The unified query optimization engine implements optimization through its built-in cost model, which includes an AI operator cost sub-model for estimating the computational cost of the at least one AI model.

6. The intelligent endogenous database system according to claim 5, characterized in that, The unified query optimization engine is configured to dynamically decide the execution order of relation filtering operations and vector similarity retrieval operations in a hybrid query based on the estimation results of the AI ​​operator cost sub-model, so as to generate a globally optimal execution plan.

7. The intelligent endogenous database system according to claim 1, characterized in that, The system also includes a resource arbitrator; The resource arbitrator is used to allocate independent resource quotas to the relational query engine and the artificial intelligence computing engine respectively, and to arbitrate and schedule computing tasks according to preset priorities when system resources are scarce.

8. The intelligent endogenous database system according to claim 7, characterized in that, The resource arbitrator creates an independent runtime sandbox for the artificial intelligence computing engine through operating system-level process isolation or containerization technology.

9. The intelligent endogenous database system according to claim 1, characterized in that, The vectorization scheduler provides a declarative interface that allows users to declare the binding of specific AI models to database tables or columns using an extended data definition language to trigger automatic vectorization. The system further includes a unified memory management component for maintaining a unified virtual address space across different physical memory domains; When vectorization is performed in response to the binding declaration, the unified memory management component enables the unstructured data to be processed to be directly transferred between the database buffer pool and the AI ​​model input buffer in a zero-copy manner.

10. A data processing method for an intelligent endogenous database system according to any one of claims 1-9, characterized in that, include: Native loading steps: Load and reside at least one AI model within the database system process; Triggering and Vectorization Steps: In response to the write transaction of unstructured data, the AI ​​model is automatically triggered, and within the context of the write transaction, it directly performs calculations on the unstructured data to generate corresponding vectorized data; Atomic association storage step: Establish an association between the unstructured data and the corresponding vectorized data in the same database transaction and persist them together.

Images