Vector database lifecycle management method and system based on SQL middleware

Abstract

The present invention provides a vector database lifecycle management method and system based on SQL middleware, which belongs to the technical field of database lifecycle management. The method defines a lifecycle strategy by receiving declarative SQL extension statements. The strategy includes drift detection, index health monitoring, and automated maintenance actions. The system periodically monitors vector data. When data drift or index performance degradation exceeds a threshold, it automatically triggers maintenance operations such as zero-downtime shadow index reconstruction, data re-embedding, or hot and cold tiering. The present invention does not require modification of the database kernel. It is compatible with multiple vector databases through an adaptation layer, significantly reducing the complexity of operation and maintenance and improving data quality and query performance.

Description

Technical Field

[0001] The present invention relates to the technical field of database lifecycle management, and in particular to a vector database lifecycle management method and system based on SQL middleware. Background Art

[0002] With the widespread adoption of artificial intelligence, high-dimensional vectors (embeddings) have become a core tool for representing unstructured data, such as text and images. For example, e-commerce platforms use the BERT model to convert product descriptions into 768-dimensional vectors to recommend similar products. Image search converts images into vectors to find visually similar content.

[0003] Vector databases (such as Milvus, Pinecone, and Weaviate) implement efficient similarity searches using approximate nearest neighbor (ANN) indexes (such as HNSW and IVF-PQ). However, as vector data scales and application scenarios become more complex, vector database management faces numerous challenges. These challenges directly impact data quality, query performance, and storage costs. Directly modifying the database kernel to address these issues is costly and technically challenging.

[0004] Challenges of vector database management:

[0005] 1. Semantic drift: The semantic representation of a vector changes over time, resulting in decreased query accuracy. For example, in a news search system, old vector representations may not accurately match newly emerging topics and contexts.

[0006] 2. Index performance degradation: As data is continuously added, deleted, and modified, the ANN index structure gradually becomes fragmented, query latency increases, and retrieval quality decreases, requiring regular optimization and reconstruction.

[0007] 3. Inefficient storage: A large amount of cold data (vectors that have not been accessed for a long time) occupies high-cost storage resources, and there is a lack of automated hot and cold data tiering / archiving mechanisms.

[0008] 4. High O&M complexity: Vector database management requires the coordination of multiple system components, relies on complex scripts and expertise, is prone to errors, and is difficult to standardize.

[0009] Disadvantages of existing solutions:

[0010] 1. Current vector database management relies primarily on manual operations or fragmented tool sets, lacking seamless integration with existing database management systems.

[0011] 2. Most maintenance operations rely on Python scripts or specific API calls, which makes it difficult to meet enterprise-level audit and compliance requirements and requires a high learning curve for operations and maintenance personnel.

[0012] 3. Developers need to learn the specific API of each vector database (such as the respective interfaces of Milvus and Pinecone), and the heterogeneity of the system increases management complexity. Summary of the Invention

[0013] The present invention provides a vector database lifecycle management method and system based on SQL middleware. This method uses SQL middleware (called "NineData"), which is built between applications and vector databases (such as Milvus and Pinecone) to achieve "one-stop" intelligent management through declarative SQL extensions.

[0014] To achieve the above object, the present invention adopts the following technical solutions:

[0015] The vector database lifecycle management method based on SQL middleware includes:

[0016] S1. Receiving and parsing: Receive an extended SQL statement containing a vector lifecycle policy declaration and parse it to obtain policy parameters, including the target table, target vector column, scheduling rules, drift threshold, and maintenance action parameters.

[0017] S2. Baseline snapshot generation; generating a first statistical snapshot based on the target vector column at an adaptive sampling rate, associating the first statistical snapshot with policy parameters and storing it in a policy directory;

[0018] S3. Strategy orchestration; based on the target vector database type, select the call sequence corresponding to the maintenance action parameters from the preset API mapping table and generate a maintenance execution plan;

[0019] S4. Periodic monitoring: regularly sample the target vector column according to the scheduling rules, calculate the drift metric between the current sample and the first statistical snapshot; obtain the index health indicator of the target table based on the target vector database type;

[0020] S5. Threshold determination: When the drift metric exceeds the drift threshold, or the index health indicator exceeds the preset performance threshold, the maintenance process is triggered;

[0021] S6. Shadow index zero-downtime maintenance: After the maintenance process is triggered, a shadow index is created in the target vector database based on the index parameters in the maintenance execution plan. The shadow index's recall rate is verified to be no less than the preset recall threshold by reading traffic mirroring. If verification passes, the shadow index is atomically switched to the primary index, and the old index is deleted after query traffic drops to zero or times out.

[0022] S7. Persistence of results: Write the threshold determination results, shadow index switching status, and related performance indicators to the policy execution log for auditing and tracing.

[0023] In this specification, the adaptive sampling rate defaults to a sampling probability of 0.01 and is dynamically adjusted according to the data size. For large tables with more than 100 million vectors, the sampling rate is automatically reduced to 0.001, and vice versa, the sampling rate is increased to ensure statistical accuracy.

[0024] In this specification, the drift metric calculation method includes: for baseline data and current data, calculating the cosine divergence of the mean vector, or calculating the KL divergence after bucketing the data, or calculating the Wasserstein distance through random projection.

[0025] In this specification, the index health indicators include query latency and fragmentation rate. When the query latency increases significantly or the fragmentation rate exceeds a preset threshold, a maintenance process is triggered.

[0026] In this specification, during the creation and switching process of the shadow index, the shadow index is built in parallel in the background, which does not affect the production query, and the switching time is controlled within 50ms.

[0027] In this specification, the maintenance actions include re-indexing, re-embedding, and layering / archiving, wherein the re-embedding operation updates the vectors of data with severe drift by calling external AI services.

[0028] In this specification, the policy directory includes the nd_policies table for recording policy definitions, the nd_policies_history table for recording policy version changes, and the nd_policy_runs table for recording policy execution details.

[0029] In this specification, the API mapping relationship table records the API call sequences of different vector database versions and maintenance actions, and supports Milvus, Pinecone, and Weaviate databases.

[0030] In this specification, in the tiering / archiving, hot and cold data are determined based on the data's time threshold, access frequency, or custom SQL conditions, cold data is migrated to low-cost storage, and a query routing table is maintained to support cold data queries.

[0031] A vector database lifecycle management system based on SQL middleware, applying any one of the above-mentioned vector database lifecycle management methods based on SQL middleware, said vector database lifecycle management system based on SQL middleware comprising:

[0032] SQL parser, used to parse extended SQL statements and output policy parameters;

[0033] Policy catalog, used to store policy parameters, baseline snapshots, and execution logs;

[0034] Statistics collectors for monitoring vector distribution and index health at an adaptive sampling rate and computing cosine divergence, KL divergence, or Wasserstein distance;

[0035] A mapping relationship table that records the API call sequences between different vector database versions and maintenance actions;

[0036] A shadow index module, configured to create shadow indexes, verify recall, and atomically switch without interrupting online queries;

[0037] Scheduler, which triggers the execution of statistics collector and shadow index modules according to the scheduling rules in the policy catalog;

[0038] The interaction results between the statistics collector and the shadow index module are written into the policy directory by the scheduler to achieve closed-loop lifecycle management.

[0039] In summary, the present invention has at least the following beneficial effects:

[0040] The present invention provides a middleware platform (NineData) that implements lifecycle management for vector databases through declarative SQL extensions, including semantic drift detection, index optimization, and hot and cold tiering / archiving. The middleware parses SQL policies, stores them in a directory, regularly monitors vector distribution, and automatically triggers maintenance operations (such as zero-downtime rebuilding of shadow indexes). It is compatible with databases such as Milvus and Pinecone through an adaptation layer. Without the need to modify the database kernel, the platform significantly reduces development and operation and maintenance costs, supports rapid adaptation to new databases, and ensures cross-platform consistency. Operations are recorded in audit tables to meet compliance requirements. As a third-party control panel, the present invention simplifies multi-database management, provides standardized interfaces, lowers technical barriers, and significantly improves data quality, query performance, and storage cost-effectiveness. It is particularly suitable for dynamic recommendation and search scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 This is a schematic diagram of the vector database lifecycle management method based on SQL middleware involved in the present invention.

[0043] Figure 2 Schematic diagram of an overview of the NineData vector lifecycle management system involved in the present invention.

[0044] Figure 3 This is a schematic diagram of the SQL parsing and strategy conversion involved in the present invention.

[0045] Figure 4 Schematic diagram of phases 1 to 2 in the shadow index zero-downtime maintenance process involved in the present invention.

[0046] Figure 5 Schematic diagram of stages 2 to 3 in the shadow index zero-downtime maintenance process involved in the present invention.

[0047] Figure 6 This is a schematic diagram of stages 3 to 4 in the shadow index zero-downtime maintenance process involved in the present invention.

[0048] Figure 7 Schematic diagram of stages 4 to 6 in the shadow index zero-downtime maintenance process involved in the present invention.

[0049] Figure 8 This is a partial schematic diagram of the drift detection and measurement calculation process involved in the present invention.

[0050] Figure 9 This is another schematic diagram of the drift detection and measurement calculation process involved in the present invention.

[0051] Figure 10 Schematic diagram of the multi-database adaptation layer architecture involved in the present invention. DETAILED DESCRIPTION

[0052] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the embodiments of the present invention. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0053] The disclosure below provides many different embodiments or examples for implementing different structures of the embodiments of the present invention. In order to simplify the disclosure of the embodiments of the present invention, the components and configurations of specific examples are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. In addition, the embodiments of the present invention may repeat reference numerals and / or reference letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or configurations discussed.

[0054] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0055] like Figure 1 As shown, this embodiment provides a vector database lifecycle management method based on SQL middleware, including:

[0056] S1. Receiving and parsing: Receive an extended SQL statement containing a vector lifecycle policy declaration and parse it to obtain policy parameters, including the target table, target vector column, scheduling rules, drift threshold, and maintenance action parameters.

[0057] S2. Baseline snapshot generation; generating a first statistical snapshot based on the target vector column at an adaptive sampling rate, associating the first statistical snapshot with policy parameters and storing it in a policy directory;

[0058] S3. Strategy orchestration; based on the target vector database type, select the call sequence corresponding to the maintenance action parameters from the preset API mapping table and generate a maintenance execution plan;

[0059] S4. Periodic monitoring: regularly sample the target vector column according to the scheduling rules, calculate the drift metric between the current sample and the first statistical snapshot; obtain the index health indicator of the target table based on the target vector database type;

[0060] S5. Threshold determination: When the drift metric exceeds the drift threshold, or the index health indicator exceeds the preset performance threshold, the maintenance process is triggered;

[0061] S6. Shadow index zero-downtime maintenance: After the maintenance process is triggered, a shadow index is created in the target vector database based on the index parameters in the maintenance execution plan. The shadow index's recall rate is verified to be no less than the preset recall threshold by reading traffic mirroring. If verification passes, the shadow index is atomically switched to the primary index, and the old index is deleted after query traffic drops to zero or times out.

[0062] S7. Persistence of results: Write the threshold determination results, shadow index switching status, and related performance indicators to the policy execution log for auditing and tracing.

[0063] In some embodiments, the adaptive sampling rate defaults to a sampling probability of 0.01 and is dynamically adjusted according to the data size. For large tables with more than 100 million vectors, the sampling rate is automatically reduced to 0.001, and vice versa, the sampling rate is increased to ensure statistical accuracy.

[0064] In some embodiments, the drift metric calculation method includes: for baseline data and current data, calculating the cosine divergence of the mean vector, or calculating the KL divergence after bucketing the data, or calculating the Wasserstein distance through random projection.

[0065] In some embodiments, the index health indicators include query latency and fragmentation rate. When the query latency increases significantly or the fragmentation rate exceeds a preset threshold, a maintenance process is triggered.

[0066] In some embodiments, during the creation and switching of the shadow index, the shadow index is built in parallel in the background, which does not affect the production query, and the switching time is controlled within 50ms.

[0067] In some embodiments, the maintenance actions include re-indexing, re-embedding, and tiering / archiving, wherein the re-embedding operation performs vector updates on data with severe drift by calling an external AI service.

[0068] In some embodiments, the policy directory includes a nd_policies table for recording policy definitions, a nd_policies_history table for recording policy version changes, and a nd_policy_runs table for recording policy execution details.

[0069] In some embodiments, the API mapping relationship table records API call sequences of different vector database versions and maintenance actions, and supports Milvus, Pinecone, and Weaviate databases.

[0070] In some embodiments, in the tiering / archiving, hot and cold data are determined based on the data's time threshold, access frequency, or custom SQL conditions, cold data is migrated to low-cost storage, and a query routing table is maintained to support cold data queries.

[0071] A vector database lifecycle management system based on SQL middleware, applying any one of the above-mentioned vector database lifecycle management methods based on SQL middleware, said vector database lifecycle management system based on SQL middleware comprising:

[0072] SQL parser, used to parse extended SQL statements and output policy parameters;

[0073] Policy catalog, used to store policy parameters, baseline snapshots, and execution logs;

[0074] Statistics collectors for monitoring vector distribution and index health at an adaptive sampling rate and computing cosine divergence, KL divergence, or Wasserstein distance;

[0075] A mapping relationship table that records the API call sequences between different vector database versions and maintenance actions;

[0076] A shadow index module, configured to create shadow indexes, verify recall, and atomically switch without interrupting online queries;

[0077] Scheduler, which triggers the execution of statistics collector and shadow index modules according to the scheduling rules in the policy catalog;

[0078] The interaction results between the statistics collector and the shadow index module are written into the policy directory by the scheduler to achieve closed-loop lifecycle management.

[0079] The technical concept of the present invention is as follows:

[0080] This paper extends standard SQL through a middleware platform (hereinafter referred to as "NineData"), adding new Data Definition Language (DDL) constructs to declaratively specify vector lifecycle policies. These policies include, but are not limited to, drift thresholds, index health thresholds, scheduling instructions, and tiering / archiving rules.

[0081] NineData parses SQL statements, verifies their semantics, stores policies in a catalog, and periodically executes monitoring queries against one or more vector databases to determine if predefined thresholds have been violated. If so, maintenance actions are automatically invoked through the native database application programming interface (API) or external embedded services. All monitoring statistics and operations are persisted in audit tables.

[0082] The invention runs entirely in user space, requiring no modifications to the vector database kernel. It provides cross-database compatibility and supports multiple vector databases, including Milvus and Pinecone. It manages the quality, performance, and cost of vector data through a unified SQL control plane.

[0083] 1. System Overview ( Figure 2 )

[0084] Client: Issue SQL statements through the GUI, CLI, or API, including:

[0085] Lifecycle management DDL (such as CREATE VECTOR_LIFECYCLE_POLICY);

[0086] Standard DML (such as INSERT) and DQL (such as SELECT);

[0087] Vector-specific DDL (such as creating a vector table) and DQL (such as similarity search).

[0088] The middleware layer includes:

[0089] Parser and planner: parse SQL, verify semantics, and generate execution plans;

[0090] Policy Catalog: stores policy definitions, snapshots, and history records;

[0091] Scheduler: manages the scheduled execution of policies;

[0092] Statistics Collector: monitors vector data and index health;

[0093] Operation engine: maps policies to database APIs.

[0094] Communication: Communicate with vector databases (such as Milvus and Pinecone) and relational databases through SQL dialects or REST / gRPC APIs, and access external AI services (such as embedding generation).

[0095] Example usage:

[0096] -- SQL1 - Create a vector table and establish an HNSW index

[0097] CREATE TABLE image_embeddings(

[0098] image_id BIGINT PRIMARY KEY,

[0099] feature_vector VECTOR(512),

[0100] category VARCHAR(100),

[0101] -- Create an approximate nearest neighbor index; PROPS contains index parameters

[0102] INDEX ann_idx(feature_vector)

[0103] PROPS(

[0104] index_type='HNSW', -- Index type: Hierarchical NSW

[0105] M=32, -- the maximum number of neighbors each node retains

[0106] efConstruction=200, -- Candidate list size during construction

[0107] metric_type='IP' -- Similarity metric: Inner Product ) );

[0110] -- SQL2 - perform vector similarity search within a specified category

[0111] SELECT image_id,category

[0112] FROM image_embeddings

[0113] WHERE category='landscape' -- retrieve only landscape images

[0114] ORDER BY

[0115] vector_search(

[0116] feature_vector,

[0117] :query_vector_placeholder )

[0119] LIMIT 10;

[0120] -- SQL3 - Declare a monthly rebuild index lifecycle policy

[0121] CREATE VECTOR_LIFECYCLE_POLICY monthly_reindex

[0122] FOR TABLE image_embeddings -- target table

[0123] ON COLUMN feature_vector -- target vector column

[0124] SCHEDULE '0 0 1 * *' -- Execute at 00:00 on the 1st of every month (CRON)

[0125] DETECT DRIFT

[0126] USING COSINE_DIVERGENCE -- Drift metric: cosine divergence

[0127] DRIFT_THRESHOLD 0.1 -- trigger threshold

[0128] ACTION REINDEX_VECTOR_COLUMN

[0129] PROPS (--Index parameters used when rebuilding

[0130] M=32, -- the number of connections, consistent with SQL1

[0131] efConstruction=256 -- Construction depth, slightly higher than SQL1

[0132] ).

[0133] 2. Parser & Planner

[0134] The parser and planner are the core components of the middleware, responsible for processing the SQL statements input by the user to ensure their correctness and executability. The following is a detailed explanation of the present invention using SQL3 as an example. The specific process is referenced Figure 3 The SQL parsing and strategy conversion process is shown.

[0135] 2.1 SQL extension identification and conversion

[0136] The parser uses a syntax tree generated based on ANTLR (or other equivalent syntax parsing tools) to distinguish between standard SQL (such as SELECT and INSERT) and the extended DDL constructs defined in the present invention (such as CREATE VECTOR_LIFECYCLE_POLICY).

[0137] The parser first scans the SQL statement for syntactical tokens and performs logical analysis. If it detects special keywords such as VECTOR_LIFECYCLE_POLICY, the system routes the statement to a dedicated extended DDL processing path. If it doesn't detect these special keywords, the standard SQL statement is directly passed to the underlying database system (such as Milvus's SQL dialect or MySQL) for processing.

[0138] During the parsing process, the parser identifies the CREATE VECTOR_LIFECYCLE_POLICY statement and parses its clauses (including FOR TABLE, ON COLUMN, SCHEDULE, DETECT DRIFT, and ACTION) into policy components.

[0139] During the conversion process, these extended DDL statements are parsed and converted into internal policy objects. These objects are broken down into several key fields, including the policy name (e.g., monthly_reindex), the target table (e.g., image_embeddings), the schedule (e.g., '0 0 1 * *') in the form of a CRON expression, the drift threshold (e.g., 0.1), the metric type (e.g., cosine_divergence), and the action to be executed (e.g., reindex_vector_column). Ultimately, these policy objects are passed to the planner in a structured format (typically JSON) for further processing.

[0140] 2.2 Semantic Verification

[0141] Before executing a policy, the planner performs a comprehensive semantic validation to ensure that the policy can be correctly applied to the target database. For example, in SQL3, the monthly_reindex policy undergoes a multi-level validation process.

[0142] First, during the table verification phase, the planner queries the vector database metadata for the FOR TABLE image_embeddings clause and uses Milvus's describe_collection API to confirm that the image_embeddings table (collection) exists and contains at least one vector (column). If the table (collection) does not exist or does not meet the requirements, the system immediately throws an error.

[0143] Secondly, during the metric validation phase, the planner checks whether COSINE_DIVERGENCE in DRIFT_THRESHOLD 0.1 USING COSINE_DIVERGENCE is a drift metric type supported by the system (the system also supports other options such as KL_DIVERGENCE and WASSERSTEIN_DISTANCE). If the specified metric type is unsupported, the system will provide the user with alternative metric type suggestions.

[0144] Third, during column validation, the planner verifies that feature_vector is a valid vector (column) in the image_embeddings table (collection) for the ON COLUMN feature_vector clause and confirms that the column supports the reindex operation specified by ACTION REINDEX_VECTOR_COLUMN. For example, the planner checks whether the column data type is a vector type and whether the current index status allows rebuilding.

[0145] Fourth, during the action validation phase, the planner verifies that the ACTION REINDEX_VECTOR_COLUMN action is applicable to the target database and the specified vector column, confirming that the action is available in the current database version.

[0146] Finally, during the parameter verification phase, the planner checks whether the parameters in PROPS (M=32, efConstruction=256) meet the index requirements of the target database. For example, it confirms that the HNSW index of Milvus supports the M and efConstruction parameters. However, for other databases such as Pinecone, different parameter sets may need to be verified. The planner confirms the validity of these parameters by querying database metadata or using preconfigured adapters.

[0147] 2.3 Pre-planning / parameterization of operations

[0148] During processing, the planner is responsible for converting user-defined declarative actions into a sequence of executable operations. This process first involves parameter mapping, which converts the parameters specified in SQL (such as PROPS(M=32,efConstruction=256)) into the format required by the target vector database's native API. For example, Milvus's create_index API requires parameters in JSON format. The planner automatically infers and generates a structured configuration similar to {"index_type":"HNSW","params":{"M":32,"efConstruction":256}}.

[0149] During the execution phase, the planner makes decisions based on the current database state. For example, if it detects that a similar index already exists in the Milvus collection and only needs to adjust parameters (such as changing the M value), the planner may choose to use the modify_index API to perform in-place parameter optimization instead of recreating the index.

[0150] To support multiple vector databases, the planner maintains a complete database adaptation layer, which stores the API signatures and specific parameter requirements of various popular vector databases (the vector databases that this invention can adapt to include but are not limited to Milvus, Pinecone, etc.). This design ensures that user-defined vector lifecycle policies can maintain compatibility across different database environments, achieving true cross-database operation capabilities. The multi-database adaptation layer architecture is as follows: Figure 10 shown.

[0151] In summary, for SQL3:

[0152] 1. The parser generates a syntax tree.

[0153] 2. The planner verifies that image_embeddings and feature_vector exist and that the cosine_divergence and hnsw parameters are valid.

[0154] 3. Generate a structured policy object and store it as JSON:

[0155] {

[0156] "policy_name":"monthly_reindex",

[0157] "table": "image_embeddings",

[0158] "schedule": "0 0 1 * *",

[0159] "drift_detection": {

[0160] "metric":"COSINE_DIVERGENCE",

[0161] "threshold": 0.1,

[0162] "enabled": true

[0163] },

[0164] "action": {

[0165] "type": "reindex",

[0166] "column": "feature_vector",

[0167] "params": {"M": 32, "efConstruction": 256}}}.

[0168] 3. Policy Catalog (124)

[0169] The policy catalog provides reliable persistence support, ensuring that all policy definitions and execution history are securely stored and protected from loss due to system restarts or failures. Furthermore, the policy catalog features comprehensive auditing capabilities, ensuring a complete operational record to meet corporate and regulatory compliance requirements and enabling administrators to track all policy changes and execution.

[0170] The policy catalog also maintains the lifecycle status and version information for each policy, enabling the system to accurately track a policy's current status, historical changes, and execution progress. Furthermore, it establishes a close link between policies, snapshots, and execution records, forming a complete data chain that facilitates analysis and troubleshooting.

[0171] In practice, NineData stores the policy catalog in an independent metadata repository (e.g., SQLite) rather than in a managed vector database. This ensures consistency across multiple vector databases while also meeting transaction, audit, backup, and high availability requirements.

[0172] With this design, users simply declare policies using SQL, and the system automatically persists them, schedules their execution, and provides complete historical tracking. The policy catalog essentially forms the infrastructure for implementing "declarative vector data lifecycle management," enabling the entire system to operate reliably, traceably, and efficiently.

[0173] 3.1 Directory table structure design

[0174] The policy catalog contains the following four core tables:

[0175] nd_policies (policy definition table): stores active vector lifecycle policies. The fields include:

[0176] -policy_id: varchar(50)

[0177] -policy_name: varchar(100)--Policy name

[0178] -table_name: varchar(100)

[0179] -column_name: varchar(100)--target column name

[0180] -schedule:varchar(50)

[0181] -drift_metric:enum('COSINE_DIVERGENCE','KL_DIVERGENCE','WASSER

[0182] STEIN_DISTANCE')

[0183] -threshold: decimal(10,6)

[0184] -action_blob:json

[0185] -status:enum('ACTIVE','PAUSED','DISABLED')

[0186] -created_time: timestamp

[0187] -updated_time: timestamp.

[0188] nd_vector_snapshots (baseline snapshot table): stores statistical features of vector data for drift detection and comparison. The fields include:

[0189] -snapshot_id: varchar(50)

[0190] -policy_id: varchar(50)--association policy

[0191] -table_name: varchar(100)

[0192] -column_name: varchar(100)

[0193] -creation_time: timestamp

[0194] -vector_count: bigint--vector number

[0195] -stats:json--statistical features

[0196] -version:int -- Snapshot version.

[0197] nd_policies_history (policy history table): records the version change history of the policy and supports auditing and rollback.

[0198] nd_policy_runs (execution log table): records policy execution details, including drift detection results, execution actions, and performance indicators.

[0199] Furthermore, the relationship between the tables is as follows:

[0200] nd_policies and nd_vector_snapshots (one-to-many: a policy can have multiple snapshot versions);

[0201] nd_policies and nd_policy_runs (one-to-many: one policy has multiple execution records);

[0202] nd_vector_snapshots and nd_policy_runs (one-to-many: a snapshot may be referenced by multiple execution records).

[0203] When creating a policy, the system automatically generates baseline snapshots. These snapshots serve as a reference for subsequent drift detection, ensuring that the system can accurately identify changes in data distribution. To optimize storage space, the system only stores statistical features rather than complete vector data. This approach significantly reduces storage overhead and enables efficient operation even on large vector datasets. Furthermore, the system supports associating policies with different versions of baseline snapshots. This multi-version support effectively addresses version differences, making management more flexible and adaptable to the needs of different business scenarios.

[0204] 4. Statistics Collector

[0205] The statistics collector is responsible for monitoring vector data and index status and generating statistics for threshold comparison.

[0206] 4.1 Vector Sampling Query

[0207] The statistics collector uses a sampling mechanism to ensure efficient and accurate monitoring of data distribution changes. The system implements dynamic sampling rate adjustment, with a default sampling probability of 0.01, which is automatically optimized based on data size: for large tables with more than 100 million vectors, the system automatically reduces the sampling rate to 0.001 to reduce the computational burden; for smaller tables, the sampling rate is increased to ensure statistical accuracy.

[0208] In addition, the system also introduces a tiered / archived sampling strategy for hot and cold data. By analyzing access frequency metadata (if supported by the database), it prioritizes sampling of frequently accessed "hot" data, or samples hot and cold data separately and generates tiered / archived statistics, thereby more accurately reflecting the data characteristics in actual usage scenarios.

[0209] At the query execution level, the system generates optimized and efficient sampling queries, making full use of the random sampling functions (such as random ID queries) or custom user functions provided by the underlying database, significantly reducing data transmission volume and improving overall sampling efficiency.

[0210] 4.2 Drift metric calculation

[0211] The system provides an intelligent metric selection mechanism that can automatically select the most suitable drift measurement method based on the characteristics and distribution characteristics of vector data. It also supports users to explicitly specify the required metric type in the SQL strategy definition. NineData implements the drift measurement algorithm in the middleware layer by sampling vectors. Drift detection and metric calculation process reference Figure 8 and Figure 9 The process shown.

[0212] For example, for text embedding vectors, COSINE_DIVERGENCE is usually used as the default metric; while for data with complex geometric characteristics such as image feature vectors, WASSERSTEIN_DISTANCE is preferred to obtain more accurate drift detection results.

[0213] In addition, to efficiently store and optimize policies, all drift calculation results are cached and persistently stored in the nd_policy_runs table. This not only facilitates administrators to analyze historical drift trends but also provides valuable data support for system performance tuning.

[0214] COSINE_DIVERGENCE: Cosine divergence is a classic vector similarity metric that detects drift by calculating the directional change in the center of the data distribution (the mean vector). The system compares the overall distribution direction of the current data with the baseline data, efficiently detecting shifts in the distribution center of gravity. It is particularly suitable for drift detection in large-scale datasets.

[0215] KL_DIVERGENCE: KL divergence (Kullback-Leibler divergence) is a classic statistical measure used to measure the difference between two probability distributions. In this system, it is used to detect changes in the shape of vector data distributions and is particularly suitable for detecting changes in complex multimodal distributions.

[0216] WASSERSTEIN_DISTANCE: Wasserstein distance (also known as Earth Mover Is Distance) is a distance metric used in classic optimal transport theory. In this system, it measures the geometric differences between vector data distributions and effectively detects geometric changes such as spatial displacement and rotation of data distributions.

[0217] 4.3 Index Health

[0218] The statistics collector monitors and evaluates the health of vector indexes in several ways.

[0219] When the database provides an index status API interface (such as Milvus's get_index_state), the collector can directly extract key indicators, including direct indicators such as fragmentation level and node imbalance rate, providing first-hand data for index performance evaluation.

[0220] For databases that don't provide a direct API, the system uses indirect inference methods. By running a series of test queries (such as random ANN searches) to measure query latency and recall (recall is a measure of search result accuracy, indicating the proportion of relevant results returned compared to all relevant results), the system can infer the health of the index. For example, if query latency increases significantly from a normal 50ms to 150ms, this may be a clear sign of index fragmentation, prompting the system to perform maintenance operations.

[0221] All collected health metrics are stored in the nd_policy_runs table and used together with drift metrics to assess thresholds, providing data support for automated maintenance decisions. This comprehensive assessment mechanism ensures that the system can promptly detect and respond to index performance degradation, maintaining query efficiency.

[0222] In actual applications, for example, when the system monitors the image_embeddings table, the collector generates and executes a sampling query such as SELECT feature_vector FROM image_embeddings USING PROBABILITY 0.001, extracting approximately 10,000 vector samples from the database.

[0223] The system then calculated a COSINE_DIVERGENCE value of 0.15 for these samples compared to the baseline and wrote this result to the nd_policy_runs table. Simultaneously, the system detected that index query latency had reached 130ms, exceeding the preset threshold. This triggered appropriate maintenance actions to ensure acceptable system performance.

[0224] 5. Action Engine

[0225] The operation engine converts policy actions into specific operations and adapts to different vector databases.

[0226] 5.1 Operation Mapping and Adaptation Layer

[0227] The operation engine implements intelligent mapping of common actions to specific database APIs through the adaptation layer. This adaptation layer is responsible for converting abstract operation instructions into database-specific API calls, such as seamlessly converting the common REINDEX_VECTOR_COLUMN instruction into Milvus's create_index function or Pinecone's create_index / delete_index interfaces.

[0228] For example, the parameters required by Milvus are:

[0229] {

[0230] "collection_name":"image_embeddings",

[0231] "field_name": "feature_vector",

[0232] "index_type": "HNSW",

[0233] "metric_type": "L2",

[0234] "params": {"M": 32, "efConstruction": 256}

[0235] }.

[0236] Based on the identified database type, the adapter generates the corresponding API call sequence. For example, for the Milvus database, the system constructs a JSON structure containing the collection name, field name, index type, and parameters. To ensure broad system compatibility, the adapter layer maintains detailed database version information and API signature libraries, supporting a variety of mainstream vector databases, including Milvus, Pinecone, and Weaviate, enabling cross-database compatible execution of operations.

[0237] 5.2 Re-embedding

[0238] During the re-embedding process, the engine first selects the data to be processed based on the policy conditions. When the drift threshold is triggered, the system prioritizes the subset of data with the most severe drift. It also supports full re-embedding of the entire table using the ACTION REEMBED_TABLE command.

[0239] Re-embedding is typically triggered automatically by a pre-set policy, but the middleware also provides SQL interfaces and internal mechanisms for manually invoking the embedding service. During automated policy execution, the system can update data using SQL operations similar to the following.

[0240] --re-embed

[0241] UPDATE image_embeddings

[0242] SETfeature_vector=EMBED_TEXT(source_text_column,'text-embedding-model-v3',768)

[0243] WHERE<condition_for_reembedding> .

[0244] In the actual automation strategy, the engine directly interacts with the external AI service according to instructions such as ACTION REEMBED_DATA USING SERVICE'external_embedding_service_v2' to obtain new vector embeddings in batches.

[0245] To improve processing efficiency, the engine uses a batch call mechanism to send vectors to embedded services (such as the OpenAI / ALIYUN API) in batches. The batch size is dynamically adjusted based on system load, with a default setting of 1,000 vectors per batch. All failures are logged in detail in the nd_policy_runs log.

[0246] Finally, all newly generated vectors are written to the database through the UPSERT operation to ensure data consistency and integrity.

[0247] 5.3 Re-indexing

[0248] Shadow index rebuilds allow vector indexes to be updated and optimized without interrupting production queries.

[0249] First, the system enters the preparation phase. Based on policy parameters (such as PROPS(M=32,efConstruction=256,shadow_copy=true)), it decides to use a shadow rebuild and generates a unique identifier for the new index, such as feature_vector_index_shadow_20241109135524. Resource reservation is then performed, assessing the resources required for the index rebuild to ensure sufficient disk space and memory. For large indexes, dedicated nodes may be allocated or the rebuild may be scheduled during off-peak hours.

[0250] During the index creation phase, the index creation API is called through the database adapter, but it is not immediately associated with the table. For example, the call method for generating the Milvus API is:

[0251] create_index(

[0252] collection_name="image_embeddings",

[0253] field_name="feature_vector",

[0254] index_name="feature_vector_index_shadow_20241109135524",

[0255] params={"index_type":"HNSW","params":{"M":32,"efConstruction":

[0256] 256}}

[0257] ).

[0258] The next step is to perform an index switch. For databases that support native index switching, the system directly uses their native API. For other databases, the classic shadow index mechanism is used (a new, identical index is created alongside the existing index, coexisting with the existing one, and an atomic switch is performed after verification). This ensures that query services are not interrupted during the switch, and the switch time is typically controlled within 50ms.

[0259] Finally, perform subsequent cleanup. After the switch is successful, wait for the old index query to complete (monitor the number of active queries to zero or timeout), and delete the old index during off-peak hours to free up resources. The shadow index method shortens the maintenance window from hours to minutes, eliminating user-perceived downtime and maintaining service availability and query performance in high-load environments. Figure 4 、 Figure 5 、 Figure 6 and Figure 7 The process shown.

[0260] It's important to note that after the shadow index is built, the middleware sends a small portion (e.g., 1%) of online read-only query requests to both the primary index and the shadow index. The system then compares the returned results (e.g., the overlap in the Top-K lists) and calculates the recall of the shadow index relative to the primary index. Only after ensuring it's at least a preset threshold (e.g., 99%) will the switch be executed.

[0261] 5.4 Layering / Archiving

[0262] The tiering / archiving feature supports hot and cold data management based on data characteristics. The system supports various criteria for determining hot and cold data, including time thresholds (for example, you can set the parameter "older_than=6months"), access frequency (inferred by analyzing database access logs or middleware behavior), and custom SQL conditions (such as "WHERE creation_date < '2024-01-01'").

[0263] Regarding migration mechanisms, the system optimizes hot and cold data tiering / archiving based on the target database's capabilities, moving infrequently accessed vector data (and optional indexes) to low-cost storage (such as Amazon S3). If the database supports native hot and cold partition migration (such as Weaviate's move_partition API), migration can be completed directly by calling the API. For databases that don't support native migration (such as Milvus), the system simulates the migration through the following steps: 1) querying cold data (based on conditions such as creation_date < '2024-01-01') and copying it to low-cost storage; 2) updating collection metadata to record the low-cost storage path for the cold data; and 3) deleting the original cold data to free up high-cost storage space. To support cold data queries, the system maintains a query routing table, dynamically redirecting queries to low-cost storage or preloading to high-performance storage (if real-time ANN search is required). Migration operations are scheduled during off-peak hours, and the duration depends on data volume and network bandwidth (for example, approximately 5-10 minutes for 1 million vectors). Metadata updates are atomic operations to ensure data consistency. If index migration is involved, the index files are stored with the cold data in low-cost storage and need to be loaded back into the local storage when used to restore query performance.

[0264] 6. Scheduler

[0265] The scheduler manages the scheduled execution of policies, ensuring the robustness and efficiency of the entire system. Regarding task management, the scheduler parses timed expressions such as "SCHEDULE CRON '0 0 1 * *'," calculates the next execution time, and stores it in an in-memory priority queue. The scheduler also sets a maximum execution time limit for each task (for example, 1 hour) and automatically terminates and logs an error message when a task times out. For tasks that fail due to reasons such as API timeouts, the system supports an automatic retry mechanism, with a default of three retries and an exponential backoff strategy.

[0266] Regarding concurrency control, the scheduler sorts tasks according to the priority defined by the policy (e.g., specified using the WITH PRIORITY HIGH directive). To prevent database overload, the system limits the number of concurrent tasks (default 10), and the engine dynamically adjusts concurrency based on database load metrics (e.g., CPU usage). When multiple policies target the same table, the scheduler serializes execution to effectively prevent lock contention.

[0267] For example, the scheduler triggers a task named monthly_reindex at 00:00 on the 1st of each month. If the Milvus database is heavily loaded at this time, the scheduler delays the execution of the task and logs the reason for the delay in the nd_policy_runs log table, ensuring transparency and traceability of system operations.

[0268] 7. Audit and Security

[0269] The system provides auditing capabilities. All policy modifications are recorded in the nd_policies_history table, preserving historical versions and change records. Furthermore, each policy execution, including details of success and failure, is recorded in the nd_policy_runs table. These logs support enterprise compliance requirements and facilitate traceability and troubleshooting.

[0270] In terms of security, the system implements a permission control mechanism. Users must possess ADMIN privileges to create or modify vector lifecycle policies. This permission control is enforced through the database's Role-Based Access Control (RBAC) system. Furthermore, all SQL statements undergo parameterized parsing before execution to prevent SQL injection attacks. Incoming JSON is fully schema-validated and escaped before being inserted into the API. This prevents string concatenation in the JSON from being mapped to the underlying REST call, which could introduce secondary injection risks.

[0271] 8. Alternative Embodiments

[0272] The system supports a variety of flexible implementation options. In terms of scheduling, in addition to the built-in scheduler, the system can also integrate with external scheduling systems such as Kubernetes CronJob or Apache Airflow. In this case, SQL statements are only used for policy declarations, while execution and scheduling are handled by these specialized tools.

[0273] For multi-tenant scenarios, the system has designed a directory table structure isolated by tenant ID, fully supporting SaaS model deployment and ensuring that data and policies between different tenants are isolated from each other.

[0274] In terms of drift measurement, the system not only provides built-in measurement methods, but also supports the extension of custom metrics through user-defined functions (UDFs) to meet the needs of specific business scenarios.

[0275] The system also supports federated query capabilities, allowing the execution of federated strategies across different vector databases and relational databases. For example, Milvus and MySQL databases can be federated simultaneously to implement more complex data management strategies.

[0276] Core innovations include:

[0277] 1. Declarative SQL extension:

[0278] Extends standard SQL and adds new commands such as CREATE VECTOR_LIFECYCLE_POLICY.

[0279] Users use simple SQL statements to manage policies, for example:

[0280] "Detecting product vector drift exceeding 10%, automatically optimizing indexes and using shadow indexes to ensure zero downtime";

[0281] “Regularly check index health and automatically rebuild inefficient indexes”;

[0282] "Data that has not been accessed for more than 6 months should be moved to low-cost storage (hot and cold tiering / archive)."

[0283] 2. Intelligent adaptation:

[0284] The middleware automatically translates common SQL strategies into native instructions for different databases, so users do not need to worry about the underlying differences.

[0285] 3. Non-invasive:

[0286] There is no need to modify the vector database kernel, making it easy to adapt to new databases or existing systems.

[0287] Main value points and innovations:

[0288] Lowering the barrier to entry: Developers can manage complex vector databases using familiar SQL without having to learn proprietary APIs.

[0289] Standardized and cross-platform: unified management interface, compatible with multiple vector databases.

[0290] Auditable: Operation records meet compliance requirements.

[0291] Declarative advantage: Users only need to define the "target state" without writing complex scripts.

[0292] Middleware value: decouples applications from databases, providing flexibility and scalability.

[0293] The difference between the present invention and the prior art

[0294] Non-simple SQL queries: Existing technologies may support SQL queries on vector databases, but the present invention uses SQL to define and automate the entire lifecycle management strategy.

[0295] Technical details of the vector drift metric:

[0296] Note: These metrics themselves are not the core innovation of this invention. Instead, they are existing technologies integrated into the declarative SQL middleware system to implement automated vector data drift detection. In other words, any unclear points in the following description can be referred to the relevant existing technologies.

[0297] A.1 Cosine Divergence

[0298] Mathematical definition:

[0299] COSINE_DIVERGENCE , )=1-cos(θ)=1-( · ) / (|| ||×|| ||);

[0300] in: represents the mean vector of baseline data; Represents the mean vector of the current data; cos(θ) is the cosine similarity between the two mean vectors.

[0301] Calculation process:

[0302] 1. Get the mean vector from the baseline snapshot ;

[0303] 2. Calculate the mean vector of the current sampling data =(1 / n)Σ ;

[0304] 3. Calculate the cosine divergence value;

[0305] 4. Compare with the preset threshold to determine whether drift has occurred.

[0306] Code example:

[0307] Python

[0308] baseline_mean=get_baseline_mean_vector(snapshot_id)

[0309] current_mean=compute_mean_vector(current_samples)

[0310] cosine_sim=dot_product(baseline_mean,current_mean) / (norm(baseline_mean)*norm(current_mean));

[0311] divergence=1-cosine_sim.

[0312] A.2KL Divergence (KL_DIVERGENCE)

[0313] Mathematical definition:

[0314] KL(P||Q)=∫p(x)log(p(x) / q(x))dx;

[0315] For high-dimensional vectors, approximate calculations are performed through dimensionality reduction or distribution fitting:

[0316] KL_approx(P||Q)=∑ip( )log(p( ) / q( ));

[0317] Calculation steps:

[0318] 1. Perform bucketing on the baseline and current vector data (e.g., bucketing after PCA dimensionality reduction);

[0319] 2. Estimate the probability density p( ) and q( );

[0320] 3. Calculate the discretized KL divergence value;

[0321] 4. Apply smoothing techniques to handle zero-probability cases.

[0322] Applicable scenarios:

[0323] Detecting changes in multimodal distributions;

[0324] A significant change in the shape of the distribution was found;

[0325] Suitable for medium-sized vector collections (avoiding the curse of dimensionality).

[0326] A.3 Wasserstein distance (WASSERSTEIN_DISTANCE)

[0327] Mathematical definition (1st-order Wasserstein distance):

[0328] (P,Q)=inf_{γ∈Γ(P,Q)}∫||xy||dγ(x,y);

[0329] Approximate calculation method: For high-dimensional vectors, use Sliced ​​Wasserstein Distance:

[0330] SWD(P,Q)=∫_{S^{d-1}} (P_θ,Q_θ)dθ;

[0331] Implementation algorithm:

[0332] 1. Randomly select multiple projection directions θ;

[0333] 2. Project the vector into one-dimensional space;

[0334] 3. Calculate the 1D Wasserstein distance of the projected distribution;

[0335] 4. Average all projection directions.

[0336] Code example:

[0337] Python

[0338] def sliced_wasserstein_distance(X,Y,num_projections=100):

[0339] distances=[]

[0340] for _ in range(num_projections):

[0341] #Randomly generate projection direction

[0342] direction=random_unit_vector(dim)

[0343] #projection

[0344] X_proj=X@direction

[0345] Y_proj=Y@direction

[0346] #Calculate 1D Wasserstein distance

[0347] distances.append(wasserstein_1d(X_proj,Y_proj))

[0348] return np.mean(distances).

[0349] A.4 Guidance for selecting measurement methods

[0350] COSINE_DIVERGENCE is applicable when:

[0351] Large-scale datasets (>106 vectors);

[0352] The main focus is on the shift in the center of the overall distribution;

[0353] Scenarios with limited computing resources;

[0354] KL_DIVERGENCE applies when:

[0355] Medium-sized dataset (10 4 -10 6 vectors);

[0356] Need to detect changes in the shape of the distribution;

[0357] Multimodal data distribution;

[0358] WASSERSTEIN_DISTANCE applies when:

[0359] Pay attention to changes in spatial geometry;

[0360] Detecting translation and rotation of distributions;

[0361] A scene that is somewhat robust to noise.

[0362] A.5 Threshold Setting Recommendations

[0363] Based on experimental experience, the recommended default thresholds are:

[0364] COSINE_DIVERGENCE: 0.05-0.15 (depending on the application scenario);

[0365] KL_DIVERGENCE: 0.1-0.5 (needs to be adjusted according to data characteristics);

[0366] WASSERSTEIN_DISTANCE: 10%-30% of the data standard deviation.

Claims

1. A vector database lifecycle management method based on SQL middleware, characterized in that: include: S1. Receiving and parsing: Receive an extended SQL statement containing a vector lifecycle policy declaration and parse it to obtain policy parameters, including the target table, target vector column, scheduling rules, drift threshold, and maintenance action parameters. S2. Baseline snapshot generation; generating a first statistical snapshot based on the target vector column at an adaptive sampling rate, associating the first statistical snapshot with a policy parameter and storing the result in a policy directory; S3. Strategy orchestration; based on the target vector database type, select the call sequence corresponding to the maintenance action parameters from the preset API mapping table and generate a maintenance execution plan; S4. Periodic monitoring; regularly sample the target vector column according to the scheduling rules and calculate the drift metric between the current sample and the first statistical snapshot; Get the index health indicators of the target table based on the target vector database type; S5. Threshold determination: When the drift metric exceeds the drift threshold, or the index health indicator exceeds the preset performance threshold, the maintenance process is triggered; S6. Shadow index zero-downtime maintenance: After the maintenance process is triggered, a shadow index is created in the target vector database based on the index parameters in the maintenance execution plan. The shadow index's recall rate is verified to be no less than the preset recall threshold by reading traffic mirroring. If verification passes, the shadow index is atomically switched to the primary index, and the old index is deleted after query traffic drops to zero or times out. S7. Persistence of results: Write the threshold determination results, shadow index switching status, and related performance indicators to the policy execution log for auditing and tracing.

2. The vector database lifecycle management method based on SQL middleware according to claim 1, characterized in that: The adaptive sampling rate dynamically adjusts the sampling probability according to the data size.

3. The vector database lifecycle management method based on SQL middleware according to claim 2, characterized in that: The adaptive sampling rate adopts a sampling probability of 0.01 by default.

4. The vector database lifecycle management method based on SQL middleware according to claim 2, characterized in that: The adaptive sampling rate dynamically adjusts the sampling probability according to the data size, including: If the database contains more than 100 million vectors, the sampling rate is automatically reduced to 0.001; Otherwise, the sampling rate is increased linearly.

5. The vector database lifecycle management method based on SQL middleware according to claim 1, characterized in that: The drift metric calculation method includes at least one of the following methods: Method 1: For baseline data and current data, calculate the cosine divergence of the mean vector; Method 2: Calculate KL divergence after bucketing the data; or Method 3: Calculate Wasserstein distance by random projection.

6. The vector database lifecycle management method based on SQL middleware according to claim 1, characterized in that: The index health indicators include query latency and fragmentation rate. When the query latency increases beyond a threshold or the fragmentation rate exceeds a preset threshold, a maintenance process is triggered.

7. The vector database lifecycle management method based on SQL middleware according to claim 1, characterized in that: During the creation and switching process of the shadow index, the shadow index is built in parallel in the background without affecting the production query.

8. The vector database lifecycle management method based on SQL middleware according to claim 7, characterized in that: The switching time of the shadow index is controlled within 50ms.

9. The vector database lifecycle management method based on SQL middleware according to claim 1, characterized in that: The maintenance actions include re-indexing, re-embedding, and tiering / archiving.

10. The vector database lifecycle management method based on SQL middleware according to claim 9, characterized in that: The re-embedding operation is as follows: a vector is updated for data whose drift metric exceeds a threshold by calling an external AI service.

11. The vector database lifecycle management method based on SQL middleware according to claim 9, characterized in that: In the tiering / archiving, hot and cold data are determined based on the data's time threshold, access frequency, or custom SQL conditions, cold data is migrated to low-cost storage, and a query routing table is maintained to support cold data queries.

12. The vector database lifecycle management method based on SQL middleware according to claim 1, characterized in that: The policy directory includes the nd_policies table for recording policy definitions, the nd_policies_history table for recording policy version changes, and the nd_policy_runs table for recording policy execution details.

13. The vector database lifecycle management method based on SQL middleware according to claim 1, characterized in that: The API mapping relationship table records different vector database versions and API call sequences of maintenance actions.

14. A vector database lifecycle management system based on SQL middleware, characterized in that: The vector database lifecycle management method based on SQL middleware according to any one of claims 1 to 13 is applied, wherein the vector database lifecycle management system based on SQL middleware comprises: SQL parser, used to parse extended SQL statements and output policy parameters; Policy catalog, used to store policy parameters, baseline snapshots, and execution logs; Statistics collectors for monitoring vector distribution and index health at an adaptive sampling rate and computing cosine divergence, KL divergence, or Wasserstein distance; A mapping relationship table that records the API call sequences between different vector database versions and maintenance actions; A shadow index module, configured to create shadow indexes, verify recall, and atomically switch without interrupting online queries; Scheduler, which triggers the execution of statistics collector and shadow index modules according to the scheduling rules in the policy catalog; The interaction results between the statistics collector and the shadow index module are written into the policy directory by the scheduler to achieve closed-loop lifecycle management.

Images