Offline vector library building method, system, electronic device, and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and provides an offline vector library building method and system, an electronic device and a storage medium, which comprise the following steps: obtaining a to-be-processed directory from an embedding queue, performing vectorization processing on each to-be-processed data under the to-be-processed directory, generating vector data and putting the vector data into a storage queue; obtaining the vector data from the storage queue and batch writing the vector data into a vector database; collecting the current queue depth of each stage queue according to a preset period, and dynamically adjusting target parameters based on the deviation of the current queue depth and a preset target interval, so that the current queue depth is maintained in the preset target interval, and the target parameters comprise the parallelism and / or batch size of the processing stage corresponding to each stage queue. Through the adaptive control strategy of dynamically adjusting the parallelism and batch size based on the queue depth deviation, load balancing and steady-state operation of the whole process of offline vector library building are realized, and the application has the advantages of high data processing throughput, strong system pressure resistance and high resource utilization rate.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an offline vector library construction method, system, electronic device, and storage medium. Background Technology

[0002] With the widespread application of artificial intelligence and Retrieval Augmented Generation (RAG) technology, transforming massive amounts of unstructured data (such as images and text) into high-dimensional vectors and storing them in databases has become a crucial step in data processing. In existing technologies, offline database construction for vector data typically employs batch encoding and importing via scripts, or leverages a general ETL (Extract, Transform, Load) framework to connect model encoding and database writing.

[0003] However, when faced with ultra-large-scale data scenarios of tens of millions or more, the above solutions lack the ability to adaptively adjust the real-time load of the system, which can easily lead to uneven loads at different stages of the pipeline, causing system overload oscillations or local bottlenecks, making it difficult to maintain a stable high throughput. Summary of the Invention

[0004] This invention provides an offline vector database construction method, system, electronic device, and storage medium to address the shortcomings of existing technologies in large-scale database construction, which suffer from low system stability and throughput due to the single flow control strategy.

[0005] This invention provides an offline vector library construction method, comprising: The process retrieves the directory to be processed from the embedding queue, performs vectorization processing on each data to be processed under the directory to be processed, generates vector data for each data to be processed, and puts the vector data into the database queue. The vector data is retrieved from the inbound queue and written into the vector database in batches. The current queue depth of each stage queue is collected according to a preset period, and the target parameters are dynamically adjusted based on the deviation between the current queue depth and the preset target interval to maintain the current queue depth within the preset target interval. Each stage queue includes the embedded queue and the inbound queue, and the target parameters include the parallelism and / or batch size of the processing stage corresponding to each stage queue.

[0006] According to an offline vector library construction method provided by the present invention, the step of dynamically adjusting the target parameters based on the deviation between the current queue depth and the preset target interval includes: Calculate the difference between the current queue depth and the target queue depth at the current sampling time to obtain the observation error. The target queue depth is determined based on the preset target interval. The proportional adjustment term is calculated based on the observation error, and the observation errors at historical sampling times are summed to calculate the integral adjustment term. The proportional adjustment term and the integral adjustment term are superimposed to calculate the initial target parameter at the next sampling time. The initial target parameter is then truncated to obtain the target parameter.

[0007] The offline vector library construction method provided by the present invention further includes: When calculating the integral adjustment term, if the target parameter reaches the preset parameter boundary value and the direction indication control quantity of the observation error continues to exceed the parameter boundary value, then the summation of the observation error at the historical sampling time is stopped or the integral adjustment term is reversed.

[0008] According to an offline vector library construction method provided by the present invention, the step of obtaining the directory to be processed from the embedding queue further includes: Perform quality gating checks on the catalog to be processed; If the directory to be processed is found to meet the quality gating conditions, the data to be processed under the directory to be processed is written to the local record library in batches, the task status of the data to be processed is initialized in the local record library, and the directory to be processed is placed into the embedding queue.

[0009] According to an offline vector library construction method provided by the present invention, the quality gating check of the directory to be processed includes: Check if a metadata file of a preset type exists in the directory to be processed, and count the number of valid data to be processed in the directory to be processed; If the metadata file exists and the number of valid files is not less than a preset threshold, the directory to be processed is determined to meet the quality gating condition.

[0010] The offline vector library construction method provided by the present invention further includes: During the process of batch writing the vector data of each data to be processed into the vector database, if the writing of the vector data of any data to be processed fails, a backoff retry will be performed within the maximum number of retries. Obtain the write result corresponding to the vector data of each data to be processed, and put the write result into the write-back queue. The queues of each stage also include the write-back queue. Based on the write results in the write-back queue, the task status of each piece of data to be processed in the local record library is updated.

[0011] The offline vector library construction method provided by the present invention further includes: After the offline vector library building task is abnormally interrupted and restarted, the task status of each data to be processed in the local record library is read. If the task status of any pending data is in the initial state or the failure state, the pending directory to which the pending data belongs is put back into the embedding queue. The pending directory is obtained from the embedded queue. Data pending in the pending directory with a task status of success is skipped, and data pending with a task status of initial or failure is vectorized and stored in the database.

[0012] According to an offline vector database construction method provided by the present invention, the step of batch writing the vector data into a vector database includes: Obtain the file storage path of any data to be processed and the index number of the data to be processed under the file storage path; Based on the file storage path and the index number, a deterministic primary key is generated for any data to be processed; Based on the deterministic primary key, the vector data of any data to be processed is written into the vector database.

[0013] According to an offline vector database construction method provided by the present invention, the step of writing the vector data of any data to be processed into the vector database based on the deterministic primary key includes: Perform a hash operation on the file storage path to obtain the group identifier of any data to be processed; Extract category information from the metadata of any data to be processed, and determine the target partition corresponding to any data to be processed in the vector database according to the preset category mapping relationship and the category information; Based on the deterministic primary key, the group identifier, and the vector data of any data to be processed, a row record is constructed and written to the target partition.

[0014] According to an offline vector database construction method provided by the present invention, the vectorization processing of each piece of data to be processed in the directory to be processed includes: The embedded model service interface is invoked to perform feature encoding on each of the data to be processed; If any data to be processed fails to be encoded, the retry waiting time is determined based on an exponential backoff strategy. The exponential backoff strategy is used to control the retry waiting time to increase exponentially with the number of retries. After the retry waiting period, any data to be processed is re-encoded until the encoding is successful or the preset maximum number of retry attempts is reached.

[0015] The offline vector library construction method provided by the present invention further includes: Obtain the remaining task volume and the average processing rate at the current moment for the offline vector database construction task, and determine the estimated remaining time for the offline vector database construction task based on the remaining task volume and the average processing rate. The instantaneous processing speed of each processing stage of the offline vector task is obtained at the current moment, and the processing stage with the lowest instantaneous processing speed is identified as the bottleneck stage at the current moment. The identifier of the bottleneck stage is used to trigger an alarm and / or assist in adjusting the target parameters.

[0016] According to an offline vector library construction method provided by the present invention, the processing stages include an embedding processing stage, a library entry stage, and a write-back stage. For the embedding processing stage, the target parameters include the parallelism of vectorized processing and / or the batch size; For the ingestion stage, the target parameters include the parallelism and / or batch size of writing to the vector database; For the write-back phase, the target parameters include the parallelism and / or batch size for updating the task status in the local record library.

[0017] This invention also provides an offline vector library building system, comprising: The embedding processing module is used to obtain the directory to be processed from the embedding queue, perform vectorization processing on each data to be processed under the directory to be processed, generate vector data of each data to be processed, and put the vector data into the database queue. The inbound module is used to retrieve the vector data from the inbound queue and write the vector data into the vector database in batches; The monitoring module is used to collect the current queue depth of each stage queue according to a preset period, and dynamically adjust the target parameters based on the deviation between the current queue depth and the preset target interval, so as to maintain the current queue depth within the preset target interval. The stage queues include the embedded queue and the inbound queue, and the target parameters include the parallelism and / or batch size of the processing stage corresponding to each stage queue.

[0018] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the offline vector library construction method as described above.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the offline vector library construction method as described above.

[0020] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the offline vector library construction method as described above.

[0021] The offline vector library construction method, system, electronic device, and storage medium provided by this invention construct a closed-loop feedback adjustment mechanism for the offline library construction process by collecting the current queue depth of each stage queue according to a preset period and calculating its deviation from a preset target range. This mechanism can adaptively and dynamically adjust the parallelism and / or batch size of each processing stage based on the real-time load of each processing stage, thereby precisely controlling the data flow speed of upstream and downstream processes. This dynamic adjustment ensures that the queue depth of each processing stage is always maintained within the preset target range, effectively balancing the rate difference between vectorization processing and vector library writing. It avoids queue backlog and system overload caused by excessively fast processing in the preceding stages, and also prevents downstream resource idleness caused by excessively slow processing in the preceding stages. It eliminates the lag and uncertainty of manual parameter tuning in the traditional fixed parameter mode, significantly improving the execution stability and overall throughput efficiency of large-scale vector library construction tasks. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the offline vector library construction method provided by the present invention; Figure 2 This is a schematic diagram of the state write-back process provided by the present invention; Figure 3 This is a schematic diagram of the breakpoint recovery process provided by the present invention; Figure 4 This is a flowchart illustrating the recoverable offline vector library construction method for multimodal ultra-large-scale data provided by the present invention; Figure 5 This is one of the structural schematic diagrams of the offline vector library building system provided by the present invention; Figure 6 This is the second schematic diagram of the offline vector library building system provided by the present invention; Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] With the rapid development of artificial intelligence, cloud computing, and big data technologies, especially against the backdrop of the increasing popularity of Large Language Models (LLM) and Retrieval Augmentation (RAG) technologies, how to efficiently manage and retrieve massive amounts of unstructured data (such as images and text) has become a focus of industry attention. In RAG systems, one of the core steps is to transform massive amounts of raw data into high-dimensional vectors (Embeddings) using deep learning models and store them in vector databases (such as Milvus and FAISS) for subsequent similarity retrieval and generation. This process is often referred to as offline vector database construction or vector data engineering.

[0026] Currently, offline database construction for vector data commonly adopts a model-generated vector and vector database storage approach. For example, scripting languages ​​like Python are used to write programs that read data serially or in simple parallel, call model encoding interfaces, and batch insert the encoded results into the vector database. Another approach is to leverage general ETL frameworks, such as a producer / consumer pattern based on queues (like Kafka and RabbitMQ), to decouple data reading, feature extraction (vectorization), and database insertion operations, utilizing general big data processing frameworks for data flow.

[0027] However, when faced with massive multimodal datasets of tens of millions or even larger scales, the above solutions have significant limitations in practical engineering applications. First, they lack adaptive load control and performance optimization strategies. In existing ETL frameworks, the processing capabilities of each stage (such as data reading, vectorization computation, and database writing) are often uneven and dynamically changing. Existing technologies typically use a fixed number of threads or batch size, lacking the ability to dynamically perceive system load. When a certain stage (such as vectorization computation) becomes a bottleneck, a fixed concurrency strategy can easily lead to upstream queue backlog or downstream resource idleness; while a simple failure retry strategy, when faced with system overload, is prone to causing an oscillating effect of overload, backoff, and further overload, making it difficult to maintain a steady-state high-throughput operation of the system.

[0028] Secondly, there is a lack of stability and breakpoint recovery mechanisms for large-scale data processing. Importing tens of millions of data points often involves lengthy tasks and is highly susceptible to network fluctuations, long-tail sample blocking, or abnormal system interruptions. Existing scripts or general ETL solutions typically lack fine-grained, engineered mechanisms for state recovery. Once a task is interrupted, a full rerun or complex manual troubleshooting is often required to pinpoint the breakpoint, leading to a significant waste of computing resources and severely impacting the timeliness of database creation. Furthermore, the lack of automatic gating for sample quality (e.g., missing metadata, corrupted files, insufficient valid data streams) easily results in a large influx of dirty data into the database, polluting index quality.

[0029] Furthermore, idempotency and consistency across batches are difficult to guarantee. Traditional solutions often rely on the database's own auto-incrementing ID generation mechanism. When a task fails and is retried or an incremental update is performed, the same data may generate new IDs, leading to duplicate data or version drift in the database, making it difficult to align with previous versions. This data inconsistency severely affects the accuracy of retrieval and causes disordered expansion of the vector index library.

[0030] Finally, there is insufficient end-to-end observability and granular operational support. When processing massive amounts of data, operations personnel struggle to monitor the precise throughput, queue depth, and estimated time of arrival (ETA) at each stage in real time. Partition routing and index optimization for multi-category data are also often inadequate, resulting in poor final retrieval efficiency.

[0031] To address this, the present invention provides an offline vector database construction method for multimodal, ultra-large-scale data, thereby overcoming the aforementioned shortcomings. It should be noted that all actions involving the acquisition of information or data in this invention are performed in accordance with the relevant data protection laws and policies of the country where the data is acquired, and with authorization from the owner of the corresponding device.

[0032] Figure 1 This is a flowchart illustrating the offline vector library construction method provided by the present invention, as shown below. Figure 1 As shown, the method includes: Step 110: Obtain the directory to be processed from the embedding queue, perform vectorization processing on each data to be processed under the directory to be processed, generate vector data for each data to be processed, and put the vector data into the database queue.

[0033] Specifically, the offline vector library construction process adopts a phased and decoupled pipeline design, in which the embedding queue serves as a buffer channel connecting the task generation stage and the vectorization processing stage (also known as the embedding processing stage). The directory to be processed can be understood as a data organization unit, such as a folder path containing multiple images or multiple text segments. Scheduling at the directory level helps maintain data locality and management consistency.

[0034] Specifically, the system initiates a vectorization processing task (i.e., the embedding phase). The execution thread or process in this phase pulls a directory of data to be processed from the embedding queue. For each piece of data to be processed in this directory (such as image files, text paragraphs, etc.), the system calls a pre-built artificial intelligence model to perform feature extraction and encoding, thereby obtaining vector data corresponding to each piece of data. It should be understood that vectorization processing refers to the process of transforming unstructured raw data (such as images and text) into high-dimensional dense vectors that computers can understand. For example, in multimodal scenarios, a pre-trained CLIP (Contrastive Language-Image Pre-training) model or a similar multimodal coding model can be loaded to map the input image data into a fixed-dimensional vector representation.

[0035] After generating vector data, the system does not write it directly to the database, but instead places it in the ingestion queue. Here, the ingestion queue acts as an intermediate buffer connecting the embedding processing stage and the database writing stage (i.e., the ingestion stage), decoupling the computationally intensive vectorization operations from the I / O (Input / Output) intensive database writing operations, so that they do not block each other.

[0036] Step 120: Obtain the vector data from the inbound queue and write the vector data into the vector database in batches.

[0037] Specifically, in this step, the execution unit of the ingestion processing stage (also known as the ingestion stage) reads the generated vector data from the ingestion queue. To improve writing efficiency and reduce network overhead and database transaction frequency, a batch writing method is adopted here, that is, accumulating a certain amount of vector data or packaging multiple vector data according to a preset time window, and calling the insertion interface of the vector database (such as Milvus, FAISS, etc.) at one time for persistent storage.

[0038] Step 130: Collect the current queue depth of each stage queue according to a preset period, and dynamically adjust the target parameters based on the deviation between the current queue depth and the preset target interval to maintain the current queue depth within the preset target interval. Each stage queue includes the embedded queue and the inbound queue. The target parameters include the parallelism and / or batch size of the processing stage corresponding to each stage queue.

[0039] Specifically, the system has a monitoring module that observes key indicators in the pipeline at a preset period (e.g., sampling once per second or every few seconds), such as collecting the current queue depth of each stage queue, so as to dynamically adjust the parallelism and batch size of the corresponding processing stage based on the queue deviation, thereby obtaining near-optimal steady-state throughput.

[0040] Here, the queues at each stage encompass the aforementioned embedding queue and inbound queue. The system can collect metrics such as the current queue depth, which is the number of pending tasks currently backlogged in the queue. For example, the number of directories waiting to be vectorized in the embedding queue, or the amount of vector data waiting to be written in the inbound queue. Queue depth reflects the degree of matching between upstream and downstream processing capabilities and the current load pressure.

[0041] The preset target range refers to an ideal range within which the system expects the queue depth to be maintained, such as 40% to 60% of the queue capacity. The purpose of setting this range is to prevent the queue from being too empty, which would cause downstream processing units to be idle and waste computing resources; on the other hand, it prevents the queue from being too full, which would cause upstream processes to be blocked or memory to overflow, leading to system instability.

[0042] Specifically, the monitoring module calculates the difference (deviation) between the currently observed queue depth and the preset target interval to determine whether the current processing capacity is insufficient or excessive. If the current queue depth is higher than the upper limit of the preset target interval, it indicates that the current processing speed is slower than the upstream data production speed, resulting in a backlog. In this case, the system will increase the target parameter. Conversely, if the queue depth is lower than the lower limit, it indicates that the processing speed is too fast or the upstream supply is insufficient. The system can appropriately decrease the target parameter to release resources.

[0043] Here, the target parameters can include the parallelism and / or batch size of the processing stages corresponding to each stage queue. Parallelism refers to the number of threads or processes executing the tasks of that stage simultaneously. For example, when the embedding queue is backlogged, the system automatically increases the number of threads in vectorized processing (i.e., the embedding processing stage) to utilize more resources to accelerate queue consumption. Batch size refers to the amount of data processed or written in a single operation. For example, when the database write queue is backlogged, the system can appropriately increase the batch size of the database write stage (i.e., the database write stage) to improve overall throughput.

[0044] Through this feedback adjustment mechanism based on queue depth deviation, the system can perceive the real-time load at each stage. For example, when the complexity of vectorized model calculations leads to accumulation in the embedding queue, the system will automatically increase the parallelism of that stage or increase the batch size of that stage; when database writes slow down, causing accumulation in the inbound queue, the system will dynamically adjust the write concurrency or batch size. This dynamic adjustment is not a one-time event, but a closed-loop control that continues throughout the acquisition cycle.

[0045] The offline vector library construction method provided in this embodiment achieves end-to-end adaptive flow control by introducing a phased queue buffering mechanism and a dynamic parameter adjustment strategy based on queue depth deviation. Specifically, by collecting the current queue depth of each stage's queue at a preset period and calculating its deviation from a preset target range, a closed-loop feedback adjustment mechanism for the offline library construction process is constructed. This mechanism can adaptively and dynamically adjust the parallelism and / or batch size of each processing stage according to the real-time load of each processing stage, thereby precisely controlling the data flow speed of upstream and downstream processes. This dynamic adjustment ensures that the queue depth of each processing stage is always maintained within the preset target range, effectively balancing the rate difference between vectorization processing and vector library writing. It avoids queue backlog and system overload caused by excessively fast processing in the preceding stages, and also prevents downstream resource idleness caused by excessively slow processing in the preceding stages. It eliminates the lag and uncertainty of manual parameter tuning in the traditional fixed parameter mode, significantly improving the execution stability and overall throughput efficiency of large-scale vector library construction tasks.

[0046] Based on the above embodiments, before step 110, the method further includes: Step 101: Perform a quality gating check on the directory to be processed.

[0047] It should be noted that in real-world offline vector database construction scenarios, data sources are often large and complex. Directly vectorizing all scanned data not only wastes computational resources but may also lead to subsequent database entry failures or poor retrieval results due to dirty data (such as missing metadata, corrupted files, or insufficient samples). Therefore, this embodiment of the invention introduces a pre-emptive quality gating and status recording mechanism.

[0048] Specifically, this step is typically performed by the task generation module. The system first scans the data root directory on the storage device, traversing to obtain each directory to be processed; for example, each directory corresponds to a specific object or image set. Before being placed into the processing queue, the directory needs to pass a quality gating check.

[0049] Furthermore, step 101 specifically includes: Check if a metadata file of a preset type exists in the directory to be processed, and count the number of valid data to be processed in the directory to be processed; If the metadata file exists and the number of valid files is not less than a preset threshold, the directory to be processed is determined to meet the quality gating condition.

[0050] Specifically, quality gating checks mainly include assessments across two key dimensions: metadata file checks and valid quantity statistics and threshold determination. Here, metadata file checks refer to the system checking whether a pre-defined type of metadata file exists in the directory to be processed. It should be understood that in multimodal RAG (Retrieval Enhanced Generation) scenarios, images alone are usually insufficient; textual information describing the image content, tags, or contextual data are also required. For example, the system will check whether a JSON-formatted description file exists in the directory. If such a file is missing, the generated vectors will lack semantic relevance and be considered invalid data; the system will directly skip the directory or mark it as missing metadata.

[0051] Furthermore, the system will further traverse the directory, counting the number of valid images (e.g., JPG, PNG) that meet the format requirements. This valid number is then compared to a preset threshold. The preset threshold can be set according to business needs and batch processing efficiency (e.g., a preset threshold of 50 images). If a directory contains only a few images (i.e., long-tail samples), starting a separate processing flow for it may lead to low batch processing efficiency. Only when the metadata file exists and the valid number is not less than the preset threshold, will the system determine that the directory meets the quality gate conditions and allow it to proceed to the next process.

[0052] Step 102: If the directory to be processed is found to meet the quality gating conditions, write each piece of data to be processed in the directory to be processed into the local record library in batches, initialize the task status of each piece of data to be processed in the local record library, and put the directory to be processed into the embedding queue.

[0053] Specifically, once the directory to be processed passes the gating check, the system does not immediately push it into the queue, but first performs a state initialization operation. The system writes each piece of data to be processed in that directory (such as the file path and unique identifier of each image) in batches to the local record database. Here, the local record database refers to a lightweight database (such as SQLite, RocksDB, or a custom key-value file structure) deployed locally on the processing node.

[0054] In the initial phase, the system uniformly marks the task status of this data as the initial state (init) to ensure task traceability. Once the local logbook successfully records this data, the system encapsulates the path or identifier of the directory to be processed into a task message and places it in the embedding queue, awaiting retrieval by the subsequent embedding processing module. This order of recording first and then enqueuing ensures that even if the system crashes after enqueuing, it can still find these incomplete data in the initial state by scanning the local logbook after restarting, thus providing a basis for resuming interrupted downloads.

[0055] The method provided in this invention filters out low-quality data with missing metadata or insufficient sample size by adding quality gating and local recording before vectorization processing, thus preventing dirty data from polluting the vector database and avoiding the waste of ineffective computing resources. Furthermore, by introducing a local record library and pre-initializing task states, a fine-grained (e.g., image-level) task state management mechanism is constructed. This not only provides data support for subsequent breakpoint recovery but also ensures the integrity of data processing in distributed or multi-threaded processing environments, preventing task omissions or duplication.

[0056] Based on any of the above embodiments, step 110, the vectorization processing of each piece of data to be processed in the directory to be processed, includes: Step 111: Call the embedded model service interface to perform feature encoding on each piece of data to be processed.

[0057] It should be noted that after the directory to be processed is placed into the embedding queue, the multiple worker threads of the embedding processing module can pull the directory from the embedding queue and call the embedding model service to generate the corresponding vector data for each piece of data to be processed (such as each image). During offline database construction, the embedding model service is typically a computationally intensive resource and can provide its capabilities as an independent service via the network (such as HTTP). However, faced with concurrent requests for massive amounts of data, the model service may experience momentary overload, timeouts, or network jitter, leading to encoding failures. To address this, this embodiment of the invention designs a robust retry strategy.

[0058] Specifically, during the embedding processing phase, the system's worker threads read the data to be processed (e.g., image files or text paragraphs) and encapsulate it into an input format supported by the model service (such as Base64-encoded image streams or JSON text). Subsequently, the worker threads call the embedding model service interface to send an inference request. This embedding model service interface can be a locally loaded model function or a remotely deployed high-performance inference cluster. The model's input is raw, unstructured data, and the output is a fixed-dimensional feature vector (e.g., a 512-dimensional or 1024-dimensional floating-point array).

[0059] Step 112: If any data to be processed fails to be encoded, determine the retry waiting time based on an exponential backoff strategy. The exponential backoff strategy is used to control the retry waiting time to increase exponentially with the number of retries.

[0060] Specifically, worker threads monitor the call results in real time. If an exception is caught, the system determines that the current data encoding has failed. To avoid further burdening the service by immediately retrying when the service is already overloaded, the system uses an exponential backoff strategy to calculate the waiting time before the next retry, i.e., the retry waiting time.

[0061] Specifically, let the base waiting time be... (For example, 100 milliseconds), if this is the nth retry (n≥1), then the retry waiting time for this time is... The calculation formula can be expressed as: ; Alternatively, random jitter can be added to the exponential growth to distribute the timing of concurrent retries. Following this strategy, as the number of retries increases, the waiting time will increase exponentially, thus giving the model service sufficient recovery time.

[0062] Step 113: After the retry waiting time has elapsed, encode and retry any data to be processed until encoding is successful or the preset maximum number of retry attempts is reached.

[0063] Specifically, the thread suspends the computed computation locally. After a set time, the same encoding request is initiated again. This process will loop until encoding is successful or the preset retry limit is reached. For example, if the server correctly returns vector data, it means the encoding was successful, and the system will exit the retry loop and proceed to the subsequent data insertion process; or, if the cumulative retries have reached 5, the system determines that the data cannot be processed in the current environment (possibly due to data corruption), stops retrying, marks the task status of the data as failed, and records an error log to avoid blocking subsequent data processing.

[0064] The method provided in this invention, by introducing an exponential backoff retry mechanism in the vectorization stage, can effectively cope with the instantaneous jitter and congestion of the model service. While improving the success rate of a single task, it protects the model service cluster from a global perspective, preventing server crashes caused by blind and frequent retries by the client, thereby improving the stability and throughput smoothness of the entire database building system.

[0065] Based on any of the above embodiments, step 120 specifically includes: Step 121: Obtain the file storage path of any data to be processed and the index number of the data to be processed under the file storage path; Step 122: Based on the file storage path and the index number, generate a deterministic primary key for any data to be processed; Step 123: Based on the deterministic primary key, write the vector data of any data to be processed into the vector database.

[0066] It should be noted that for all data to be processed in the directory, after generating the corresponding vectors, these vectors will be placed in the database ingestion queue for batch and parallel writing to the vector database. In traditional database write processes, the database's own auto-incrementing ID or random UUID (Universally Unique Identifier) ​​is typically used as the primary key. However, in large-scale scenarios such as offline vector database construction where frequent retries are possible, if the task is interrupted and restarted, the same data may be assigned a new ID, resulting in duplicate data in the database and making it difficult to establish a correspondence with the original file. To address this issue, this embodiment of the invention solves the problem by constructing a deterministic primary key.

[0067] Specifically, during the data ingestion phase, the system typically processes specific data (such as an image) located in a directory to be processed. The system first obtains the file storage path of this data (path, e.g., / data / dataset / Rose_001) and its index number within the current directory (e.g., index i for the i-th image, counting from 0). These two parameters, combined, uniquely locate a specific data fragment on physical storage.

[0068] Specifically, the system uses a hash algorithm to convert the file storage path and index number into a unique integer or string ID. The specific construction logic can be represented as follows: ; Wherein, to_int64 represents a function that truncates or reversibly maps a 64-bit hash to a signed / unsigned 64-bit space; hash can be MurmurHash, xxHash, or a fast hash algorithm with an engineering-proven acceptable collision rate; concat represents concatenation. This means concatenating the file path (path) and the image's index number (i) in that directory to calculate a 64-bit integer. The resulting integer ID is called a deterministic primary key.

[0069] Finally, when calling the vector database's insert interface, the system explicitly specifies the ID of the data as the deterministic primary key generated above, instead of letting the database generate it automatically. Therefore, as long as the source file (such as an image) and its location (i.e., the file storage path) remain unchanged, no matter how many times the system reruns, the calculated ID will always be consistent. New records will directly overwrite old records instead of being added, thus ensuring that the database does not expand or duplicate.

[0070] The method provided in this invention generates a deterministic primary key based on file attributes, ensuring data idempotency and consistency. Even if a breakpoint resumes or a full rerun occurs, newly written data will directly overwrite or be recognized as already existing, thus avoiding duplicate data issues in the database and facilitating subsequent direct querying of the original file's storage location by ID.

[0071] Based on any of the above embodiments, step 123 specifically includes: Step 1231: Perform a hash operation on the file storage path to obtain the group identifier of any data to be processed; Step 1232: Extract category information from the metadata of any data to be processed, and determine the target partition corresponding to any data to be processed in the vector database according to the preset category mapping relationship and the category information; Step S233: Based on the deterministic primary key, the group identifier, and the vector data of any data to be processed, construct a row record and write the row record into the target partition.

[0072] It should be noted that, in order to improve the efficiency of large-scale vector retrieval and the convenience of management, the data storage structure has been further optimized in this embodiment of the invention.

[0073] Specifically, in addition to the unique primary key for each image, the system also calculates a group identifier. This identifier is generated based on the path of the directory to be processed, and the specific generation logic can be represented as follows: ; Here, SHA256 refers to a 256-bit cryptographic hash algorithm. This means that all images in the same directory (e.g., the 50 rose images in the "Rose_001" directory), although they have different primary keys, share the same group identifier. This design facilitates subsequent cross-stage tracking and aggregation management.

[0074] In multimodal RAG scenarios, data typically includes category labels such as plants, animals, and buildings. The system reads the accompanying metadata and extracts the category information. Then, the system queries a pre-defined category partitioning mapping table. For example, the categories "Rose" and "Peony" both map to the Plant_Partition in the vector database. If category information is missing from the metadata, the system routes the data to a default partition. By determining the target partition, the system can physically isolate the data, avoiding the scanning of animal data when retrieving plant data, thus significantly reducing interference from inverted indexes and improving retrieval speed.

[0075] Next, the system assembles all the above information into a complete row record. This row record typically includes a primary key (i.e., id=phi(path,i)), a group identifier (i.e., uuid=SHA256(path)), vector data (denoted as embedding), and metadata (denoted as meta, such as path, timestamp, etc.). Finally, the system writes this row record in batches to the designated target partition.

[0076] The method provided in this invention introduces group identifiers, enabling the system to manage data at the directory level. For example, when all data in a directory needs to be deleted, it is not necessary to traverse every image in that directory; simply deleting all records with the same group identifier in the database is sufficient, greatly simplifying maintenance operations. Furthermore, by introducing category-based partitioned routing, the search space for vector retrieval is reduced, minimizing interference from irrelevant data.

[0077] Based on any of the above embodiments Figure 2 This is a schematic diagram of the state write-back process provided by the present invention, as follows: Figure 2 As shown, the method also includes: Step 210: During the process of batch writing the vector data of each data to be processed into the vector database, if the writing of the vector data of any data to be processed fails, a backoff retry will be performed within the maximum number of retries. Step 220: Obtain the write result corresponding to the vector data of each data to be processed, and put the write result into the write-back queue. The queues of each stage also include the write-back queue. Step 230: Update the task status of each piece of data to be processed in the local record library according to the write results in the write-back queue.

[0078] It should be noted that for each piece of data to be processed, after the corresponding row records are constructed, they can be routed to the corresponding vector database partition for storage according to category. However, in an offline database construction pipeline, the vector database usually runs as an independent service, and network fluctuations or sudden high service loads may cause write operations to fail. To ensure eventual data consistency and system throughput, this embodiment of the invention introduces a retry mechanism and a write-back queue.

[0079] Specifically, during the data insertion phase, the worker threads of the insertion module attempt to send a batch of vector data to the vector database server via an interface. The system has a built-in exception handling mechanism; if a write failure is detected (e.g., timeout, connection refused), the data will not be discarded immediately, but retry logic will be triggered.

[0080] To avoid crashes due to frequent retries under high server load, the system employs a backoff retry strategy. Before each retry, a specific time interval is waited, which typically increases with the number of retries. Only after the preset maximum number of retries (e.g., 3 or 5) has been reached, and the write operation still fails, will the system consider the batch or the data entry to have ultimately failed. It should be understood that, assuming a single write success rate of p, and r retries (approximately independent and identically distributed), the overall success probability is p_succ = 1 - (1 - p). r+1 Therefore, an optimal r can be selected to maximize the product of throughput and success rate.

[0081] Regardless of whether the write operation succeeds or ultimately fails, the worker thread retrieves the write result, including the data ID, status (such as success or failure), and possible error information. To avoid blocking the input thread's resources, the system does not directly operate on the local database (local databases like SQLite typically have weak concurrent write capabilities and can easily become a bottleneck) within the input thread. Instead, a third crucial queue, the write-back queue, is introduced. The input thread encapsulates the write result into a message and quickly delivers it to the write-back queue. This design achieves asynchronous decoupling between vector-based input and status write-back.

[0082] The system launches a dedicated log database module or background thread to consume messages from the write-back queue. Based on the message content, this module locates the corresponding record in the local log database and updates the task status. For example, if the vector of an image is successfully inserted into the database, its status is updated from "init" to "success"; if it fails after multiple retries, it is updated to "failed".

[0083] This invention improves the data writing success rate under network jitter or server-side instantaneous jitter by introducing a database retry mechanism, thereby enhancing the robustness of the system. The write-back queue design decouples high-concurrency vector database operations from low-concurrency local record database operations, preventing delays in record status from affecting the processing speed of the entire database construction pipeline and ensuring maximum system throughput.

[0084] Based on any of the above embodiments, the processing stages of the offline vector library building task include an embedding processing stage, a library entry stage, and a write-back stage. For the embedding processing stage, the target parameters include the parallelism of vectorized processing and / or the batch size; For the ingestion stage, the target parameters include the parallelism and / or batch size of writing to the vector database; For the write-back phase, the target parameters include the parallelism and / or batch size for updating the task status in the local record library.

[0085] Specifically, during the execution of the offline vector library construction task, the system continuously collects the queue depth of each processing stage at a fixed sampling period, so as to adaptively adjust the target parameters of the processing stage according to the queue deviation. Here, the target parameters are not abstract concepts, but specific configuration items designed for different computational characteristics of the links in the pipeline.

[0086] Specifically, for the embedding processing stage, the target parameters can include the parallelism of vectorized processing and / or the batch size. Since this stage typically calls the model service via HTTP requests, the parallelism corresponds to the number of concurrent threads that initiate the HTTP requests on the client side; the batch size corresponds to the number of images or text contained in a single HTTP request. Dynamically adjusting these two parameters can balance the load on the model service and prevent service timeouts or memory overflows due to excessive requests.

[0087] For the data insertion phase, target parameters can include the parallelism and / or batch size of the writes to the vector database. Parallelism corresponds to the number of client connections established with the vector database or the number of concurrent write threads; batch size corresponds to the number of data rows submitted in a single API call. Adjusting these two parameters primarily accommodates network bandwidth and the write queue limitations of the database server.

[0088] For the write-back phase, the target parameters can include the degree of parallelism and / or batch size for updating the task status in the local record store. Here, the degree of parallelism corresponds to the number of threads operating on the local record store; the batch size corresponds to the size of the database transaction.

[0089] This invention enables targeted and refined control by clearly defining the specific physical meaning of target parameters at different processing stages. Since the performance bottlenecks at different stages originate from different sources, by defining and adjusting their respective parallelism and batch size, the system can find the optimal parameter combination on heterogeneous hardware resources, thereby maximizing the overall processing potential of the server while ensuring the stable operation of each component.

[0090] Based on any of the above embodiments, step 130, which involves dynamically adjusting the target parameters based on the deviation between the current queue depth and the preset target interval, includes: Step 131: Calculate the difference between the current queue depth and the target queue depth at the current sampling time to obtain the observation error. The target queue depth is determined based on the preset target interval. Step 132: Calculate the proportional adjustment term based on the observation error, and sum the observation errors at historical sampling times to calculate the integral adjustment term; Step 133: The proportional adjustment term and the integral adjustment term are superimposed to calculate the initial target parameter at the next sampling time, and the initial target parameter is truncated to obtain the target parameter.

[0091] It should be noted that the embodiments of the present invention provide a detailed explanation of the specific implementation logic of dynamically adjusting parameters, namely the parameter adjustment process based on the proportional-integral (PI) control algorithm.

[0092] Specifically, for each processing stage k, the system first determines its target queue depth (denoted as k). ). Here, The target range can be determined based on the preset target range of the queue corresponding to this processing stage (denoted as...). This is determined by, for example, 40% to 60% of the queue capacity. Typically, the midpoint of this range (e.g., 50%) or a specific value within the range is set as [value missing]. .

[0093] Next, the system obtains the actual current queue depth (denoted as t) at the current sampling time t. ), and calculate the difference between its depth and the target queue depth, i.e., the observation error. This observation error The sign and magnitude of the sign directly reflect the current state of the system. For example, if This indicates queue backlog and insufficient processing capacity, requiring adjustment of target parameters, such as increasing parallelism or decreasing batch size; if This indicates that the queue is idle, but resources may be excessive. The target parameter can be reduced to save resources, such as decreasing the parallelism or increasing the batch size. This indicates that the system is in an ideal steady state.

[0094] Subsequently, the system calculates the two core components of the PI controller: the proportional control term and the integral control term. Here, the proportional control term is proportional to the current observation error, and its calculation formula can be expressed as follows: ; in, This is the proportional gain coefficient. The function of the proportional adjustment term is to quickly respond to changes in the current load. When the error suddenly increases, this term will quickly generate a large adjustment.

[0095] The integral adjustment term is proportional to the cumulative amount of historical error, and its calculation formula can be expressed as follows: in, This is the integral gain coefficient. This represents all sampling points from the start of control to the current time. The integral adjustment term is used to eliminate steady-state error. For example, if there is only a proportional adjustment term, the adjustment strength will weaken when the error decreases to a certain extent, causing the queue depth to never fully return to the target value; while the integral adjustment term will accumulate small errors over time, continuously increasing the adjustment strength until the error is completely eliminated.

[0096] The system superimposes the proportional and integral adjustment terms calculated above to obtain the theoretical initial target parameter value. In some implementations, a reference value is also superimposed, thus determining the initial target parameter at the next sampling time. It can be represented as follows: ; in, This represents the target parameter at the current sampling time (t), i.e., the parallelism or batch size at the current sampling time.

[0097] Since system resources (CPU, memory, number of database connections) are limited, the calculated theoretical parameter values... This may exceed the system's actual capacity or allowed configuration range. Therefore, clipping is required. ; in, This represents the target parameter obtained at time t+1, which can be the parallelism (such as the number of threads) or batch size that the system expects to set at the next time step. and These are the minimum and maximum allowed values ​​for this parameter, respectively.

[0098] Once the final target parameters are obtained, the system immediately applies them to the corresponding processing stage. For example, if the parallelism of the embedding processing stage is adjusted, the system will dynamically increase or decrease the number of worker threads; if the batch size of the data entry stage is adjusted, the system will update the batch write configuration.

[0099] By adjusting the parameters based on PI closed-loop control as described above, this embodiment of the invention can achieve precise control of queue depth. It utilizes the proportional term to achieve a rapid response to sudden traffic surges, and the integral term to ensure that the system can stably maintain the target load level, thus avoiding the overload-idle oscillation problem caused by simple threshold-triggered adjustment.

[0100] Based on any of the above embodiments, step 132 further includes: When calculating the integral adjustment term, if the target parameter reaches the preset parameter boundary value and the direction indication control quantity of the observation error continues to exceed the parameter boundary value, then the summation of the observation error at the historical sampling time is stopped or the integral adjustment term is reversed.

[0101] It should be noted that a phenomenon known as integral saturation may occur during PI control. For example, when the system load is extremely high, the calculated target parameter (such as the required number of threads) may reach the maximum value allowed by the system. (e.g., 32 threads), but the queue is still backed up (i.e., observation error). If no restrictions are placed on the integral term, the integral adjustment term... The error will continue to accumulate and expand due to its persistent existence.

[0102] When subsequent traffic decreases and queue backlog eases, the calculated target parameter will still remain at a certain value for a long time due to the excessively large accumulation of the integral term. The above issues prevent the system from reducing parallelism in a timely manner, leading to resource waste or subsequent overreaction. To address these problems, this embodiment of the invention adds an anti-integral saturation mechanism when calculating the integral adjustment term; that is, clip simultaneously implements boundary constraints and anti-integral saturation.

[0103] Specifically, before or during the integration and accumulation operation (i.e., summing historical errors), the system performs the following checks: First, it checks the target parameter status, that is, it determines whether the target parameter calculated in the previous moment or currently has reached the preset parameter boundary value (e.g., it has reached...). or Then, check the direction of the error, that is, determine the current observation error. Does the sign indicate that the control variable is attempting to exceed the boundary? For example, when the parameter has reached... and (Attempting to continue increasing parameters), or the number of parameters has been reached. and (Attempting to continue reducing the parameters).

[0104] If both of the above conditions are met, the system will take measures such as stopping the summation or reversing the calculation. Here, stopping the summation means temporarily halting the calculation of the current summation. The value of the integral term is added to the integral adjustment term, keeping it unchanged. Reverse correction refers to reducing the integral adjustment term in the opposite direction, causing it to exit the saturation region.

[0105] By introducing the aforementioned anti-integral saturation mechanism, this invention ensures that the controller does not accumulate ineffective over-regulation when the system is at its capacity limit (saturation state). Once the system load returns to the normal range, the controller can quickly exit the saturation state and sensitively revert to the target parameters, thereby improving the system's recovery speed and stability under extreme conditions.

[0106] Based on any of the above embodiments, in order to realize the PI adaptive dynamic adjustment mechanism, the system also needs to have a fine-grained observability module (i.e., a monitoring module). This embodiment of the invention quantifies the processing capacity of each stage of the system through periodic sampling and statistics. The specific processing flow is as follows: According to the preset period, the throughput and average processing latency of each processing stage at the current sampling time are collected, wherein the throughput and the average processing latency are used to evaluate the processing capability of each processing stage.

[0107] Specifically, a global monitoring module runs in the background of the system. This module accesses the context objects of each processing stage (embedding stage, database entry stage, etc.) in the pipeline at fixed intervals (e.g., every 5 seconds or every 10 seconds) and reads thread-safe statistical counters. During this process, in addition to collecting the current queue depth of the stage queues (such as the embedding queue, database entry queue, write-back queue, etc.), the system also collects throughput and average processing latency.

[0108] Here, for each processing stage, throughput represents the amount of data successfully processed per unit of time, reflecting the current productivity of that stage. Average processing latency represents the average time taken to process a unit amount of data in that processing stage, which can be calculated by maintaining a sliding window or a cumulative time counter. For example, if the ingestion stage wrote 1000 data entries in the past 10 seconds, with a total time of 2000 milliseconds, then the average processing latency is 2 milliseconds per entry.

[0109] Understandably, the collected throughput and average processing latency are not merely for display purposes, but also serve as the basis for evaluating the system's capabilities. For example, the controller compares the throughput of the embedding processing stage with that of the data entry stage. If the throughput of the embedding processing stage is significantly greater than that of the data entry stage, and the latency of the data entry stage is significantly higher, it indicates that the data entry stage is the current bottleneck and its processing capacity is insufficient. This evaluation data can be used as input signals to feed back to the PI controller, triggering dynamic adjustments to the parallelism or batch size.

[0110] The method provided in this invention, by establishing a periodic performance sampling mechanism, can not only help operation and maintenance personnel understand the system health status in real time, but also provide quantitative data support for automated scaling and flow control, enabling the system to optimize resource allocation based on actual processing capacity.

[0111] Based on any of the above embodiments, the method further includes: Obtain the remaining task volume and the average processing rate at the current moment for the offline vector database construction task, and determine the estimated remaining time for the offline vector database construction task based on the remaining task volume and the average processing rate. The instantaneous processing speed of each processing stage of the offline vector task is obtained at the current moment, and the processing stage with the lowest instantaneous processing speed is identified as the bottleneck stage at the current moment. The identifier of the bottleneck stage is used to trigger an alarm and / or assist in adjusting the target parameters.

[0112] It should be noted that in large-scale database construction scenarios (e.g., processing tens of millions of images), tasks often run for hours or even days. Operations personnel or users need an accurate progress indicator to schedule subsequent work or assess whether manual intervention (such as adding machines) is required. To address this, this invention provides an ETA calculation method based on the global average rate.

[0113] Specifically, the system first obtains the total number of offline vector database construction tasks, the number of tasks processed at the current moment, and the task's runtime. Here, the total number of tasks (denoted as T) refers to the total number of images or files that pass the gating check, calculated by the system during the quality gating and scanning phases, after traversing all directories to be processed at the initial task startup. The number of tasks processed (denoted as P) refers to the total number of records marked as "success" in the local database, read in real-time by the system, or the amount of data currently completed and added to the database, obtained through a global counter. The task's runtime (denoted as h) refers to the time difference between the current system time and the task startup time.

[0114] Then, based on the amount of tasks processed and the duration of task execution, the global average processing rate v_bar can be calculated using the formula v_bar = P / h. The global average processing rate reflects the overall average performance of the task from start to finish. It smooths out network jitter or short-term congestion in the middle and better represents the long-term processing trend of the system.

[0115] Next, the difference between the total task volume and the processed task volume is calculated to obtain the remaining task volume (denoted as R), i.e., R = T – P. Then, the ratio of the remaining task volume to the global average processing rate is calculated to obtain the estimated remaining time (ETA) for the offline vector database construction task. ETA = R / v_bar = (T - P) / (P / h); The system will output the ETA value to the log or monitoring dashboard in real time for easy viewing. It should be understood that, in order to prevent the v_bar from fluctuating drastically when h is too small at the beginning of the task, the ETA is usually output only after P>0 and h exceeds a certain threshold (such as 1 minute), or weighted smoothing is performed by combining the rate of the recent sliding window to provide a more user-friendly prediction experience.

[0116] This invention provides predictable progress management for long-cycle offline tasks by calculating the global average processing rate, enabling operations and maintenance personnel to determine whether a task can be completed on schedule based on the ETA, thereby improving the planning and controllability of task management.

[0117] Furthermore, in complex distributed pipeline systems, the overall throughput often depends on the slowest processing stage. To avoid blindly increasing resources in non-bottleneck stages, this embodiment of the invention also introduces real-time bottleneck detection logic to reduce manual intervention.

[0118] Specifically, the instantaneous processing speed of each processing stage at the current moment is recorded separately. The system monitoring module maintains a set of real-time speed indicators. For example, during the embedding phase, the number of vectors generated per second is recorded; during the database insertion phase, the number of vectors successfully written to the database per second is recorded; and during the write-back phase, the number of times the local record database is updated per second is recorded.

[0119] In each sampling period, the controller sorts and compares the above speed values ​​to find the stage corresponding to the minimum value. For example, if the system detects that the instantaneous processing speed is lowest during the inbound stage, it determines that the inbound stage is the bottleneck stage at the current moment.

[0120] Once the bottleneck is identified, the system can adopt a two-tiered response strategy: operational alerts and auxiliary parameter tuning. For example, the system can send alerts to operations personnel via SMS, email, or instant messaging tools, indicating that the data entry module may be blocked. Furthermore, the bottleneck identifier can also be used as an auxiliary signal input to the PID controller. For instance, when the controller attempts to increase the concurrency of the embedding processing stage, if it finds that the data entry stage is already a bottleneck, the controller will suppress the scaling operation on the embedding processing stage and instead prioritize adjusting the parameters of the data entry stage. This collaborative mechanism avoids overall imbalance caused by local optimization.

[0121] Based on any of the above embodiments Figure 3 This is a schematic diagram of the breakpoint recovery process provided by the present invention, as shown below. Figure 3 As shown, the method also includes: Step 310: After the offline vector library building task is abnormally interrupted and restarted, read the task status of each data to be processed in the local record library; Step 320: If the task status of any data to be processed is in the initial state or the failure state, put the processing directory to which the data to be processed belongs back into the embedding queue. Step 330: Obtain the pending directory from the embedded queue, skip the pending data in the pending directory whose task status is successful, and vectorize and store the pending data whose task status is initial or failed.

[0122] It should be noted that in large-scale data database construction tasks, the task often lasts for hours or even days, and machine crashes, power outages, or abnormal program exits are unavoidable. To address this, this embodiment of the invention utilizes the task status recorded in a local database to achieve breakpoint recovery capabilities accurate to the data item level.

[0123] Specifically, when the system restarts, it first enters recovery mode. The system scans and reads the local logbook. At this time, the logbook stores the last known state of all data before the interruption. These states may be success, failed, or init. Success indicates that the data has been successfully stored in the vector library and does not require further processing; failed indicates that the data processing failed previously and needs to be retried; and init indicates that the data has been scanned but has not yet been completely stored in the library.

[0124] The system queries the local record database to filter out directories containing incomplete data (status: init or failed). To maintain consistency in task scheduling, the system re-adds these directories to the embedding queue. It's important to note that although the entire directory object is added, the system explicitly knows which data within that directory is incomplete.

[0125] When the worker thread in the embedding phase retrieves the directory from the queue again, it does not blindly vectorize all data (such as images) in the directory. Instead, it checks the status in the local record library again. For data with a status of "success," the worker thread skips it directly, without performing model inference, consuming computing resources, or generating an ingestion request. For data with a status of "init" or "failed," the worker thread loads, vectorizes, and ingests it.

[0126] This mechanism ensures the idempotency of operations. For example, if a directory contains 100 images, and the 50 images were already in the database before an interruption, after a restart, the system only processes the remaining 50 images. Since the primary keys generated by vectors are usually deterministic, reprocessing will not result in duplicate data in the database; instead, it will either overwrite or add new data.

[0127] This invention, by combining the state of the local record library with the filtering logic after restart, achieves fine-grained breakpoint resume functionality. This not only reduces the recovery cost after a system crash and avoids repeated calculations and writes of successfully processed data, saving valuable computing power and time, but also ensures data consistency across batches. It solves the pain point of traditional solutions where a full rerun is required once an interruption occurs, making it particularly suitable for offline processing scenarios of ultra-large-scale datasets.

[0128] Based on any of the above embodiments Figure 4 This is a flowchart illustrating the recoverable offline vector library construction method for multimodal ultra-large-scale data provided by the present invention, as shown below. Figure 4 As shown, the system adopts a phased, decoupled pipeline architecture, combined with end-to-end monitoring and local state management, to achieve efficient and stable vector library construction. Specifically, it includes the following steps: Step S1, Data Scanning and Quality Gating: Traverse the object directory. Only if the target RAG metadata file exists in the directory and the number of images is not less than the threshold, the directory is determined to be a valid task.

[0129] Step S2, Production Enqueue: For directories that have passed the gate, the task status of all images to be processed in the local record library is initialized to init in batches, and the directory is encapsulated as a task object and put into the embedded queue, following the mechanism of triggering backpressure by the upper limit of the queue.

[0130] Step S3, Embedding Processing Stage: Worker threads pull directories from the embedding queue and, using a multi-threaded parallel processing mode, call remote or local embedding model services for each image in the directory to generate corresponding feature vectors. Failed samples are retried or skipped according to a strategy. The aggregated directory-level results are then placed into the database queue.

[0131] Step S4, Inbound Stage: Batch retrieve vector data from the inbound queue, construct row records containing primary keys, group identifiers, vectors, and metadata, and route them to the vector library partitions by category. Perform batch inbound. During this process, capture exceptions and back off retries within the retry limit.

[0132] Step S5, Write-back stage: Write back the record library according to the image-level results. If successful, update the status to "success"; otherwise, update it to "failed".

[0133] Step S6, Monitoring and Adaptation: During the entire process, the throughput, queue depth, and completion rate of each stage are collected in real time, and the ETA is calculated. When a certain stage becomes a bottleneck, its parallelism or batch size is adjusted through PI control to achieve adaptive back pressure.

[0134] Step S7, Breakpoint Recovery: If the process is interrupted, the system will restart and read the record database, automatically skip completed items, and only process incomplete items to ensure idempotency and consistency.

[0135] This invention improves the system's concurrent processing capabilities and resource utilization by breaking down the database construction process into four decoupled stages: scanning, embedding, data entry, and write-back, connected by three buffer queues. Specifically, PI-based parameter adjustment maintains the target queue depth under ultra-large data volumes and long-task scenarios, ensuring balanced throughput across stages, avoiding significant blocking and long tails, and achieving overall performance close to theoretical optimality. Deterministic primary keys, grouping, and record database alignment prevent duplicate data and drift during cross-batch repeated construction. The record database state machine enables rapid recovery from long-running tasks, reducing failure recovery time. Stage decoupling and backpressure based on the target queue depth stabilize the queue within a suitable working range, significantly reducing insertion failures and oscillations. RAG metadata and threshold gating suppress dirty data at the source, improving data reliability within the database. Furthermore, multidimensional counting and ETA estimation assist in operation and maintenance, bottleneck location, and reduce manual intervention.

[0136] The offline vector library building system provided by the present invention is described below. The offline vector library building system described below can be referred to in correspondence with the offline vector library building method described above.

[0137] Based on any of the above embodiments Figure 5 This is one of the structural diagrams of the offline vector library building system provided by the present invention, such as... Figure 5 As shown, the system includes: The embedding processing module 510 is used to obtain the directory to be processed from the embedding queue, perform vectorization processing on each data to be processed under the directory to be processed, generate vector data of each data to be processed, and put the vector data into the database queue. The inbound module 520 is used to retrieve the vector data from the inbound queue and write the vector data into the vector database in batches; The monitoring module 530 is used to collect the current queue depth of each stage queue according to a preset period, and dynamically adjust the target parameters based on the deviation between the current queue depth and the preset target interval, so as to maintain the current queue depth within the preset target interval. The stage queues include the embedded queue and the inbound queue, and the target parameters include the parallelism and / or batch size of the processing stage corresponding to each stage queue.

[0138] The system provided in this invention constructs a closed-loop feedback adjustment mechanism for the offline database building process by collecting the current queue depth of each stage queue according to a preset period and calculating its deviation from a preset target range. This mechanism can adaptively and dynamically adjust the parallelism and / or batch size of each processing stage based on the real-time load of each processing stage, thereby precisely controlling the data flow speed of upstream and downstream processes. This dynamic adjustment ensures that the queue depth of each processing stage is always maintained within the preset target range, effectively balancing the rate difference between vectorization processing and vector library writing. It avoids queue backlog and system overload caused by excessively fast processing in the preceding stages, and also prevents downstream resource idleness caused by excessively slow processing in the preceding stages. It eliminates the lag and uncertainty of manual parameter tuning in the traditional fixed parameter mode, significantly improving the execution stability and overall throughput efficiency of large-scale vector database building tasks.

[0139] Based on any of the above embodiments, the monitoring module 530 includes a parameter adjustment unit, which is specifically used for: Calculate the difference between the current queue depth and the target queue depth at the current sampling time to obtain the observation error. The target queue depth is determined based on the preset target interval. The proportional adjustment term is calculated based on the observation error, and the observation errors at historical sampling times are summed to calculate the integral adjustment term. The proportional adjustment term and the integral adjustment term are superimposed to calculate the initial target parameter at the next sampling time. The initial target parameter is then truncated to obtain the target parameter.

[0140] Based on any of the above embodiments, the parameter adjustment unit is further configured to: When calculating the integral adjustment term, if the target parameter reaches the preset parameter boundary value and the direction indication control quantity of the observation error continues to exceed the parameter boundary value, then the summation of the observation error at the historical sampling time is stopped or the integral adjustment term is reversed.

[0141] Based on any of the above embodiments Figure 6This is the second structural schematic diagram of the offline vector library building system provided by the present invention, as shown below. Figure 6 As shown, the system also includes a task generation module 500 and a record library module 540, wherein the task generation module 500 is used for: Perform quality gating checks on the catalog to be processed; If the directory to be processed is found to meet the quality gating conditions, the data to be processed under the directory to be processed is written to the local record library in batches, the task status of the data to be processed is initialized in the local record library, and the directory to be processed is placed into the embedding queue.

[0142] Based on any of the above embodiments, the task generation module 500 is specifically used for: Check if a metadata file of a preset type exists in the directory to be processed, and count the number of valid data to be processed in the directory to be processed; If the metadata file exists and the number of valid files is not less than a preset threshold, the directory to be processed is determined to meet the quality gating condition.

[0143] Based on any of the above embodiments, the data entry module 520 is used to perform a backoff retry within the maximum number of retries if it detects that the vector data of any data to be processed has failed to be written into the vector database in batches during the process of writing the vector data of each data to be processed into the vector database. The record library module 540 is used to obtain the write result corresponding to the vector data of each data to be processed, and put the write result into the write-back queue. The queues at each stage also include the write-back queue. Based on the write result in the write-back queue, the task status of each data to be processed in the local record library is updated.

[0144] Based on any of the above embodiments, the record library module 540 is further configured to read the task status of each data to be processed in the local record library after the offline vector library building task is abnormally interrupted and restarted. The task generation module 500 is also used to put the processing directory to which any data to be processed belongs back into the embedding queue when the task status of any data to be processed is in the initial state or the failure state. The embedding processing module 510 is further configured to obtain the processing directory from the embedding queue, skip the processing data in the processing directory whose task status is successful, and vectorize the processing data whose task status is initial or failed. The data entry module 520 is also used to perform data entry processing on pending data whose task status is in the initial state or the failure state.

[0145] Based on any of the above embodiments, the warehousing module 520 is specifically used for: Obtain the file storage path of any data to be processed and the index number of the data to be processed under the file storage path; Based on the file storage path and the index number, a deterministic primary key is generated for any data to be processed; Based on the deterministic primary key, the vector data of any data to be processed is written into the vector database.

[0146] Based on any of the above embodiments, the warehousing module 520 is specifically used for: Perform a hash operation on the file storage path to obtain the group identifier of any data to be processed; Extract category information from the metadata of any data to be processed, and determine the target partition corresponding to any data to be processed in the vector database according to the preset category mapping relationship and the category information; Based on the deterministic primary key, the group identifier, and the vector data of any data to be processed, a row record is constructed and written to the target partition.

[0147] Based on any of the above embodiments, the embedding processing module 510 is specifically used for: The embedded model service interface is invoked to perform feature encoding on each of the data to be processed; If any data to be processed fails to be encoded, the retry waiting time is determined based on an exponential backoff strategy. The exponential backoff strategy is used to control the retry waiting time to increase exponentially with the number of retries. After the retry waiting period, any data to be processed is re-encoded until the encoding is successful or the preset maximum number of retry attempts is reached.

[0148] Based on any of the above embodiments, the monitoring module 530 is further configured to: Obtain the remaining task volume and the average processing rate at the current moment for the offline vector database construction task, and determine the estimated remaining time for the offline vector database construction task based on the remaining task volume and the average processing rate. The instantaneous processing speed of each processing stage of the offline vector task is obtained at the current moment, and the processing stage with the lowest instantaneous processing speed is identified as the bottleneck stage at the current moment. The identifier of the bottleneck stage is used to trigger an alarm and / or assist in adjusting the target parameters.

[0149] Based on any of the above embodiments, each processing stage includes an embedding processing stage, a database entry stage, and a write-back stage; For the embedding processing stage, the target parameters include the parallelism of vectorized processing and / or the batch size; For the ingestion stage, the target parameters include the parallelism and / or batch size of writing to the vector database; For the write-back phase, the target parameters include the parallelism and / or batch size for updating the task status in the local record library.

[0150] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute an offline vector library construction method. This method includes: obtaining a directory to be processed from an embedding queue; performing vectorization processing on each data to be processed under the directory to be processed to generate vector data for each data to be processed; and placing the vector data into an inbound queue; obtaining the vector data from the inbound queue and batch writing the vector data into a vector database; collecting the current queue depth of each stage queue according to a preset period, and dynamically adjusting the target parameters based on the deviation between the current queue depth and a preset target interval to maintain the current queue depth within the preset target interval. Each stage queue includes the embedding queue and the inbound queue, and the target parameters include the parallelism and / or batch size of the processing stage corresponding to each stage queue.

[0151] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the offline vector library building method provided by the above methods. The method includes: obtaining a directory to be processed from an embedding queue; performing vectorization processing on each data to be processed under the directory to be processed to generate vector data for each data to be processed; and placing the vector data into an inbound queue; obtaining the vector data from the inbound queue and writing the vector data into a vector database in batches; collecting the current queue depth of each stage queue according to a preset period, and dynamically adjusting the target parameters based on the deviation between the current queue depth and a preset target interval to maintain the current queue depth within the preset target interval. Each stage queue includes the embedding queue and the inbound queue, and the target parameters include the parallelism and / or batch size of the processing stage corresponding to each stage queue.

[0153] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the offline vector library construction method provided by the above methods. The method includes: obtaining a directory to be processed from an embedding queue; performing vectorization processing on each data to be processed under the directory to be processed to generate vector data for each data to be processed; and placing the vector data into an inbound queue; obtaining the vector data from the inbound queue and batch writing the vector data into a vector database; collecting the current queue depth of each stage queue according to a preset period, and dynamically adjusting the target parameter based on the deviation between the current queue depth and a preset target interval to maintain the current queue depth within the preset target interval. The stage queues include the embedding queue and the inbound queue, and the target parameter includes the parallelism and / or batch size of the processing stage corresponding to each stage queue.

[0154] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An offline vector library construction method, characterized in that, include: The process retrieves the directory to be processed from the embedding queue, performs vectorization processing on each data to be processed under the directory to be processed, generates vector data for each data to be processed, and puts the vector data into the database queue. The vector data is retrieved from the inbound queue and written into the vector database in batches. The current queue depth of each stage queue is collected according to a preset period, and the target parameters are dynamically adjusted based on the deviation between the current queue depth and the preset target interval to maintain the current queue depth within the preset target interval. Each stage queue includes the embedded queue and the inbound queue, and the target parameters include the parallelism and / or batch size of the processing stage corresponding to each stage queue.

2. The offline vector library construction method according to claim 1, characterized in that, The step of dynamically adjusting the target parameters based on the deviation between the current queue depth and the preset target interval includes: The difference between the current queue depth and the target queue depth at the current sampling time is calculated to obtain the observation error, wherein the target queue depth is determined based on the preset target interval; The proportional adjustment term is calculated based on the observation error, and the observation errors at historical sampling times are summed to calculate the integral adjustment term. The proportional adjustment term and the integral adjustment term are superimposed to calculate the initial target parameter at the next sampling time. The initial target parameter is then truncated to obtain the target parameter.

3. The offline vector library construction method according to claim 2, characterized in that, Also includes: When calculating the integral adjustment term, if the target parameter reaches the preset parameter boundary value and the direction indication control quantity of the observation error continues to exceed the parameter boundary value, then the summation of the observation error at the historical sampling time is stopped or the integral adjustment term is reversed.

4. The offline vector library construction method according to claim 1, characterized in that, The step of retrieving the directory to be processed from the embedded queue also includes: Perform quality gating checks on the catalog to be processed; If the directory to be processed is found to meet the quality gating conditions, the data to be processed under the directory to be processed is written to the local record library in batches, the task status of the data to be processed is initialized in the local record library, and the directory to be processed is placed into the embedding queue.

5. The offline vector library construction method according to claim 4, characterized in that, The quality gating check of the directory to be processed includes: Check if a metadata file of a preset type exists in the directory to be processed, and count the number of valid data to be processed in the directory to be processed; If the metadata file exists and the number of valid files is not less than a preset threshold, the directory to be processed is determined to meet the quality gating condition.

6. The offline vector library construction method according to claim 4, characterized in that, Also includes: During the process of batch writing the vector data of each data to be processed into the vector database, if the writing of the vector data of any data to be processed fails, a backoff retry will be performed within the maximum number of retries. Obtain the write result corresponding to the vector data of each data to be processed, and put the write result into the write-back queue. The queues of each stage also include the write-back queue. Based on the write results in the write-back queue, the task status of each piece of data to be processed in the local record library is updated.

7. The offline vector library construction method according to claim 6, characterized in that, Also includes: After the offline vector library building task is abnormally interrupted and restarted, the task status of each data to be processed in the local record library is read. If the task status of any pending data is in the initial state or the failure state, the pending directory to which the pending data belongs is put back into the embedding queue. The pending directory is obtained from the embedded queue. Data pending in the pending directory with a task status of success is skipped, and data pending with a task status of initial or failure is vectorized and stored in the database.

8. The offline vector library construction method according to claim 1, characterized in that, The step of writing the vector data into the vector database in batches includes: Obtain the file storage path of any data to be processed and the index number of the data to be processed under the file storage path; Based on the file storage path and the index number, a deterministic primary key is generated for any data to be processed; Based on the deterministic primary key, the vector data of any data to be processed is written into the vector database.

9. The offline vector library construction method according to claim 8, characterized in that, The step of writing the vector data of any data to be processed into the vector database based on the deterministic primary key includes: Perform a hash operation on the file storage path to obtain the group identifier of any data to be processed; Extract category information from the metadata of any data to be processed, and determine the target partition corresponding to any data to be processed in the vector database according to the preset category mapping relationship and the category information; Based on the deterministic primary key, the group identifier, and the vector data of any data to be processed, a row record is constructed and written to the target partition.

10. The offline vector library construction method according to any one of claims 1 to 9, characterized in that, The vectorization process for each piece of data to be processed in the directory to be processed includes: The embedded model service interface is invoked to perform feature encoding on each of the data to be processed; If any data to be processed fails to be encoded, the retry waiting time is determined based on an exponential backoff strategy. The exponential backoff strategy is used to control the retry waiting time to increase exponentially with the number of retries. After the retry waiting period, any data to be processed is re-encoded until the encoding is successful or the preset maximum number of retry attempts is reached.

11. The offline vector library construction method according to any one of claims 1 to 9, characterized in that, Also includes: Obtain the remaining task volume and the average processing rate at the current moment for the offline vector database construction task, and determine the estimated remaining time for the offline vector database construction task based on the remaining task volume and the average processing rate. The instantaneous processing speed of each processing stage of the offline vector task is obtained at the current moment, and the processing stage with the lowest instantaneous processing speed is identified as the bottleneck stage at the current moment. The identifier of the bottleneck stage is used to trigger an alarm and / or assist in adjusting the target parameters.

12. The offline vector library construction method according to claim 11, characterized in that, Each processing stage includes an embedding processing stage, a database entry stage, and a write-back stage; For the embedding processing stage, the target parameters include the parallelism of vectorized processing and / or the batch size; For the ingestion stage, the target parameters include the parallelism and / or batch size of writing to the vector database; For the write-back phase, the target parameters include the parallelism and / or batch size for updating the task status in the local record library.

13. An offline vector library building system, characterized in that, include: The embedding processing module is used to obtain the directory to be processed from the embedding queue, perform vectorization processing on each data to be processed under the directory to be processed, generate vector data of each data to be processed, and put the vector data into the database queue. The inbound module is used to retrieve the vector data from the inbound queue and write the vector data into the vector database in batches; The monitoring module is used to collect the current queue depth of each stage queue according to a preset period, and dynamically adjust the target parameters based on the deviation between the current queue depth and the preset target interval, so as to maintain the current queue depth within the preset target interval. The stage queues include the embedded queue and the inbound queue, and the target parameters include the parallelism and / or batch size of the processing stage corresponding to each stage queue.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the offline vector library construction method as described in any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the offline vector library construction method as described in any one of claims 1 to 12.

Images