Vector migration method and apparatus

Abstract

The present disclosure provides a vector migration method and device, the method comprising: in response to detecting a vector model replacement event, querying, from an original vector set associated with a vector migration task, a first text slice corresponding to each to-be-migrated document associated with the vector migration task; calling a second vector model to vectorize the first text slice to obtain a second vectorized representation; generating a first target vector record according to the second vectorized representation, the text content of the first text slice, and the corresponding metadata, and writing the first target vector record to an intermediate vector set that has been created; in response to successful writing, deleting the original vector set and creating a target vector set to migrate all target vector records in the intermediate vector set to the target vector set, thereby avoiding the semantic space inconsistency problem caused by mixing new and old vectors, ensuring that the entire knowledge base maintains a unified vector representation and retrieval behavior after model switching, and effectively ensuring service stability.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a vector transfer method and apparatus. Background Technology

[0002] In Retrieval Augmentation (RAG)-based systems, vector databases store text slice vectors encoded by vector models for semantic similarity retrieval. However, vector models are highly coupled with their semantic space, output dimensions, distance metrics (such as cosine similarity), and index structure. When a model is replaced for domain adaptation or performance optimization—for example, switching from a general model to a vertical domain model such as healthcare or law, or upgrading from a low-dimensional model to a high-dimensional model—the old and new vectors are usually no longer in the same semantic space. Mixing them can lead to inaccurate similarity calculations, causing issues such as missed recall, false recall, ranking anomalies, and retrieval instability. Therefore, a reliable, efficient, and application-insensitive vector migration mechanism is urgently needed when replacing models to ensure data consistency and service continuity. Summary of the Invention

[0003] This disclosure provides a vector transfer method and apparatus to at least partially solve one of the technical problems in related technologies. The technical solution of this disclosure is as follows: According to a first aspect of the present disclosure, a vector migration method is provided, comprising: in response to detecting a vector model change event, querying a first text slice corresponding to each document to be migrated associated with the vector migration task from an original vector set associated with the vector migration task; wherein the original vector set stores multiple text slices and corresponding first vectorized representations and metadata; the first vectorized representation is obtained by vectorizing the text slices using a first vector model; calling a second vector model to vectorize the first text slices to obtain a second vectorized representation of the first text slices; wherein the positive evaluation index of the second vector model is higher than that of the first vector model; generating a first target vector record based on the second vectorized representation, the text content of the first text slices and the corresponding metadata, and writing the first target vector record into an existing intermediate vector set; in response to the successful writing of the first target vector record, deleting the original vector set and creating a target vector set to migrate all target vector records in the intermediate vector set to the target vector set.

[0004] According to a second aspect of the present disclosure, a vector migration apparatus is provided, comprising: a query module, configured to, in response to detecting a vector model change event, query a first text slice corresponding to each document to be migrated associated with the vector migration task from an original vector set associated with the vector migration task; wherein the original vector set stores multiple text slices and corresponding first vectorized representations and metadata; the first vectorized representation is obtained by vectorizing the text slices using a first vector model; a calling module, configured to call a second vector model to vectorize the first text slices to obtain a second vectorized representation of the first text slices; wherein the positive evaluation index of the second vector model is higher than that of the first vector model; a generation module, configured to generate a first target vector record based on the second vectorized representation, the text content of the first text slices and the corresponding metadata, and write the first target vector record into an existing intermediate vector set; and a migration module, configured to, in response to the successful writing of the first target vector record, delete the original vector set and create a target vector set to migrate all target vector records in the intermediate vector set to the target vector set.

[0005] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the vector migration method as described in the first aspect of the present disclosure.

[0006] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the vector migration method as described in the first aspect of the present disclosure.

[0007] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the vector migration method as described in the first aspect of the present disclosure.

[0008] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: In this technical solution, upon detecting a vector model change event, the system does not directly overwrite or update the original vector set in place. Instead, it first re-vectorizes the text slices corresponding to the document to be migrated using a higher-performance second vector model, generating a second vectorized representation with higher semantic consistency. This second vectorized representation, along with the original text content and metadata, is then written into an independent intermediate vector set. This avoids the semantic space inconsistency problem caused by mixing old and new vectors. Furthermore, only after confirming that the first target vector record has been successfully written into the intermediate vector set is the original vector set deleted and a structurally compatible target vector set created. This completely migrates the intermediate data to the target set, effectively isolating the migration process. The system mitigates data risks during the migration process, ensuring recovery through intermediate or original vector sets even in the event of interruption or failure, significantly improving migration reliability and fault tolerance. Simultaneously, upper-layer RAG applications can connect to new model data without configuration modifications, achieving seamless service switching during model upgrades and guaranteeing retrieval accuracy and system continuity. Specifically, upon detecting a vector model change event, the system creates a migration task and records its creation time. Using this time as a boundary, text slices from the original set with an entry time no later than this moment are selected as historical stock data, clearly distinguishing between incremental and stock data and reducing the risk of data contamination or state inconsistencies. Furthermore, through a time window defined by the first creation time and the target set creation time, the system accurately identifies documents added during the migration but not processed by the new model, calls the second vector model to re-vectorize them, generates semantically consistent vectors, and writes them into the target set. This mechanism ensures the target set fully covers historical data and new data added within the window period, effectively avoiding semantic fragmentation and retrieval inconsistencies caused by mixing old and new vectors, significantly improving the completeness, consistency, and stability of online services during migration.

[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0011] Figure 1 This is a flowchart illustrating the vector transfer method shown in the first embodiment of this disclosure; Figure 2 This is a schematic flowchart of the vector transfer method shown in the second embodiment of this disclosure; Figure 3 This is a flowchart illustrating the vector transfer method shown in the third embodiment of this disclosure; Figure 4 This is a schematic diagram illustrating the principle of the vector transfer method shown in the embodiments of this disclosure; Figure 5 This is a schematic diagram of the structure of the vector transfer device shown in the fourth embodiment of this disclosure; Figure 6 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of the present disclosure. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0013] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0014] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0015] In the current RAG system, vector model replacement is typically achieved through a "new collection migration" approach. Specifically, the system creates a new collection in the vector database to store vector data generated by the new vector model. Then, the configuration of the file parsing service, vector generation service, or database insertion service is adjusted so that vectors generated by subsequently added, updated, or re-parsed files are written to this new collection. Simultaneously, batch processing tasks re-vectorize existing text slices in the old collection using the new vector model, and the reconstructed data is gradually migrated to the new collection. Once the data migration to the new collection is complete and passes verification, the system switches the query entry point of the retrieval service to the new collection, and finally deletes, archives, or takes the old collection offline. This approach achieves a certain degree of physical isolation between the old and new vector data, reducing the risk of semantic inconsistencies caused by mixing vectors from different models. However, the above-mentioned solutions still have several obvious shortcomings: First, it is highly intrusive to upper-level business systems and has high collaboration costs. This approach typically requires modifying or reconfiguring multiple backend modules (such as file parsing, vector insertion, and retrieval routing) to ensure that new data is written to the new set and query requests are directed to the new set. For RAG systems already in operation, if file parsing, incremental updates, vector writing, and retrieval services are handled independently by different modules or task queues, set switching must ensure that the configurations of each module are updated synchronously. If the configurations are not synchronized, some services may still be writing to the old set while others have switched to the new set, leading to inconsistent data states and even causing different slices of the same document to use different models to generate vectors. Second, the migration process has poor fault tolerance and weak ability to recover from anomalies. Existing solutions mostly rely on offline scripts or batch tasks to complete the migration, involving multiple stages such as reading the original text, calling the new model, batch writing, index building, query switching, and old set deletion. An error in any stage (such as model call failure, network interruption, write exception, index building failure, or task abort) can lead to incomplete new set data. If the retrieval entry point has already been switched, some knowledge will be unrecoverable; repeating the migration task may cause duplicate writes, primary key conflicts, metadata mismatches, or data overwriting. Furthermore, most solutions lack robust mechanisms for resuming interrupted downloads, retrying on failure, and rolling back, making it difficult to accurately track the migration status of each knowledge base, file, or slice, resulting in unrecoverable and non-idempotent tasks. Third, it is difficult to effectively coordinate dynamic data entry tasks. Vector data in a RAG system is typically generated dynamically—users continuously upload, update, or re-parse files, with related tasks executed asynchronously. If old tasks are still being processed during the migration process, while the system has initiated set switching or deleted old sets, the following may occur: some slices are written based on the old model, while other slices remain unprocessed; or subsequent tasks continue to write data to deprecated sets. This not only leads to inconsistent slice model versions within the same document but may also result in residual old model vectors, compromising overall semantic consistency. Fourth, deleting from old collections carries high risks. Deletion operations in vector databases directly impact search results and index status. Incorrect settings for deletion conditions, set names, knowledge base identifiers, primary key ranges, or batch boundaries can easily lead to the accidental deletion of non-target data. This risk is particularly pronounced in scenarios involving multiple knowledge bases, multiple tenants, or multiple services sharing the same vector database. Furthermore, the sheer volume of vector data means that accidental deletion often necessitates re-executing the entire parsing-slicing-vectorization-indexing process, resulting in high recovery costs and potential disruption to online service stability. Fifth, the migration result verification mechanism is inadequate. Successful vector writing does not equate to stable data retrieval. Vector databases typically require additional background operations such as index building, data loading, logical deletion activation, cache refresh, and query visibility synchronization. Without comprehensive validation of data before and after migration, including document count, text slice count, vector dimensions, model version, primary key consistency, metadata integrity, and sampling recall effectiveness, it is difficult to promptly detect issues such as missing data, duplication, residual old vectors, or inconsistencies between the old and new sets.

[0016] To address any of the above problems, this disclosure proposes a vector transfer method and apparatus.

[0017] The vector migration method and apparatus of this disclosure are described below with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart illustrating the vector transfer method shown in the first embodiment of this disclosure.

[0019] It should be noted that the embodiments disclosed herein are illustrated by the example of a vector migration method configured in a vector migration device, which can be applied to an electronic device to enable the electronic device to perform vector migration functions.

[0020] like Figure 1 As shown, this vector transfer method includes: Step 110: In response to the detection of a vector model change event, query the first text slice corresponding to each document to be migrated associated with the vector migration task from the original vector set associated with the vector migration task.

[0021] The original vector set stores multiple text slices and their corresponding first vectorized representations and metadata; the first vectorized representation is obtained by vectorizing the text slices using a first vector model.

[0022] As one possible implementation, when the system detects that the vector model needs to be replaced (e.g., due to model upgrades, domain adaptation, or performance optimization), or receives a vector model replacement instruction from the administrator, it triggers a vector model replacement event and creates a vector migration task to centrally manage the data migration work involved in this model switch. Subsequently, the system queries the original vector set currently in use in the production environment for the first text slices corresponding to all documents to be migrated that are associated with this migration task. These text slices of documents to be migrated were written to the original vector set earlier than or equal to the creation time of the vector migration task, thus ensuring that the migration scope covers all historical data already stored before the task started, avoiding omissions.

[0023] It should be noted that the original vector set not only stores the vector data generated by the first vector model, but also retains the text slices on which these vectors depend and their associated metadata. The metadata may include, but is not limited to, contextual attributes such as document identifier, the position information of the document slice in the original document, creation time, source path, business tags, highlight information, and the identifier of the knowledge base to which it belongs.

[0024] In some embodiments, such as Figure 2 As shown, step 110 may include the following steps: Step 1101: In response to the detection of a vector model change event, a vector migration task is created, and the first creation time of the vector migration task is recorded.

[0025] When the system detects the need to change the currently used vector model, it generates a vector model change event. This could be achieved through automatic monitoring to identify the deployment of a new model, or by receiving a manual replacement command from an administrator. In response to this event, the system creates a vector migration task to manage the data reconstruction and migration work involved in this model switch. Simultaneously, the system records the creation time of this vector migration task and marks it as the first creation time.

[0026] Step 1102: Based on the first creation time, select the first text slice from multiple text slices in the original vector set.

[0027] The time when the first text slice is written into the original vector set is earlier than or equal to the first creation time.

[0028] To accurately define the data scope of this migration, as a possible approach, the first creation time is used as the baseline time point. All text slices stored in the original vector set are traversed, and each slice is filtered based on its entry time. This operation ensures that only text slices that were successfully written to the original vector set before or at the exact moment of creation of the vector migration task are processed, excluding dynamic data added after the task starts or still being processed asynchronously. This ensures that the main migration phase processes stable and complete existing data. The excluded incremental data will be uniformly identified and reprocessed later, thereby ensuring migration consistency while achieving complete coverage of all data.

[0029] In summary, when a vector model change event is detected, the system creates a vector migration task and records the task creation time. Then, using this time as a boundary, it selects all text slices from the original vector set whose entry time is earlier than or equal to that time as migration objects. This mechanism clearly distinguishes between historical existing data and incremental data generated after the task starts, effectively reducing the risk of data mixing, repeated migration, or state disorder, making the entire migration process more accurate and complete.

[0030] To improve the integrity of the migration data, after step 1102, as follows: Figure 3 As shown, it may also include the following steps: Step 1103: Based on the first creation time and the second creation time of the target vector set, determine the incremental document whose slice parsing start time is located between the first creation time and the second creation time.

[0031] It is important to understand that during the time window between the start of the migration task and the creation of the target vector set, the RAG system may still be operating normally, continuously receiving user uploads, file updates, or re-parsing requests, and generating new text slices. The vectorization processing of these text slices generated during the window period may still be performed by the old model. If they are not identified and supplemented, they may end up in a different semantic space from other migrated data in subsequent searches, leading to problems such as inconsistent recall, abnormal ranking, or decreased search quality.

[0032] Therefore, precise time boundaries are defined based on the creation time of the migration task and the creation time of the target set, accurately identifying documents that have been parsed but not yet processed by the new model within this window period and marking them as incremental documents.

[0033] Step 1104: Call the second vector model to vectorize the second text slice of the incremental document to obtain the third vectorized representation of the second text slice.

[0034] To ensure consistency in semantic space and vector representation between incremental documents and migrated data, the system invokes a second vector model to re-vectorize the second text slices corresponding to the incremental documents. It's important to note that these second text slices are generated within a time window after the vector migration task begins but before the target vector set is created. Their original vectors may have been generated by the old first vector model or have not yet been processed by the new model. Directly retaining the original vectors would lead to inconsistencies with the newly generated vectors in the target set in terms of dimension, semantic distribution, or similarity measurement, thus affecting retrieval performance. Therefore, by using the currently effective and higher-performing second vector model to re-encode these slices, a third vectorized representation with a unified format and semantic space is generated. This ensures that all data is generated based on the same vector model before being written into the target vector set, thereby guaranteeing the semantic consistency and retrieval reliability of the entire vector database.

[0035] Step 1105: Generate a second target vector record based on the third vectorized representation, the text content of the second text slice and the corresponding metadata, and write the second target vector record into the target vector set.

[0036] Next, the newly generated third vectorized representation, the original text content of the corresponding second text slice, and the complete metadata associated with the slice are structurally assembled to form a complete second target vector record. This record maintains consistency with the generated first target vector record in terms of data structure, ensuring that the data format of the entire target vector set is uniform and semantically complete. Subsequently, this record is directly written into the created and usable target vector set. This operation ensures that the target vector set ultimately contains complete, consistent, and high-quality vector data, thereby achieving a complete migration without omissions or misuse, and providing a stable and reliable semantic retrieval foundation for the RAG system.

[0037] To ensure the integrity, consistency, and availability of the vector migration results, after the second target vector record is successfully written, the system performs multi-dimensional data verification on the target vector set. If the verification passes, the intermediate set is cleaned up and the migration is marked as complete; if the verification fails, the intermediate set is retained and the task is marked as abnormal for later processing.

[0038] As one possible implementation, in response to the successful writing of the second target vector record, data verification is performed on the target vector set; wherein, the data verification includes at least one of the following: document quantity verification, text slice quantity verification, vector dimension verification, vector model version verification, index status verification, and sampling retrieval verification; in response to the data verification passing, the intermediate vector set is deleted, and the status of the vector migration task is updated to migration completed; in response to the data verification failing, the intermediate vector set is retained, and the status of the vector migration task is updated to verification failed or pending manual processing.

[0039] In other words, after the second target vector record is successfully written into the target vector set, the system will immediately initiate a comprehensive data verification process to verify the correctness and availability of the migration results. This data verification can cover multiple key dimensions, including but not limited to: whether the number of documents in the target set matches the expected value, whether the total number of text slices is complete, whether the dimensions of all vectors conform to the new model specifications, whether the model version identifier associated with the vectors is unified as the second vector model, whether the underlying index has been successfully built and is in a queryable state, and whether the semantic recall effect meets expectations through sampling retrieval tests. If all the above verifications pass, it indicates that the migration data is complete, the structure is compliant, and the service is available. The system will safely delete the intermediate vector set used for the transition and update the status of the vector migration task to "migration completed," marking the successful completion of the entire model switching process. If any verification item fails, it indicates that there is an anomaly or incompleteness in the migration results. In this case, the system will proactively retain the intermediate vector set as a basis for recovery and update the task status to "verification failed" or "pending manual processing" so that operations and maintenance personnel can intervene to investigate the problem and avoid erroneous data from affecting the retrieval quality and stability of the RAG system.

[0040] In summary, by defining a precise time window based on the first creation time of the vector migration task and the second creation time of the target vector set, the system can accurately identify incremental documents newly generated during the migration process but not yet processed by the new model. Then, it calls the second vector model to re-vectorize the second text slices, generating a semantically consistent third vectorized representation. This is combined with the original text and metadata to construct a second target vector record, which is then written into the target vector set. This not only ensures that the target vector set contains all the data, covering both historical stock and new content added during the migration window, but also fundamentally eliminates the risks of semantic space fragmentation, inconsistent search results, or abnormal sorting caused by mixing old and new vectors. This significantly improves the completeness, consistency, and stability of online services in vector migration.

[0041] Step 120: Call the second vector model to vectorize the first text slice to obtain the second vectorized representation of the first text slice.

[0042] Among them, the positive evaluation index of the second vector model is higher than that of the first vector model.

[0043] Furthermore, the first text slice extracted from the original vector set is re-vectorized using the second vector model to generate a new vector representation, namely the second vectorized representation. The second vector model outperforms the original first vector model in all positive evaluation metrics, including but not limited to accuracy and precision. The second vector model can be an improved model obtained by updating the first vector model, or it can be a completely new vector model with better performance or more suitable for a specific domain.

[0044] Step 130: Generate a first target vector record based on the second vectorized representation, the text content of the first text slice and the corresponding metadata, and write the first target vector record into the created intermediate vector set.

[0045] Furthermore, after re-vectorizing the original text slice, the system structurally integrates the second vectorized representation, the original content of the text slice, and its associated metadata to generate a complete first target vector record. This first target vector record not only contains high-quality vectors generated by the new model but also retains readable original text and rich contextual information (such as document identifier, slice location, knowledge base affiliation, creation time, source path, business tags, and highlighted information), providing sufficient data sources for subsequent operations such as retrieval, debugging, auditing, or visualization. Then, the first target vector record is written into a set of intermediate vectors that has already been created. It should be noted that this intermediate vector set serves as temporary storage during the migration process and is physically isolated from the original vector set in the production environment. This avoids interference with online services during the migration operation and provides a secure and controllable data foundation for subsequent data verification, failure retries, exception rollbacks, or incremental supplementation.

[0046] Furthermore, if the creation of the intermediate vector set fails due to insufficient permissions, insufficient storage resources, abnormal database connection, or configuration errors, the system will immediately record the specific reasons for the failure, such as error codes, exception information, or operation logs, and proactively terminate the current vector migration task, ceasing subsequent data processing steps. During the termination process, the system will ensure that the original vector set remains intact and has not been deleted or modified, maintaining the original data state and retrieval capabilities of the RAG service unaffected. This effectively prevents data loss or service interruption that may occur when the basic environment is not met, ensuring the stability and reliability of the system.

[0047] Step 140: In response to the successful writing of the first target vector record, delete the original vector set and create a target vector set to migrate all target vector records in the intermediate vector set to the target vector set.

[0048] Finally, after confirming that the first target vector record has been successfully written to the intermediate vector set, the old model vectors stored in the original vector set are no longer needed. The original vector set is then deleted to completely eliminate the risk of mixing old and new vectors and to release existing storage resources. Subsequently, the system creates a new target vector set that maintains consistency with the original set in key attributes such as set name, field structure, index type, and vector dimensions. In particular, the same set name can be reused, ensuring seamless integration of the upper-layer retrieval service without requiring configuration modifications. Next, the system migrates all verified and complete target vector records from the intermediate vector set to this new target vector set in one go, and completes the necessary index building or data loading operations, enabling the target vector set to have online retrieval capabilities. The entire model upgrade process is completely transparent to users and business systems, ensuring data consistency and service continuity while improving the user experience.

[0049] To ensure transparency and data consistency for upper-layer applications during vector model switching, the target vector set is designed to align with existing interface specifications and new model capabilities. The field structure of the target vector set is consistent with that of the intermediate vector set, ensuring that all text content, metadata, and vector fields can be migrated completely and without loss. The name of the target vector set reuses the name of the original vector set, allowing the retrieval service to seamlessly point to the new data without modifying the configuration. At the same time, the dimensions of the vector fields in the target vector set precisely match the output dimensions of the second vector model, ensuring the effectiveness of semantic representation and the correctness of index construction.

[0050] To achieve fine-grained status management of the vector migration process, after the first target vector record of each document to be migrated is successfully written, the system can update the migration status of each document to be migrated to "migration completed". If any document fails to be written, the migration status of the corresponding document to be migrated will be updated to "migration failed and awaiting retry", and the corresponding reason for the writing failure will be recorded.

[0051] To achieve fault tolerance and automatic recovery capabilities for vector migration tasks, the system automatically retryes vectorization and writing operations when the document to be migrated is in the state of migration failure pending retry and the set period has been reached. If the cumulative number of retries reaches the upper limit and still fails, the entire migration task is determined to have failed, the intermediate vector set is deleted, and the task status is updated to migration failure.

[0052] As an example, in response to the migration status of any document to be migrated being updated to migration failed and pending retry, and the set period being reached, the vectorization processing of the first text slice of the corresponding document to be migrated is re-executed, and the newly generated first target vector record is written into the intermediate vector set; if the cumulative number of retries of the corresponding document to be migrated reaches the maximum number of retries and the writing still fails, then the vector migration task is determined to have failed, the intermediate vector set is deleted, and the status of the vector migration task is updated to migration failed.

[0053] In other words, when a document to be migrated fails to write its corresponding first target vector record to the intermediate vector set due to network fluctuations, vector model service anomalies, database write conflicts, or insufficient resources, the system updates the migration status of the document to "migration failed, pending retry" and records the specific reason for the failure. Subsequently, the system does not immediately terminate the entire migration task but introduces a delayed retry strategy. After a preset retry period (e.g., 5 minutes, 30 minutes), it automatically calls the second vector model again to vectorize the first text slice corresponding to the document and attempts to write the newly generated first target vector record back to the intermediate vector set. This process can be repeated, but is limited to a maximum number of retries (e.g., 3 or 5). If the document still cannot be successfully written after reaching the maximum number of retries, the system determines that the entire vector migration task has failed, indicating a serious, unrecoverable problem. Continuing to proceed may lead to data inconsistency. At this point, to avoid incomplete or erroneous data affecting subsequent operations, the system proactively deletes the created intermediate vector set and explicitly updates the status of the entire vector migration task to migration failure. Simultaneously, it can trigger an alarm to notify maintenance personnel to intervene and investigate. This design not only ensures self-healing capability under temporary failures but also prevents invalid data from remaining for a long time, ensuring clear and controllable system status.

[0054] To ensure disaster recovery and reproducibility when the creation of the target set fails in a vector migration task, the system supports automatic rollback and reconstruction of the original set and data recovery, or retention of the intermediate set and manual intervention by operations and maintenance personnel.

[0055] As an example, in response to the failure to create the target vector set after deleting the original vector set, a vector set with the same data structure as the original vector set is recreated based on the metadata of the original vector set, and the data in the intermediate vector set or the pre-stored backup is restored to the recreated vector set; In other words, since the original vector set has been deleted, if the new vector set fails to be created successfully, the system will face data loss or service unavailability, directly affecting the availability of the RAG service. To ensure service continuity and data recoverability, the system will use the metadata of the original vector set that has been saved in advance to recreate a new vector set that is completely consistent with the original set in terms of data structure and interface behavior. Subsequently, the system can restore the successfully processed vector records from the intermediate vector set or the pre-stored backup data and write them into the reconstructed set, so as to restore the system to a complete and usable state before the migration as much as possible. This avoids the knowledge base becoming unavailable or the data being permanently lost due to operation interruption, thereby improving the robustness and security of the entire vector model switching process.

[0056] As another example, in response to the failure to create the target vector set after deleting the original vector set, the intermediate vector set is retained, and the status of the vector migration task is updated to pending manual recovery or pending continuation, so that the operations and maintenance personnel can perform subsequent recovery operations on the vector migration task based on the migration task log.

[0057] In other words, if the system fails to create the target vector set after deleting the original vector set due to insufficient permissions, resource quota limitations, database anomalies, or configuration errors, and automatic rollback and reconstruction are not feasible or too risky, the system will proactively retain the generated intermediate vector set. At the same time, the system will explicitly update the status of the vector migration task to "awaiting manual recovery" or "awaiting continued execution" and record the complete migration context and failure logs. Based on this information, operations and maintenance personnel can diagnose the root cause of the problem, manually repair the environment or configuration, and choose to continue to complete the set creation and data migration, or safely write the intermediate data back to the reconstructed original vector set. This avoids the loss of generated data and preserves the necessary operational space for the operations and maintenance team, ensuring that the migration task can still be tracked, intervened, and recovered even in complex fault situations.

[0058] The vector migration method of this disclosure, upon detecting a vector model change event, does not directly overwrite or update the original vector set in place. Instead, it first re-vectorizes the text slices corresponding to the document to be migrated using a second vector model with better performance, generating a second vectorized representation with higher semantic consistency. This second vectorized representation, along with the original text content and metadata, is written into an independent intermediate vector set, thus avoiding the semantic space inconsistency problem caused by mixing old and new vectors. Furthermore, only after confirming that the first target vector record has been successfully written into the intermediate vector set is the original vector set deleted and a structurally compatible target vector set created, completely migrating the intermediate data to the target set. This effectively isolates data risks during the migration process and ensures that even if the migration is interrupted or fails, it can still be recovered through the intermediate vector set or the original vector set, significantly improving the reliability and fault tolerance of the migration. At the same time, the upper-layer RAG application can seamlessly connect to the new model data without modifying the configuration, truly realizing seamless service switching during the model upgrade process and ensuring retrieval accuracy and system continuity.

[0059] Based on any of the above embodiments, such as Figure 4 As shown, the vector transfer method of this disclosure embodiment may include the following steps: S100: Create a migration task and record the first time point t1 When the system detects that the vector model needs to be replaced, or receives a vector model replacement instruction initiated by the administrator, a migration task is created, and the following information is recorded: migration task identifier, knowledge base identifier, original set (original vector set) name, original vector model identifier (identifier of the first vector model), new vector model identifier (identifier of the second vector model), original vector dimension, new vector dimension, task creation time (i.e., the first time point), task status, failure reason, number of retries, and operation log, etc. It should be noted that before the migration officially begins, the system records the current time as the first time point (t1). This time point can be used to define the migration scope, that is, to identify documents that already exist in the original vector set before the migration starts, and to provide a time boundary for subsequent identification of incremental documents that need compensation processing. In a specific embodiment, the system queries all documents currently existing in the original vector set on a document-by-document basis and extracts their document identifiers (which may include document ID, knowledge base ID, business ID, file version number, parsing task ID, or a combination thereof) to form a clear list of documents to be migrated. This list is written to the migration task table or migration details table, and the migration status of each document is initialized to "pending migration". In this way, the system avoids uncontrollable batch deletion operations on the entire vector database, and instead locks in a clear and traceable migration scope before the migration starts, laying a solid foundation for subsequent document-level migration control, failure retries, progress tracking, and rollback operations when necessary. S200: Establish an intermediate set (intermediate vector set) Based on the output vector dimensions of the new vector model, the system creates an intermediate set in the vector database to temporarily store the text vector data regenerated using the new vector model. The field structure of this intermediate set is largely consistent with the original vector set, including but not limited to: text content fields, document identifier fields, text slice identifier fields, knowledge base identifier fields, paragraph position fields, page number fields, business metadata fields, creation time fields, update time fields, and vector fields. The main difference is that the vector field dimensions in the intermediate set are consistent with the output dimensions of the new vector model to ensure the correctness of subsequent writes and index construction. In one embodiment, the name of the intermediate set can be automatically generated based on the original set name, migration task identifier, and timestamp. For example, if the original set name is kb_vector_collection and the migration task identifier is migration_001, then the intermediate set name can be set to kb_vector_collection_migration_001_temp. This naming method not only effectively avoids set name conflicts between different migration tasks, but also facilitates subsequent operation traceability, status management, and resource cleanup. Once the intermediate set is successfully created, the system will update the status of the corresponding migration task to "Intermediate set created". If the creation process fails due to issues such as permissions, resources, or configuration, the system will record the specific reason for the failure and immediately terminate the migration task. At the same time, it will ensure that no deletion or modification operations are performed on the original vector set, thereby guaranteeing the stability of online services and data security.

[0060] S300: Migrate the data from the original set (original vector set) to the intermediate set. The system queries the text slice data corresponding to the document to be migrated from the original vector set on a document-by-document basis. During the query process, it is possible to choose not to read the original vector fields, but only to obtain other necessary fields besides the vectors, including text content, document ID, slice ID, knowledge base ID, paragraph position, page number information, and business metadata, thereby reducing unnecessary I / O overhead.

[0061] For each text slice, the system calls a new vector model to vectorize it, generating a new text vector adapted to the new model. Then, this new vector is combined with the text content and corresponding metadata of the original text slice to form a complete vector record, which is then written to an intermediate set.

[0062] In one specific embodiment, the system performs document-level migration according to the following process: First, select a document from the list of documents to be migrated that is in a state of "to be migrated" or "migration failed and to be retried"; Next, based on its document ID, query all text slices of that document in the original set; Then, for each text slice, a new vector model is called to generate a new vector, and the new vector, the original text content, and related metadata are written into an intermediate set; If all text slices of the document have been successfully written, then its migration status will be updated to "migration complete"; If any text slice fails during vectorization or writing, the document status is updated to "migration failed, please retry", and the specific reason for the failure is recorded.

[0063] This approach performs migration and retries at the document level, rather than redoing the entire collection uniformly. This design avoids repeated rollbacks of the entire task due to anomalies in individual documents, significantly improving migration efficiency and resource utilization.

[0064] Furthermore, the system can configure a maximum number of retries for each document. For example, if a document migration fails, the system will automatically retry after a preset time interval; if the number of retries reaches the threshold and still fails, the entire vector migration task is deemed a failure. In this case, the system will delete the created intermediate set, retain the original vector set, and update the migration task status to "migration failed" to ensure that online services are not affected.

[0065] Through the above mechanism, before deleting the original set, all old data has been converted into a format adapted to the new vector model and safely written into the intermediate set. Only when the data in the intermediate set is verified to be complete and usable will the system proceed to the next step, delete the original set and create the target set. This process effectively reduces the risk of data loss or service interruption and ensures the security and reliability of the vector model upgrade process. S400: Delete the original set and create a new set. Once all documents to be migrated have been successfully written to the intermediate set, the system enters the set replacement phase. This phase first deletes the original vector set and then creates a new set with the exact same name. The field structure of the new set is consistent with the intermediate set, including text content, metadata fields, and vector fields, where the dimensions of the vector fields strictly match the output dimensions of the new vector model. By retaining the name of the original set, the upper-layer RAG retrieval service, knowledge base question-and-answer module, or business call interface do not need to modify the set access address or configuration, thus significantly reducing system integration and transformation costs and achieving transparent switching at the application layer. After the new set is successfully created, the system records the current time as the second time point (t2) to mark the time boundary when the new set is officially ready. It should be noted that between the first time point (t1) (before migration begins) and the second time point (t2) (new set creation complete), the system may still be receiving user uploads or triggering new document parsing tasks. Since the original set is in a migration, deletion, or reconstruction state during this period, documents that begin parsing during these time periods may be written to the old set, fail to be written, or be in an uncertain intermediate state. To ensure data consistency, this solution treats all documents that initiate parsing between (t1) and (t2) as "parsing-required" objects (incremental documents). After the new set is created, the parsing and vectorization processes are re-executed to ensure that the documents use vectors generated by the new model and are included in the target set. If the creation of the new set fails due to permission, resource, or configuration issues after deleting the original set, the system will perform exception handling according to a preset strategy: In one embodiment, the system recreates a vector set with the same structure based on the metadata of the original set, and attempts to restore the data generated in the intermediate set or the pre-stored backup to the new set in order to restore the service as soon as possible; In another embodiment, the system retains the intermediate set unchanged and updates the migration task status to "awaiting manual recovery" or "awaiting continued execution," allowing maintenance personnel to perform diagnosis and subsequent operations based on the migration logs.

[0066] The above design balances automated fault tolerance with human intervention capabilities, ensuring service continuity while effectively avoiding data loss or state chaos, and providing reliable support for the smooth upgrade of the vector model.

[0067] S500: Migrate intermediate set data back to the new set After the new set is successfully created, the system migrates all data from the intermediate set to the new set. This migration process can be performed using batch insertion, segmented insertion, or streaming insertion, depending on system load and performance requirements, to balance efficiency and resource consumption. In one specific embodiment, the system reads data from the intermediate set in batches by document ID or slice ID and writes each batch to the new set. To prevent duplicate writes, the system uses a combination of document ID, slice ID, knowledge base ID, and vector model version number as a unique identifier. If the underlying vector database supports Upsert operations (i.e., "update if it exists, insert if it doesn't"), the system can directly use Upsert to write; if Upsert is not supported, the system checks if a record with the same unique identifier already exists in the new set before writing, and only performs the insertion operation if it is confirmed that the record does not exist, thus ensuring data idempotency and consistency. Once all data in the intermediate set has been successfully written to the new set, the system updates the migration task status to "basic data migration complete". At this point, all historical documents that existed in the original set before the migration began have been vectorized and reconstructed based on the new vector model, and fully restored to a new set with the same name as the original set. Upper-layer services can continue to access them seamlessly. In one optional embodiment, the system can proactively trigger index building or index loading operations after the data migration is complete. For example, in vector databases such as Milvus that support explicit index management, the system will call the index creation and loading interfaces after data writing is complete to ensure that the new set has efficient vector retrieval capabilities, thereby guaranteeing the normal operation of subsequent RAG question answering, semantic search, and other services.

[0068] S600: Re-parse documents within the time period t1 to t2 After the new collection is established and the basic data migration is completed, the system queries the document parsing task records and filters out all documents whose parsing start time falls within the time period t1 to t2. For documents within this time period, the system uniformly re-executes the document parsing, text slicing, vectorization processing, and vector storage process.

[0069] Specifically, the system can perform compensation analysis according to the following steps: (1) Query the parsing task table for documents whose parsing start time is greater than or equal to t1 and less than or equal to t2; (2) Mark the original parsing result of the above document as invalid, or delete its corresponding vector data; (3) Reread the document source file; (4) Regenerate text slices according to the current RAG system's text slicing rules; (5) Use the new vector model to generate text vectors; (6) Write the text content, new vector, document ID, slice ID, knowledge base ID, paragraph position and business metadata into a new set; (7) Update the compensation resolution status to "Completed".

[0070] This step addresses the issues of documents being parsed during migration potentially being written to the old set, failing to write, or being written incompletely. In other words, this invention does not require the system to completely halt document parsing during migration. Instead, by recording two time points, t1 and t2, it uniformly compensates for uncertain parsing results within the migration window, thereby ensuring the integrity and consistency of the data in the final new set.

[0071] S700: Clean up the intermediate set and complete the migration task. Once the intermediate set data has been migrated to the new set, and all documents within the time period t1 to t2 have undergone compensation parsing, the system performs data validation on the new set. Validation may include: document count validation, text slice count validation, vector dimension validation, vector model version validation, index status validation, and sampling retrieval validation.

[0072] If the verification passes, the system deletes the intermediate set and updates the migration task status to "Migration Complete". Simultaneously, the system records information such as migration completion time, total number of documents, total number of slices, number of retries on failure, and number of documents parsed as compensation.

[0073] If the verification fails, the system retains the intermediate set and updates the migration task status to "Verification Failed" or "Pending Manual Processing". Since the intermediate set contains complete intermediate data generated based on the new vector model, the system can continue to investigate and recover without reprocessing all historical documents.

[0074] Corresponding to the vector migration method provided in the above embodiments, this disclosure also provides a vector migration apparatus. Since the vector migration apparatus provided in this disclosure corresponds to the vector migration method provided in the above embodiments, the implementation of the vector migration method is also applicable to the vector migration apparatus provided in this disclosure, and will not be described in detail in this disclosure.

[0075] Figure 5 This is a schematic diagram of the structure of the vector transfer device shown in the fourth embodiment of this disclosure.

[0076] like Figure 5 As shown, the vector migration device 500 includes: a query module 510, a calling module 520, a generation module 530, and a migration module 540.

[0077] The query module 510, in response to the detection of a vector model change event, queries the original vector set associated with the vector migration task to retrieve the first text slices corresponding to each document to be migrated. The original vector set stores multiple text slices, their corresponding first vectorized representations, and metadata. The first vectorized representation is obtained by vectorizing the text slices using the first vector model. The calling module 520 calls a second vector model to vectorize the first text slices, obtaining a second vectorized representation of the first text slices. The second vector model has a higher positive evaluation index than the first vector model. The generation module 530 generates a first target vector record based on the second vectorized representation, the text content of the first text slice, and the corresponding metadata, and writes the first target vector record into the created intermediate vector set. The migration module 540, in response to the successful writing of the first target vector record, deletes the original vector set and creates a target vector set to migrate all target vector records from the intermediate vector set to the target vector set.

[0078] As one possible implementation, the query module 510 is used to create a vector migration task in response to the detection of a vector model change event, and record the first creation time of the vector migration task; based on the first creation time, the first text slice is selected from multiple text slices in the original vector set; wherein the time when the first text slice is written into the original vector set is earlier than or equal to the first creation time.

[0079] As one possible implementation, the migration module 540 is also used to determine the incremental document whose slice parsing start time is between the first creation time and the second creation time based on the first creation time and the second creation time of the target vector set; call the second vector model to perform vectorization processing on the second text slice of the incremental document to obtain the third vectorized representation of the second text slice; generate the second target vector record based on the third vectorized representation, the text content of the second text slice and the corresponding metadata, and write the second target vector record into the target vector set.

[0080] As one possible implementation, the vector transfer device 500 also includes a verification module.

[0081] The verification module is used to perform data verification on the target vector set in response to the successful writing of the second target vector record. The data verification includes at least one of the following: document quantity verification, text slice quantity verification, vector dimension verification, vector model version verification, index status verification, and sampling retrieval verification. In response to the data verification passing, the intermediate vector set is deleted, and the status of the vector migration task is updated to migration completed. In response to the data verification failing, the intermediate vector set is retained, and the status of the vector migration task is updated to verification failed or pending manual processing.

[0082] As one possible implementation, the vector migration device 500 also includes an update module.

[0083] The update module is used to update the migration status of each document to be migrated to "migration completed" when all the first target vector records corresponding to each document to be migrated are successfully written; and to update the migration status of the corresponding document to be migrated to "migration failed and awaiting retry" when the first target vector record corresponding to any document to be migrated fails to be written, and to record the corresponding reason for the writing failure.

[0084] As one possible implementation, the vector migration device 500 also includes an execution module.

[0085] The execution module is used to respond to the migration status of any document to be migrated being updated to migration failure pending retry, and the set period is reached. It re-executes the vectorization processing of the first text slice of the corresponding document to be migrated, and writes the newly generated first target vector record into the intermediate vector set. If the cumulative number of retries for the corresponding document to be migrated reaches the maximum number of retries and the writing still fails, it determines that the vector migration task has failed, deletes the intermediate vector set, and updates the status of the vector migration task to migration failure.

[0086] As one possible implementation, the vector migration device 500 also includes a recovery module.

[0087] The recovery module is used to respond to the failure to create a target vector set after deleting the original vector set. Based on the metadata of the original vector set, it recreates a vector set with the same data structure as the original vector set and restores the data in the intermediate vector set or the pre-stored backup to the recreated vector set. Alternatively, it retains the intermediate vector set and updates the status of the vector migration task to pending manual recovery or pending continued execution, so that the operation and maintenance personnel can perform subsequent recovery operations on the vector migration task based on the migration task log.

[0088] As one possible implementation, the field structure of the target vector set is consistent with that of the intermediate vector set, the set name of the target vector set is the same as that of the original vector set, and the dimension of the vector field in the target vector set is consistent with the dimension of the output vector of the second vector model.

[0089] As one possible implementation, the vector migration device 500 also includes a termination module.

[0090] The termination module is used to respond to the failure of creating the intermediate vector set, record the corresponding failure reason, and terminate the vector migration task, wherein the original vector set is retained during the termination process.

[0091] The vector migration apparatus of this disclosure, upon detecting a vector model change event, does not directly overwrite or update the original vector set in place. Instead, it first re-vectorizes the text slices corresponding to the document to be migrated using a second vector model with better performance, generating a second vectorized representation with higher semantic consistency. This second vectorized representation, along with the original text content and metadata, is then written into an independent intermediate vector set, thus avoiding the semantic space inconsistency problem caused by mixing old and new vectors. Furthermore, only after confirming that the first target vector record has been successfully written into the intermediate vector set is the original vector set deleted and a structurally compatible target vector set created. The intermediate data is then completely migrated to the target set, effectively isolating data risks during the migration process. This ensures that even if the migration is interrupted or fails, it can still be recovered through the intermediate vector set or the original vector set, significantly improving the reliability and fault tolerance of the migration. At the same time, the upper-layer RAG application can connect to the new model data without modifying the configuration, truly realizing seamless service switching during the model upgrade process and ensuring retrieval accuracy and system continuity.

[0092] In an exemplary embodiment, an electronic device is also proposed.

[0093] The electronic devices include: processor; Memory used to store processor-executable instructions; The processor is configured to execute instructions to implement the vector migration method as proposed in any of the foregoing embodiments.

[0094] As an example, Figure 6 This is a schematic diagram of the structure of an electronic device 600 as shown in an exemplary embodiment of this disclosure, as follows: Figure 6 As shown, the aforementioned electronic device 600 may further include: The memory 610 and processor 620 are connected by a bus 630, which connects the different components (including the memory 610 and the processor 620). The memory 610 stores a computer program, which implements the vector migration method described in the embodiments of this disclosure when the processor 620 executes the program.

[0095] Bus 630 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0096] Electronic device 600 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 600, including volatile and non-volatile media, removable and non-removable media.

[0097] Memory 610 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 640 and / or cache memory 650. Electronic device 600 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 660 can be used to read and write non-removable, non-volatile magnetic media (… Figure 6 Not shown; usually referred to as a "hard drive"). Although Figure 6 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 630 via one or more data media interfaces. Memory 610 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0098] A program / utility 680 having a set (at least one) of program modules 670 may be stored, for example, in memory 610. Such program modules 670 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 670 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0099] Electronic device 600 can also communicate with one or more external devices 690 (e.g., keyboard, pointing device, display 691, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 692. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1093. As shown, network adapter 693 communicates with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0100] The processor 620 executes various functional applications and data processing by running programs stored in the memory 610.

[0101] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the vector migration method of this disclosure embodiment, and will not be repeated here.

[0102] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of an electronic device to perform the vector migration method proposed in any of the above embodiments. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0103] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implement the vector migration method proposed in any of the above embodiments.

[0104] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0105] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A vector transfer method, characterized in that, include: In response to the detection of a vector model change event, the first text slice corresponding to each document to be migrated associated with the vector migration task is queried from the original vector set associated with the vector migration task; wherein, the original vector set stores multiple text slices and corresponding first vectorized representations and metadata; the first vectorized representation is obtained by vectorizing the text slices using a first vector model; The first text slice is vectorized by calling the second vector model to obtain the second vectorized representation of the first text slice; wherein, the positive evaluation index of the second vector model is higher than that of the first vector model. Based on the second vectorized representation, the text content of the first text slice and the corresponding metadata, a first target vector record is generated and written into the created intermediate vector set. In response to the successful writing of the first target vector record, the original vector set is deleted and a target vector set is created to migrate all target vector records in the intermediate vector set to the target vector set.

2. The method according to claim 1, characterized in that, In response to detecting a vector model change event, querying the first text slice corresponding to each document to be migrated associated with the vector migration task from the original vector set associated with the vector migration task includes: In response to the detection of a vector model change event, a vector migration task is created, and the first creation time of the vector migration task is recorded; Based on the first creation time, the first text slice is selected from multiple text slices in the original vector set; wherein the time when the first text slice is written into the original vector set is earlier than or equal to the first creation time.

3. The method according to claim 2, characterized in that, The method further includes: Based on the first creation time and the second creation time of the target vector set, determine the incremental documents whose slice parsing start time is located between the first creation time and the second creation time; The second vector model is invoked to vectorize the second text slice of the incremental document, resulting in a third vectorized representation of the second text slice. Based on the third vectorized representation, the text content of the second text slice, and the corresponding metadata, a second target vector record is generated and written into the target vector set.

4. The method according to claim 3, characterized in that, The method further includes: In response to the successful writing of the second target vector record, data verification is performed on the target vector set; wherein, the data verification includes at least one of the following: document quantity verification, text slice quantity verification, vector dimension verification, vector model version verification, index status verification, and sampling retrieval verification; In response to the data verification passing, the intermediate vector set is deleted, and the status of the vector migration task is updated to migration complete; In response to the data verification failing, the intermediate vector set is retained, and the status of the vector migration task is updated to verification failed or pending manual processing.

5. The method according to claim 1, characterized in that, The method further includes: In response to the successful writing of the first target vector record corresponding to each of the documents to be migrated, the migration status of each of the documents to be migrated is updated to migration complete; In response to a failure to write the first target vector record corresponding to any document to be migrated, the migration status of the corresponding document to be migrated is updated to "migration failed and awaiting retry", and the corresponding reason for the write failure is recorded.

6. The method according to claim 5, characterized in that, The method further includes: In response to the migration status of any document to be migrated being updated to migration failed and pending retry, and the set period being reached, the vectorization processing of the first text slice of the corresponding document to be migrated is re-executed, and the newly generated first target vector is recorded and written into the intermediate vector set. If the cumulative number of retries for the corresponding document to be migrated reaches the maximum number of retries and the write still fails, then the vector migration task is determined to have failed, the intermediate vector set is deleted, and the status of the vector migration task is updated to migration failed.

7. The method according to claim 1, characterized in that, The method further includes: In response to the failure to create the target vector set after deleting the original vector set, a new vector set with the same data structure as the original vector set is recreated based on the metadata of the original vector set, and the data in the intermediate vector set or the pre-stored backup is restored to the newly created vector set; or, The intermediate vector set is retained, and the status of the vector migration task is updated to await manual recovery or await continuation, so that the operation and maintenance personnel can perform subsequent recovery operations on the vector migration task based on the migration task log.

8. The method according to claim 1, characterized in that, The field structure of the target vector set is consistent with that of the intermediate vector set, the set name of the target vector set is the same as that of the original vector set, and the dimension of the vector field in the target vector set is consistent with the dimension of the output vector of the second vector model.

9. The method according to claim 1, characterized in that, The method further includes: In response to the failure to create the intermediate vector set, the corresponding failure reason is recorded, and the vector migration task is terminated, wherein the original vector set is retained during the termination process.

10. A vector transfer device, characterized in that, include: The query module is used to, in response to the detection of a vector model change event, query the first text slices corresponding to each document to be migrated associated with the vector migration task from the original vector set associated with the vector migration task; wherein, the original vector set stores multiple text slices and their corresponding first vectorized representations and metadata; the first vectorized representations are obtained by vectorizing the text slices using a first vector model; The calling module is used to call the second vector model to perform vectorization processing on the first text slice to obtain the second vectorized representation of the first text slice; wherein, the positive evaluation index of the second vector model is higher than that of the first vector model; The generation module is used to generate a first target vector record based on the second vectorized representation, the text content of the first text slice and the corresponding metadata, and write the first target vector record into the created intermediate vector set; The migration module is used to, in response to the successful writing of the first target vector record, delete the original vector set and create a target vector set, so as to migrate all target vector records in the intermediate vector set to the target vector set.

Images