Database updating method and device, electronic equipment and storage medium

Abstract

The invention provides a database updating method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of databases. The method comprises the steps of obtaining an original data table; performing full-amount snapshot on data in the original data table, generating a second vector for the full-amount snapshot data by adopting a new embedding model, and storing the second vector and the original data into a new data table; for newly written data, adopting a new embedding model to generate a new vector, and storing the new vector into the newly-built message queue, so as to write the new vector into the newly-built data table through the message queue; and when the number of the to-be-processed messages in the message queue reaches a preset threshold value, switching the table alias of the data table from the original data table to the new data table so as to switch the read-write operation from the original data table to the new data table. According to the method and the device, the whole upgrading process is non-sensitive, and the influence on service operation is greatly reduced.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the field of database technology, and in particular to database update methods, apparatus, electronic devices, and storage media. Background Technology

[0002] With the rapid development of large-scale language models (LLM), embedding models are also being rapidly updated, which places higher demands on the performance and functionality of databases (such as vector databases). Embedding models are a key technology for converting complex data (such as text, images, etc.) into fixed-dimensional vectors, and are widely used in similarity search, recommender systems, and other fields.

[0003] However, existing vector databases have significant shortcomings when facing embedding model upgrades. Specifically, when users want to regenerate all data in the vector database using a completely new embedding model, they typically have no choice but to stop the service. This process leads to prolonged service interruptions, severely impacting normal business operations. Such extended service disruptions not only degrade the user experience but can also result in data loss or inconsistencies. Summary of the Invention

[0004] This disclosure provides a database update method, apparatus, electronic device, and storage medium.

[0005] According to one aspect of this disclosure, a database update method is provided, the method comprising: Obtain the original data table, which contains original data and a first vector, the first vector being a vector generated based on the original data using an old embedding model; A full snapshot of the data in the original data table is taken, and a second vector is generated from the full snapshot data using a new embedding model. The second vector and the original data are then stored in a newly created data table. For newly written data, a new vector is generated using the new embedding model and stored in a newly created message queue, so that the new vector can be written into the newly created data table through the message queue; When the number of messages to be processed in the message queue reaches a preset threshold, the table alias of the data table is switched from the original data table to the newly created data table, so that read and write operations are switched from the original data table to the newly created data table.

[0006] According to another aspect of this disclosure, a database update apparatus is provided, the apparatus comprising: An acquisition module is used to acquire an original data table, which contains original data and a first vector, the first vector being a vector generated based on the original data using an old embedding model. The full embedding update module is used to take a full snapshot of the data in the original data table, generate a second vector from the full snapshot data using a new embedding model, and store the second vector and the original data into a newly created data table. The incremental embedding update module is used to generate a new vector using the new embedding model for newly written data and store it in a newly created message queue, so that the new vector can be written into the newly created data table through the message queue. The switching module is used to switch the table alias of the data table from the original data table to the newly created data table when the number of pending messages in the message queue reaches a preset threshold, so as to switch the read and write operations from the original data table to the newly created data table.

[0007] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in any of the above technical solutions.

[0008] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any one of the methods described above.

[0009] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in any one of the above technical solutions.

[0010] This disclosure provides a database update method, apparatus, device, and storage medium. By acquiring the original data table and performing a full snapshot, this disclosure ensures data integrity and consistency during the upgrade process, avoiding the risk of data corruption and business interruption that may occur from direct operations on the original table, thereby significantly improving the security and reliability of data upgrades. Next, a new embedding model is used to generate new vectors from the full snapshot data, and this data is stored in a newly created data table, providing a data foundation for subsequent seamless switching. For newly written data, a new embedding model is used to generate vectors and temporarily stored in a message queue. This process ensures real-time processing and synchronous updates of incremental data, enabling the newly created data table to reflect the latest data status in a timely manner, further enriching the real-time performance and accuracy of the data. Finally, when the number of pending messages in the message queue reaches a preset threshold, by switching the table alias, read and write operations are seamlessly switched from the original data table to the newly created data table, achieving a seamless upgrade process, greatly reducing the impact on business operations, improving system availability and user experience, and also reducing the operational costs and risks caused by upgrade operations.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram illustrating an exemplary vector database data writing process; Figure 2 This is a schematic diagram of the steps of the database update method in the embodiments of this disclosure; Figure 3 This is a schematic diagram of the process of full embedding update and incremental embedding update in the embodiments of this disclosure; Figure 4 This is a schematic diagram of the table alias switching process in an embodiment of this disclosure; Figure 5 This is a schematic diagram of the read operation process performed to update the new data table in an embodiment of this disclosure; Figure 6 A schematic block diagram of the database update device in this disclosure embodiment; Figure 7 This is a block diagram of an electronic device used to implement the database update method of the embodiments of this disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] For a better understanding of the technical solution in this application, please refer to [link / reference]. Figure 1 As shown, Figure 1 This is a schematic diagram illustrating an exemplary data writing process for a vector database. Specifically, firstly, the text generated upstream is segmented into multiple text chunks. These text chunks are then fed into an embedding model for vectorization, generating corresponding vector representations [0.1, 0.2, 0.3...]. These vectors, along with the original text, are stored in a table. Therefore, it can be considered that the vector database stores the original text and the embedded vectors. Now, if the embedding module needs to be upgraded, and a user wants to regenerate all data in the vector database using a completely new embedding model, typically, the user can only choose to stop the service, delete the old data, and then regenerate the new embedding vectors. If the data volume is large, this process will cause the service to be interrupted for a long time, seriously affecting the normal operation of the business. Such a long service interruption not only degrades the user experience but may also lead to data loss or inconsistency. Therefore, how to efficiently upgrade the embedding model without affecting the normal operation of the business has become an urgent technical problem to be solved.

[0015] To address the aforementioned technical problems, this disclosure provides a database update method, see [link to relevant documentation]. Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the steps of a database update method in an embodiment of this disclosure, the method including: Step S201: Obtain the original data table, which contains the original data and the first vector, which is a vector generated based on the original data using an old embedding model.

[0016] Specifically, in a vector database, the original data table is a table structure that stores user data and its corresponding vectors. The original data can be text, images, or other types of data, while the first vector refers to the vector representation generated after processing this original data using an older embedding model. An embedding model is an algorithm that transforms complex data into fixed-dimensional vectors for efficient similarity searching in vector databases. Specifically, the older embedding model maps the original data to a vector space, and the generated vectors can be used for subsequent retrieval and analysis. For example, if the original data is text, the embedding model converts the text into a numerical vector, making similar text closer together in the vector space.

[0017] In this scheme, by obtaining the original data table containing the original data and its first vector, the basic data support for subsequent upgrade operations is provided, ensuring that new vectors can be generated based on these data during the upgrade process, thereby achieving seamless upgrade of the embedding model.

[0018] Step S202: Take a full snapshot of the data in the original data table, and use a new embedding model to generate a second vector from the full snapshot data. Store the second vector and the original data in a newly created data table.

[0019] Specifically, during the vector database upgrade process, to ensure data consistency and integrity while avoiding the risks associated with directly manipulating the original data table, this solution employs a full snapshot technique. A full snapshot involves making a complete copy of all data in the original data table (including the original data and its corresponding first vector), generating a data replica. This process ensures that the original data table remains unchanged during the upgrade, thus avoiding the risk of data corruption or loss. Next, a new embedding model is used to process the original data in the full snapshot, generating a new vector representation, i.e., the second vector. New embedding models typically have better performance or superior feature representation capabilities, providing more accurate support for subsequent retrieval and analysis. The generated second vector, along with the original data, is stored in a newly created data table, which will serve as the target table for subsequent operations.

[0020] This approach not only preserves the original data but also updates it using a new embedded model, laying the foundation for seamless switching and data upgrades. This process ensures data integrity and consistency, and facilitates subsequent incremental data processing and table switching, making the entire upgrade process more efficient and secure.

[0021] In step S203, for newly written data, a new vector is generated using a new embedding model and stored in a newly created message queue, so that the new vector can be written into a newly created data table through the message queue.

[0022] Specifically, during the vector database upgrade process, to ensure that newly written data reflects the latest embedding model in a timely manner, and to avoid conflicts or inconsistencies caused by directly writing to the new table, this solution employs a message queue mechanism to handle incremental data. Specifically, when new data is written, it is first sent to a newly created message queue. The message queue is a temporary storage area used to temporarily store newly written data and its corresponding vectors until they are synchronized to the new table. For this newly written data, new vectors are generated using the new embedding model. The new embedding model typically has better performance or superior feature representation capabilities than the old model, providing more accurate support for subsequent retrieval and analysis. The generated new vectors, along with the original data, are stored in the message queue. Subsequently, this data is read from the message queue and written to the newly created data table.

[0023] This process ensures that the data in the new table is up-to-date and consistent with the data in the old table. Through the message queue mechanism, not only can incremental data be processed efficiently, but direct write operations to the new table are also avoided during full data updates, thus ensuring data consistency and integrity. Furthermore, message queues provide buffering and order guarantees, further improving the stability and reliability of the system.

[0024] Step S204: When the number of messages to be processed in the message queue reaches a preset threshold, the table alias of the data table is switched from the original data table to the newly created data table, so that the read and write operations are switched from the original data table to the newly created data table.

[0025] Specifically, during the vector database upgrade process, to achieve seamless switching and ensure data consistency and business continuity, this solution employs a table alias mechanism. A table alias is a reference to an actual data table, allowing for flexible switching between tables without modifying the business system's code or configuration. During the upgrade process, the number of pending messages in the message queue is monitored. The message queue is a temporary storage area used to temporarily store newly written data and its corresponding vectors until they are synchronized to the newly created data table. When the number of pending messages in the message queue decreases to a preset threshold, it indicates that the data in the new data table is essentially synchronized with the original data table. At this point, the table alias is switched from the original data table to the new data table. This switching operation is seamless, and users will hardly perceive any interruption.

[0026] This approach not only ensures data consistency and integrity but also enables a smooth transition between read and write operations, greatly reducing the impact on business operations and improving system availability and user experience.

[0027] This disclosure provides a database update method, apparatus, electronic device, and storage medium. By acquiring the original data table and taking a full snapshot of it, this disclosure ensures data integrity and consistency during the upgrade process, avoiding the risk of data corruption and business interruption that might occur if operations were performed directly on the original table, thus significantly improving the security and reliability of data upgrades. Next, a new embedding model is used to generate new vectors from the full snapshot data, and this data is stored in a newly created data table, providing a data foundation for subsequent seamless switching. For newly written data, a new embedding model is used to generate vectors and temporarily stored in a message queue. This process ensures real-time processing and synchronous updates of incremental data, enabling the newly created data table to reflect the latest data status in a timely manner, further enriching the real-time performance and accuracy of the data. Finally, when the number of pending messages in the message queue reaches a preset threshold, by switching the table alias, read and write operations are seamlessly switched from the original data table to the newly created data table, achieving a seamless upgrade process, greatly reducing the impact on business operations, improving system availability and user experience, and also reducing the operational costs and risks caused by upgrade operations.

[0028] In some optional embodiments, a full snapshot of the data in the original data table is taken, and a second vector is generated from the full snapshot data using a new embedding model. The second vector and the original data are then stored in a newly created data table, including: Create a new data table; Read all data from the original data table to obtain a full snapshot of the data; A new embedding model is used to perform vector transformation on the original data in the full snapshot data to obtain a second vector; Store the original data and the second vector from the snapshot data into a newly created data table.

[0029] Specifically, the process begins with creating a new data table to store the upgraded data, providing a foundation for a seamless transition. Next, all data is read from the original data table to generate a full snapshot. A full snapshot involves a complete copy of all data in the original data table (including the original data and its corresponding old vectors), ensuring data integrity during the upgrade. Then, a new embedding model is used to vectorize the original data in the full snapshot, generating a second vector. New embedding models typically offer better performance or superior feature representation capabilities, providing more accurate support for subsequent retrieval and analysis. Finally, the original data and the second vector from the snapshot are stored in the newly created data table, completing the data migration and update. This process not only ensures data integrity and consistency but also lays the foundation for a seamless transition and incremental data processing, making the entire upgrade process more efficient and secure.

[0030] In this way, a full snapshot is generated by creating a new data table and reading all data from the original data table. Then, the data is vectorized using a new embedding model and stored in the new data table. This process not only ensures data integrity and consistency, avoiding the risk of data corruption and business interruption that could occur from direct operations on the original table, but also improves the feature representation capabilities of the data by using the new embedding model, providing more accurate support for subsequent retrieval and analysis. Furthermore, the mechanism of full snapshots and new data tables allows for flexible handling of incremental data during the upgrade process, further enhancing the real-time performance and accuracy of the data.

[0031] In some optional embodiments, for newly written data, a new vector is generated using a new embedding model and stored in a newly created message queue, so that the new vector is written to a newly created data table via the message queue, including: Establish a message component, which is used to temporarily store newly written data and perform data synchronization; Retrieve the newly written data to obtain the incremental data; The incremental data is used to generate a third vector using a new embedding model, and the incremental data and the third vector are stored in a message queue so that the third vector can be written into a newly created data table through the message queue.

[0032] Specifically, during the vector database upgrade process, to ensure that newly written data can be processed and synchronized to the new data table in a timely manner, this solution introduces a message component. The message component is a key middleware used for temporarily storing newly written data (i.e., incremental data) and for data synchronization. Specifically, when new data is written, it is first identified as incremental data and sent to the message component. Subsequently, this incremental data is processed using a new embedding model to generate corresponding third vectors. The new embedding model typically has better performance or superior feature representation capabilities than the old model, providing more accurate support for subsequent retrieval and analysis. The generated third vectors, along with the incremental data, are stored in a message queue. The message queue is a first-in, first-out (FIFO) storage structure used to process and synchronize this data sequentially. By reading the incremental data and its third vectors from the message queue and writing them to the newly created data table, this process not only ensures that the data in the new table is up-to-date but also avoids direct write operations to the new table during a full data update, thus ensuring data consistency and integrity. By using message components and message queues, incremental data can be processed efficiently, avoiding data loss or inconsistency, while improving the stability and reliability of the system.

[0033] By establishing a message component for temporarily storing newly written data (incremental data) and synchronizing it, this solution achieves efficient processing and synchronous updates of incremental data during vector database upgrades. Specifically, newly written data is acquired and identified as incremental data, then a third vector is generated from this incremental data using a new embedding model. The generated incremental data and its corresponding third vector are stored in a message queue, and through the message queue mechanism, this data is gradually synchronized to the newly created data table. This process not only ensures that the new data table reflects the latest data status in a timely manner, avoiding data loss or inconsistency, but also improves the stability and reliability of the system through the buffering and order guarantee functions of the message queue. Furthermore, the introduction of the message component allows for the continued reception and processing of new data during the full data update process, thus achieving a seamless upgrade process, greatly reducing the impact on business operations, and improving user experience and overall system performance.

[0034] In some optional embodiments, the incremental data and the third vector are stored in a message queue so that the third vector can be written to a newly created data table via the message queue, including: After the full snapshot data update is complete, the third vector is read from the message queue and written to the newly created data table.

[0035] Specifically, during the vector database upgrade process, the completion of the full snapshot data update means that all data in the original data table has been processed using the new embedding model, and the generated second vector has been stored in a newly created data table along with the original data. At this point, it's necessary to ensure that all incremental data written after the full snapshot (i.e., newly written data) is also synchronized to the new data table to maintain data consistency and integrity. To this end, the third vector corresponding to this incremental data is read from the message queue. The message queue is a first-in, first-out (FIFO) storage structure used to temporarily store newly written data and its corresponding vectors, awaiting synchronization. Then, according to the order in the message queue, the third vector and its corresponding incremental data are written to the new data table. This process ensures that the data in the new data table is up-to-date and consistent with the data in the original data table. In this way, not only can incremental data be processed efficiently, but direct write operations to the new table during the full data update process can also be avoided, thus ensuring data consistency and integrity, while improving system stability and reliability.

[0036] In this way, after the full snapshot data update is complete, the third vector corresponding to the incremental data is read from the message queue and written to the newly created data table. This operation not only ensures that the data in the newly created data table is up-to-date, but also improves the stability and reliability of the system through the buffering and order guarantee functions of the message queue. In addition, this method allows new data to continue to be received and processed during the full data update process, thereby achieving a seamless upgrade process, greatly reducing the impact on business operations, and improving user experience and overall system performance.

[0037] For a better understanding of the solutions in the embodiments of this application, please refer to... Figure 3 , Figure 3 This is a schematic diagram of the full embedding update and incremental embedding update process in this embodiment of the disclosure. First, the raw data (e.g., text data) generated by the upstream system is processed into text chunks. Then, these raw data are used to generate old vectors [0.1, 0.2, 0.3...] using the old embedding model, i.e., the first vector. Then, the raw data and the first vector are stored in the raw data table (Table1). Next, a full snapshot of the data in the raw data table is taken, and a second vector is generated using the new embedding model (NewEmbedding) on ​​the full snapshot data. The second vector and the raw data are then stored in a newly created data table (Table2).

[0038] For newly written data, a new vector [0.4, 0.5, 0.6...] is generated using a new model (New Embedding). The new vector and the newly written data are temporarily stored in a message queue and written to a newly created data table (Table2) after a full data update is completed. During this process, by maintaining a table alias (Table1_Alias) pointing to the currently active data table, the read / write operations of the business system are decoupled from the data table updates. This "full embedding update" strategy ensures data consistency and integrity while minimizing the impact on business operations during the upgrade process.

[0039] In some optional embodiments, when the number of messages to be processed in the message queue reaches a preset threshold, the table alias of the data table is switched from the original data table to a newly created data table, so that read and write operations are switched from the original data table to the newly created data table, including: When the number of pending messages in the message queue reaches a preset threshold, read and write operations are stopped, and the remaining pending messages are written to a newly created data table. After the remaining messages to be processed are written to the newly created data table, the old embedding model is replaced with the new embedding model, and the table alias is changed from the original data table to the newly created data table, so as to enable read and write operations through the newly created data table.

[0040] Specifically, during the vector database upgrade process, the message queue is a key component used to temporarily store newly written data and its corresponding vectors (i.e., incremental data) to ensure that this data can be gradually synchronized to the newly created data tables. The preset threshold refers to a set value for the number of pending messages in the message queue. When the number of pending messages in the message queue reaches this threshold, a series of operations are triggered to complete the data table switch and the update of the embedded model.

[0041] The specific implementation process of this scheme includes: when the number of pending messages in the message queue reaches a preset threshold, read and write operations are first stopped to ensure data consistency and integrity during the switchover process. Then, the remaining pending messages in the message queue (including incremental data and its corresponding vectors) are written to a newly created data table. This process ensures that the data in the newly created data table is up-to-date and consistent with the data in the original data table.

[0042] After the remaining pending messages are written to the newly created data table, two key operations are performed: First, the old embedding model is replaced with a new one. This means that subsequent vector generation will use the new model, providing superior feature representation capabilities. Second, the table alias is switched from the original data table to the newly created data table. The table alias is a reference to the actual data table. By switching the table alias, the read and write operations of the business system will seamlessly switch to the newly created data table without modifying the business system's code or configuration.

[0043] This approach not only enables seamless switching of data tables but also ensures data consistency and integrity. Simultaneously, the switch to the new embedded model provides more accurate support for subsequent retrieval and analysis. This process significantly reduces the impact on business operations, improves user experience and overall system performance, and solves the problem of balancing data consistency and business continuity during upgrades in existing technologies.

[0044] In this way, by stopping read and write operations when the number of pending messages in the message queue reaches a preset threshold, and writing the remaining pending messages to a newly created data table, data consistency and integrity are ensured during data switching. This process avoids data conflicts or inconsistencies caused by real-time writing. Simultaneously, by replacing the old embedding model with a new one, the feature representation capability of the data is improved, providing more accurate support for subsequent retrieval and analysis. Furthermore, by switching the table alias from the original data table to the newly created data table, a seamless switch between read and write operations is achieved, greatly reducing the impact on business operations. This seamless switching mechanism not only improves system stability but also enhances the user experience, making the entire upgrade process virtually imperceptible to the user, and solving the problem of balancing data consistency and business continuity during upgrades in existing technologies.

[0045] For a better understanding of the solutions in the embodiments of this application, please refer to... Figure 4 , Figure 4 This is a schematic diagram of the table alias switching process in this embodiment. First, the raw data (e.g., text data) generated by the upstream system is processed into text chunks. Then, these raw data are used to generate old vectors [0.1, 0.2, 0.3...] using the old model (Embedding), which is the first vector. Then, the raw data and the first vector are stored in the raw data table (Table1). Next, a full snapshot of the data in the raw data table is taken, and a second vector is generated from the full snapshot data using the new embedding model (New Embedding). The second vector and the raw data are then stored in the newly created data table (Table2).

[0046] For newly written data, a new vector [0.4, 0.5, 0.6...] is generated using the new embedding. The new vector and the newly written data are temporarily stored in a message queue and written to a newly created data table (Table2) after the full data update is complete. Once Table2 is fully ready and all data in the message queue has been processed, a re-alias operation is used to point Table1_Alias ​​to Table2, completing the data table upgrade process.

[0047] The entire flowchart illustrates an efficient data upgrade strategy. It achieves a smooth transition and upgrade of data tables by processing vectors generated from the old and new embedding models in parallel and utilizing message queues and alias switching mechanisms, while ensuring business continuity and data consistency.

[0048] In some optional embodiments, obtaining the original data table includes: Obtain the raw data and the old embedding model; The first vector is generated based on the original data using an old embedding model, and the original data table is constructed based on the first vector and the original data.

[0049] Specifically, in the construction and upgrading of vector databases, raw data refers to the basic data that needs to be processed, such as text, images, or other forms of data. The older embedding model is an algorithm or model used to transform raw data into fixed-dimensional vector representations, which can then be used for subsequent similarity searches and analysis.

[0050] The specific implementation process of this scheme includes: First, acquiring the raw data and the old embedding model, which forms the basis for constructing the raw data table. Next, processing the raw data using the old embedding model generates corresponding first vectors. The first vector is the representation of the raw data in vector space, capturing its features and semantic information. Based on these first vectors and the raw data, a raw data table is constructed, storing the raw data and its corresponding vector representations, providing a data foundation for subsequent retrieval and analysis. This process not only ensures structured data storage but also provides necessary preparation for subsequent model upgrades and data synchronization, enabling efficient data processing and retrieval, while laying the foundation for future upgrades and optimizations.

[0051] In this way, by acquiring the original data and the old embedding model, and using the old embedding model to generate the first vector from the original data, the original data table is constructed, thus laying a solid foundation for the initialization and subsequent upgrade operations of the vector database. This process not only ensures the structured storage of the data, enabling the original data and its corresponding vector representation to be retrieved and analyzed efficiently, but also provides the necessary data preparation for subsequent model upgrades and data synchronization.

[0052] In some optional embodiments, the method further includes, before acquiring the raw data: Create a table alias and point the alias to the original data table so that read and write operations can be performed on the original data table through the table alias.

[0053] Specifically, in the management and upgrade process of vector databases, table aliases are an important mechanism used to simplify referencing and manipulating data tables. In essence, a table alias is a reference to the actual data table, similar to a "pointer" or "alias," allowing access to and manipulation of the original data table through a simplified name. By creating a table alias and pointing it to the original data table, subsequent read and write operations on the original table can be performed using the alias. This method not only simplifies code and configuration but also improves system flexibility and maintainability. When upgrading or migrating data, only the table alias needs to be modified, without changing the business logic code, thus achieving a seamless switch. For example, during a vector database upgrade, when it's necessary to switch the read and write operations of the business system from the old table to the new table, simply re-point the table alias to the new table. This mechanism not only reduces the impact on business operations but also improves system stability and user experience.

[0054] By creating a table alias and pointing it to the original data table, read and write operations on the original data table can be performed through the alias, greatly improving the flexibility and stability of the vector database during upgrades and maintenance. The table alias, acting as a reference to the actual data table, allows for data read and write operations without directly manipulating the original data table. This mechanism not only simplifies code and configuration and reduces the risk of errors that might occur from directly manipulating the original data table, but also facilitates seamless switching later. During upgrades, by modifying the table alias, business logic can be easily switched from the old table to the new table without changing a large amount of business code, thus reducing the impact on business operations and improving system availability and user experience.

[0055] In some optional embodiments, the method further includes: When taking a full snapshot of the data in the original data table, read operations are performed from the original data table.

[0056] Specifically, during the vector database upgrade process, the execution of a full snapshot and read operations on the original data tables are performed in parallel. This design has significant advantages and is necessary. First, a full snapshot means performing a complete copy of all data in the original data tables to create a data replica, which provides a foundation for data consistency and integrity. Second, while the full snapshot operation is underway, read operations can still be performed on the original data tables, ensuring the continuity of business operations. This ensures that business activities such as querying and data retrieval are not affected, thereby minimizing the impact of the upgrade process on the user experience.

[0057] The implementation of parallel read operations allows for continued support of new queries and data requests during snapshotting. These requests can directly retrieve information from the original data table, ensuring data real-time performance and accuracy. Simultaneously, this read data can be used to process new embedding models to generate new vector representations. These new vectors, along with the original data, are stored in a newly created data table, preparing for subsequent system switches.

[0058] The advantage of this parallel operation lies in its provision of real-time data backup while maintaining the accessibility of data tables, enabling efficient data management and service response during the upgrade process. Finally, when the new data table is ready and the number of pending messages in the message queue reaches a preset threshold, read and write operations can be smoothly switched to the new data table, completing the upgrade process. This process not only improves the flexibility of data management and the maintainability of the system, but also significantly enhances the user experience and overall system performance by reducing service downtime.

[0059] This approach leverages the key advantages of vector database upgrades by performing read operations while simultaneously taking a full snapshot of the data in the original tables. This allows for the creation of data copies while continuing to respond to new data requests, ensuring service continuity and real-time data access. This parallel processing mechanism not only reduces the impact on business operations during the upgrade process but also guarantees data integrity and consistency because all read operations are performed directly from the original tables. Furthermore, this strategy supports the introduction of new embedding models, allowing newly generated vectors to be stored alongside the original data in newly created tables, facilitating subsequent system switchover and data management.

[0060] In some optional embodiments, the method further includes: After switching read / write operations from the original data table to the newly created data table, the old embedded model, message queue, and original data table are deleted to free up resources.

[0061] Specifically, during the vector database upgrade process, the old embedding model refers to the model previously used to convert raw data into vectors, while the message queue is a temporary storage area used to temporarily store newly written data and its corresponding vectors, so that they can be synchronized to the newly created data table after the full data update is completed. The raw data table is the table that stores the old vectors and raw data, used to support system read and write operations before the upgrade. After completing read and write operations and switching from the raw data table to the newly created data table, the old embedding model, message queue, and raw data table are deleted to release resources. This process not only cleans up unnecessary resources but also optimizes system storage and performance, ensuring that the system can run efficiently after the upgrade. In this way, not only is storage space usage reduced, but the overall performance and stability of the system are also improved, providing convenience for subsequent operation and maintenance.

[0062] By deleting the old embedded model, message queue, and original data table after completing read / write operations and switching from the original data table to the new data table, system resources are effectively freed up and storage space usage is optimized. This process not only cleans up unnecessary components and data, reducing storage space usage, but also improves the overall performance and stability of the system. Deleting the old embedded model and message queue eliminates redundant resources associated with the old system, while removing the original data table further simplifies the data structure, enabling the system to operate more efficiently. This resource cleanup mechanism ensures that the system operates in a more optimized state after the upgrade, providing more efficient support for subsequent business operations, while also reducing system maintenance costs.

[0063] For a better understanding of the process described in this application, please refer to [link / reference]. Figure 5 , Figure 5This is a schematic diagram of the read operation process performed to update a new data table in an embodiment of this disclosure. The flowchart illustrates the working principle of a query processing and search system, where the query text sent from the upstream is first sent to the Embedding module for vectorization, generating a corresponding vector representation [0.1, 0.2, 0.3...]. This vector is then used to construct a HybridSearchRequest, which is sent to a data table pointed to by a database table alias (Table1_Alias) for searching. The data table returns a list of the most matching results based on vector similarity, for example, id:1,"Approximate Result 1", id:2,"Approximate Result 2", id:3,"Approximate Result 3". This process achieves efficient vector-based search, enabling the rapid discovery of the most relevant items to the query from large amounts of data.

[0064] The following describes an apparatus embodiment of this application, which can be used to execute the database update method in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the database update method described above.

[0065] This disclosure also provides a database update apparatus 600, such as... Figure 6 As shown, it includes: The acquisition module 601 is used to acquire the original data table, which contains original data and a first vector, which is a vector generated based on the original data by using an old embedding model. The full embedding update module 602 is used to take a full snapshot of the data in the original data table, and generate a second vector from the full snapshot data using a new embedding model. The second vector and the original data are then stored in a newly created data table. The incremental embedding update module 603 is used to generate a new vector using a new embedding model for newly written data and store it in a newly created message queue so that the new vector can be written to a newly created data table through the message queue. The switching module 604 is used to switch the table alias of the data table from the original data table to the newly created data table when the number of pending messages in the message queue reaches a preset threshold, so as to switch the read and write operations from the original data table to the newly created data table.

[0066] In some optional embodiments, the full embedding update module 602 performs a full snapshot of the data in the original data table, generates a second vector from the full snapshot data using a new embedding model, and stores the second vector and the original data into a newly created data table, including: Create a new data table; Read all data from the original data table to obtain a full snapshot of the data; A new embedding model is used to perform vector transformation on the original data in the full snapshot data to obtain a second vector; Store the original data and the second vector from the snapshot data into a newly created data table.

[0067] In some optional embodiments, the incremental embedding update module 603 generates a new vector using a new embedding model for newly written data and stores it in a newly created message queue, so as to write the new vector into a newly created data table through the message queue, including: Establish a message component, which is used to temporarily store newly written data and perform data synchronization; Retrieve the newly written data to obtain the incremental data; The incremental data is used to generate a third vector using a new embedding model, and the incremental data and the third vector are stored in a message queue so that the third vector can be written into a newly created data table through the message queue.

[0068] In some optional embodiments, the incremental embedding update module 603 stores the incremental data and the third vector into a message queue, so as to write the third vector into a newly created data table through the message queue, including: After the full snapshot data update is complete, the third vector is read from the message queue and written to the newly created data table.

[0069] In some optional embodiments, when the number of pending messages in the message queue reaches a preset threshold, the switching module 604 switches the table alias of the data table from the original data table to a newly created data table, so as to switch read and write operations from the original data table to the newly created data table, including: When the number of pending messages in the message queue reaches a preset threshold, read and write operations are stopped, and the remaining pending messages are written to a newly created data table. After the remaining messages to be processed are written to the newly created data table, the old embedding model is replaced with the new embedding model, and the table alias is changed from the original data table to the newly created data table, so as to enable read and write operations through the newly created data table.

[0070] In some optional embodiments, the acquisition module 601 acquires the original data table, including: Obtain the raw data and the old embedding model; The first vector is generated based on the original data using an old embedding model, and the original data table is constructed based on the first vector and the original data.

[0071] In some optional embodiments, the acquisition module 601 is used to: Create a table alias and point the alias to the original data table so that read and write operations can be performed on the original data table through the table alias.

[0072] In some optional embodiments, the incremental embedding update module 603 is further configured to: When taking a full snapshot of the data in the original data table, read operations are performed from the original data table.

[0073] In some optional embodiments, the switching module is further configured to: After switching read / write operations from the original data table to the newly created data table, the old embedded model, message queue, and original data table are deleted to free up resources.

[0074] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0075] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0076] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0077] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0078] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 708, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0079] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the database update method. For example, in some embodiments, the database update method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the applet distribution described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the database update method by any other suitable means (e.g., by means of firmware).

[0080] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0081] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable database update device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0082] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0083] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0084] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0085] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0086] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0087] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A database update method, wherein, The method includes: Obtain the original data table, which contains original data and a first vector, the first vector being a vector generated based on the original data using an old embedding model; A full snapshot of the data in the original data table is taken, and a second vector is generated from the full snapshot data using a new embedding model. The second vector and the original data are then stored in a newly created data table. For newly written data, a new vector is generated using the new embedding model and stored in a newly created message queue, so that the new vector can be written into the newly created data table through the message queue; When the number of messages to be processed in the message queue reaches a preset threshold, the table alias of the data table is switched from the original data table to the newly created data table, so that read and write operations are switched from the original data table to the newly created data table.

2. The method according to claim 1, wherein, The step of taking a full snapshot of the data in the original data table, generating a second vector from the full snapshot data using a new embedding model, and storing the second vector and the original data in a newly created data table includes: Create a new data table to obtain the newly created data table; Read all data from the original data table to obtain full snapshot data; The new embedding model is used to perform vector transformation on the original data in the full snapshot data to obtain the second vector; The original data from the snapshot data and the second vector are stored in the newly created data table.

3. The method according to claim 1, wherein, For newly written data, a new vector is generated using the new embedding model and stored in a newly created message queue. The new vector is then written into the newly created data table via the message queue, including: Establish a message component, which is used to temporarily store newly written data and perform data synchronization; Retrieve the newly written data to obtain the incremental data; The incremental data is used to generate a third vector using the new embedding model, and the incremental data and the third vector are stored in the message queue so that the third vector can be written into the newly created data table through the message queue.

4. The method according to claim 3, wherein, The step of storing the incremental data and the third vector into the message queue, so as to write the third vector into the newly created data table through the message queue, includes: After the full snapshot data update is completed, the third vector is read from the message queue and written into the newly created data table.

5. The method according to claim 1, wherein, When the number of pending messages in the message queue reaches a preset threshold, the table alias of the data table is switched from the original data table to the newly created data table, so as to switch read and write operations from the original data table to the newly created data table, including: When the number of pending messages in the message queue reaches a preset threshold, read and write operations are stopped, and the remaining pending messages are written to the newly created data table. After the remaining pending messages are written to the newly created data table, the old embedding model is replaced with the new embedding model, and the table alias is switched from the original data table to the newly created data table, so as to enable read and write operations through the newly created data table.

6. The method according to claim 1, wherein, The process of obtaining the original data table includes: Obtain the raw data and the old embedding model; Based on the original data, a first vector is generated using the old embedding model, and the original data table is constructed based on the first vector and the original data.

7. The method according to any one of claims 1 to 6, wherein, Prior to acquiring the raw data, the method further includes: Create a table alias and point the alias to the original data table so that read and write operations can be performed on the original data table through the alias.

8. The method according to any one of claims 1 to 6, wherein, The method further includes: When taking a full snapshot of the data in the original data table, a read operation is performed from the original data table.

9. The method according to any one of claims 1 to 6, wherein, The method further includes: After switching the read / write operation from the original data table to the newly created data table, the old embedded model, the message queue, and the original data table are deleted to release resources.

10. A database update apparatus, wherein, The device includes: An acquisition module is used to acquire an original data table, which contains original data and a first vector, the first vector being a vector generated based on the original data using an old embedding model. The full embedding update module is used to take a full snapshot of the data in the original data table, generate a second vector from the full snapshot data using a new embedding model, and store the second vector and the original data into a newly created data table. The incremental embedding update module is used to generate a new vector using the new embedding model for newly written data and store it in a newly created message queue, so that the new vector can be written into the newly created data table through the message queue. The switching module is used to switch the table alias of the data table from the original data table to the newly created data table when the number of pending messages in the message queue reaches a preset threshold, so as to switch the read and write operations from the original data table to the newly created data table.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images