Data storage method and device of vector database, data recovery method and device and storage medium

Abstract

The invention discloses a data storage method, recovery method and device of a vector database and a storage medium, the data storage method comprises the following steps: calling a storage interface of the vector database according to attribute information of to-be-stored vector data, and judging whether the storage interface exists or not; if the storage interface exists, performing local cache database storage, vector database storage and persistent database storage on the to-be-stored vector data in sequence; the data recovery method comprises the following steps: after a service is started, checking whether data in a vector database needs to be subjected to data recovery operation or not; if data recovery operation needs to be carried out, data recovery is carried out on missing data in the vector database according to the local cache database and / or the persistent database. The embodiment of the invention improves the data security and recovery efficiency, and can be widely applied to the technical field of databases.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data storage method, recovery method, device and storage medium for a vector database. Background Art

[0002] With the rapid development of AI applications and big data technology, vector databases have been widely used, covering almost many fields and industries such as urban governance, campus, and transportation. However, the current Milvus stand-alone version or self-developed memory vector library may encounter various failures during operation, resulting in data loss or damage. Therefore, the fast recovery technology of vector databases is particularly important. When a database fails, it is necessary to quickly restore the database to avoid affecting the use of retrieval services. Summary of the invention

[0003] In view of this, an object of the embodiments of the present invention is to provide a data storage method, recovery method, device and storage medium for a vector database to improve data security and recovery efficiency.

[0004] On the one hand, an embodiment of the present invention provides a data storage method of a vector database, comprising:

[0005] Calling the storage interface of the vector database according to the attribute information of the vector data to be stored, and determining whether the storage interface exists;

[0006] If the storage interface exists, the vector data to be stored is sequentially stored in the local cache database, the vector database and the persistent database.

[0007] Optionally, the storage method further includes:

[0008] Calculate vector instance fragmentation information according to the vector data to be stored;

[0009] The vector data to be stored is processed by sharding data according to the vector instance sharding information, and vectors are added in batches to complete storage.

[0010] The implementation of the embodiment of the present invention includes the following beneficial effects: the data storage method of the vector database in the embodiment of the present invention first calls the storage interface of the vector database according to the attribute information of the vector data to be stored, and determines whether the storage interface exists; if the storage interface exists, the vector data to be stored is stored in the local cache database, the vector database and the persistent database in sequence, and the vector data to be stored is backed up in the vector database's own memory, the local cache database, the remote persistent database and other ways to ensure data security, provide a consistency guarantee mechanism for the data recovery process to ensure data security, and improve the data security of the vector database.

[0011] On the other hand, an embodiment of the present invention provides a data recovery method for a vector database, comprising:

[0012] After the service is started, check whether data recovery operations need to be performed on the data in the vector database;

[0013] If a data recovery operation is required, the missing data in the vector database is recovered according to the local cache database and / or the persistent database.

[0014] Optionally, the checking whether it is necessary to perform a data recovery operation on the data in the vector database includes:

[0015] Query the first feature sequence table information of the vector database, the second feature sequence table information of the local cache database, and the third feature sequence table information of the persistent database;

[0016] If the first feature sequence table information is consistent with the second feature sequence table information and the third feature sequence table information, no data recovery operation is required;

[0017] If the first feature sequence table information is inconsistent with the second feature sequence table information or the third feature sequence table information, a data recovery operation is required.

[0018] Optionally, recovering the missing data in the vector database according to the local cache database and / or the persistent database includes:

[0019] If the first feature sequence list information is inconsistent with the third feature sequence list information, determining missing feature library list information according to the first feature sequence list information and the third feature sequence list information;

[0020] The missing data in the vector database is recovered from the local cache database according to the missing feature library list information.

[0021] Optionally, the data recovery method further includes:

[0022] If there are still missing features to be restored in the missing feature library list information, data recovery is performed on the missing data in the vector database from the persistent database according to the missing features to be restored.

[0023] Optionally, the data recovery method further includes:

[0024] The missing data in the local cache database is recovered from the persistent database according to the missing features to be recovered.

[0025] Optionally, the data recovery method further includes:

[0026] During the data recovery process, data storage is suspended;

[0027] If a data recovery exception occurs, the abnormal data recovery information is recorded. After the service is restarted, data recovery continues based on the abnormal data recovery information.

[0028] On the other hand, an embodiment of the present invention provides a data storage and / or recovery device for a vector database, comprising:

[0029] at least one processor;

[0030] at least one memory for storing at least one program;

[0031] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned data storage and / or recovery method.

[0032] On the other hand, an embodiment of the present invention provides a storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to execute the above-mentioned data storage and / or recovery method.

[0033] Implementation of the embodiments of the present invention includes the following beneficial effects: in the data recovery method of the vector database in the embodiment, after the service is started, it is checked whether it is necessary to perform data recovery operation on the data in the vector database; if data recovery operation is required, the missing data in the vector database is recovered according to the local cache database and / or the persistent database, and the data consistency is checked by service startup, and rapid recovery is preferentially performed through the local cache database. If the local cache database recovery conditions are not met, data recovery can also be performed through the remote persistent database, thereby improving data recovery efficiency; in addition, the service is started by self-checking and self-recovering mode, which simplifies the recovery operation process and reduces the recovery operation cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a structural schematic diagram of a data storage and / or recovery system for a vector database provided by an embodiment of the present invention;

[0035] Figure 2 It is a schematic diagram of the structure of another vector database data storage and / or recovery system provided by an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the steps of a vector database data storage method provided by an embodiment of the present invention;

[0037] Figure 4It is a schematic diagram of the steps of another method for storing data in a vector database provided by an embodiment of the present invention;

[0038] Figure 5 It is a schematic diagram of the steps of a method for recovering data from a vector database provided by an embodiment of the present invention;

[0039] Figure 6 It is a timing diagram of a data recovery method for a vector database provided by an embodiment of the present invention;

[0040] Figure 7 It is a schematic diagram of the steps of another method for recovering data from a vector database provided by an embodiment of the present invention;

[0041] Figure 8 It is a structural schematic diagram of a data recovery method for a vector database provided by an embodiment of the present invention;

[0042] Fig. 9 It is a structural schematic diagram of a vector data ID generation rule provided by an embodiment of the present invention;

[0043] Fig.10 It is a structural schematic diagram of another method for recovering data from a vector database provided by an embodiment of the present invention;

[0044] Fig.11 It is a structural block diagram of a data storage system of a vector database provided by an embodiment of the present invention;

[0045] Fig.12 It is a structural block diagram of a data recovery system for a vector database provided by an embodiment of the present invention;

[0046] Fig.13 It is a structural block diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0048] The system architecture diagram of the implementation platform in the embodiment of the present invention is as follows Figure 1 and Figure 2As shown in the figure, the platform includes Agent service and PaaS components. One instance of Agent service corresponds to one vector library instance, which mainly manages the vector library feature library and CPU / GPU index. Agent service includes functional modules such as multi-vector database adaptation, CPU / GPU, and multiple memory vector library docking. Agent service provides Docker images and helm resource files, which can be directly uploaded to the image warehouse and then started with one click through helminstall. PaaS components include multiple databases, such as vector database, local cache database, and persistent database. Different databases need to be deployed, including but not limited to MongoDB, ClickHouse, Milvus (self-developed memory vector database), Redis, Kafka, RocksDB, etc.

[0049] An agent is a software or service running in a computer system, whose main function is to communicate and coordinate between the server and the client. An agent can run as a module or process of a server, responsible for receiving requests from the client and processing them; it can collect, analyze and transmit data, perform specific tasks or provide specific services. "Agent" is a broad and flexible concept in computer terminology, which can refer to any computing entity with autonomous perception, decision-making and action capabilities. These entities can be physical, virtual or software, and they can interact with the environment and pursue preset goals. In practical applications, agents can refer to many types of entities. For example, in software development, an agent may be a software that processes queries on behalf of an application and returns results. In network management, an agent may be a process that resides in a managed device and reports the value of a specified variable to a management station. In hardware architecture, such as Cisco hardware, an agent may be an independent processor card that provides one or more media interfaces. In addition, a multi-agent system is a computing society composed of multiple agents, which reach an acceptable solution through some negotiation mechanism to complete complex tasks or goals.

[0050] The CPU (Central Processing Unit) is the computing and control core of the computer system and the final execution unit for information processing and program execution. Its main function is to interpret computer instructions and process data in computer software. The CPU is the core component in the computer that is responsible for reading instructions, decoding instructions and executing instructions. The central processing unit mainly consists of two parts, namely the controller and the arithmetic unit, which also includes a cache memory and a data and control bus that realizes the connection between them.

[0051] GPU (graphics processing unit), also known as display core, visual processor, display chip, is a microprocessor specially used for image and graphics related calculations on computers, workstations, game consoles and some mobile devices (such as tablets, smart phones, etc.).

[0052] Docker images are built in a layered manner, and each image is composed of a series of "image layers". Docker images use a copy-on-write strategy to share images between multiple containers. Each container does not need to copy a separate image file when it starts. Instead, all image layers are mounted to a mount point in a read-only manner, and then a read-write container layer is overlaid on top. All changes during runtime, including writing and updating data and files, will be saved in this container layer. Therefore, multiple containers running from the same image contain different container layers.

[0053] Helm is a package management tool in the Kubernetes ecosystem that automates the creation, packaging, configuration, and deployment of applications and services to Kubernetes clusters.

[0054] PaaS (Platform as a Service) is a service model in cloud computing, usually referred to as platform as a service. It provides a basic platform framework on which users can build, deploy and run their own applications. Users can focus on application development and innovation without having to consider the infrastructure. PaaS components are part of the PaaS platform. They refer to a series of services or tools that help users develop, test, deploy and expand applications more efficiently. In order to meet the needs of different users, PaaS components are usually designed in a modular form, and users can flexibly select the required components according to their needs. PaaS components usually include but are not limited to the following aspects: operating environment, database service, message queue service, Web application framework, application monitoring, automated deployment, security control, etc. These components provide users with powerful capabilities and tools, allowing users to quickly build applications and respond to market changes quickly, saving a lot of time and resource costs for enterprises and individuals.

[0055] MongoDB is a database based on distributed file storage. It aims to provide a scalable, high-performance data storage solution for WEB applications. MongoDB is a product between relational databases and non-relational databases. It is the most feature-rich and relational database among non-relational databases. The biggest feature of Mongo is that it supports a very powerful query language. Its syntax is somewhat similar to an object-oriented query language. It can almost realize most of the functions of single-table queries in relational databases, and it also supports indexing data. A record in MongoDB is a document, which is a data structure consisting of field and value pairs; multiple keys and their associated values ​​are put together in order to form a document; the value of a field can include other documents, arrays, and document arrays. A collection is a group of documents, similar to a table in a relational database. A collection is schema-less, and the documents in a collection can be of various types. They have different keys and different value types, but they can be stored in the same collection, that is, documents of different modes can be placed in the same collection. In practical applications, documents are classified and stored in different collections to facilitate the management of collections, such as improving the efficiency of operations such as querying collections. In MongoDB, multiple documents constitute a collection, and multiple collections constitute a database. A MongoDB instance can host multiple databases, which are independent of each other, and each database has independent permission control. A MongoDB instance can contain a group of databases, a database can contain a group of collections, and a collection can contain a group of documents.

[0056] ClickHouse is a completely columnar distributed database management system that allows you to create tables and databases, load data, and run queries at runtime without reconfiguring and restarting the server. It supports linear expansion, simplicity, convenience, high reliability, and fault tolerance. ClickHouse supports sharding, which depends on clusters. Each cluster consists of 1 to multiple shards, and each shard corresponds to a service node of ClickHouse. The upper limit of the number of shards depends on the number of nodes.

[0057] Milvus is a vector database that provides multiple index types to sort field values ​​for efficient similarity search. In the context of a Milvus collection, an entity is a single, identifiable instance in a collection. Entities in a collection share a common set of properties that outline the structure that each entity must adhere to, including field names, data types, and any other constraints. The data provided to successfully insert an entity into a collection should contain all the schema-defined fields of the target collection.

[0058] Redis (Remote Dictionary Server), or remote dictionary service, is an open source in-memory data structure storage system that can be used as a database, cache, and message middleware. Redis supports a variety of data structures, such as strings, hash tables, lists, sets, ordered sets, bitmaps, geospatial indexes, and streams. The core of Redis is a key-value pair storage system, where each data is identified by a unique key and associated with a value, which can be one of multiple types, such as strings, lists, hashes, sets, and so on.

[0059] Kafka is an open source, high-throughput distributed messaging middleware. Message middleware is a software system based on queue and message passing technology that provides synchronous or asynchronous, reliable message transmission support for application systems in a network environment. Message middleware uses efficient and reliable message passing mechanisms for platform-independent data exchange and integrates distributed systems based on data communication. By providing message passing and message queuing models, it can expand inter-process communication in a distributed environment.

[0060] RocksDB is a high-performance, scalable, embedded, persistent, reliable, easy-to-use and customizable key-value storage library. It uses the LSM tree data structure, supports high-throughput writes and fast range queries, can be embedded in applications to achieve persistent storage, supports horizontal expansion, can be deployed on multiple servers to achieve clustered storage, has high reliability and stability, is easy to use, and can be customized and optimized according to needs.

[0061] like Figure 3 As shown, an embodiment of the present invention provides a data storage method of a vector database, which is applied to an Agent service, including:

[0062] S110 . Calling the storage interface of the vector database according to the attribute information of the vector data to be stored, and determining whether the storage interface exists.

[0063] The attribute information is determined according to the actual application, and this embodiment does not impose any specific restrictions. For example, the attribute information includes but is not limited to the data unit and partition information of the vector data to be stored. For the Milvus vector database, the attribute information includes collection and partition. A collection is the basic unit for storing vector data in Milvus. Each collection consists of multiple fields, which can include vector fields (for storing vector data) and scalar fields (such as numbers, text, etc., for storing descriptive data or metadata).

[0064] S120: If the storage interface exists, the vector data to be stored is sequentially stored in the local cache database, the vector database, and the persistent database.

[0065] If the storage interface exists, the vector data to be stored will be stored in the local cache database, the vector database and the persistent database in sequence. The specific method of data storage is determined according to the actual application, and this embodiment does not make any specific restrictions, such as shard storage.

[0066] Optionally, the storage method further includes:

[0067] S121, calculating vector instance fragmentation information according to the vector data to be stored;

[0068] S122. Slice the vector data to be stored according to the vector instance slicing information, and add vectors in batches to complete storage.

[0069] The vector data to be stored needs to be stored in different databases. Therefore, the vector data to be stored needs to calculate the vector instance sharding information corresponding to different types of databases. The vector data to be stored is processed according to the vector instance sharding information to obtain the sharding data corresponding to different databases, and the sharding data corresponding to different databases are added in vector batches to be stored in different databases.

[0070] See also Figure 4 In a specific embodiment, after obtaining the vector data to be stored, the database entry interface is called according to collectionld+partiton+feature vector to determine whether the database entry interface exists. If the database entry interface does not exist, the data storage operation is terminated; if the database entry interface exists, the vector instance sharding information is calculated, and the sharding data is processed according to the vector instance sharding information, and then the vector data to be stored is saved in the local cache database, the vector database and the persistent database in sequence in the form of vector batch addition, wherein the local cache database adopts RocksDB, the vector database adopts Milvus, and the persistent database adopts MongoDB.

[0071] The implementation of the embodiment of the present invention includes the following beneficial effects: the data storage method of the vector database in the embodiment of the present invention first calls the storage interface of the vector database according to the attribute information of the vector data to be stored, and determines whether the storage interface exists; if the storage interface exists, the vector data to be stored is stored in the local cache database, the vector database and the persistent database in sequence, and the vector data to be stored is backed up in the vector database's own memory, the local cache database, the remote persistent database and other ways to ensure data security, provide a consistency guarantee mechanism for the data recovery process to ensure data security, and improve the data security of the vector database.

[0072] See also Figure 5 The embodiment of the present invention provides a data recovery method for a vector database, which is applied to an Agent service and includes:

[0073] S210: After the service is started, check whether it is necessary to perform data recovery operation on the data in the vector database.

[0074] The service refers to the Agent service. If the data in the vector database is the same as that in the local cache database or the persistent database, it indicates that the vector database has no missing data and no data recovery operation is required. If the data in the vector database is different from that in the local cache database or the persistent database, it indicates that the vector database has missing data and data recovery operation is required. The specific method for checking whether the data in the vector database is missing is determined according to the actual application, and this embodiment does not make any specific restrictions.

[0075] S220: If data recovery is required, recover the missing data in the vector database according to the local cache database and / or the persistent database.

[0076] If data recovery is required, firstly, the missing data in the vector database is recovered according to the local cache database. If there is still missing data in the vector database, the persistent database is further used to recover the missing data in the vector database.

[0077] Optionally, check whether data recovery operations need to be performed on the data in the vector database, including:

[0078] S211, query the first feature sequence table information of the vector database, the second feature sequence table information of the local cache database, and the third feature sequence table information of the persistent database.

[0079] The first feature sequence list information represents the vector data features of the vector database, the second feature sequence list information represents the vector data features of the local cache database, and the third feature sequence list information represents the vector data features of the persistent database. The amount and type of data in each database can be determined based on the feature sequence list information.

[0080] S212A: If the first feature sequence table information is consistent with the second feature sequence table information and the third feature sequence table information, no data recovery operation is required.

[0081] If the data information represented by the first feature sequence table information is consistent with the data information represented by the second feature sequence table information, it means that the data in the vector database and the local cache database are consistent. If the data information represented by the first feature sequence table information is consistent with the data information represented by the third feature sequence table information, it means that the data in the vector database and the persistent database are consistent. Therefore, if the first feature sequence table information is consistent with the second feature sequence table information and the third feature sequence table information, it means that the data in the vector database is consistent with the data in the local cache database and the persistent database, and no data recovery operation is required.

[0082] S212B: If the first feature sequence table information is inconsistent with the second feature sequence table information or the third feature sequence table information, a data recovery operation needs to be performed.

[0083] If the data information represented by the first feature sequence table information is inconsistent with the data information represented by the second feature sequence table information, it means that the data in the vector database is inconsistent with the data in the local cache database. If the data information represented by the first feature sequence table information is inconsistent with the data represented by the third feature sequence table information, it means that the data in the vector database is inconsistent with the data in the persistent database. Therefore, if the first feature sequence table information is consistent with the second feature sequence table information or the third feature sequence table information, it means that the data in the vector database is inconsistent with the data in the local cache database or the persistent database, and data recovery operations are required.

[0084] Optionally, recovering the missing data in the vector database according to the local cache database and / or the persistent database includes:

[0085] S221, if the first feature sequence list information is inconsistent with the third feature sequence list information, determining missing feature library list information according to the first feature sequence list information and the third feature sequence list information;

[0086] S222. Recover the missing data in the vector database from the local cache database according to the missing feature library list information.

[0087] The missing feature library list information represents the data features missing from the vector database. If the first feature sequence list information is inconsistent with the third feature sequence list information, the missing feature library list information is determined according to the first feature sequence list information and the third feature sequence list information; and the missing data in the vector database is recovered from the local cache database according to the missing feature library list information. The data recovery method is determined according to the actual application, and this embodiment does not make specific restrictions. For example, the data is recovered in a concurrent manner by library, and each library reads the local cache data in batches in sequence.

[0088] See also Figure 6 In a specific embodiment, the data recovery method of the vector database is as follows:

[0089] S1, Agent service queries vector library shard library information;

[0090] S2. The vector library returns the shard library information of the vector library to the Agent service;

[0091] S3,Agent service queries the metadata information of the persistent database (MongoDB);

[0092] S4. The persistent database returns metadata information of the persistent database to the Agent service;

[0093] S5, Agent service queries the total amount of data in the cache database (RocksDB);

[0094] S6. Return the total amount of cache database (RocksDB) data to the Agent service;

[0095] S7, Agent service queries the total amount of data in the persistent database;

[0096] S8. Return the total amount of persistent database data to the Agent service;

[0097] S9. Determine whether to perform data recovery operation on the vector library according to the returned shard library information, metadata information of the persistent database, total amount of data in the cache database, and total amount of data in the persistent database;

[0098] S10, if a recovery operation is required, batch query the cache corresponding library data from the cache database according to the missing data in the vector database;

[0099] S11, return the cache corresponding library data to the Agent service;

[0100] S12, the Agent service saves and caches the vector data of the corresponding library data in batches for the vector library;

[0101] S13. The vector library returns the saved result to the Agent service.

[0102] Optionally, the data recovery method further includes:

[0103] S223: If there are still missing features to be restored in the missing feature library list information, restore the missing data in the vector database from the persistent database according to the missing features to be restored.

[0104] If there are still missing features to be restored in the missing feature library list information, it means that the data missing in the vector database is also missing in the local cache database. Therefore, consider restoring it from the persistent database. The specific method of restoring the missing data in the vector database from the persistent database is determined according to the actual application. This embodiment does not make any specific restrictions. It can be considered to use the method of restoring data from the cache database to restore the data.

[0105] Optionally, the data recovery method further includes:

[0106] S224: Recover the missing data in the local cache database from the persistent database according to the missing features to be recovered.

[0107] It is more efficient to recover the missing data in the vector database from the local cache database. Therefore, the missing data in the local cache database is recovered from the persistent database according to the missing features to be recovered. This can keep the data of the local cache database and the persistent database synchronized, and also ensure that the next data recovery can quickly recover the data from the cache.

[0108] Optionally, the data recovery method further includes:

[0109] S225, during the data recovery process, suspending data storage;

[0110] S226. If a data recovery exception occurs, record the abnormal data recovery information. After the service is restarted, continue to recover the data based on the abnormal data recovery information.

[0111] During the process of batch recovery of data in the vector database, the writing of external incremental vector data is suspended first to ensure the consistency of the recovered data; if an exception occurs during the incremental recovery process, the abnormal data recovery information is recorded. After the service is restarted, data recovery is continued based on the abnormal data recovery information to reduce the amount of data recovery for missing data.

[0112] See also Figure 7 In a specific embodiment, the process of vectorizing the data in the database is as follows:

[0113] S21. After the Agent service is started, check whether to restore the feature library vector data; query the feature library list information of the vector library (need to adapt to the self-developed memory vector library, Milvus, etc.), and query the feature library list information in the persistent database;

[0114] S22, judging whether there is data that exists in the feature library list information of the persistent database but does not exist in the feature library list information of the vector library, and if there is a missing feature library, recording the missing feature library list, and further judging and processing;

[0115] S23, query whether the feature library list information in the local cache database is consistent with the feature library list information in the remote persistent feature library. If they are consistent, refer to Figure 8 , the vector data in the missing feature library is restored through the vector cache data in the localized feature library first; during the recovery process, the shard status is set to synchronized and Kafka data consumption is not started; the vector database contains data corresponding to feature library 1, the local cache database contains data corresponding to feature 1 and feature 2, and the remote persistent database contains data corresponding to feature 1 and feature 2. Therefore, the feature library list information in the local cache database is consistent with the feature list information in the remote persistent feature library, and the data corresponding to feature 2 in the vector database is restored through the local cache database;

[0116] S24. According to the missing feature library list, restore by library concurrency (maximum 100 concurrency). Each library reads the local cache data in batches according to the id order. The local cache database synchronizes the vector database, and then saves the data in batches to the vector library in turn. Fig. 9 , where the vector data id generation rule is: 6-digit feature library code + 4-digit date (the number of days of the creation time for January 1, 2023) + 9-digit incremental serial number. The id range is obtained based on the library information and date information, and the vector data is quickly restored according to the id range.

[0117] S25. If the cached data is inconsistent with the remote persistent feature library list, refer to Fig.10, then this part of the database is restored from the vector data in the remote persistent feature database. During the recovery process, the shard status is set to being synchronized and Kafka data consumption is not started; the vector database contains data corresponding to feature database 1; the remote persistent database contains data corresponding to feature 1 and feature 2; the local cache database does not contain data corresponding to feature 2; therefore, the feature database list information in the local cache database is inconsistent with the feature database list information in the remote persistent feature database, and the data corresponding to feature 2 in the vector database is restored through the remote persistent database; similarly, based on the missing feature database list, the database is restored concurrently (maximum 100 concurrent), and each database reads data in the order of cursor id, and 1000 batches are taken and saved in the vector database in batches, and the remote persistent database synchronizes the vector database; the overall cache database recovery performance exceeds the remote persistent data recovery by 1000QPS; and at the same time, another thread is started to handle cache and persistent data synchronization to ensure that the next recovery can quickly restore data from the cache; refer to Fig.10 , start another thread to synchronize the data corresponding to feature 2 in the remote persistent database to the local cache database.

[0118] S26. If an exception occurs during the incremental recovery process, the library ID and the starting ID of the abnormal data will be recorded. After the service is restarted, self-recovery can continue from the breakpoint according to the library ID and data ID.

[0119] Implementation of the embodiments of the present invention includes the following beneficial effects: in the data recovery method of the vector database in the embodiment, after the service is started, it is checked whether it is necessary to perform data recovery operation on the data in the vector database; if data recovery operation is required, the missing data in the vector database is recovered according to the local cache database and / or the persistent database, and the data consistency is checked by service startup, and rapid recovery is preferentially performed through the local cache database. If the local cache database recovery conditions are not met, data recovery can also be performed through the remote persistent database, thereby improving data recovery efficiency; in addition, the service is started by self-checking and self-recovering mode, which simplifies the recovery operation process and reduces the recovery operation cost.

[0120] See also Fig.11 , an embodiment of the present invention provides a data storage system for a vector database, comprising:

[0121] The first module is used to call the storage interface of the vector database according to the attribute information of the vector data to be stored, and determine whether the storage interface exists;

[0122] The second module is used to store the vector data to be stored in the local cache database, vector database and persistent database in sequence if the storage interface exists.

[0123] It can be seen that the contents of the above-mentioned data storage method embodiments are all applicable to the present data storage system embodiments, the functions specifically implemented by the present data storage system embodiments are the same as those of the above-mentioned data storage method embodiments, and the beneficial effects achieved are also the same as those achieved by the above-mentioned data storage method embodiments.

[0124] See also Fig.12 , an embodiment of the present invention provides a data recovery system for a vector database, comprising:

[0125] The third module is used to check whether data recovery operation is needed for the data in the vector database after the service is started;

[0126] The fourth module is used to recover the missing data in the vector database according to the local cache database and / or the persistent database if data recovery operation is required.

[0127] It can be seen that the contents of the above data recovery method embodiments are all applicable to the present data recovery system embodiments, the functions specifically implemented by the present data recovery system embodiments are the same as those of the above data recovery method embodiments, and the beneficial effects achieved are also the same as those achieved by the above data recovery method embodiments.

[0128] An embodiment of the present invention provides a data storage and / or recovery device for a vector database, comprising:

[0129] at least one processor;

[0130] at least one memory for storing at least one program;

[0131] When at least one program is executed by at least one processor, the at least one processor implements the above-mentioned data storage and / or recovery method.

[0132] It can be seen that the contents of the above-mentioned data storage and / or recovery method embodiments are all applicable to the embodiments of the present device. The functions specifically implemented by the embodiments of the present device are the same as those of the above-mentioned data storage and / or recovery method embodiments, and the beneficial effects achieved are also the same as those achieved by the above-mentioned method embodiments.

[0133] In addition, the embodiments of the present application also disclose a computer program product or a computer program, and the computer program product or the computer program is stored in a computer-readable storage medium. The processor of the computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned data storage and / or recovery method. Similarly, the contents of the above-mentioned data storage and / or recovery method embodiments are all applicable to the present storage medium embodiment, and the functions specifically implemented by the present storage medium embodiment are the same as those of the above-mentioned method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned method embodiment.

[0134] Specifically, the computer device may be a user terminal or a server.

[0135] The present application embodiment takes the computer device as a user terminal as an example, and the details are as follows:

[0136] like Fig.13 As shown, the computer device 1300 may include an RF (Radio Frequency) circuit 1310, a memory 1320 including one or more computer-readable storage media, an input unit 1330, a display unit 1340, a sensor 1350, an audio circuit 1360, a short-range wireless transmission module 1370, a processor 1380 including one or more processing cores, and a power supply 1390. Those skilled in the art will appreciate that Figure 6 The device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0137] The RF circuit 1310 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors 1380 for processing; in addition, the data related to the uplink is sent to the base station. Generally, the RF circuit 1310 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a user identity module (SIM) card, a transceiver, a coupler, an LNA (Low Noise Amplifier), a duplexer, etc. In addition, the RF circuit 1310 can also communicate with the network and other devices through wireless communication. Wireless communication can use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, SMS (Short Messaging Service), etc.

[0138] The memory 1320 can be used to store software programs and modules. The processor 1380 executes various functional applications and data processing by running the software programs and modules stored in the memory 1320. The memory 1320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the device 1300 (such as audio data, a phone book, etc.), etc. In addition, the memory 1320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 1320 may also include a memory controller to provide the processor 1380 and the input unit 1330 with access to the memory 1320. Although Figure 6 The RF circuit 1310 is shown, but it is understandable that it is not a necessary component of the device 1300 and can be omitted as needed without changing the essence of the invention.

[0139] The input unit 1330 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 1330 may include a touch-sensitive surface 1331 and other input devices 1332. The touch-sensitive surface 1331, also known as a touch display screen or touch pad, can collect user touch operations on or near it (such as operations performed by the user using any suitable object or accessory such as a finger, stylus, etc. on or near the touch-sensitive surface 1331), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface 1331 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 1380, and can receive and execute commands sent by the processor 1380. In addition, the touch-sensitive surface 1331 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 1331, the input unit 1330 may further include other input devices 1332. Specifically, the other input devices 1332 may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0140] The display unit 1340 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the control 1300, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 1340 may include a display panel 1341. Optionally, the display panel 1341 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 1331 may be covered on the display panel 1341. When the touch-sensitive surface 1331 detects a touch operation on or near it, it is transmitted to the processor 1380 to determine the type of touch event. Subsequently, the processor 1380 provides a corresponding visual output on the display panel 1341 according to the type of touch event. Although in Figure 6 In the embodiment, the touch-sensitive surface 1331 and the display panel 1341 are implemented as two independent components to implement input and output functions, but in some embodiments, the touch-sensitive surface 1331 and the display panel 1341 can be integrated to implement input and output functions.

[0141] The computer device 1300 may also include at least one sensor 1350, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1341 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1341 and / or the backlight when the device 1300 is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used for applications that recognize the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the device 1300, they will not be repeated here.

[0142] The audio circuit 1360, the speaker 1361, and the microphone 1362 can provide an audio interface between the user and the device 1300. The audio circuit 1360 can transmit the received audio data to the speaker 1361 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1362 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1360 and converted into audio data, and then the audio data is processed by the output processor 1380 and sent to another control device through the RF circuit 1310, or the audio data is output to the memory 1320 for further processing. The audio circuit 1360 may also include an earplug jack to provide communication between an external headset and the device 1300.

[0143] The short-distance wireless transmission module 1370 may be a WIFI (wireless fidelity) module, a Bluetooth module, an infrared module, etc. The device 1300 may transmit information with a wireless transmission module provided on a competing device through the short-distance wireless transmission module 1370 .

[0144] The processor 1380 is the control center of the device 1300. It uses various interfaces and lines to connect various parts of the entire control device. By running or executing software programs and / or modules stored in the memory 1320, and calling data stored in the memory 1320, it executes various functions of the device 1300 and processes data, thereby monitoring the control device as a whole. Optionally, the processor 1380 may include one or more processing cores; optionally, the processor 1380 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1350.

[0145] The device 1300 also includes a power supply 1390 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 1380 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 1390 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0146] Although not shown, the device 1300 may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0147] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0148] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0149] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0150] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0153] The step numbers in the above method embodiment are only provided for the convenience of explanation and description, and no limitation is imposed on the order of the steps. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0154] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A data storage method for a vector database, characterized in that: include: Calling the storage interface of the vector database according to the attribute information of the vector data to be stored, and determining whether the storage interface exists; If the storage interface exists, the vector data to be stored is sequentially stored in the local cache database, the vector database and the persistent database.

2. The data storage method according to claim 1, characterized in that: The storage method further comprises: Calculate vector instance fragmentation information according to the vector data to be stored; The vector data to be stored is processed by sharding data according to the vector instance sharding information, and vectors are added in batches to complete storage.

3. A data recovery method for a vector database, characterized in that: include: After the service is started, check whether data recovery operations need to be performed on the data in the vector database; If a data recovery operation is required, the missing data in the vector database is recovered according to the local cache database and / or the persistent database.

4. The data recovery method according to claim 3, characterized in that: The checking whether it is necessary to perform a data recovery operation on the data in the vector database includes: Query the first feature sequence table information of the vector database, the second feature sequence table information of the local cache database, and the third feature sequence table information of the persistent database; If the first feature sequence table information is consistent with the second feature sequence table information and the third feature sequence table information, no data recovery operation is required; If the first feature sequence table information is inconsistent with the second feature sequence table information or the third feature sequence table information, a data recovery operation is required.

5. The data recovery method according to claim 4, characterized in that: Performing data recovery on the missing data in the vector database according to the local cache database and / or the persistent database includes: If the first feature sequence list information is inconsistent with the third feature sequence list information, determining missing feature library list information according to the first feature sequence list information and the third feature sequence list information; The missing data in the vector database is recovered from the local cache database according to the missing feature library list information.

6. The data recovery method according to claim 5, characterized in that: The data recovery method further includes: If there are still missing features to be restored in the missing feature library list information, data recovery is performed on the missing data in the vector database from the persistent database according to the missing features to be restored.

7. The data recovery method according to claim 6, characterized in that: The data recovery method further includes: The missing data in the local cache database is recovered from the persistent database according to the missing features to be recovered.

8. The data recovery method according to any one of claims 3 to 7, characterized in that: The data recovery method further comprises: during the data recovery process, suspending data storage; If a data recovery exception occurs, the abnormal data recovery information is recorded. After the service is restarted, data recovery continues based on the abnormal data recovery information.

9. A data storage and / or recovery device for a vector database, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 8.

10. A storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 8 when executed by the processor.

Images