Vector database scene-oriented updating method and device and electronic equipment
By obtaining the data difference information of newly added and existing vectors in the vector database and selecting the appropriate update mode, the problem of slow clustering algorithm during the vector database update process is solved, and more efficient vector index reconstruction and resource utilization are achieved.
Patent Information
- Application Number
- CN202411978105.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-29
AI Technical Summary
The clustering algorithm is extremely slow during the vector database update process, resulting in inefficient vector index reconstruction.
By obtaining the data difference information of the new vector and the existing vector, select the appropriate update mode, including adding new vectors to the existing class cluster in the vector database, indexing and reconstruction with the center point of the cluster as the starting center point, or randomly selecting the starting center point from the new vector and vector database for clustering.
It improves the flexibility and efficiency of vector database updates, ensures that new vectors can be updated in time to the vector database, and at the same time optimizes resource utilization and improves update speed.
Smart Images
Figure CN119903066A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, specifically to the field of vector database technology, and in particular to an update method, device and electronic device for a vector database scenario. Background Art
[0002] Clustering is widely used in vector index construction. However, the clustering algorithm itself is extremely slow. For the update of vector database, how to improve the reconstruction of vector index is a pain point that needs to be solved. Summary of the invention
[0003] The present disclosure provides an update method, device and electronic device for a vector database scenario.
[0004] According to one aspect of the present disclosure, an update method for a vector database scenario is provided, including: obtaining a newly added vector and a first data volume of the newly added vector; obtaining a second data volume of an existing vector in the vector database; obtaining difference information between the first data volume and the second data volume; and updating the vector database based on the difference information and the newly added vector.
[0005] According to another aspect of the present disclosure, an update device for a vector database scenario is provided, including: a first acquisition module, used to acquire a newly added vector and a first data volume of the newly added vector; a second acquisition module, used to acquire a second data volume of an existing vector in the vector database; a third acquisition module, used to acquire difference information between the first data volume and the second data volume; and an update module, used to update the vector database according to the difference information and the newly added vector.
[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the update method for the vector database scenario described in the above-mentioned first embodiment.
[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, on which a computer program / instructions are stored, and the computer instructions are used to enable the computer to execute the update method for the vector database scenario described in the above-mentioned embodiment.
[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the updating method for the vector database scenario described in the above-mentioned first embodiment is implemented.
[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0011] Figure 1 A flowchart of an update method for a vector database scenario provided by an embodiment of the present disclosure;
[0012] Figure 2 A flowchart of another updating method for a vector database scenario provided by an embodiment of the present disclosure;
[0013] Figure 3 A flowchart of another updating method for a vector database scenario provided by an embodiment of the present disclosure;
[0014] Figure 4 A schematic diagram of the structure of an updating device for a vector database scenario provided by an embodiment of the present disclosure;
[0015] Figure 5 The present invention is a block diagram of an electronic device for implementing the updating method for the vector database scenario according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0016] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0017] The following describes the updating method, device and electronic device for the vector database scenario according to the embodiments of the present disclosure with reference to the accompanying drawings.
[0018] Figure 1A flow chart of an update method for a vector database scenario provided in an embodiment of the present application. It should be noted that the update method for a vector database scenario in this embodiment can be applied to an update device for a vector database scenario. In a feasible implementation, the device can be configured in an electronic device or a chip so that the electronic device or the chip can perform updates for the vector database scenario. The electronic device can be a terminal device, or a server, a database manager, etc. In a feasible implementation, the device can also be software in an electronic device, etc.
[0019] like Figure 1 As shown, the updating method for the vector database scenario may include but is not limited to the following steps:
[0020] S101: Obtain a newly added vector and a first data volume of the newly added vector.
[0021] In some embodiments, the vector database updating device may obtain an update request for the vector database.
[0022] In some embodiments, the update request may include at least a newly added vector and a newly added vector.
[0023] In some embodiments, the update request may include at least a newly added vector and a first data volume of the newly added vector.
[0024] In some embodiments, the updating device of the vector database may receive an update request sent by the target device.
[0025] In some embodiments, the updating device of the vector database may periodically receive an update request sent by the target device.
[0026] In some embodiments, the updating device of the vector database may receive an update request sent by the target device in real time.
[0027] In some embodiments, the updating device of the vector database may receive an update request sent by the target device when a set condition is met.
[0028] In some embodiments, the target device can determine the number of newly added vectors. In response to the number of newly added vectors reaching a set number, the target device can send an update request to the update device of the vector database. Accordingly, the update device of the vector database can receive the update request sent by the target device when the set conditions are met.
[0029] In some embodiments, the target device may be a sample collection device, and after collecting a set number of samples, an update request may be sent to an update device of the vector database.
[0030] In some embodiments, the update request may instruct an update device of the vector database to obtain a newly added vector and / or a first data volume of the newly added vector from other devices.
[0031] In some embodiments, the target device monitors the update operation of the vector database, and in response to monitoring the update operation, may send an update request to the update device of the vector database, and the update device of the vector database may obtain the newly added vector based on the update request.
[0032] In some embodiments, the target device may be a terminal device or a search engine device.
[0033] In some embodiments, after receiving the update request, the updating device of the vector database may pull the newly added samples from the sample collection device and determine the newly added vectors according to the newly added samples.
[0034] In some embodiments, after receiving the update request, the updating device of the vector database can directly pull the vector corresponding to the newly added sample from the sample collection device as the newly added vector. In other words, the sample collection device can perform feature extraction and vector encoding on the sample to obtain the vector corresponding to the newly added sample.
[0035] In some embodiments, the updating device of the vector database can collect samples by itself. After the samples are collected for a set time or a set number, the vector corresponding to the newly added sample, that is, the newly added vector, can be directly obtained, and the first data volume of the newly added vector can be determined.
[0036] S102, obtaining a second amount of data of existing vectors in the vector database.
[0037] In some embodiments, the updating device of the vector database can determine the second data volume of the existing vectors according to the log. Optionally, each time the vector database vector is updated, the number of vectors existing in the vector database recorded in the log can be updated. For example, the update operation can include operations such as deletion, addition, and modification. After deletion and addition, the number of vectors existing in the vector database will be updated, so that the second data volume can be obtained from the log.
[0038] In some embodiments, the updating device of the vector database may perform real-time statistics on the existing vectors to obtain the second data volume.
[0039] S103: Obtain difference information between the first data volume and the second data volume.
[0040] S104: Update the vector database according to the target difference information and the newly added vector.
[0041] In some embodiments, the difference information between the first data volume and the second data volume is obtained, and the size relationship between the first data volume and the second data volume and the difference in data volume are determined through the difference information. That is, based on the difference information, it can be determined whether the first data volume is much smaller than the second data volume, or whether the first data volume is much larger than the second data volume, or whether the first data volume is not much different from the second data volume.
[0042] In some embodiments, the size of the first data volume and the second data volume, as well as the data volume difference between the two, can be determined based on the difference information. Further, different update modes can be selected under different gap conditions to adapt to specific scenarios, thereby improving the flexibility and efficiency of vector database update.
[0043] In some embodiments, a target mode selection condition satisfied by the difference information is determined, and further, a target update mode of the vector database is determined based on the target mode selection condition satisfied by the difference information.
[0044] In some embodiments, multiple update modes can be configured for the vector database, and corresponding mode selection conditions can be configured for each update mode. Optionally, the difference information can be compared with each mode selection condition, and in response to satisfying one of the mode selection conditions, the update mode corresponding to the mode selection condition is used as the target update mode to be adopted when the vector database is updated.
[0045] Further, after the target update mode is obtained, the vector database can be updated based on the target update mode and the newly added vector. It is understandable that updating the vector database can include adding the newly added vector to the vector database, and further, the clustering of the vector database can be updated, and the index of the vector database can be updated according to the result of the clustering update.
[0046] In an embodiment of the present application, a suitable update mode can be selected based on the difference in data volume between the newly added vectors and the existing vectors, which can not only ensure that the newly added vectors can be updated to the vector database, but also make the update of the vector database more flexible and diverse, which is conducive to the rational use of resources.
[0047] Figure 2 The following is a flow chart of an update method for a vector database scenario provided by an embodiment of the present application. Figure 2 As shown, the updating method for the vector database scenario may include but is not limited to the following steps:
[0048] S201, obtaining a newly added vector and a first data volume of the newly added vector.
[0049] S202, obtaining a second amount of data of existing vectors in the vector database.
[0050] For a detailed description of steps S201 to S202, please refer to the relevant contents in the above embodiment, which will not be repeated here.
[0051] S203: Obtain difference information between the first data volume and the second data volume.
[0052] In some embodiments, a difference between the first data amount and the second data amount may be determined, and the difference may be used as difference information between the first data amount and the second data amount.
[0053] In some embodiments, a ratio between the first data amount and the second data amount may be determined, and the ratio may be used as difference information between the first data amount and the second data amount.
[0054] S204: Determine the target mode selection condition satisfied by the difference information.
[0055] In some embodiments, newly added vectors are directly added to existing clusters in the vector database as the first update mode.
[0056] In some embodiments, the vector database is updated starting from the existing clusters in the vector database as the second update mode.
[0057] In some embodiments, the vector database is globally rebuilt as a third update mode.
[0058] In some embodiments, each update mode may be pre-configured with a corresponding mode selection condition. Optionally, the first update mode corresponds to the first mode selection condition; the second update mode corresponds to the second mode selection condition; and the third update mode corresponds to the third mode selection condition.
[0059] S205, determining a target update mode of the vector database according to a target mode selection condition satisfied by the difference information.
[0060] In some embodiments, after the difference information is acquired, the difference information may be compared with the mode selection condition of each update mode configuration to determine the target mode selection condition satisfied by the difference information.
[0061] In some embodiments, if the difference information is a ratio, the first mode selection condition of the first update mode is: the ratio is less than the first set value; the second mode selection condition of the second update mode is: the ratio is greater than or equal to the first set value and less than the second set value; the third mode selection condition of the third update mode is: the ratio is greater than or equal to the third set value.
[0062] In some embodiments, if the difference information is a difference value, the first mode selection condition of the first update mode is: the difference value is less than the first set difference value; the second mode selection condition of the second update mode is: the difference value is greater than or equal to the first set difference value, and less than the second set difference value; the third mode selection condition of the third update mode is: the ratio is greater than or equal to the third set difference value.
[0063] By setting different mode selection conditions for different update modes, not only can the update methods of the vector database be enriched, but also different update modes occupy different resources. The appropriate update mode can be selected according to the actual situation, thereby achieving reasonable use of resources.
[0064] S206: Update the vector database according to the target update mode and the newly added vectors.
[0065] For a detailed description of step S206, please refer to the relevant contents in the above embodiment, which will not be repeated here.
[0066] In an embodiment of the present application, a suitable update mode can be selected based on the difference in data volume between the newly added vectors and the existing vectors, which can not only ensure that the newly added vectors can be updated to the vector database, but also make the update of the vector database more flexible and diverse, which is conducive to the rational use of resources.
[0067] Figure 3 The following is a flow chart of an update method for a vector database scenario provided by an embodiment of the present application. Figure 3 As shown, the updating method for the vector database scenario may include but is not limited to the following steps:
[0068] S301: Obtain a newly added vector and a first data volume of the newly added vector.
[0069] S302, obtaining a second amount of data of existing vectors in the vector database.
[0070] S303: Obtain a ratio of the first data volume to the second data volume.
[0071] For a detailed description of steps S301 to S303, please refer to the relevant contents in the above embodiment, which will not be repeated here.
[0072] S304: In response to the ratio being less than a first set value, adding a new vector to an existing cluster in the vector database.
[0073] In some embodiments, the cluster center of the existing cluster is determined, and the distance between the newly added vector and the cluster center is obtained. Further, based on the distance, the target cluster to which the newly added vector belongs is determined from the existing cluster, and the newly added vector is added to the target cluster, and the index of the vector database is maintained. Optionally, the similarity between the newly added vector and the cluster center can be calculated as the distance between the two. For example, cosine similarity, Euclidean distance, etc.
[0074] In some embodiments, based on the distance between the new vector and the cluster center point, the existing cluster closest to the new vector, that is, the existing cluster with the greatest similarity to the new vector, is determined as the target cluster to which the new vector belongs.
[0075] In some embodiments, the existing clusters are sorted from near to far according to the distance, and the existing cluster ranked first is selected as the target cluster to which the newly added vector belongs.
[0076] In an embodiment of the present application, when the number of newly added vectors is relatively small compared to the existing vectors, the newly added vectors will not have a large impact on the clusters of the vector database. The newly added vectors can be directly added to the corresponding existing clusters, thereby increasing the update speed of the vector database without occupying more resources.
[0077] S305, in response to the ratio being greater than or equal to the first set value and less than the second set value, obtaining the cluster center point of the existing clusters in the vector database, and reindexing the newly added vectors and the existing vectors in the vector database with the cluster center point as the starting center point.
[0078] In some embodiments, the ratio is greater than or equal to the first set value and less than the second set value, indicating that the number of newly added vectors will not be too large, and the existing clusters have a certain reference for the newly added vectors. In this case, the cluster center point can be used as the starting center point to re-index the newly added vectors and the existing vectors in the vector database.
[0079] In some embodiments, the number of clustering iterations during index reconstruction is determined according to the ratio. That is, a larger ratio indicates that the data volume of the two is far different, and the number of iterations can be more, while a smaller ratio indicates that the data volume of the two is less different, and the number of iterations can be less.
[0080] In an embodiment of the present application, in a scenario with a certain amount of newly added vectors, the newly added vectors will have a certain impact on the clusters of the vector database, but the impact will not be too great. In this scenario, the newly added vectors can be updated to the vector database based on the existing clusters. Without occupying more resources, the update speed of the vector database can be improved, and the clusters of the vector database will also be updated, thereby ensuring the index accuracy of the vector database.
[0081] S306, in response to the ratio being greater than or equal to the third set value, randomly selecting a starting center point from the newly added vector and the vector database, and reindexing the newly added vector and the existing vectors in the vector database based on the randomly selected starting center point.
[0082] In some embodiments, if the ratio is greater than or equal to the third set value, it means that the number of newly added vectors is large, far exceeding the existing vectors. At this time, the existing clusters are no longer representative of the newly added vectors. In this case, a starting center point is randomly selected from the newly added vectors and the vector database, and based on the randomly selected starting center point, a new clustering process is performed on the newly added vectors and the existing vectors in the vector database to obtain the final clustering result, and the index of the vector database is re-established based on the final clustering result. In the embodiment of the present application, since there are many newly added vectors, re-clustering can ensure that the index construction of the vector database is more accurate.
[0083] It can be understood that the first setting value is smaller than the second setting value, and the second setting value is smaller than the third setting value.
[0084] In an embodiment of the present application, a suitable update mode can be selected based on the difference in data volume between the newly added vectors and the existing vectors, which can not only ensure that the newly added vectors can be updated to the vector database, but also make the update of the vector database more flexible and diverse, which is conducive to the rational use of resources.
[0085] Corresponding to the update methods for vector database scenarios provided in the above-mentioned embodiments, an embodiment of the present disclosure also provides an update device for vector database scenarios. Since the update device for vector database scenarios provided in the embodiment of the present disclosure corresponds to the update methods for vector database scenarios provided in the above-mentioned embodiments, the implementation methods of the above-mentioned update methods for vector database scenarios are also applicable to the update device for vector database scenarios provided in the embodiment of the present disclosure, and will not be described in detail in the following embodiments.
[0086] Figure 4 A schematic diagram of the structure of an updating device for a vector database scenario provided in an embodiment of the present disclosure.
[0087] like Figure 4 As shown, the updating device 400 for the vector database scenario according to the embodiment of the present disclosure includes a first acquisition module 401 , a second acquisition module 402 , a third acquisition module 403 and an updating module 404 .
[0088] A first acquisition module 401, configured to acquire a newly added vector and a first data volume of the newly added vector;
[0089] A second acquisition module 402 is used to acquire a second amount of data of existing vectors in the vector database;
[0090] A third acquisition module 403 is used to acquire difference information between the first data volume and the second data volume;
[0091] The updating module 404 is used to update the vector database according to the difference information and the newly added vector.
[0092] In one embodiment of the present disclosure, the update module 404 is further used to: determine a target update mode of the vector database according to the difference information; and update the vector database according to the target update mode and the newly added vector.
[0093] In one embodiment of the present disclosure, the update module 404 is further used to: determine a target mode selection condition satisfied by the difference information; and determine a target update mode of the vector database according to the target mode selection condition satisfied by the difference information.
[0094] In one embodiment of the present disclosure, the updating module 404 is further configured to: in response to the difference information being smaller than the first mode selection condition, add the newly added vector to the existing clusters in the vector database.
[0095] In one embodiment of the present disclosure, the update module 404 is also used to: determine the cluster center point of the existing cluster and obtain the distance between the newly added vector and the cluster center point; determine the target cluster to which the newly added vector belongs from the existing cluster based on the distance; add the newly added vector to the target cluster and maintain the index of the vector database.
[0096] In one embodiment of the present disclosure, the update module 404 is also used to: in response to the difference information satisfying the second mode selection condition, obtain the cluster center point of the existing cluster in the vector database; and use the cluster center point as the starting center point to reindex the newly added vector and the existing vector in the vector database.
[0097] In one embodiment of the present disclosure, the update module 404 is also used to: determine the number of clustering iterations in the index reconstruction process based on the difference information; take the cluster center point as the starting center point, re-cluster the newly added vectors and the existing vectors in the vector database according to the number of clustering iterations, and rebuild the index based on the clustering results.
[0098] In one embodiment of the present disclosure, the update module 404 is also used to: in response to the difference information satisfying the third mode selection condition, randomly select a starting center point from the newly added vector and the vector database, and reindex the newly added vector and the existing vectors in the vector database based on the randomly selected starting center point.
[0099] In an embodiment of the present application, a suitable update mode can be selected based on the difference in data volume between the newly added vectors and the existing vectors, which can not only ensure that the newly added vectors can be updated to the vector database, but also make the update of the vector database more flexible and diverse, which is conducive to the rational use of resources.
[0100] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0101] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0102] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0103] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program / instruction stored in a read-only memory (ROM) 502 or a computer program / instruction loaded from a storage unit 506 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0104] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506 such as a keyboard, a mouse, etc.; an output unit 507 such as various types of displays, speakers, etc.; a storage unit 508 such as a disk, an optical disk, etc.; and a communication unit 509 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0105] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the update method for the vector database scenario. For example, in some embodiments, the update method for the vector database scenario may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 506. In some embodiments, part or all of the computer program / instructions may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program / instructions are loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the update method for the vector database scenario described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the updating method for the vector database scenario in any other appropriate manner (for example, by means of firmware).
[0106] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs / instructions that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0107] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0108] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0110] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.
[0111] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs / instructions running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the disclosure can be achieved, and this document does not limit them here.
[0113] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. An update method for a vector database scenario, wherein: The method comprises: Acquire a newly added vector and a first data amount of the newly added vector; Acquire a second amount of data of the vectors already in the vector database; Acquire difference information between the first data volume and the second data volume; The vector database is updated according to the difference information and the newly added vector.
2. The method according to claim 1, wherein: The updating of the vector database according to the difference information and the newly added vector includes: Determining a target update mode of the vector database according to the difference information; The vector database is updated according to the target update mode and the newly added vector.
3. The method according to claim 2, wherein: Determining a target update mode of the vector database according to the difference information includes: Determining a target mode selection condition satisfied by the difference information; The target update mode of the vector database is determined according to the target mode selection condition satisfied by the difference information.
4. The method according to any one of claims 1 to 3, wherein: The updating of the vector database according to the difference information and the newly added vector includes: In response to the difference information being less than a first mode selection condition, the newly added vector is added to an existing cluster in the vector database.
5. The method according to claim 4, wherein: The adding the newly added vector to the existing cluster in the vector database includes: Determine the cluster center point of the existing cluster, and obtain the distance between the newly added vector and the cluster center point; Determining, from the existing clusters, a target cluster to which the newly added vector belongs according to the distance; The newly added vector is added to the target cluster, and the index of the vector database is maintained.
6. The method according to any one of claims 1 to 3, wherein: The updating of the vector database according to the difference information and the newly added vector includes: In response to the difference information satisfying the second mode selection condition, obtaining the cluster center point of the existing cluster in the vector database; Taking the cluster center point as the starting center point, reindexing is performed on the newly added vector and the existing vectors in the vector database.
7. The method according to claim 5, wherein: The step of reindexing the newly added vector and the existing vectors in the vector database with the cluster center point as the starting center point includes: Determining the number of clustering iterations during index reconstruction according to the difference information; Taking the cluster center point as the starting center point, the newly added vector and the existing vectors in the vector database are re-clustered according to the number of clustering iterations, and index reconstruction is performed based on the clustering result.
8. The method according to any one of claims 1 to 3, wherein: The updating of the vector database according to the difference information and the newly added vector includes: In response to the difference information satisfying the third mode selection condition, a starting center point is randomly selected from the newly added vector and the vector database, and based on the randomly selected starting center point, the newly added vector and the existing vectors in the vector database are indexed and rebuilt.
9. An updating device for a vector database scenario, wherein: The device comprises: A first acquisition module, used to acquire a newly added vector and a first data volume of the newly added vector; A second acquisition module, used to acquire a second amount of data of existing vectors in the vector database; A third acquisition module, used to acquire difference information between the first data volume and the second data volume; An updating module is used to update the vector database according to the difference information and the newly added vector.
10. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.
11. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.
12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.