Incremental data processing method, device and server in vector retrieval system

Abstract

The application discloses a kind of vector retrieval system incremental data processing method, device and electronic equipment, the method includes: the vector space of vector retrieval system is split processing, obtains multiple subspaces;Utilize clustering algorithm to the multiple subspaces respectively clustering processing;Locking processing is carried out to the write request of each sub-space after clustering processing, the working mode of lock is the use of optimistic lock and pessimistic lock switching, wherein, when the number of write request meets preset condition, the lock is switched to optimistic lock, when the number of write request does not meet the preset condition, the lock is switched to pessimistic lock. Embodiments of the present application can be used to quickly update the vector retrieval index of incremental data in vector retrieval system.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to an incremental data processing method, apparatus, server and storage medium in a vector retrieval system. Background Technology

[0002] With the rapid development of internet technology, various new functions and services based on vector retrieval (or vector search) technology are constantly being launched, such as image search, image and text recommendation, product recommendation, video search, etc., striving to meet the diverse needs of users. These functional applications are generally characterized by large data processing volumes and high dimensionality. In typical use cases, the incremental data accumulation of vector retrieval indexes is relatively slow. Therefore, the design of vector retrieval indexing algorithms tends to build indexes based on existing data, and rebuild the index after incremental data has accumulated to a certain extent. (See reference...) Figure 1 While this approach is low-cost and easy to design indexes, its processing performance deteriorates significantly in scenarios with large amounts of incremental data. For example, in specific scenarios such as second-hand e-commerce, incremental data accumulates rapidly, and existing data needs to be replaced quickly, making the existing processing methods unable to meet the needs. Summary of the Invention

[0003] In view of this, embodiments of this application provide an incremental data processing method, apparatus, server, and storage medium in a vector retrieval system to solve at least one of the above technical problems.

[0004] In a first aspect, embodiments of this application provide an incremental data processing method in a vector retrieval system, including:

[0005] The vector space of the vector retrieval system is divided into multiple subspaces.

[0006] Clustering algorithms are used to cluster the multiple subspaces respectively;

[0007] Write requests to each subspace after clustering are locked. The lock operates by switching between optimistic and pessimistic locking modes. When the number of write requests meets a preset condition, the lock switches to optimistic locking; when the number of write requests does not meet the preset condition, the lock switches to pessimistic locking.

[0008] Optionally, the sum of the probability that the lock is an optimistic lock and the probability that the lock is a pessimistic lock is 1, and the probability that the lock is an optimistic lock is related to the number of write requests.

[0009] Optionally, the probability that the lock is an optimistic lock for:

[0010]

[0011] Where the subscript i represents the i-th subspace, t is the time interval, and n is the time window containing the current time interval. For the number of write requests, The number of conflicts that occur in optimistic locking.

[0012] Optionally, for the i-th subspace, the preset condition is the number of write requests to the i-th subspace at the current time. The probability that the lock is an optimistic lock Greater than or equal to 0.5.

[0013] Optionally, the vector retrieval system includes a product search system of an e-commerce transaction platform.

[0014] Secondly, embodiments of this application provide an incremental data processing device for a vector retrieval system, comprising:

[0015] The segmentation module is used to segment the vector space of the vector retrieval system to obtain multiple subspaces;

[0016] The clustering processing module is used to perform clustering processing on the multiple subspaces respectively using a clustering algorithm;

[0017] The lock processing module is used to lock write requests for each subspace after clustering. The lock operates by switching between optimistic and pessimistic locking modes. When the number of write requests meets a preset condition, the lock switches to optimistic locking; when the number of write requests does not meet the preset condition, the lock switches to pessimistic locking.

[0018] Thirdly, embodiments of this application provide a server, which includes a processor and a memory storing computer program instructions; the server executes the computer program instructions to implement the method described above.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the method described above.

[0020] In this embodiment, the lock is configured to switch between optimistic and pessimistic locking. When the number of write requests meets the condition, it switches to optimistic locking; otherwise, it switches to pessimistic locking. This allows the vector retrieval system to automatically adjust the lock mode according to the number of write requests, maximizing its role at different stages and improving the overall performance of the vector retrieval system. Furthermore, the locking granularity in this embodiment is reduced from a general global space to a subspace. Multiple locks implement the aforementioned switching control on each subspace, which can significantly improve the overall concurrent read and write capabilities of the system. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This diagram illustrates a process of building a vector retrieval index based on existing data.

[0023] Figure 2 A flowchart of an incremental data processing method in a vector retrieval system according to an embodiment of this application is shown.

[0024] Figure 3 This illustration shows a process of updating the vector retrieval index using incremental data according to an embodiment of this application.

[0025] Figure 4 A structural block diagram of the incremental data processing device in the vector retrieval system according to an embodiment of this application is shown.

[0026] Figure 5 A schematic diagram of a server used to implement the incremental data processing method in the vector retrieval system of this application is shown. Detailed Implementation

[0027] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0028] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0029] Before describing the implementation methods of this application, a brief introduction will be given on the concepts and related content that may be involved in the embodiments of this application.

[0030] In computer technology, when concurrent transactions may access a resource at the same time, it may lead to data inconsistency. A mechanism is needed to sequence data access to ensure the consistency of database data. "Lock" is one such mechanism. Generally, "lock" is considered to be a synchronization mechanism used to forcibly restrict resource access when performing multi-threaded operations. There are many types of "locks". For example, according to different granularities, there are table locks and row locks. According to different ways of using locks, there are optimistic locks and pessimistic locks, and so on.

[0031] Among them, pessimistic locking, also known as pessimistic concurrency control (PCC), refers to a concurrency control method that takes a pessimistic attitude towards data modification. Every time data is read, it is assumed that other threads will modify the data and there will be data conflicts. Therefore, the data is locked every time it is read or written, and other threads will be blocked and suspended when accessing the data.

[0032] Optimistic locking, also known as optimistic concurrency control (OCC), is a concurrency control method that takes an optimistic attitude towards data modifications. It assumes that data modifications will not cause conflicts in most cases, and data conflicts are only detected when the data is committed for update, thus resulting in high data throughput.

[0033] Therefore, it is generally believed that optimistic locking is suitable for scenarios with few writes and few conflicts, while pessimistic locking is suitable for scenarios with many writes and many conflicts.

[0034] The purpose of vector space partitioning (or vector segmentation) is to discretize a continuous vector space, which can optimize the speed of vector distance calculation. Vectors in vector retrieval scenarios are usually high-dimensional (e.g., 256-dimensional, 512-dimensional, etc.). To improve the speed of vector retrieval and recall, vector partitioning can be performed. This can typically be done as follows: each high-dimensional vector in the vector space is divided into M segments (M must be divisible by the vector's dimension W), resulting in M ​​W / M dimensional subspaces. Then, clustering (e.g., K-means clustering) is performed on the data in each subspace to obtain the quantized results of each subspace, which are used to calculate vector distance during vector search.

[0035] To address the problem that vector retrieval indexes for incremental data cannot be updated in a timely manner due to the rapid accumulation of incremental data in certain application scenarios, embodiments of this application propose a method for processing incremental data in a vector retrieval system. Figure 1 A flowchart of the method is shown, including:

[0036] S102, the vector space of the vector retrieval system is divided to obtain multiple subspaces;

[0037] S104, Clustering algorithms are used to perform clustering processing on the multiple subspaces respectively;

[0038] S106, lock the write requests of each subspace after clustering. The lock works by switching between optimistic locking and pessimistic locking. When the number of write requests meets the preset condition, the lock switches to optimistic locking. When the number of write requests does not meet the preset condition, the lock switches to pessimistic locking.

[0039] The main objective of this application embodiment is to add locks to write requests in a vector retrieval system. By controlling the processing of write requests through a locking mechanism, the vector space is preprocessed as follows: the vector space in the vector retrieval system is divided into several low-dimensional vector subspaces, and then clustering algorithms are used to cluster these subspaces, preparing for subsequent locking processing. During locking, write requests in each preprocessed subspace are locked, and the lock operates in a "switching between optimistic and pessimistic locking" mode. That is, the lock in this application embodiment is configured to switch between optimistic and pessimistic locking, with the switching condition related to the number of write requests. For ease of description, this lock is referred to as a "hybrid lock." When the number of write requests meets the condition, the hybrid lock switches to optimistic locking; when the number of write requests does not meet the condition, the hybrid lock switches to pessimistic locking. In this way, the vector retrieval system can automatically adjust the lock mode or type according to the number of write requests, allowing the hybrid lock to maximize its role at different stages and improving the overall performance of the vector retrieval system.

[0040] For example, in some scenarios, data read / write state transitions are frequent. When the number of write requests is small or there are more reads than writes, the hybrid lock is an optimistic lock, contributing high throughput. When the number of write requests increases rapidly, or there are more reads than writes or fewer reads than writes, the optimistic lock switches to a pessimistic lock to reduce the overhead caused by data conflicts. This allows for reasonable concurrency control of data read / write in both the initial and final stages, thus optimizing and improving the overall performance of the vector retrieval system. Furthermore, in this embodiment, the locking granularity of the hybrid lock is reduced from a general global space to multiple low-dimensional subspaces. Multiple hybrid locks implement the aforementioned switching control on each subspace, which can significantly improve the overall concurrent read / write capability of the system.

[0041] Based on the above advantages, in application scenarios where incremental data accumulates rapidly (such as vector retrieval systems of second-hand goods trading platforms), the hybrid locking mechanism of this application embodiment can quickly update incremental data and complete the process of updating the vector retrieval index of incremental data without having to replace existing data with incremental data and then rebuild the index. This can significantly shorten the waiting time for updating the index of the vector retrieval system and improve the overall performance of the system.

[0042] According to an embodiment of this application, optionally, the sum of the probability that the lock is an optimistic lock and the probability that the lock is a pessimistic lock is 1, and the probability that the lock is an optimistic lock is related to the number of write requests. Here, setting the sum of the probabilities that the hybrid lock is an optimistic lock and a pessimistic lock to 1 allows the hybrid lock mode to switch between the two types of locks. Furthermore, the probability that the hybrid lock is an optimistic lock can be obtained based on the number of write requests, so the hybrid lock mode switching can be triggered by changes in the number of write requests. This achieves automatic and rapid switching between the two types of locks, and is particularly suitable for application scenarios where the number of write requests changes frequently and incremental data accumulates rapidly.

[0043] According to an embodiment of this application, optionally, the probability that the lock is an optimistic lock can be calculated according to the following formula:

[0044]

[0045] Where the subscript i represents the i-th subspace, t is the time interval, and n is the time window containing the current time interval. For the number of write requests, This represents the number of data conflicts that occur during optimistic locking.

[0046] Specifically, for a single subspace, considering that the probability range is between [0,1] and the probability change should be smooth to prevent service instability caused by sharp increases or decreases, the state within a time window before the current time t should also be considered. Therefore, assuming n is the length of the time window, the current time t can be the last time of window n; the number of data collisions occurring in the i-th subspace at time t due to optimistic locking is... This can be detected through lock detection, from which the probability of using optimistic locking can be calculated.

[0047] According to an embodiment of this application, optionally, for the i-th subspace, the preset condition is the number of write requests to the i-th subspace at the current time. The probability that the lock is an optimistic lock Greater than or equal to 0.5. In other words, if the calculated probability value for using optimistic locking is... Then the lock for that subspace should be switched to an optimistic lock; otherwise, Switch to pessimistic locking to achieve automatic switching between the two types of locking.

[0048] Optionally, according to an embodiment of this application, the vector retrieval system includes a product search system of an e-commerce transaction platform, and the incremental data processing method is used to update the vector retrieval index of incremental data in the product search system.

[0049] For example, in a second-hand goods trading platform, the search system not only provides product search functionality but also allows sellers to upload information about second-hand goods for sale. Therefore, data read / write demands are frequent, and incremental data accumulates extremely quickly. If the vector retrieval index cannot be updated in a timely manner, inconsistencies between online product data and actual data can easily arise. Incorrect online data negatively impacts user experience. The incremental data processing method described in this application implements an optimistic and pessimistic locking switching strategy for system data read / write, enabling rapid updates to the vector retrieval index for incremental data. This provides users with highly real-time and accurate vector retrieval services, improving user experience and enhancing the overall competitiveness of the system.

[0050] The embodiments of this application propose a mechanism for segmented locking of vector subspaces and automatic adjustment of "optimistic locking-pessimistic locking". By adopting at least one of the above embodiments of this application, the locking mode can be quickly adjusted, improving system performance. It is especially suitable for data maintenance of second-hand trading systems, providing a high real-time and high-precision vector retrieval system for second-hand trading platforms.

[0051] The above describes various implementations of the incremental data processing method in the vector retrieval system of this application through multiple embodiments. The following describes the processing process of the embodiments of this application through specific examples.

[0052] The embodiments of this application can be implemented based on the open-source code library "Faiss". Faiss is a computational library commonly used in the field for vector search. Using Faiss, reliable and efficient vector retrieval can be performed on massive amounts of data in high-dimensional space, which can meet the performance requirements of recall modules in most recommendation systems. Using the embodiments of this application, improvements are made by adding segment locks to subspaces, etc., to update the vector retrieval index of incremental data in real time. The processing procedure is as follows.

[0053] The vector data is indexed using an index building algorithm (such as the IndexIVF method). The cluster center IDs corresponding to the vectors generated by the K-means clustering algorithm in the index building result can be stored in an array. The array is a thread-unsafe data structure, which may cause performance degradation during concurrent read and write operations.

[0054] To address the issue of thread unsafety in arrays, a high-performance optimistic lock is added to each subspace based on the spatial division results of the K-means algorithm. The subspace is represented by the cluster ID of each cluster center. The optimistic lock can achieve high concurrency performance under read-heavy and write-light conditions.

[0055] When the performance of optimistic locking changes from read-heavy to write-heavy or read-light to write-heavy, the performance of optimistic locking degrades significantly. In this case, an optimistic-to-pessimistic locking switch is implemented, degenerating the optimistic lock into a pessimistic lock. The switching strategy is as follows:

[0056] (1) At the current time t, the probability that the write lock of subspace i is an optimistic lock is: The probability of being a pessimistic lock is The number of write requests is The number of read requests is The number of data conflicts that occur with optimistic locking is

[0057] (2) Considering the state within a time window n before the current time t, we get:

[0058]

[0059] Where n represents the number of moments within the time window. This represents the frequency of data conflicts occurring at the corresponding time scale.

[0060] The following is a brief comparison of the concurrent read process before and after the improvement:

[0061] A. Before the improvement (before adding hybrid lock to the subspace):

[0062] Check if there are any write requests in the global space. If there are no write requests, read the array data directly. If there are write requests, completely block and wait for the write requests to finish.

[0063] B. Improved version (after adding a hybrid lock to the subspace):

[0064] Lock granularity is devolved to each subspace, and a hybrid mode of optimistic and pessimistic locking is used to accommodate the uncertain number of write requests. When there are few write requests, optimistic locking is more likely to be used; when there are many write requests, pessimistic locking is more likely to be used. The hybrid use of the two types of locks improves the overall throughput.

[0065] The following provides test data for vector recall of similar products in a product recommendation scenario to verify the effect. The test conditions are as follows: the data volume is 50 million (tens of millions), the vector dimension is 100, and the write QPS (data written per second) is about 1 / 15 of the read QPS (data read per second). The test results are shown in Table 1:

[0066]

[0067] Table 1

[0068] As can be seen, in tests without the embodiments of this application, data processing time was long and data throughput was low. After adopting the embodiments of this application, processing time was significantly reduced and throughput was significantly improved, achieving the effect of real-time addition of incremental data to the vector retrieval index. A / B test experiments verified that the solution of the embodiments of this application significantly improves the product sales rate.

[0069] Corresponding to the method provided in this application, embodiments of this application also provide an incremental data processing apparatus in a vector retrieval system, referencing... Figure 4 The incremental data processing device 100 includes:

[0070] The segmentation processing module 110 is used to segment the vector space of the vector retrieval system to obtain multiple subspaces;

[0071] Clustering processing module 120 is used to perform clustering processing on the multiple subspaces respectively using a clustering algorithm;

[0072] The lock processing module 130 is used to lock the write requests of each subspace after clustering. The lock works by switching between optimistic locking and pessimistic locking. When the number of write requests meets the preset condition, the lock switches to optimistic locking. When the number of write requests does not meet the preset condition, the lock switches to pessimistic locking.

[0073] This application also provides a server, which includes: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the object data processing method of any of the above embodiments.

[0074] Figure 5 A schematic diagram of the hardware structure of an embodiment of the server provided in this application is shown.

[0075] like Figure 5 As shown, the server may include a processor 601 and a memory 602 storing computer program instructions.

[0076] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0077] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 602 is non-volatile solid-state memory.

[0078] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.

[0079] The processor 601 implements any of the product search methods described in the above embodiments by reading and executing computer program instructions stored in the memory 602.

[0080] In one example, the server may also include a communication interface 603 and a bus 610. Wherein, as... Figure 5 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 610 and complete communication with each other.

[0081] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0082] Bus 610 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0083] Furthermore, in conjunction with the product search methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the product search methods in the above embodiments.

[0084] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0085] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Machine-readable media can include non-transitory computer-readable storage media, such as electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, and can also include radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0086] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0087] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0088] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method of incrementally processing data in a vector retrieval system, characterized in that, The method comprises: performing segmentation processing on a vector space of a vector retrieval system to obtain a plurality of subspaces; performing clustering processing on the plurality of subspaces respectively by using a clustering algorithm; performing locking processing on write requests of each of the subspaces after the clustering processing, and a working mode of the lock is switched between an optimistic lock and a pessimistic lock, wherein when a number of the write requests meets a preset condition, the lock is switched to the optimistic lock, and when the number of the write requests does not meet the preset condition, the lock is switched to the pessimistic lock. wherein the probability that the lock is an optimistic lock is: Where the subscript i represents the i-th subspace, t is the time step, and n is the time window including the current time step. For the number of write requests, The number of data collisions that occur during optimistic locking; for the i-th subspace, the preset condition is the number of write requests to the i-th subspace at the current time. The probability that the lock is an optimistic lock Greater than or equal to 0.

5.

2. The method of claim 1, wherein a sum of a probability that the lock is the optimistic lock and a probability that the lock is the pessimistic lock is 1, and the probability that the lock is the optimistic lock is related to the number of the write requests. The vector retrieval system comprises a commodity search system of an e-commerce transaction platform, and the processing method of the incremental data is used to update a vector retrieval index of incremental data in the commodity search system.

3. The method of claim 1, wherein, The method comprises:

4. An incremental data processing apparatus in a vector retrieval system, characterized by comprising: performing segmentation processing on a vector space of a vector retrieval system to obtain a plurality of subspaces; performing clustering processing on the plurality of subspaces respectively by using a clustering algorithm; performing locking processing on write requests of each of the subspaces after the clustering processing, and a working mode of the lock is switched between an optimistic lock and a pessimistic lock, wherein when a number of the write requests meets a preset condition, the lock is switched to the optimistic lock, and when the number of the write requests does not meet the preset condition, the lock is switched to the pessimistic lock.

5. The apparatus of claim 4, wherein a sum of a probability that the lock is the optimistic lock and a probability that the lock is the pessimistic lock is 1, and the probability that the lock is the optimistic lock is related to the number of the write requests. wherein the probability that the lock is an optimistic lock is: Where the subscript i represents the i-th subspace, t is the time step, and n is the time window including the current time step. For the number of write requests, The number of data collisions that occur during optimistic locking; for the i-th subspace, the preset condition is the number of write requests to the i-th subspace at the current time. The probability that the lock is an optimistic lock Greater than or equal to 0.

5. The vector retrieval system comprises a commodity search system of an e-commerce transaction platform, and the processing method of the incremental data is used to update a vector retrieval index of incremental data in the commodity search system. The server comprises a processor and a memory storing computer program instructions; 6. The apparatus of claim 4, wherein, The server implements the method of any one of claims 1-3 when executing the computer program instructions.

7. A server, characterized by The computer storage medium stores computer program instructions, and the computer program instructions are executed by the processor to implement the method of any one of claims 1-3. ​ 8. A computer-readable storage medium, characterized in that, ​

Images