Retrieval method, device, storage medium and computer equipment

Abstract

The present application discloses a retrieval method, device, storage medium and computer equipment. The method includes: obtaining a scalar data file and an index identifier associated with a vector retrieval result; extracting candidate scalar data from the scalar data file according to the index identifier; matching the scalar retrieval result in the candidate scalar data according to the scalar retrieval condition; and outputting target data according to the vector retrieval result and the scalar retrieval result. The method of the present application places the scalar retrieval in the vector database after the vector index retrieval for processing, thereby reducing the amount of scalar data that needs to be traversed, effectively reducing the number and amount of disk reads, reducing the time delay in executing the retrieval instruction and the disk performance loss, and improving the retrieval performance. In addition, the scalar data and the vector data are stored separately, and the index identifier is used as the basis for locating the scalar data, so that the selection of the scalar field involved in the scalar retrieval condition has high flexibility, and does not generate the time overhead of rebuilding the vector index.

Description

Technical Field

[0001] The present application relates to the technical field of data query, and in particular to a retrieval method, device, storage medium and computer equipment. Background Art

[0002] A vector database is a database that stores and manages vector data in a vector-embedded manner. The vectors it stores are mainly generated by vectorization of unstructured data such as text, speech, images, and videos. In a vector database, each vector has a unique identifier and can be stored in a continuous vector space. Vector databases usually use vector retrieval as the main retrieval method and scalar retrieval as the auxiliary retrieval method, and provide users with the final hybrid retrieval results. However, the index of scalar data is usually a B+ data structure, and disk files are more often used for storage and access. Frequent disk access is required during retrieval, and the retrieval performance is very poor on data volumes in the billions, billions, or even higher levels, which often slows down the overall performance of the entire hybrid retrieval.

[0003] In the related art, when generating a vector index, scalar data is attached to the vector index data as an attribute in a labeling manner. When searching, the vector index is first searched, and when a hit is found, the scalar data attached to the hit vector is filtered according to the scalar search condition. The disadvantage of this method is that the scalar data used for filtering needs to be attached to the vector index data when the vector index is established. If a scalar field for filtering is added later, the index needs to be re-established. In retrieval products such as papers, journals, and patents, there are many fields for splitting a piece of data, generally reaching dozens or even more than a hundred fields, such as author, institution, document number, pinyin journal name, year, issue, journal, classification number, topic code, abstract, publication date, source database, full text, etc., and many fields usually become fields for filtering. Moreover, if the same set of data tables is provided to upper-level products with different needs for design, each product will select the scalar fields it needs for filtering, and most of the time their choices are different. In this scenario, it is not very suitable to attach scalar data to the vector index by labeling, which will generate additional time overhead and maintenance costs for frequent re-creation of indexes. Summary of the invention

[0004] In view of this, the present application provides a retrieval method, apparatus, storage medium and computer device, which reduces the scalar data that needs to be obtained by placing the scalar retrieval after the vector index retrieval for processing, reduces disk access, and has high flexibility in the selection of the scalar fields involved, does not incur the time overhead of rebuilding the vector index, and has high applicability.

[0005] According to one aspect of the present application, a retrieval method is provided, comprising:

[0006] Obtaining a scalar data file and an index identifier associated with a vector search result, wherein the vector search result is obtained by searching a vector search condition, and the index identifier is used to indicate a position of scalar data associated with the vector search result in the scalar data file;

[0007] Extracting candidate scalar data from the scalar data file according to the index identifier;

[0008] matching the scalar search result in the candidate scalar data according to the scalar search condition;

[0009] Target data is output according to the vector search result and the scalar search result.

[0010] Optionally, the obtaining of the scalar data file and index identifier associated with the vector search result includes:

[0011] Parsing the search instruction to obtain the vector search condition and the scalar search condition;

[0012] Retrieving the vector search result in a vector database according to the vector search condition;

[0013] The scalar data file and the index identifier associated with the vector search result are extracted, and the scalar data file and the index identifier are cached.

[0014] Optionally, searching the vector search result in a vector database according to the vector search condition includes:

[0015] Determine the matching degree between the feature vectors of different vector data in the vector database and the vector search condition;

[0016] Screening candidate vector data that meets the vector search condition from the vector database according to the matching degree;

[0017] If the number of the candidate vector data is greater than or equal to the number threshold, the candidate vector data located before the position sequence corresponding to the number threshold in the vector sequence is used as the vector search result, wherein the vector sequence is obtained by arranging the candidate vector data according to the matching degree;

[0018] If the number of the candidate vector data is less than the number threshold, the candidate vector data is used as the vector search result.

[0019] Optionally, extracting candidate scalar data from the scalar data file according to the index identifier includes:

[0020] Calculating an offset according to the index identifier and a data structure parameter of the scalar data file;

[0021] The scalar data of the field corresponding to the offset is read from the scalar data file as the candidate scalar data.

[0022] Optionally, the index identifier is a sequence number of a field of the scalar data in a scalar data file;

[0023] The data structure parameters of the scalar data file include data length, data precision, data type, number of fields or timestamp;

[0024] The offset is the product of the sequence number and the data length.

[0025] Optionally, the retrieval method further includes:

[0026] In response to a data storage operation, determining a preset feature vector corresponding to the data to be stored indicated by the data storage operation in a vector database, wherein the preset feature vector is a feature vector of the vector data stored in the vector database;

[0027] According to the data structure parameters of the scalar data file associated with the preset feature vector, the scalar data corresponding to the data to be stored is stored in the scalar data file associated with the preset feature vector;

[0028] Determine the index identifier of the scalar data corresponding to the data to be stored in the scalar data file associated with the preset feature vector;

[0029] The vector data of the data to be stored, the feature vector of the data to be stored and the index identifier are associated and stored in the vector database.

[0030] Optionally, the retrieval method further includes:

[0031] Comparing the feature vector of the data to be stored with the preset feature vector;

[0032] If the preset feature vector matching the feature vector of the data to be stored does not exist in the vector database, marking the feature vector as the preset feature vector;

[0033] A scalar data file of the feature vector is created, or the feature vector is associated with an already created scalar data file.

[0034] According to another aspect of the present application, a retrieval device is provided, comprising:

[0035] an acquisition module, used to acquire a scalar data file and an index identifier associated with a vector search result, wherein the vector search result is obtained by searching a vector search condition, and the index identifier is used to indicate a position of the scalar data associated with the vector search result in the scalar data file;

[0036] A retrieval module is used to extract candidate scalar data from the scalar data file according to the index identifier; and

[0037] matching the scalar search result in the candidate scalar data according to the scalar search condition;

[0038] A feedback module is used to output target data according to the vector search result and the scalar search result.

[0039] Optionally, the retrieval device further includes:

[0040] A parsing module, used for parsing the search instruction and obtaining the vector search condition and the scalar search condition;

[0041] The retrieval module is further used to retrieve the vector retrieval result in the vector database according to the vector retrieval condition;

[0042] The acquisition module is specifically used to extract the scalar data file and index identifier associated with the vector search result;

[0043] A cache module is used to cache the scalar data file and the index identifier.

[0044] Optionally, the retrieval module is specifically used to determine the matching degree between the feature vectors of different vector data in the vector database and the vector retrieval condition; based on the matching degree, filter the candidate vector data that meets the vector retrieval condition from the vector database; if the number of the candidate vector data is greater than or equal to the quantity threshold, use the candidate vector data in the vector sequence that is located before the corresponding position sequence of the quantity threshold as the vector retrieval result, wherein the vector sequence is obtained by arranging the candidate vector data according to the matching degree; if the number of the candidate vector data is less than the quantity threshold, use the candidate vector data as the vector retrieval result.

[0045] Optionally, the retrieval module is specifically used to calculate the offset based on the index identifier and the data structure parameters of the scalar data file; and read the scalar data of the field corresponding to the offset from the scalar data file as the candidate scalar data.

[0046] Optionally, the index identifier is the serial number of the field of scalar data in the scalar data file; the data structure parameters of the scalar data file include data length, data precision, data type, number of fields or timestamp; the offset is the product of the serial number and the data length.

[0047] Optionally, the retrieval device further includes:

[0048] A determination module, configured to determine, in response to a data storage operation, a preset feature vector corresponding to the data to be stored indicated by the data storage operation in a vector database, wherein the preset feature vector is a feature vector of the vector data stored in the vector database;

[0049] A storage module, used for storing the scalar data corresponding to the data to be stored in the scalar data file associated with the preset feature vector according to the data structure parameters of the scalar data file associated with the preset feature vector;

[0050] The determination module is further used to determine the index identifier of the scalar data corresponding to the data to be stored in the scalar data file associated with the preset feature vector;

[0051] The storage module is further used to associate the vector data of the data to be stored, the feature vector of the data to be stored and the index identifier and store them in the vector database.

[0052] Optionally, the retrieval device further includes:

[0053] A matching module, used for comparing the feature vector of the data to be stored with the preset feature vector;

[0054] An association module is used to mark the feature vector as the preset feature vector if the preset feature vector matching the feature vector of the data to be stored does not exist in the vector database; and to create a scalar data file of the feature vector, or to associate the feature vector with a created scalar data file.

[0055] According to another aspect of the present application, a readable storage medium is provided, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the above-mentioned retrieval method are implemented.

[0056] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, and the processor implements the steps of the above-mentioned retrieval method when executing the program.

[0057] By means of the above technical solution, when a user initiates a search, the vector index is first searched according to the vector search conditions to obtain the vector search results. A scalar data file and an index identifier for storing scalar data related to the vector search results are obtained. And the candidate scalar data related to the vector search results are read in the scalar data file through the index identifier. According to the scalar search conditions, a logical judgment is made on the candidate scalar data read out, and the data that does not meet the scalar search conditions is removed from the cache. The target data after the final filtering is returned to the user. Thus, the scalar search in the vector database is placed after the vector index search for processing, and data of multiple scalar search conditions are obtained from the results of the vector search. Since the number of vector searches is generally small, the scalar data that needs to be obtained is very limited, which reduces the number of scalar data that needs to be traversed, reduces the scalar search range, and effectively reduces the number of disk reads and the amount of disk reads, reduces the time delay and disk performance loss of executing the search instruction, and improves the search performance. In addition, scalar data and vector data are stored separately, and the index identifier is used as the basis for locating the scalar data, so that the selection of scalar fields involved in the scalar search conditions has high flexibility, does not incur the time overhead of rebuilding the vector index, and has high applicability, thereby solving the problem of poor performance of mixed retrieval of vectors and scalars in vector databases. In addition, since scalar data is obtained based on vector retrieval results, it can prevent the situation where the intersection of scalar retrieval results and vector retrieval results is empty when scalar retrieval and vector retrieval are performed separately.

[0058] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0060] Figure 1 A schematic diagram of the process of the search method provided in the embodiment of the present application is shown;

[0061] Figure 2 A structural block diagram of a retrieval device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0062] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0063] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0064] Those skilled in the art will appreciate that, unless expressly stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, or there may be intermediate elements. In addition, the "connection" or "connection" used herein may include wireless connection or wireless fusion. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0065] Now, exemplary embodiments according to the present application will be described in more detail with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in a variety of different forms and should not be interpreted as being limited to the embodiments set forth herein. It should be understood that these embodiments are provided to make the disclosure of the present application thorough and complete, and to fully convey the concepts of these exemplary embodiments to those of ordinary skill in the art.

[0066] In this embodiment, a retrieval method is provided, such as Figure 1 As shown, the method includes:

[0067] Step 101: Obtain the scalar data file and index identifier associated with the vector search result.

[0068] Among them, the vector search result is obtained by searching the vector search conditions, and the vector search can be implemented by at least one method of the prior art, which is not specifically limited in the embodiments of the present application. The scalar data file is used to store scalar data, and the characteristic vectors of different vector data can be associated with the same scalar data file or different scalar data files, thereby reducing the data volume of a single scalar data file and saving data storage space. At the same time, the isolation of scalar data between different vector data is ensured, which facilitates the rapid query of scalar data related to the vector search result. The index identifier is used to indicate the position of the scalar data related to the vector search result in the scalar data file. Exemplarily, in order to increase the amount of data referred to by the index identifier, the index identifier can be the serial number of the field of the scalar data in the scalar data file, such as the row number and the column number. Scalar data of at least one field is associated by the serial number.

[0069] It can be understood that scalar data is data with only numerical value but no direction. Vector data is data with data size and direction. Feature vectors are used to describe the low-dimensional features of vector data. A low-dimensional feature space can be constructed through feature vectors, and high-dimensional vector data can be projected into this low-dimensional space to obtain a more concise feature representation for subsequent classification and retrieval tasks. For example, for published articles, their names and journal numbers are used as scalar data, and their contents are used as vector data.

[0070] In this embodiment, the scalar data of the vector data is stored in advance in units of index identifiers in a scalar data file associated with the feature vector of the vector data, and multiple field values ​​can be stored together and associated with one index identifier. In this way, after completing the vector search, the part of the scalar data file that is closely associated with the vector search result can be located by the index identifier, thereby achieving the purpose of not traversing the entire storage disk and reducing the search time.

[0071] In one embodiment, before step 101, the retrieval method further includes:

[0072] Step 201 : In response to a data storage operation, determining a preset feature vector corresponding to the data to be stored indicated by the data storage operation in a vector database.

[0073] The preset feature vector is a feature vector of vector data stored in the vector database.

[0074] Step 202 , according to the data structure parameters of the scalar data file associated with the preset feature vector, the scalar data corresponding to the data to be stored is stored in the scalar data file associated with the preset feature vector.

[0075] Specifically, the data structure parameter of a scalar data file can be the data length, that is, the fixed length of the data sequence itself in the structured data; it can also be the number of fields that the data sequence can contain, the timestamp of the stored data, the data precision, the data type, etc.

[0076] In this embodiment, different scalar data can be classified and arranged according to certain rules through data structure parameters and stored in a scalar data file associated with a preset feature vector, which helps to save storage space and reduce the generation of storage fragments.

[0077] Step 203: determine the index identifier of the scalar data corresponding to the data to be stored in the scalar data file associated with the preset feature vector.

[0078] Step 204: associate the vector data of the data to be stored, the feature vector of the data to be stored, and the index identifier and store them in a vector database.

[0079] In this embodiment, when new data to be stored is stored in the database, the data to be stored is converted into vector data, and the characteristic vector of the vector data and the related scalar data are extracted. A preset characteristic vector that matches the characteristic vector of the data to be stored is searched in the vector database, so as to determine whether there is a related scalar data file in the vector database. According to the data structure parameters, the scalar data corresponding to the data to be stored is stored in the scalar data file associated with the preset characteristic vector. After the scalar data is stored, the index identifier of the scalar data in the scalar data file can be determined. The vector data of the data to be stored, the characteristic vector of the data to be stored and the index identifier are associated and stored in the vector database. Thus, a complete data storage system is constructed. When scalar retrieval is required, the scalar data related to the vector data can be quickly located through the index identifier, so that while ensuring the connection between the vector data and the scalar data, there is no need to store the scalar data and the vector data together, which improves the flexibility when expanding the scalar data and facilitates the unified management and use of data.

[0080] Furthermore, before determining the preset feature vector of the data to be stored indicated by the data storage operation in step 201, the retrieval method also includes: comparing the feature vector of the data to be stored with the preset feature vector; if there is no preset feature vector matching the feature vector of the data to be stored in the vector database, marking the feature vector as a preset feature vector; creating a scalar data file of the feature vector, or associating the feature vector with a scalar data file that has been created.

[0081] In this embodiment, by comparing the feature vector of the data to be stored with the existing preset feature vector, effective classification of the data can be achieved. When there is no matching preset feature vector, the new feature vector is marked as the preset feature vector. At the same time, a new scalar data file is created or an existing scalar data file is associated, so that the data storage structure is clearer, which not only ensures the data volume of the scalar data file, but also controls the number of scalar data files, which helps to optimize the data storage structure.

[0082] Specifically, if the similarity between the feature vector of the data to be stored and the preset feature vector is greater than the preset similarity, it can be determined that the two match.

[0083] Exemplarily, whether to create a new scalar data file is determined based on the space occupied by the scalar data file. When the space occupied by an existing scalar data file is less than a preset value, that is, the scalar data file stores less data. In order to make full use of the scalar data file, the scalar data file can be associated with a feature vector. Conversely, when the space occupied by an existing scalar data file is greater than or equal to the preset value, a new scalar data file can be created to store related scalar data.

[0084] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, step 101, i.e., the step of obtaining the scalar data file and index identifier associated with the vector search result, specifically includes:

[0085] Step 101 - 1 , parse the search instruction to obtain vector search conditions and scalar search conditions.

[0086] Step 101 - 2 , searching the vector search result in the vector database according to the vector search condition.

[0087] Step 101 - 3 , extracting the scalar data file and index identifier associated with the vector search result, and caching the scalar data file and index identifier.

[0088] In this embodiment, when the user triggers the generation of a search instruction, the system obtains the vector search conditions and scalar search conditions required by the user by parsing the search instruction. A targeted vector search is performed in the vector database through the vector search conditions. Since the vector data and the scalar data have been linked through the scalar data file and the index identifier when the data is stored, the scalar data file and the index identifier associated with the vector search result can be directly extracted at this time. The obtained scalar data file and the index identifier are cached to form a cache result set. Thus, by caching the scalar data file and the index identifier of the vector search result of the user in a targeted manner, under the premise of ensuring the integrity of the information carried in the vector search result, too much cache space is not occupied, and the role of the cache in improving the search speed is maximized. And when there is the same or similar search demand in a short period of time, the scalar data file and the index identifier can be directly obtained from the cache without having to perform the complete vector search process again, especially for the scenario of frequently performing the same or similar search operations, which can greatly improve the search speed and reduce the response time of the system.

[0089] Specifically, step 101-2, that is, searching the vector search results in the vector database based on the vector search conditions, specifically includes the following steps: determining the matching degree between the feature vectors of different vector data in the vector database and the vector search conditions; screening the candidate vector data that meets the vector search conditions from the vector database based on the matching degree; if the number of candidate vector data is greater than or equal to the quantity threshold, taking the candidate vector data that is located before the corresponding position sequence of the quantity threshold in the vector sequence as the vector search result; if the number of candidate vector data is less than the quantity threshold, taking the candidate vector data as the vector search result.

[0090] The vector sequence is obtained by arranging the candidate vector data according to the matching degree. The quantity threshold can be set by the technician according to the needs or experience, or determined through a large number of experiments, and is not specifically limited in the embodiments of the present application.

[0091] In this embodiment, the current vector retrieval condition is determined to be matched with the feature vector of the stored vector data for similarity matching, so as to judge the correlation between the vector retrieval condition and the feature vector of the stored vector data through the similarity. A specified number of candidate vector data with high similarity is retained as the result of the vector retrieval. On the one hand, it can reduce redundant vector calculations and filtering operations, avoid the excessive complexity of the returned results due to too much vector data that meets the retrieval conditions, and enable users to obtain relatively more important and more representative partial data within an acceptable range. On the other hand, it can reduce the search basis for subsequent scalar retrieval and thus reduce the amount of scalar data obtained, avoid indiscriminate and comprehensive traversal of the entire disk, and significantly reduce unnecessary calculations and data processing, thereby improving the efficiency of retrieval, while also saving the system's computing resources, storage resources, and network transmission resources, so that the system can run more efficiently.

[0092] Step 102: extract candidate scalar data from the scalar data file according to the index identifier.

[0093] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, step 102, that is, the step of extracting candidate scalar data from the scalar data file according to the index identifier, specifically includes:

[0094] Step 102-1, calculating the offset according to the index identifier and the data structure parameters of the scalar data file.

[0095] Exemplarily, the product of the row number in the sequence number and the data length is calculated as the offset for locating the scalar data, so that the scalar data locating task is completed with a relatively small data operation, which helps to improve the accuracy and efficiency of scalar retrieval.

[0096] Step 102 - 2 , reading scalar data of a field corresponding to the offset from the scalar data file as candidate scalar data.

[0097] In this embodiment, the approximate area where the scalar data associated with the vector search result is located can be quickly located through the index identifier, and the offset can be calculated in combination with the data structure parameters of the scalar data file, so that the specific field position to be read can be further accurately determined. Thereby, the precise positioning of the scalar data is achieved, and blind search in the entire scalar data file is avoided, which greatly improves the efficiency of data access. Moreover, considering that different scalar data files may have different data structures, the required scalar data can be read by combining the index identifier and the data structure parameters, which has strong versatility and compatibility and can play a role in various types of scalar data file processing scenarios.

[0098] Step 103, matching the scalar search results in the candidate scalar data according to the scalar search condition.

[0099] Among them, the scalar search results are obtained by scalar search condition retrieval, and the scalar search can be implemented using at least one method of the existing technology, such as exact matching, inclusion matching, front consistent matching, wildcard matching, etc., and the embodiments of the present application do not make specific limitations on this.

[0100] It should be noted that the retrieval system for performing vector retrieval and the retrieval system for performing scalar retrieval may be the same or different. In order to avoid time delay caused by data transmission, the same retrieval system may be used for scalar retrieval and vector retrieval.

[0101] Step 104, outputting target data according to the vector search result and the scalar search result.

[0102] It can be understood that by taking the intersection of the scalar search result and the vector search result and merging them, the final target data is obtained as the data search result.

[0103] The retrieval scheme provided in the embodiment of the present application, when the user initiates a search, first searches the vector index according to the vector search condition to obtain the vector search result. Obtain a scalar data file and index identifier for storing scalar data related to the vector search result. And read the candidate scalar data related to the vector search result in the scalar data file through the index identifier. According to the scalar search condition, perform logical judgment on the read candidate scalar data, and remove the data that does not meet the scalar search condition from the cache. Return the final filtered target data to the user. Thereby, the scalar search in the vector database is placed after the vector index search for processing, and the data of multiple scalar search conditions are obtained from the results of the vector search. Since the number of vector searches is generally small, the scalar data that needs to be obtained is very limited, which reduces the number of scalar data that needs to be traversed, reduces the scalar search range, and effectively reduces the number of disk reads and the amount of disk reads, reduces the time delay and disk performance loss of executing the search instruction, and improves the search performance. In addition, scalar data and vector data are stored separately, and the index identifier is used as the basis for locating the scalar data, so that the selection of scalar fields involved in the scalar search conditions has high flexibility, does not incur the time overhead of rebuilding the vector index, and has high applicability, thereby solving the problem of poor performance of mixed retrieval of vectors and scalars in vector databases. In addition, since scalar data is obtained based on vector retrieval results, it can prevent the situation where the intersection of scalar retrieval results and vector retrieval results is empty when scalar retrieval and vector retrieval are performed separately.

[0104] The retrieval method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud vector databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the retrieval method, etc., but is not limited to the above forms.

[0105] It can be understood that the retrieval method provided in the embodiments of the present application can be applied to text retrieval and video retrieval or multimodal retrieval scenarios, for example, cross-modal retrieval scenarios such as searching for videos with pictures and searching for pictures with text.

[0106] Specifically, for example, the retrieval method includes the following steps:

[0107] Step S1, retrieve the first K nearest vector data through the vector index, obtain the record row number associated with the vector data, and construct a vector search result set.

[0108] Step S2, retrieve the record row number of the Ith (I is initialized to 1, I is less than or equal to K) position in the vector retrieval set, and obtain the corresponding scalar data from the memory cache or disk file.

[0109] Step S3, according to the scalar field definition, parse the scalar data required in the specific scalar search condition; if the scalar search condition involves N fields, obtain the data of the corresponding N fields.

[0110] Step S4, according to the Mth (M is initialized to 1, M is less than or equal to N) scalar search condition, such as exact match, inclusion match, front consistent match, wildcard match, etc., logical judgment is performed on the scalar data.

[0111] Step S5: If the data meets the scalar search condition, no processing is performed; if the data does not meet the scalar search condition, the record row number is removed from the vector result set.

[0112] Step S6: if there are multiple scalar search conditions, repeat steps S4-S6 for the M+1th scalar search condition.

[0113] Step S7: If there are multiple hit records in the vector search result set, repeat steps S2-S7 for the row number of the record at the I+1th position in the set.

[0114] Step 80, returning the final filtered result set to the user.

[0115] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0116] Furthermore, if Figure 2 As shown, as a specific implementation of the above retrieval method, an embodiment of the present application provides a retrieval device 300, which includes: an acquisition module 301, a retrieval module 302 and a feedback module 303.

[0117] The acquisition module 301 is used to acquire a scalar data file and an index identifier associated with a vector search result, wherein the vector search result is obtained by searching a vector search condition, and the index identifier is used to indicate the position of the scalar data associated with the vector search result in the scalar data file;

[0118] A search module 302 is used to extract candidate scalar data from the scalar data file according to the index identifier; and to match the scalar search result in the candidate scalar data according to the scalar search condition;

[0119] The feedback module 303 is used to output target data according to the vector search result and the scalar search result.

[0120] Furthermore, the retrieval device 300 further includes:

[0121] A parsing module (not shown in the figure), which is used to parse the search instruction and obtain the vector search condition and the scalar search condition;

[0122] The search module 302 is further used to search the vector search result in the vector database according to the vector search condition;

[0123] The acquisition module 301 is specifically used to extract the scalar data file and index identifier associated with the vector search result;

[0124] A cache module (not shown in the figure) is used to cache scalar data files and index identifiers.

[0125] Furthermore, the retrieval module 302 is specifically used to determine the matching degree between the feature vectors of different vector data in the vector database and the vector retrieval condition; based on the matching degree, the candidate vector data that meets the vector retrieval condition is screened from the vector database; if the number of candidate vector data is greater than or equal to the quantity threshold, the candidate vector data that is located before the corresponding position of the quantity threshold in the vector sequence is used as the vector retrieval result, wherein the vector sequence is obtained by arranging the candidate vector data according to the matching degree; if the number of candidate vector data is less than the quantity threshold, the candidate vector data is used as the vector retrieval result.

[0126] Furthermore, the search module 302 is specifically configured to calculate an offset according to an index identifier and a data structure parameter of a scalar data file; and read scalar data of a field corresponding to the offset from the scalar data file as candidate scalar data.

[0127] Furthermore, the index identifier is the serial number of the field of the scalar data in the scalar data file; the data structure parameters of the scalar data file include data length, data precision, data type, number of fields or timestamp; the offset is the product of the serial number and the data length.

[0128] Furthermore, the retrieval device 300 further includes:

[0129] A determination module (not shown in the figure), the determination module is used to determine, in response to the data storage operation, a preset feature vector corresponding to the data to be stored indicated by the data storage operation in the vector database, wherein the preset feature vector is a feature vector of the vector data stored in the vector database;

[0130] A storage module (not shown in the figure), the storage module is used to store the scalar data corresponding to the data to be stored in the scalar data file associated with the preset feature vector according to the data structure parameters of the scalar data file associated with the preset feature vector;

[0131] The determination module is further used to determine the index identifier of the scalar data corresponding to the data to be stored in the scalar data file associated with the preset feature vector;

[0132] The storage module is also used to associate the vector data of the data to be stored, the feature vector of the data to be stored and the index identifier and store them in the vector database.

[0133] Furthermore, the retrieval device 300 further includes:

[0134] A matching module (not shown in the figure), which is used to compare the feature vector of the data to be stored with a preset feature vector;

[0135] An association module (not shown in the figure) is used to mark the feature vector as a preset feature vector if there is no preset feature vector matching the feature vector of the data to be stored in the vector database; and to create a scalar data file for the feature vector, or to associate the feature vector with a scalar data file that has been created.

[0136] For the specific definition of the retrieval device, please refer to the definition of the retrieval method above, which will not be repeated here. Each module in the above retrieval device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0137] Based on the above Figure 1 The method shown in the embodiment of the present application accordingly provides a readable storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned Figure 1 The search method shown.

[0138] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0139] Based on the above Figure 1 The method shown, and Figure 2 In order to achieve the above-mentioned purpose, the embodiment of the present application further provides a computer device, which can be a personal computer, a server, a network device, etc. The computer device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 The search method shown.

[0140] Furthermore, the computer device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a Bluetooth interface, a WI-FI interface), etc.

[0141] Those skilled in the art will appreciate that the computer device structure provided in this embodiment does not limit the computer device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0142] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages and saves the hardware and software resources of the computer device, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to realize communication between the components inside the storage medium, and communication with other hardware and software in the physical device.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware platforms, or by hardware to obtain scalar data files and index identifiers associated with vector search results, wherein the vector search results are retrieved by vector search conditions, and the index identifier is used to indicate the position of the scalar data related to the vector search results in the scalar data file; extract candidate scalar data from the scalar data file according to the index identifier; match the scalar search results in the candidate scalar data according to the scalar search conditions; output the target data according to the vector search results and the scalar search results. In the embodiment of the present application, when a user initiates a search, the vector index is first searched according to the vector search conditions to obtain the vector search results. A scalar data file and an index identifier for storing scalar data related to the vector search results are obtained. And the candidate scalar data related to the vector search results are read in the scalar data file through the index identifier. According to the scalar search conditions, the read candidate scalar data are logically judged, and the data that does not meet the scalar search conditions is removed from the cache. The final filtered result set is returned to the user. Therefore, the scalar search in the vector database is placed after the vector index search for processing, and the data of multiple scalar search conditions are obtained from the results of the vector search. Since the number of vector searches is generally small, the scalar data that needs to be obtained is very limited, which reduces the number of scalar data that needs to be traversed, effectively reduces the number of disk reads and the amount of disk reads, reduces the time delay of executing search instructions and disk performance loss, and improves the search performance. In addition, scalar data and vector data are stored separately, and the index identifier is used as the basis for locating scalar data, so that the selection of scalar fields involved in the scalar search condition has high flexibility, does not generate the time overhead of rebuilding the vector index, and has high applicability, thereby solving the problem of poor performance of mixed vector and scalar searches in vector databases.

[0144] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0145] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. A search method, characterized in that: The method comprises: Obtaining a scalar data file and an index identifier associated with a vector search result, wherein the vector search result is obtained by searching a vector search condition, and the index identifier is used to indicate a position of scalar data associated with the vector search result in the scalar data file; Extracting candidate scalar data from the scalar data file according to the index identifier; matching the scalar search result in the candidate scalar data according to the scalar search condition; Target data is output according to the vector search result and the scalar search result.

2. The search method according to claim 1, characterized in that: The step of obtaining the scalar data file and index identifier associated with the vector search result includes: Parsing the search instruction to obtain the vector search condition and the scalar search condition; Retrieving the vector search result in a vector database according to the vector search condition; The scalar data file and the index identifier associated with the vector search result are extracted, and the scalar data file and the index identifier are cached.

3. The search method according to claim 2, characterized in that: The step of searching the vector search result in a vector database according to the vector search condition includes: Determine the matching degree between the feature vectors of different vector data in the vector database and the vector search condition; Screening candidate vector data that meets the vector search condition from the vector database according to the matching degree; If the number of the candidate vector data is greater than or equal to the number threshold, the candidate vector data located before the position sequence corresponding to the number threshold in the vector sequence is used as the vector search result, wherein the vector sequence is obtained by arranging the candidate vector data according to the matching degree; If the number of the candidate vector data is less than the number threshold, the candidate vector data is used as the vector search result.

4. The search method according to claim 1, characterized in that: The step of extracting candidate scalar data from the scalar data file according to the index identifier includes: Calculating an offset according to the index identifier and a data structure parameter of the scalar data file; The scalar data of the field corresponding to the offset is read from the scalar data file as the candidate scalar data.

5. The search method according to claim 4, characterized in that: The index identifier is a sequence number of a field of the scalar data in a scalar data file; The data structure parameters of the scalar data file include data length, data precision, data type, number of fields or timestamp; The offset is the product of the sequence number and the data length.

6. The search method according to any one of claims 1 to 5, characterized in that: The method further comprises: In response to a data storage operation, determining a preset feature vector corresponding to the data to be stored indicated by the data storage operation in a vector database, wherein the preset feature vector is a feature vector of the vector data stored in the vector database; According to the data structure parameters of the scalar data file associated with the preset feature vector, the scalar data corresponding to the data to be stored is stored in the scalar data file associated with the preset feature vector; Determine the index identifier of the scalar data corresponding to the data to be stored in the scalar data file associated with the preset feature vector; The vector data of the data to be stored, the feature vector of the data to be stored and the index identifier are associated and stored in the vector database.

7. The search method according to claim 6, characterized in that: The method further comprises: Comparing the feature vector of the data to be stored with the preset feature vector; If the preset feature vector matching the feature vector of the data to be stored does not exist in the vector database, marking the feature vector as the preset feature vector; A scalar data file of the feature vector is created, or the feature vector is associated with an already created scalar data file.

8. A search device, characterized in that: The device comprises: an acquisition module, used to acquire a scalar data file and an index identifier associated with a vector search result, wherein the vector search result is obtained by searching a vector search condition, and the index identifier is used to indicate a position of the scalar data associated with the vector search result in the scalar data file; A retrieval module is used to extract candidate scalar data from the scalar data file according to the index identifier; and matching the scalar search result in the candidate scalar data according to the scalar search condition; A feedback module is used to output target data according to the vector search result and the scalar search result.

9. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the steps of the search method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the program, the search method according to any one of claims 1 to 7 is implemented.

Images