Vector database composite retrieval method

Abstract

The invention discloses a vector database composite retrieval method, which comprises the following steps of S1, vector database index entry construction: generating an initial index entry according to a data form of submitted original data and an index entry type; performing multiple processing on the source matching key of the index entry to generate a new matching key; and S2, vector database index entry retrieval: sorting the previous information, and searching and returning a mapping value corresponding to the matching key from a preset retrieval entry of the vector database according to the previous information. The method has the advantages that the data retrieval precision and coverage can be improved, more effective prompt content is provided for the large language model, and the application difficulty of the large language model is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer software technology, and in particular to a vector database composite retrieval method. Background Art

[0002] At present, with the rise of large language model technology, the application demand for large language models is gradually increasing. The large language model is essentially a conditional probability model that records the frequency of occurrence of subsequent language units when the previous language unit sequence appears. When a large language model is given, the previous language unit sequence determines the probability distribution of subsequent language units. Based on this distribution, the sampling strategy selects the output language unit. We call the previous language unit sequence a prompt, and the subsequent output is called the completion of the sequence. In the specific application of large language models, the prompt determines the degree of adaptation to the application scenario, so the accuracy of the prompt content plays a key role in the actual application effect of the large language model. The process of preparing prompts and the research and development of related technologies are called prompt engineering. Related technologies include prompt generation technology based on vector databases.

[0003] Using the semantic vector of the conversation context as a search item and retrieving the corresponding preset content in the vector database is the main way for the vector database to provide prompts. Currently, mature large language model capability providers usually provide this technology, which is called a knowledge base. The knowledge in the knowledge base can be understood as a mapping of preset search items and feedback. The construction method is to first directly segment the unstructured language text based on format symbols such as punctuation, and limit the maximum length of the segment with a preset value. This process is called slicing, and then select the semantic vector of the text segment as the search item, and the adjacent content of the segment is used as feedback. The retrieval method based on semantic vectors requires the overall context of the conversation to match the search item, and defining the context involved in the retrieval is called a difficulty. Therefore, recently, a solution that uses traditional keyword matching methods as a supplement has also emerged, but traditional keyword matching methods lack the ability to match synonymous language units, and therefore cannot effectively match longer language units such as sentences and paragraphs. In short, existing methods cannot solve the problem of partial semantic approximation. Summary of the invention

[0004] The object of the present invention is to provide a vector database composite retrieval method to solve the above-mentioned problems existing in the prior art.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A vector database composite retrieval method comprises the following steps:

[0007] S1. Construction of vector database index items:

[0008] Generate initial index items according to the data format and index item type of the submitted original data; perform multiple processing on the source matching key of the index item to generate a new matching key;

[0009] S2. Vector database index item retrieval:

[0010] The above information is sorted out, and according to the above information, a mapping value corresponding to a matching key is searched from a preset search item of a vector database and returned.

[0011] Preferably, step S1 specifically includes the following contents:

[0012] S10, determining the partition to which the generated index item belongs, and storing it as an attribute of the index item in the vector database in a logical structure; the partition is the basis for defining the search scope during retrieval, and is determined by the scenario to which the index item belongs and the customer;

[0013] S11, identifying the data form and index item type of the original data, and determining the initial index item according to the data form and the index item type, wherein the matching key of the initial index item is the source matching key;

[0014] S12. Perform synonym generalization on the source matching key of the index item as a whole, perform semantic completion on the matching key after synonym generalization, split the matching key after semantic completion into multi-granularity language units, perform secondary synonym generalization on the matching key after splitting into multi-granularity language units, calculate the regularization weight of the matching key after secondary synonym generalization, so as to generate a new matching key and index structure.

[0015] Preferably, step S11 specifically includes the following contents:

[0016] S111. The data form of the original data includes unstructured language text and semi-structured data table; the index item type is the usage mode after the data is finally formed into index items, and the usage mode includes using the mapping value as a large language model prompt or directly as a final answer;

[0017] S112. For unstructured language text, segmentation is performed based on the common method, and the text fragments are selected by moving windows as matching keys, and the adjacent fragments are used as mapping values. For unstructured tables, the information source needs to be added to the mapping value. If the index item is not used as a large model prompt but as a direct output, it is called the question and answer type. Other index items that are output as large model prompts are called ordinary types. The type is stored in the vector database as a feature of the index item. The index item structure is: {key: (source) matching key, value: mapping value [URL], type: [ordinary|question and answer|reference]}.

[0018] Preferably, step S12 specifically includes the following contents:

[0019] S121, combining the synonymous content generation requirement and the source matching key into a prompt, and using the large language model to perform synonym expansion on the source matching key; when the number of source matching keys is N and the number of synonym expansions is M-1, an N-by-M array A of the source matching keys and the expanded matching keys is obtained;

[0020] S122, using the dependency grammar analysis model to perform grammar analysis on the matching key components in array A, obtain word segmentation and corresponding part of speech, grammar, semantic unit tags and structures, and perform semantic completion according to the completion rules;

[0021] S123, in the process of splitting multi-granularity language units, remove non-key words. If there is a negative word, the negative word and its central word are not split as a whole. The language unit retained after the split is called a matching subkey. Each matching key is split into an L-dimensional vector, and the array A becomes an N-by-M-by-L array B = [B1 B2 ... B N ];

[0022] S124, the synonymous content generation requirement and the matching subkey are combined into a prompt, the matching subkey is synonymously expanded using a large language model, K candidate synonymous expansions are randomly selected at each matching subkey position under each matching key, and the combination is put into a set, and the sampling, combination, and putting into the set are repeated until the cardinality of the set reaches a preset value; when the preset value is J-1, the array B becomes an N-J-L array C=[C1 C2 ... C N ];

[0023] S125, sequentially concatenate the matching subkeys of each matching key in the array C into a string, use the embedding model to convert the string into a semantic vector, use the semantic vector corresponding to the source matching key as a reference, calculate the semantic similarity between the semantic vector of the matching key and the corresponding source matching key reference semantic vector, obtain a similarity matrix S, calculate the regularization weight based on the similarity matrix, and obtain a normalized weight matrix W;

[0024] S126, the N source matching keys in the array C correspond one-to-one to the N source matching keys in the index item structure in step S112, and the J matching keys under each source matching key and the corresponding source matching key index item are in a many-to-one relationship, and each matching key corresponds to a normalized weight. At this time, the logical structure of each index item is: {key: (source) matching key, value: mapping value [URL], type: [normal|question and answer|reference];

[0025] match_array corresponds to C n and match_weights corresponds to W n =[w n,1 w n,2 … w n,J ]; n=1~N.

[0026] Preferably, the completion rules in step S122 specifically include the following contents:

[0027] S1221, extract the central words of parallel grammatical and semantic units together;

[0028] S1222, determine whether the central word and the parallel word have the same part of speech, if they are the same, proceed to step S1223, otherwise proceed to step S1224;

[0029] S1223, according to the position of the component words on the left and right sides of the central word, copy the component words directly pointing to the central word and insert them into the corresponding side of the parallel word;

[0030] S1224. When Chinese is used as the scenario, the core word is divided into three types of characters: Chinese, non-Chinese, and numbers. If the parallel language only contains English or numbers, the missing Chinese characters are inserted into the parallel language, and the insertion position is consistent with the position of the part in the core word.

[0031] Preferably, step S2 specifically includes the following contents:

[0032] S21, sorting out the existing conversation records and the current conversation, selecting the most recent contexts of different lengths, and semantically completing the contexts;

[0033] S22. Synthesize a regularized discriminant combination based on the matching keys, construct an indirect index matrix, generate a regularized discriminant combination hit vector, calculate the regularized approximation of the matching keys based on the hit vector, the indirect index matrix and the regularized weight, and determine the retrieval return content of the index item based on the regularized approximation.

[0034] Preferably, step S21 specifically includes the following contents:

[0035] S211, select the most recent context of different lengths, search for contexts of multiple lengths, and use different context lengths for different types of index items; in human-computer dialogue, for question-and-answer index items, use the content of the most recent dialogue round to search; for general index items, use the content of the most recent 1 to 3 dialogue rounds; for dialogue scenarios, the context finally submitted for search is spliced ​​from the content in the dialogue;

[0036] S212, using the method based on dependency grammar analysis and completion rules in step S122 to complete the parallel terms at the word level according to their corresponding central word forms; for multi-role multi-round dialogues, this method is used to complete the dialogue content of each role in each round.

[0037] Preferably, step S22 specifically includes the following contents:

[0038] S220, determining the partition to be searched according to the scenario and customer information;

[0039] S221. Take out the matching subkeys of all index items in the partition, sort them in ascending order according to the length of the subkey characters, and reconstruct them into an ordered set of subkeys SKEY = {skey1, skey2, ..., skey l ,…,skey L}, the subkey ordered set records the source of the subkey, that is, the index item n to which the subkey belongs and the row number j in match_array, l is the sequence number of the subkey in the subkey ordered set, l, n and j construct the dictionary D l :l→(n,j), the structure is {key:l,value:(n,j)};

[0040] Convert the ordered set of subkeys SKEY into a regular discriminant combination REXS, and construct a dictionary D according to the mapping relationship between the serial number ri of the regular discriminant in the regular discriminant combination and the serial number rj and l of the subkey in the regular discriminant. r :(ri,rj)→l, the structure is {key:(ri,rj),value:l};

[0041] S222. Construct an indirect index matrix R according to the ordered set of subkeys SKEY. The row index in the indirect index matrix R is x, and there is a mapping relationship between x and the tuple of index item number n and match_array row number j, D x :x→(n,j); the row number x of the matrix R needs to be mapped to the index item number n and the match_array row number j to form the upper discriminant element r x,l The value of

[0042] S223, combining hit vectors based on regular discriminant REXS;

[0043] S224, calculating the regularized approximation based on the hit vector, the indirect index matrix, and the regularized weight;

[0044] S225. According to the index item to which the matching key belongs, the regularized approximation is grouped by index item, and the maximum value in each group is used as the approximation between the above text and the index item, and the mapping value corresponding to the matching key is determined based on the approximation of the index item.

[0045] Preferably, step S223 specifically includes the following contents:

[0046] S2231, using the regular discriminant combination REXS to discriminate the above texts involved in the retrieval, and generating a discriminant matrix D;

[0047] S2232, build hit vector The initial element value is 0; use sparse matrix operation method or directly through dictionary D r:(ri,rj)→l, convert the position (ri,rj) with element value 1 to serial number l, and assign values ​​to the components of the hit vector.

[0048] Preferably, step S224 specifically includes the following contents:

[0049] S2241, multiplying the hit vector by the transposed matrix of the indirect index matrix to obtain a proximity vector;

[0050] S2242. According to the mapping relationship in step S222, the proximity vector is folded into a proximity matrix S, and the proximity matrix S is multiplied point by point with the regularized weight matrix W to obtain a regularized proximity matrix RS.

[0051] The beneficial effects of the present invention are: 1. The method of the present invention can improve the accuracy and coverage of data retrieval, provide more effective prompt content for large language models, and reduce the application difficulty of large language models. 2. The method of the present invention completes, screens and limits the split granularity of text through grammatical analysis, improves the matching accuracy at the semantic level, and avoids serious synthetic errors. 3. The method of the present invention simplifies the processing process and improves the scalability of the method by combining regular discriminant hit vectors, indirect index matrices and other mathematical modeling and calculation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flowchart of the retrieval method in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] like Figure 1 As shown, in this embodiment, a vector database composite retrieval method is provided, and the method specifically includes the following two parts:

[0055] 1. Vector Database Index Item Construction

[0056] This includes the processing from raw data to preset index items, which mainly adopts the asynchronous processing method of the service program after the information to be added is submitted online. In this stage, the submitted raw data is processed in various ways to form the index items of the database for use in the retrieval stage.

[0057] 1.1. Identification of submitted data and determination of index items

[0058] After submitting the data, you need to determine the partition to which the generated index item belongs. Partition is the basis for defining the search scope during retrieval. It is generally determined by the scenario to which the index item belongs and the customer, and is stored in the vector database as an attribute of the index item in the logical structure. The processing after partitioning can be divided into two steps:

[0059] 1.1.1 Identify different data formats and index item types

[0060] There are two main data forms: unstructured language text, such as customer background introduction, application scenario information, conversation records, etc.; semi-structured data tables, such as sorted question and answer pairs, term explanations, and conversation reference dictionaries, etc. Index item type refers to the way the data is used after it is finally formed into index items, including using the mapping value as a large language model prompt, or directly as the final answer.

[0061] 1.1.2. Determine index items according to different forms and types

[0062] According to different forms and types, the organization generates the initial index items, where the matching key of the initial index item becomes the source matching key. For unstructured language text, it is segmented based on punctuation, maximum length limit and other common methods, and the text fragments are selected as matching keys by moving windows, and the adjacent fragments are used as mapping values. If the type is reference type (semi-structured data table), it is also necessary to add the information source (such as URL) to the mapping value for post-retrieval processing. If the index item is not directly output as a large model prompt, it is called question and answer type, and other index items that are output as large model prompts are called ordinary type. The type is stored in the database as a feature (marker bit) of the index item. The index item structure is: {key: (source) matching key, value: mapping value [URL], type: [ordinary|question and answer|reference]}.

[0063] 1.2 Multi-process the source matching key of the index item to generate a new matching key

[0064] 1.2.1 Synonymize the source matching key as a whole

[0065] The synonymous content generation requirement and the source matching key are combined into a prompt, and the source matching key is synonymously expanded using a large language model. When the number of source matching keys is N and the number of synonymous expansions is M-1, an N-by-M array A of source matching keys and expanded matching keys is obtained:

[0066]

[0067] Among them, key is the matching key; n is the serial number of the index item in the vector database; m is the serial number of the matching key in the index item. The corresponding meanings of each element in array A are:

[0068]

[0069] 1.2.2 Semantic completion of generalized matching keys

[0070] There are cases where components are omitted in the language, which affects the accuracy of keyword matching in specific scenarios, so completion is required. A dependency grammar analysis model (such as LTP) is used to perform grammatical analysis on the matching key components in A to obtain word segmentation and corresponding part of speech, grammar, semantic unit markers and structures, and then process them through a set of rules. Due to differences in requirements and the pertinence of rules, here we take parallel completion as an example to explain only the basic algorithm:

[0071] (1) Extract the core words (HED / Root) of the parallel grammatical and semantic units (COO / eCOO) at the same time.

[0072] (2) Determine whether the central word and the parallel word have the same part of speech. If they do, proceed to 1.2.2.3; if they do not, proceed to 1.2.2.4.

[0073] (3) Copy the component that directly points to the central word and insert it to the left side of the parallel word (if the component is on the left side of the central word) or to the right side of the parallel word (if the component is on the right side of the central word).

[0074] (4) When Chinese is used as the scenario, the core word is divided into three types of characters: Chinese, non-Chinese, and numbers. If the parallel language only contains English or numbers, the missing Chinese characters are inserted into the parallel language, and the insertion position is consistent with the position of the part in the core word.

[0075] Using the above algorithm, "Shenzhou 15 and 16" can be completed as "Shenzhou 15 and Shenzhou 16".

[0076] 1.2.3 Split the matching key after completion into multiple granularity language units

[0077] The splitting is based on the results of dependency grammar analysis, and the splitting is different for specific scenarios and languages. For example, "Shenzhou 15 and Shenzhou 16" will be split into "Shenzhou 15" and "Shenzhou 16", removing non-key words such as "and" (such as the semantic marker mRELA). If there is a negation word (mNEG), the negation word and its central word are not split as a whole, for example, "don't like" is not split into "don't" and "like". The language unit retained after the split is called a matching subkey. Matching key 1.1 is decomposed into an L-dimensional vector: [matching subkey 1.1.1, matching subkey 1.1.2,…, matching subkey 1.1.L]. Array A becomes an N times M times L array B, B = [B1 B2 …B N ],in,

[0078]

[0079] Among them, skey is the matching subkey after the matching key is split; n is the sequence number of the index item in the vector database; m is the sequence number of the matching key in the index item; l is the sequence number of the matching subkey in the matching key.

[0080] 1.2.4 Perform secondary synonym generalization on the split language units

[0081] The synonymous content generation requirements and matching subkeys are combined into prompts, and the matching subkeys are synonymously expanded using a large language model. Each matching subkey selects K-1 synonymous expansions. At this time, full expansion will produce K to the power of L possible combinations. The present invention does not expand the combination, but randomly selects K candidates at each matching subkey position under each matching key, puts the combination into a set, and repeats sampling, combination, and putting into the set until the cardinality of the set reaches a preset value. When the preset value is J-1, array B becomes an array C of N times J times L, C = [C1 C2 ... C N ],in,

[0082]

[0083] At this time, there are J matching keys (including the source matching key) under each of the N index items, and each matching key is composed of L matching sub-keys obtained by sampling.

[0084] 1.2.5 Verify the semantics of ordered combinations of generalized units (generate regularized weights)

[0085] The matching subkeys of each matching key in array C are concatenated into strings in order, and the strings are converted into semantic vectors using an embedding model (such as SBERT, etc.). The semantic vector corresponding to the source matching key is used as a benchmark, and the semantic similarity between the semantic vector of the matching key and the corresponding source matching key benchmark semantic vector is calculated to obtain the similarity matrix S.

[0086]

[0087] The weight is: s n,j =sim(jkey n,1 ,jkey n,j ), jkey is the ordered concatenation of all matching subkeys under the same matching key: represents splicing; the approximation uses the normalized angle cosine value of the semantic vector, and the calculation function is,

[0088]

[0089] Among them, x and y are language texts, and vec returns the semantic vector corresponding to the string, that is, embedding is performed on x and y. Then the normalized weight is calculated to obtain the normalized weight matrix W.

[0090]

[0091] Serving size:

[0092] Among them, s is the component in the proximity matrix S. The role of the regularization weight is to quantify the semantic proximity obtained by matching keys after distinguishing generalization, which is used in 2.2.4.

[0093] 1.2.6 Constructing a new matching key index structure

[0094] The N source matching keys in array C correspond one-to-one to the N source matching keys in the index item structure in 1.1.2. The J matching keys under each source matching key and the corresponding source matching key index item are in a many-to-one relationship. Each matching key corresponds to a normalized weight. At this time, the logical structure of each index item is:

[0095]

[0096] match_array corresponds to C n and match_weights corresponds to W n , W n =[w n,1 w n,2 … w n,J ], n is the serial number of the index item in the vector database.

[0097] 2. Vector Database Index Item Retrieval

[0098] Before the big language model processes the prompt, the process of searching and returning matching content from the preset search items in the vector database according to the above. This stage searches and returns the mapping value corresponding to the matching key from the preset search items in the vector database according to the above. When the big language model is used for human-computer dialogue, a complete conversation includes multiple rounds of dialogue content, each round of dialogue corresponds to a big model request, and also includes at least one vector database search.

[0099] 2.1 Organize the above information

[0100] The existing conversation records and the current conversation are sorted and prepared to be submitted to the vector database for retrieval. The sorting process can be divided into two steps:

[0101] 2.1.1 Selecting the nearest context of different lengths

[0102] Different lengths of preceding text will produce different matching results in the search. Longer preceding text will produce more matches, but the content of the mapping value may not be helpful for the question that needs to be answered. It is actually very difficult to reasonably choose the length of preceding text. When the length is limited, preceding texts of multiple lengths are searched, and different preceding text lengths are used on different types of index items. In human-computer dialogue, for question-and-answer index items, the content of the most recent round of dialogue is used for retrieval; for ordinary index items, the content of the most recent 1 to 3 rounds of dialogue is used. In a two-role multi-round dialogue, the selected dialogue content is in the form of,

[0103] Context t =[(A t-2 ,B t-2 ),(A t-1 ,B t-1 ),(A t ,B t )]

[0104] A and B represent the conversation between two characters, usually A represents the human character and B represents the program character. t is the current round, t-1 is the previous round, and so on. t In general, it is empty because the current round of the program role has not yet generated feedback content. For the dialogue scenario, the final text submitted for retrieval is spliced ​​from the content in the Context, for example

[0105] 2.1.2 Semantic completion of the above text

[0106] The method based on dependency grammar analysis and matching rules in 1.2.2 is used to complete the parallel sentences at the word level according to their corresponding central word forms. For multi-role multi-round dialogues, this completion method is used to complete the dialogue content of each role in each round.

[0107] 2.2 Retrieve preset index items

[0108] When searching, first determine the partition to be searched based on the scenario, customer, and other information. The processing can then be divided into five steps:

[0109] 2.2.1 Matching keys to synthesize regular discriminant combinations

[0110] Take out the matching subkeys of all index items in the partition, sort them in ascending order by subkey character length, and reconstruct them into an ordered set of subkeys SKEY. In this process, record the source of the subkey, that is, the index item n to which the subkey belongs and the row j in match_array. The set is in the form of:

[0111] SKEY={skey1,skey2,…,skey l ,…,skey L}

[0112] Among them, l represents the sequence number of the subkey in the ordered set, and l, n and m construct the dictionary D l :l→(n,j) structure is {key:l,value:(n,j)}. Due to the large number of subkeys, a single regular discriminant cannot carry all subkeys (for example, in Python's re package, we limit the regular pattern to no more than 60,000 characters). Therefore, the ordered set of subkeys is converted into a regular discriminant combination REXS.

[0113]

[0114] The subkeys are connected by |, ri is the serial number of the discriminant in the combination, rj is the serial number of the subkey in the discriminant, and the serial number l of the subkey in the ordered set can be obtained through ri and rj. According to the mapping relationship between ri, rj and l, the dictionary D is constructed. r :(ri,rj)→l The structure is {key:(ri,rj),value:l}.

[0115] 2.2.2 Constructing the indirect index matrix

[0116] Construct the indirect index matrix R according to the ordered set of subkeys obtained in 2.2.1,

[0117]

[0118] Where the row index is x, there is a mapping D from x to the index item number n and the match_array row number j tuple x :x→(n,j), the column index is l, the element value is,

[0119]

[0120] That is, when there is a position subkey of the ordered set l in the j-th matching key under the n-th index item in the partition, the corresponding r x,l is 1, otherwise 0. X is the total number of matching keys in the index item under the specified partition, and L is the cardinality of the ordered set in 2.2.1. "Indirect" means that the row number x of the matrix R needs to be mapped to the index item number n and the match_array row number j to identify the element r. x,l The value of .

[0121] 2.2.3 Generate Regular Discriminant Combination Hit Vector

[0122] (1) Generate a discriminant matrix. Use the regular discriminant combination REXS to discriminate the previous text involved in the retrieval and generate a discriminant matrix D.

[0123]

[0124] When there is a matching subkey corresponding to the position (ri, rj) in the above text, the element is 1, otherwise it is 0. In practical applications, most of the elements are 0, and the discriminant matrix is ​​very sparse.

[0125] (2) Constructing a hit vector, The initial element value is 0. Since the discriminant matrix is ​​sparse, sparse matrix operation methods (such as those provided by Python's SciPy) can be used, or matrix operations can be omitted and the dictionary D can be used directly. r :(ri,rj)→l, convert the position (ri,rj) with element value 1 to the serial number l, and assign the components of the hit vector. Logically, we can directly get Component value, that is, the lth subkey of the subkey ordered set exists above, The lth position component of is 1, otherwise it is 0:

[0126]

[0127] 2.2.4 Calculate the normalized approximation based on the hit vector, index matrix, and normalized weight

[0128] The normalized weights obtained in 1.2.6 are taken from all index items in the partition in order to form a normalized weight matrix, and the order of obtaining is consistent with the order of obtaining the matching subkeys in 2.2.1. If the partitions in the retrieval phase and the index item construction phase are the same, the normalized weight matrix is ​​in the form of W in 1.2.5.

[0129] (1) Based on the hit vector in 2.2.3 and the index matrix in 2.2.2, calculate the approximation vector

[0130]

[0131] (2) According to the mapping relationship D in 2.2.2 x :x→(n,j), the proximity vector can be folded into a matrix S and point-wise multiplied with the regularized weight matrix W to obtain the regularized proximity matrix RS.

[0132]

[0133] Because the value range of the approximation and the normalized weight are both real numbers between 0 and 1, the product is also a real number between 0 and 1, and must not be greater than the original approximation, that is, the normalized approximation is less than or equal to the original approximation. Therefore, the approximation can be truncated before normalization according to the preset minimum approximation threshold, that is, the approximation below the threshold is returned to 0. In practical applications, there are many components of the original approximation below the threshold. At this time, the folded proximity matrix is ​​a sparse matrix, and the sparse matrix algorithm can be used to calculate point-by-point multiplication, or the weighted proximity can be directly calculated without folding.

[0134] 2.2.5 Determine the returned content

[0135] Through 2.2.4, we can get the normalized similarity of all matching keys in the specified partition. According to the index item to which the matching key belongs, the similarity can be grouped by index item, that is, the row vector of RS. The maximum value of each group is selected as the similarity sim between the above text and the index item. RS (n),

[0136] sim RS (n) = max(RS n )

[0137] The process of selecting index items is:

[0138]

[0139] Where th is the similarity threshold under a specific context and partition. Different types of index items have different similarity thresholds and maximum return count settings, and the settings for different customers and scenarios are also different. In principle, the similarity threshold for question-and-answer type is higher than that for ordinary type, and the number of returns is less.

[0140] The method of the present invention divides the preset index items (entries) in the vector database into two parts, namely, matching keys and mapping values. The matching keys are used in the matching calculation with the above text. When the matching key matches the above text to a high enough degree, the mapping value is returned. The method of the present invention performs matching calculations on all preset matching keys and the above text, and determines whether the matching key matches the above text to a high degree.

[0141] The method of the present invention integrates semantic vectors and keyword retrieval methods to provide a new method for partial semantic approximation between the preceding text and the matching key, which functionally covers the traditional vector database retrieval method based on semantic vectors and keywords. In the traditional vector database retrieval method, semantic vector retrieval can only determine whether the vectorized preceding text as a whole matches a preset matching key as a whole, while keyword retrieval is limited to the presence of preset keywords in the preceding text. The semantic vector solves the problem of semantic approximation and cannot determine whether the preceding text contains the meaning of the matching key. The latter can only determine whether the preceding text literally contains the keyword and cannot solve the problem of partial semantic approximation. The method of the present invention increases the matching hit rate through two generalizations, and quantifies the approximation deviation brought by generalization through regularization weights and introduces it into the calculation.

[0142] When the context is semantically similar to the matching key part, the combination of parts may contradict their meanings, for example, part of it is positive, but in the overall context it is a negative semantic fragment. This kind of composition error is a difficulty in dealing with partial semantic similarity. The present invention completes, screens and limits the splitting granularity of the text through grammatical analysis, improves the matching accuracy at the semantic level, and avoids serious composition errors.

[0143] In addition, when solving some semantic similarity problems, generalization brings higher computational complexity. How to control computational complexity is also a problem that must be considered in practical applications. The present invention simplifies the processing process and improves the scalability of the method by combining mathematical modeling and calculation methods such as hit vectors and indirect index matrices with regular discriminants.

[0144] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0145] The present invention provides a vector database composite retrieval method, which can improve data retrieval accuracy and coverage, provide more effective prompt content for large language models, and reduce the application difficulty of large language models. The method of the present invention completes, screens and limits the split granularity of text through grammatical analysis, improves the matching accuracy at the semantic level, and avoids serious synthetic errors. The method of the present invention simplifies the processing process and improves the scalability of the method by combining mathematical modeling and calculation methods such as hit vectors and indirect index matrices with regular discriminants.

[0146] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A vector database composite retrieval method, characterized by: The following steps are included: S1. Construction of vector database index items: Generate initial index items according to the data format and index item type of the submitted original data; perform multiple processing on the source matching key of the index item to generate a new matching key; S2. Vector database index item retrieval: The above information is sorted out, and according to the above information, a mapping value corresponding to a matching key is searched from a preset search item of a vector database and returned.

2. The vector database composite retrieval method according to claim 1, characterized in that: Step S1 specifically includes the following contents: S10, determining the partition to which the generated index item belongs, and storing it as an attribute of the index item in the vector database in a logical structure; The partition is the basis for defining the search scope during retrieval, which is determined by the scenario to which the index item belongs and the customer; S11, identifying the data form and index item type of the original data, and determining the initial index item according to the data form and the index item type, wherein the matching key of the initial index item is the source matching key; S12. Perform synonym generalization on the source matching key of the index item as a whole, perform semantic completion on the matching key after synonym generalization, split the matching key after semantic completion into multi-granularity language units, perform secondary synonym generalization on the matching key after splitting into multi-granularity language units, calculate the regularization weight of the matching key after secondary synonym generalization, so as to generate a new matching key and index structure.

3. The vector database composite retrieval method according to claim 2, characterized in that: Step S11 specifically includes the following contents: S111. The data form of the original data includes unstructured language text and semi-structured data table; the index item type is the usage mode after the data is finally formed into index items, and the usage mode includes using the mapping value as a large language model prompt or directly as a final answer; S112. For unstructured language text, segmentation is performed based on the common method, and the text fragments are selected by moving windows as matching keys, and the adjacent fragments are used as mapping values. For unstructured tables, the information source needs to be added to the mapping value. If the index item is not used as a large model prompt but as a direct output, it is called the question and answer type. Other index items that are output as large model prompts are called ordinary types. The type is stored in the vector database as a feature of the index item. The index item structure is: {key: (source) matching key, value: mapping value [URL], type: [ordinary|question and answer|reference]}.

4. The vector database composite retrieval method according to claim 2, characterized in that: Step S12 specifically includes the following contents: S121, combining the synonymous content generation requirement and the source matching key into a prompt, and using the large language model to perform synonym expansion on the source matching key; when the number of source matching keys is N and the number of synonym expansions is M-1, an N-by-M array A of the source matching keys and the expanded matching keys is obtained; S122, using the dependency grammar analysis model to perform grammar analysis on the matching key components in array A, obtain word segmentation and corresponding part of speech, grammar, semantic unit tags and structures, and perform semantic completion according to the completion rules; S123, in the process of splitting multi-granularity language units, remove non-key words. If there is a negative word, the negative word and its central word are not split as a whole. The language unit retained after the split is called a matching subkey. Each matching key is split into an L-dimensional vector, and the array A becomes an N-by-M-by-L array B = [B1 B2 ... B N ]; S124, the synonymous content generation requirement and the matching subkey are combined into a prompt, the matching subkey is synonymously expanded using a large language model, K candidate synonymous expansions are randomly selected at each matching subkey position under each matching key, and the combination is put into a set, and the sampling, combination, and putting into the set are repeated until the cardinality of the set reaches a preset value; when the preset value is J-1, the array B becomes an N-J-L array C=[C1 C2 ... C N ]; S125, sequentially concatenate the matching subkeys of each matching key in the array C into a string, use the embedding model to convert the string into a semantic vector, use the semantic vector corresponding to the source matching key as a reference, calculate the semantic similarity between the semantic vector of the matching key and the corresponding source matching key reference semantic vector, obtain a similarity matrix S, calculate the regularization weight based on the similarity matrix, and obtain a normalized weight matrix W; S126, the N source matching keys in array C correspond one to one with the N source matching keys in the index item structure in step S112, and the J matching keys under each source matching key and the corresponding source matching key index item are in a many-to-one relationship, and each matching key corresponds to a normalized weight. At this time, the logical structure of each index item is: {key: (source) matching key, value: mapping value [URL], type: [normal|question and answer|reference]; match_array corresponds to C n and match_weights corresponds to W n =[w n,1 w n,2 … w n，J ]; n=1~N.

5. The vector database composite retrieval method according to claim 4, characterized in that: The completion rules in step S122 specifically include the following: S1221, extract the central words of parallel grammatical and semantic units together; S1222, determine whether the central word and the parallel word have the same part of speech, if they are the same, proceed to step S1223, otherwise proceed to step S1224; S1223, according to the position of the component words on the left and right sides of the central word, copy the component words directly pointing to the central word and insert them into the corresponding side of the parallel word; S1224. When Chinese is used as the scenario, the core word is divided into three types of characters: Chinese, non-Chinese, and numbers. If the parallel language only contains English or numbers, the missing Chinese characters are inserted into the parallel language, and the insertion position is consistent with the position of the part in the core word.

6. The vector database composite retrieval method according to claim 4, characterized in that: Step S2 specifically includes the following contents: S21, sorting out the existing conversation records and the current conversation, selecting the most recent contexts of different lengths, and semantically completing the contexts; S22. Synthesize a regularized discriminant combination based on the matching keys, construct an indirect index matrix, generate a regularized discriminant combination hit vector, calculate the regularized approximation of the matching keys based on the hit vector, the indirect index matrix and the regularized weight, and determine the retrieval return content of the index item based on the regularized approximation.

7. The vector database composite retrieval method according to claim 6, characterized in that: Step S21 specifically includes the following contents: S211, select the most recent context of different lengths, search for contexts of multiple lengths, and use different context lengths for different types of index items; in human-computer dialogue, for question-and-answer index items, use the content of the most recent dialogue round to search; for general index items, use the content of the most recent 1 to 3 dialogue rounds; for dialogue scenarios, the context finally submitted for search is spliced ​​from the content in the dialogue; S212, using the method based on dependency grammar analysis and completion rules in step S122 to complete the parallel terms at the word level according to their corresponding central word forms; for multi-role multi-round dialogues, this method is used to complete the dialogue content of each role in each round.

8. The vector database composite retrieval method according to claim 6, characterized in that: Step S22 specifically includes the following contents: S220, determining the partition to be searched according to the scenario and customer information; S221. Take out the matching subkeys of all index items in the partition, sort them in ascending order according to the length of the subkey characters, and reconstruct them into an ordered set of subkeys SKEY = {skey1, skey2, ..., skey l , ..., skey L }, the subkey ordered set records the source of the subkey, that is, the index item n to which the subkey belongs and the row number j in match_array, l is the sequence number of the subkey in the subkey ordered set, l, n and j construct the dictionary D l :l→(n, j), the structure is {key: l, value: (n, j)}; Convert the ordered set of subkeys SKEY into a regular discriminant combination REXS, and construct a dictionary D according to the mapping relationship between the serial number ri of the regular discriminant in the regular discriminant combination and the serial number rj and l of the subkey in the regular discriminant. r (ri, rj)→l, the structure is {key: (ri, r·j), value: l}; S222. Construct an indirect index matrix R according to the ordered set of subkeys SKEY. The row index in the indirect index matrix R is x, and there is a mapping relationship between x and the tuple of index item number n and match_array row number j, D x :x→(n, j); the row number x of the matrix R needs to be mapped to the index item number n and the match_array row number j to form the upper discriminant element r x，l The value of S223, combining hit vectors based on regular discriminant REXS; S224, calculating the regularized approximation based on the hit vector, the indirect index matrix, and the regularized weight; S225. According to the index item to which the matching key belongs, the regularized approximation is grouped by index item, and the maximum value in each group is used as the approximation between the above text and the index item, and the mapping value corresponding to the matching key is determined based on the approximation of the index item.

9. The vector database composite retrieval method according to claim 8, characterized in that: Step S223 specifically includes the following contents: S2231, using the regular discriminant combination REXS to discriminate the above texts involved in the retrieval, and generating a discriminant matrix D; S2232, build hit vector The initial element value is 0; use sparse matrix operation method or directly through dictionary D r : (ri, rj)→l, convert the position (ri, rj) with element value 1 into serial number l, and assign values ​​to the components of the hit vector.

10. The vector database composite retrieval method according to claim 8, characterized in that: Step S224 specifically includes the following contents: S2241, multiplying the hit vector by the transposed matrix of the indirect index matrix to obtain a proximity vector; S2242. According to the mapping relationship in step S222, the proximity vector is folded into a proximity matrix S, and the proximity matrix S is multiplied point by point with the regularized weight matrix W to obtain a regularized proximity matrix RS.

Images