Vector knowledge base retrieval method and device based on artificial intelligence, equipment and medium

Abstract

The invention discloses a vector knowledge base retrieval method and device based on artificial intelligence, equipment and a medium, and relates to the technical field of artificial intelligence and financial science and technology. The embedding model is used for mapping information of different importance degrees of the retrieval item to different dimensions of a vector space to obtain a first embedding vector of the retrieval item, then the first embedding vector is cut into first sub-vectors according to retrieval granularity, the first sub-vectors with small dimensions are selected to participate in retrieval of a vector knowledge base, and the second sub-vectors with small dimensions are selected to participate in retrieval of the vector knowledge base; after the candidate vector is retrieved, the retrieval range of the next retrieval task is reduced according to the candidate vector, so that when the first sub-vector with the larger dimension is used for performing vector knowledge base retrieval, retrieval can be performed in the smaller retrieval range, the similarity calculation task participated by the vector with the larger dimension is reduced, and the retrieval efficiency is improved. The similarity calculation amount of the high-dimensional vector is reduced, and the retrieval efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and financial technology, and in particular to a vector knowledge base retrieval method, device, equipment and medium based on artificial intelligence. Background Art

[0002] The development of vector knowledge base provides a feasible and fruitful idea for the enhanced generation of large model retrieval. Vector knowledge base can be applied to a variety of fields such as economic analysis and financial technology. For example, vector knowledge base can be used in the research and development and application of large economic scenario models to achieve economic data question and answer, analysis and report writing. For example, in some financial business scenarios, users can use the question and answer model built based on the vector knowledge base to ask for information about a certain insurance product, credit, stock, or fund.

[0003] The current construction of vector knowledge bases in the industry generally follows the process of feature extraction, pre-embedding, and large-scale storage, and implements similarity retrieval on this basis. Some optimized similarity retrieval can be calculated on graphics computing cards, and retrieval results can be obtained relatively quickly.

[0004] However, although there are some optimized retrieval techniques that can be quickly calculated on graphics cards, when faced with large-scale data sets, similarity retrieval requires calculating and comparing a large number of vectors in high-dimensional space. The retrieval process will still become relatively slow and inefficient. Summary of the invention

[0005] The present invention provides a vector knowledge base retrieval method, device, equipment and medium based on artificial intelligence to solve the problem of low retrieval efficiency.

[0006] In a first aspect, a vector knowledge base retrieval method based on artificial intelligence is provided, the method comprising: inputting a search item into a trained embedding model so that the embedding model maps information of different importance levels corresponding to the search item to different dimensions of a vector space to obtain a first embedding vector; obtaining a search granularity for the vector knowledge base; according to the search granularity, trimming the first embedding vector to obtain a trimmed first sub-vector; selecting one of the first sub-vectors from small to large in order according to the number of dimensions of the first sub-vector to participate in a single search task of the vector knowledge base, so as to retrieve a candidate vector matching the first sub-vector from the vector knowledge base; wherein the search range of the next search task in the vector knowledge base is determined based on the candidate vector retrieved by the previous search task; and the candidate vector retrieved by the last search task is used as a target vector matching the search item.

[0007] In a second aspect, a vector knowledge base retrieval device based on artificial intelligence is provided, the device comprising: a vector embedding module, used to input the search terms into a trained embedding model, so that the embedding model maps the information of different importance levels corresponding to the search terms to different dimensions of the vector space to obtain a first embedding vector; an acquisition module, used to obtain the search granularity of the vector knowledge base; a vector cropping module, used to crop the first embedding vector according to the search granularity to obtain a cropped first sub-vector; a retrieval module, used to select one of the first sub-vectors in order from small to large according to the number of dimensions of the first sub-vector to participate in a single retrieval task of the vector knowledge base, so as to retrieve a candidate vector matching the first sub-vector from the vector knowledge base; wherein the search range of the next retrieval task in the vector knowledge base is determined based on the candidate vector retrieved by the previous retrieval task; the candidate vector retrieved by the last retrieval task is used as the target vector matching the search term.

[0008] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned artificial intelligence-based vector knowledge base retrieval method when executing the computer program.

[0009] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned artificial intelligence-based vector knowledge base retrieval method are implemented.

[0010] In the scheme implemented by the above-mentioned artificial intelligence-based vector knowledge base retrieval method, device, equipment and medium, the search items that need to be referenced when searching the vector knowledge base are input into the trained embedding model, and the embedding model is used to map the information of different importance levels of the search item to different dimensions of the vector space to obtain the first embedding vector of the search item, and then the first embedding vector is cut into the first sub-vector according to the retrieval granularity, and the first sub-vector with a small dimension is first selected to participate in the retrieval of the vector knowledge base, and then after the candidate vector is retrieved, the retrieval scope of the next retrieval task is narrowed according to the candidate vector. In this way, when the first sub-vector with a larger dimension is used to search the vector knowledge base, the retrieval can be carried out within a smaller retrieval range, thereby reducing the similarity calculation tasks involving vectors with a larger dimension, reducing the similarity calculation amount of high-dimensional vectors, and improving retrieval efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0012] Figure 1 It is a schematic diagram of an application environment of a vector knowledge base retrieval method based on artificial intelligence in one embodiment of the present invention; Figure 2 It is a flow chart of a vector knowledge base retrieval method based on artificial intelligence in one embodiment of the present invention; Figure 3 is a flow chart of a vector knowledge base retrieval method based on artificial intelligence in another embodiment of the present invention; Figure 4 is a schematic diagram of cropping the second embedding vector in another embodiment of the present invention; Figure 5 is a flow chart of a vector knowledge base retrieval method based on artificial intelligence in another embodiment of the present invention; Figure 6 is a flow chart of a vector knowledge base retrieval method based on artificial intelligence in another embodiment of the present invention; Figure 7 It is a structural schematic diagram of a vector knowledge base retrieval device based on artificial intelligence in one embodiment of the present invention; Figure 8 It is a structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] The vector knowledge base retrieval method based on artificial intelligence provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the client communicates with the server through a network. The server can obtain the search item input by the user in the client interactive interface, and the server uses the trained embedding model to obtain the first embedding vector corresponding to the search item, and then cuts the first embedding vector with reference to the search granularity to obtain first sub-vectors with different dimensions, and uses these first sub-vectors to carry out multiple rounds of searches on the vector knowledge base. As the number of dimensions of the first sub-vectors involved in the search continues to increase, the search range will continue to narrow. When using the first sub-vector with a large dimension to search the vector knowledge base, there is no need to search the entire vector knowledge base, but only to search within the range of the candidate vector retrieved by the previous first sub-vector. Further, the retrieved target vector can be returned to the client so that the client generates a response message based on the target vector. The computational complexity and similarity matching between high-dimensional vectors are much higher than those of low-dimensional vectors. The present application uses sub-vectors with different dimensional numbers to carry out multiple rounds of searches on the vector knowledge base, gradually narrowing the search range of high-dimensional sub-vectors, reducing the amount of similarity calculation tasks between high-dimensional vectors, improving overall computational efficiency, and improving search efficiency.

[0015] The client may include but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server may be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0016] See also Figure 2 As shown, Figure 2 A flowchart of a vector knowledge base retrieval method based on artificial intelligence provided by an embodiment of the present invention includes the following steps: S201, inputting the search terms into the trained embedding model, so that the embedding model maps information of different importance levels corresponding to the search terms to different dimensions of the vector space, and obtains a first embedding vector.

[0017] Vector knowledge base retrieval can be understood as calculating the similarity between a certain embedding vector A and the embedding vectors stored in the vector knowledge base, thereby selecting the one with the highest similarity to the embedding vector A from the embedding vectors stored in the vector knowledge base.

[0018] The search term can be understood as the text, picture and other data that needs to be referenced when searching the vector knowledge base. The trained embedding model generates an embedding vector for the search term, that is, the above-mentioned embedding vector A. For the sake of distinction, this application expresses it as the first embedding vector.

[0019] For example, when the vector knowledge base is applied to a retrieval system, users can enter text information related to their query requirements in the retrieval system so that the system can find relevant knowledge items in the vector knowledge base. In this scenario, the search terms can be understood as all types of information entered in the retrieval system, whether it is text, keywords, phrases, more complex query statements or images, etc.

[0020] For example, the search term may be "Please recommend insurance products suitable for people aged 20-30."

[0021] The trained embedding model processes the search term and finally outputs the first embedding vector of the search term.

[0022] The embedding model is used to map information of different importance levels corresponding to the search terms to different dimensions of the vector space to obtain a first embedding vector.

[0023] For example, in the search term "Please recommend insurance products suitable for people aged 20-30", the following order can be obtained according to the importance of the words: "insurance products" > "20-30 years old" > "recommend" > "suitable" > "people" > "of" > "a" > "please". Then the information corresponding to the words with higher importance can be mapped to the low dimension, and as the importance decreases, the information can be gradually mapped to the high dimension.

[0024] If the dimension of the vector output by the fixed embedding model is 2048, then the embedding model can map the text information such as "insurance product", "20-30 years old", and "recommendation" to the first 521 dimensions, and map the text information such as "suitable" and "population" to the first 522-first 1024 dimensions, and so on. The search item "Please recommend insurance products suitable for people aged 20-30" is completely mapped into the vector space to obtain the first embedding vector of 2048 dimensions.

[0025] That is to say, for the first 2048-dimensional embedding vector, the information "insurance product", "20-30 years old", and "recommended" can be obtained from the first 512-dimensional vector elements; the information "insurance product", "20-30 years old", "recommended", "suitable", and "population" can be obtained from the first 1024-dimensional vector elements.

[0026] It should be noted that the "front" in the "front 512 dimensions", "front 1024 dimensions", "front 512 dimensions", etc. described in this embodiment can be understood by the following explanation: the two vector elements x1 and x2 in the first embedded vector, the vector element close to the head element of the first embedded vector (that is, the first vector element of the first embedded vector) is the front element, and the vector element far from the head element is the back element.

[0027] In this way, the embodiment of the present application can use elements at different positions in the first embedded vector to represent information of different importance. Preferably, information with high importance can be represented first, and then as the dimension increases, information with gradually decreasing importance is added to the first embedded vector, so that the first embedded vector can fully represent the content of the search item. In this way, the key content / approximate content of the search item can also be obtained through the low-dimensional data of the first embedded vector. In general, the lower the dimension of the vector element in the first embedded vector, the higher the importance of the information it represents.

[0028] Among them, the embedding model maps information of different importance to different dimensions of the vector space. This ability can be acquired through model training.

[0029] S202, obtaining the search granularity of the vector knowledge base.

[0030] The retrieval granularity of the vector knowledge base can be pre-set and flexibly switched according to scenario requirements or application requirements.

[0031] For example, retrieval granularity can be roughly divided into coarse-grained and fine-grained.

[0032] In some embodiments, the search granularity may be quantified to divide the search granularity into multiple types between coarse granularity and fine granularity.

[0033] Coarse-grained: usually refers to a classification or division in which each category or division unit covers a larger scope but has fewer details.

[0034] Fine-grained: On the contrary, it means that in classification or division, each category or division unit covers a smaller scope and has more details.

[0035] "Coarse-grained" appears to be more "rough" in terms of granularity because it covers a large range and has few details. On the contrary, "fine-grained" appears to be more "fine" in terms of granularity because it covers a small range and has many details.

[0036] Based on the above analysis, it is clear that the granularity of "coarse-grained" is greater than that of "fine-grained". The "large" here means that it covers a wider range and has fewer details.

[0037] For example, when you need to roughly search for sentences with similar meanings to a certain text from the vector knowledge base, you can perform a coarse-grained search. When you need to accurately search for answer information that matches a certain question and answer from the vector knowledge base, you can perform a fine-grained search.

[0038] S203: Crop the first embedded vector according to the retrieval granularity to obtain a cropped first sub-vector.

[0039] When the search granularity is large, the maximum number of dimensions of the first sub-vector cut out may be relatively reduced; and / or the number of the first sub-vector cut out may also be relatively reduced; When the search granularity is small, the maximum number of dimensions of the cut-out first sub-vector may be relatively increased; and / or the number of the cut-out sub-vectors may also be relatively increased.

[0040] For example, when performing coarse-grained retrieval, the 2048-dimensional first embedding vector is cropped once to obtain a 128-dimensional first sub-vector; when performing fine-grained retrieval, the 2048-dimensional second embedding vector is cropped four times to obtain a 64-dimensional first sub-vector, a 128-dimensional first sub-vector, a 512-dimensional first sub-vector, and a 2048-dimensional first sub-vector, respectively.

[0041] S204, according to the number of dimensions of the first sub-vector, select one of the first sub-vectors in order from small to large to participate in a single retrieval task of the vector knowledge base, so as to retrieve a candidate vector matching the first sub-vector from the vector knowledge base; wherein, the retrieval range of the next retrieval task in the vector knowledge base is determined according to the candidate vector retrieved in the previous retrieval task; the candidate vector retrieved in the last retrieval task is used as the target vector matching the retrieval item.

[0042] According to the order of the number of dimensions of the first sub-vector, a first sub-vector is selected from small to large to participate in a single retrieval task of the vector knowledge base.

[0043] The size of the above vector dimension can be understood as the size of the vector size, the number of vector elements included. For example, a 64-dimensional first sub-vector may include 64 vector elements, and a 128-dimensional first sub-vector may include 128 vector elements; compared with the 64-dimensional first sub-vector and the 128-dimensional first sub-vector, the 64-dimensional first sub-vector is the first sub-vector with a smaller dimension, and the 128-dimensional first sub-vector is the first sub-vector with a larger dimension.

[0044] For example, the first sub-vectors cut out in step S203 include a 64-dimensional first sub-vector, a 128-dimensional first sub-vector, a 512-dimensional first sub-vector, and a 2048-dimensional first sub-vector. When performing a vector knowledge base search, the following process can be used: First, the 64-dimensional first sub-vector is used to perform the first search task in the vector knowledge base, and a candidate vector matching the 64-dimensional first sub-vector is retrieved from the vector knowledge base, that is, the candidate vector corresponding to this search task.

[0045] Then, the search range of the second search task in the vector knowledge base is determined based on the candidate vector matching the 64-dimensional first sub-vector, and then the second search task is continued within the search range using the 128-dimensional first sub-vector. For example, similarity calculation is performed between the 128-dimensional first sub-vector and the vectors belonging to the search range in the vector knowledge base, and the candidate vector matching the 128-dimensional first sub-vector is screened out as the candidate vector corresponding to the second search task.

[0046] The search range of the third search task in the vector knowledge base is determined based on the candidate vectors that match the 128-dimensional first sub-vector, and then the third search task is continued within the search range using the 512-dimensional first sub-vector to screen out the candidate vectors that match the 512-dimensional first sub-vector.

[0047] Repeat the above process, and use the 2048-dimensional first sub-vector to carry out the fourth retrieval task in the retrieval range determined by the candidate vector retrieved by the third retrieval task, and obtain the candidate vector matching the 2048-dimensional first sub-vector as the candidate vector retrieved by the fourth retrieval task.

[0048] In this example, the fourth retrieval task is the last retrieval task, so the candidate vector retrieved by the fourth retrieval task is used as the target vector matching the retrieval item.

[0049] In some embodiments, if only N target vectors matching the search terms are required, the candidate vectors retrieved by the fourth search task are ranked according to their similarity to the 2048-dimensional first sub-vector, and the top N candidate vectors are used as target vectors.

[0050] In summary, the artificial intelligence-based vector knowledge base retrieval method proposed in the embodiment of the present application inputs the search terms that need to be referenced when searching the vector knowledge base into a trained embedding model, and the embedding model is used to map the information of different importance levels of the search terms to different dimensions of the vector space to obtain the first embedding vector of the search term, and then the first embedding vector is cut into first sub-vectors according to the retrieval granularity, and the first sub-vector with a small dimension is first selected to participate in the retrieval of the vector knowledge base, and then after the candidate vector is retrieved, the retrieval scope of the next retrieval task is narrowed according to the candidate vector. In this way, when the first sub-vector with a larger dimension is used to search the vector knowledge base, the retrieval can be carried out within a smaller retrieval range, thereby reducing the similarity calculation tasks involving vectors with a larger dimension, reducing the similarity calculation amount of high-dimensional vectors, and improving the retrieval efficiency.

[0051] Based on the above embodiments, Figure 3 As shown, the embedding model is trained in the following way: S301, input the training sample into the embedding model to obtain a second embedding vector.

[0052] S302, respectively calculating loss functions between second sub-vectors with different dimensions corresponding to the second embedding vector and the labels of the training samples.

[0053] S303, importance scaling is performed on each of the loss functions, and all the loss functions after importance scaling are aggregated to obtain a target loss function corresponding to the training sample.

[0054] S304: Train the embedding model based on the target loss function until the loss value of the trained embedding model is less than a preset loss threshold.

[0055] The embedding model can be a word embedding model, a recurrent neural network, a convolutional neural network, a Transformer model, etc., and this application does not limit it.

[0056] When training the embedding model, a set of labeled training samples can be selected, and the training samples can be input into the embedding model to generate an embedding vector corresponding to the training sample. For ease of distinction, this application describes it as a second embedding vector.

[0057] The second embedding vector of each training sample is split to obtain second sub-vectors with different dimensions, and then the loss function between each second sub-vector and the label of the training sample is calculated respectively; then importance scaling is performed on all loss functions corresponding to the training sample, for example, each loss function is multiplied by the corresponding importance weight, and then all loss functions after importance scaling are aggregated to obtain the loss function corresponding to the training sample.

[0058] Leveraging Embedding Models Generate a second embedding vector Z of a training sample. The second embedding vector Z is a d-dimensional vector, expressed as ; Then the elements of the first m dimensions of the second embedding vector are taken as the second subvector, expressed as , m is any value in the set M, such as the set .

[0059] The loss function is then calculated between each second subvector and the label of the training sample, and the loss is optimized using standard empirical minimization.

[0060] For example, select a labeled dataset , embedding model F, multi-class loss function Expand the training of the embedding model F to obtain the minimum value shown in the following formula through training, and then you can get the trained embedding model F: in, Denotes the embedding model F consisting of learnable weights Parameterization, represents the i-th training sample in the dataset D, Represents training samples Tags, are the parameters of the linear classifier, is the total number of the second sub-vector, is the importance weight corresponding to the mth second sub-vector.

[0061] In the embodiment of the present application, during model training, by focusing on the loss between the information represented by the elements of different dimensions of the second embedding vector and the labels of the training samples, and assigning different attention levels to the elements of different dimensions (i.e., performing importance scaling on the loss functions corresponding to different second sub-vectors), the trained model can map the information of different importance levels of the search terms to different dimensions of the vector space. For example, by training the embedding model, the embedding model learns to map the key information in the search terms to low dimensions, and gradually maps the remaining information to high dimensions, so that the final embedded vector can preferentially represent important information on low-dimensional elements. In this way, when performing coarse-grained retrieval of a vector knowledge base, only the first 1 / 2 of the first embedding vector can be used. In the case of fine-grained retrieval, the first embedded vector can be used to preferentially select candidate vectors that match the key information of the search item through the low-dimensional vector elements in the first embedded vector, and then the search scope can be narrowed down based on the candidate vectors. Then, some high-dimensional vector elements are superimposed on these vector elements, and further search is performed within the search scope to obtain candidate vectors that highly match most of the information of the search item. This avoids performing a large number of similarity calculations in the vector knowledge base using the high-dimensional first embedded vector, reduces the similarity calculation tasks of high-dimensional vectors, improves the overall calculation efficiency, and thus improves the retrieval efficiency.

[0062] Based on the above embodiment, the importance scaling of each loss function in step S303 may include the following steps: Multiply each loss function by the importance weight corresponding to the loss function to complete the importance scaling of the loss function; Among them, the smaller the dimension of the second sub-vector corresponding to the loss function, the greater the importance weight corresponding to the loss function.

[0063] During model training, the embodiment of the present application pays more attention to the second sub-vector with low dimensionality, and reduces the loss of the second sub-vector with low dimensionality through training, so that the model learns to map the important information of the input data to the elements of fixed dimensionality in the first embedding vector. The number of these elements is small, and the second sub-vector containing these elements is used to perform vector knowledge base retrieval. Through fast similarity matching, vectors matching the important information of the input data can be retrieved from the vector knowledge base.

[0064] Based on the above embodiment, the second sub-vectors with different dimensions in the above step S302 can be obtained in the following way: The second embedded vector is subjected to multiple vector cropping starting from the head element with different cropping step sizes to obtain the second sub-vector including the head element.

[0065] like Figure 4 As shown, for a second embedding vector , the dimension of the second embedding vector is d, that is, the second embedding vector includes d vector elements, then it is cut once from the head element with a step length of d / 16 to obtain the second sub-vector , the vector elements of the second sub-vector are all vector elements between the head element to the d / 16th vector element in the second embedded vector.

[0066] You can use d / 8 as the cutting step to perform another cutting to get the second sub-vector ; You can perform another trimming with a trimming step of d / 4 to get the second sub-vector ; You can perform another trimming with a trimming step length of d / 2 to obtain the second sub-vector ; You can perform another cut with d as the cutting step to get the second sub-vector .

[0067] In this way, we can get Figure 4 The four second sub-vectors are shown.

[0068] In the embodiment of the present application, the step size is calculated each time from the head element of the second embedded vector when the second embedded vector is cropped. In this way, in the obtained second sub-vector, as the number of dimensions of the second sub-vector increases, the information represented by the second sub-vector is continuously accumulated, and the second sub-vector with a higher dimension will contain all the vector elements of the second sub-vector with a lower dimension, and the information that can be represented is becoming more and more complete.

[0069] Based on the above embodiments, Figure 5As shown, in the above S203, “according to the retrieval granularity, the first embedded vector is cropped to obtain a cropped first sub-vector” may include the following steps: S501, obtaining a maximum cropping step length corresponding to the search granularity.

[0070] In this embodiment, the number of dimensions of the first sub-vector required for the last search task may be set in advance for different search granularities, and the number of dimensions serves as the maximum search step corresponding to the search granularity.

[0071] For example, when performing coarse-grained retrieval, the dimension of the first subvector required for the last retrieval task is 512, so when the first embedding vector is pruned, the maximum pruned step is 512. Vector elements 1 to 512 in the first embedding vector can be used as the first subvector of 512.

[0072] S502: Perform vector pruning starting from the head element on the second embedded vector multiple times with different pruning steps that are less than or equal to the maximum pruning step to obtain the first sub-vector including the head element.

[0073] After determining the maximum cropping step corresponding to the retrieval granularity, multiple cropping steps may be selected without exceeding the maximum cropping step, and the second embedding vector may be cropped multiple times using the cropping steps.

[0074] In some embodiments, when performing vector clipping, the total number of clipping steps used may also be pre-configured for the search granularity. For example, the maximum clipping step corresponding to different search granularities and the total number of clipping steps are pre-stored; or all clipping steps corresponding to different search granularities are directly stored.

[0075] For example: One configuration is: the search granularity is 50%, the maximum cutting step is 512, and the total number of cutting steps is 4; Thus, when the 2048-dimensional first embedding vector is cropped, a 512-dimensional first sub-vector, a 256-dimensional first sub-vector, a 64-dimensional first sub-vector, or a 32-dimensional first sub-vector can be cropped; or a 512-dimensional first sub-vector, a 64-dimensional first sub-vector, a 32-dimensional first sub-vector, an 8-dimensional first sub-vector can be cropped, and multiple cropping methods can be used.

[0076] Another configuration method is: the search granularity is 50%, and all the cropping steps include: 512, 256, 64, and 32. In this method, the first sub-vector of a certain combination of dimensional numbers can be fixedly cropped.

[0077] When the trained embedding model preferentially maps high-importance information to low-dimensional vector space, each trimming of the first embedding vector starts from the head element of the first embedding vector, so that each trimmed second sub-vector can represent important information of the search item, and as the number of dimensions of the first sub-vector increases, other information of the search item is continuously expanded based on the important information.

[0078] In an embodiment of the present application, when performing vector knowledge base search at different search granularities, the amount of information required for reference is different. For example, the amount of information required for reference is small at coarse granularity, while the amount of information required for reference is large at fine granularity. Therefore, the number of dimensions of the first sub-vector used in the last search task when performing vector knowledge base search at coarse granularity is smaller than the number of dimensions of the first sub-vector used in the last search task when performing vector knowledge base search at fine granularity.

[0079] In other embodiments, the number of first sub-vectors may be 1, and the search task performed using the first sub-vector is both the first search task and the last search task. The number of dimensions of the first sub-vector is equal to the maximum clipping step corresponding to the search granularity. The first sub-vector can represent important information of the search item, and combined with the requirements of the search granularity, expands other information to a certain extent on the basis of the important information. The candidate vector retrieved from the vector knowledge base using the first sub-vector can be used as a target vector that meets the requirements of the search granularity and can provide response information for the search item.

[0080] In an embodiment of the present application, after determining the maximum cropping step, the first embedded vector can be cropped multiple times without exceeding the maximum cropping step to obtain multiple first sub-vectors with increasing dimensionality. When using these first sub-vectors to search a vector knowledge base, a large-scale search can be performed using the first sub-vector with a small dimensionality first, and then the search range of the next search task can be determined based on the candidate vectors retrieved in the previous search task. As a result, as the number of dimensions of the first sub-vectors participating in the search task increases, the search range will continue to shrink. When the first sub-vector with the largest dimensionality is used for search, the search range will be much smaller than that of the above-mentioned first sub-vector. In this way, the amount of similarity calculation for the first sub-vector with the largest dimensionality will also become smaller, thereby improving the search efficiency.

[0081] Based on the above embodiments, Figure 6 As shown, the above vector knowledge base is constructed in the following way: S601, using the trained embedding model, obtaining a third embedding vector required for constructing the vector knowledge base; S602, performing vector cropping starting from the head element on the third embedded vector multiple times with different cropping step sizes to obtain the third sub-vector including the head element; S603: Store the third sub-vectors with different dimensions in different storage spaces to construct the vector knowledge base.

[0082] In constructing a vector knowledge base, data in the form of text, pictures, etc. that need to be stored in the vector knowledge base are obtained, and these data are input into the trained embedding model to obtain the corresponding embedding vector. For ease of distinction, the embodiment of the present application expresses it as a third embedding vector.

[0083] For example, when building an intelligent response system for fund products, the basic information of each fund product, the latest market analysis, the applicable population, etc. can be constructed into text for input into the embedding model.

[0084] The embedding model generates a third embedding vector for each text. The embedding model can map the high-importance information corresponding to the text to the low-dimensional vector space first, and then expand to the high-dimensional mapping as the importance decreases. In the obtained third embedding vector, the vector elements in the low dimension can represent the key information of the text.

[0085] When storing the third embedded vector, the third embedded vector may be cropped first, and a cropping step is selected each time, and the cropping starts from the head element of the third embedded vector, and each cropping obtains a third sub-vector.

[0086] For example, for the third embedding vector of 2048 dimensions, first use the 8-dimensional cropping step to crop it, and obtain the third sub-vector including the 1st to 8th vector elements in the third embedding vector; then use the 16-dimensional cropping step to crop it, and obtain the third sub-vector including the 1st to 16th vector elements in the third embedding vector, until the cropping step is increased to 2048 dimensions, and the remaining cropping processes repeat the above steps, which will not be repeated here.

[0087] All the third sub-vectors obtained after trimming are stored separately: the third sub-vectors with the same dimension are stored in the same position, and the third sub-vectors with different dimensions are stored separately.

[0088] It should be noted that different storage locations can be different storage spaces divided from the same storage device, such as storing third subvectors of different dimensions in different directories; or different storage locations can also be different storage devices. Different methods can be selected to implement separate storage as needed, and this application does not limit this.

[0089] For example, in the final vector knowledge base, 8-dimensional embedded vectors are stored in the same space, 16-dimensional embedded vectors are stored in the same space... and 2048-dimensional embedded vectors are stored in the same space.

[0090] When performing vector knowledge base retrieval, first use the 8-dimensional first sub-vector to search in the space storing the 8-dimensional embedded vector, sort the retrieved candidate vectors by similarity, select the top 50% candidate vectors, and filter out the 16-dimensional embedded vectors corresponding to the top 50% candidate vectors in the storage space storing the 16-dimensional embedded vectors as the retrieval range corresponding to the 16-dimensional first sub-vector, then use the 16-dimensional first sub-vector and the filtered 16-dimensional embedded vector to calculate the similarity, retrieve a batch of candidate vectors, and then select the top 50% candidate vectors in similarity from this batch of candidate vectors, repeat the above steps to determine the retrieval range corresponding to the 32-dimensional first sub-vector..., until all the first sub-vectors are used to complete the retrieval task.

[0091] It should be noted that the above similarity sorting refers to sorting the candidate vectors according to the similarity between the candidate vectors and the first subvector. The ranking percentage for screening the candidate vectors can be set as needed, and the percentage selected for each screening can be different, which is not limited in this application.

[0092] In some embodiments, a set of cropping steps may be pre-set, and the first embedding vector, the second embedding vector, and the third embedding vector may be cropped based on the set of cropping steps during model training, vector knowledge base establishment, and vector knowledge base retrieval.

[0093] For example, a set of cropping step sizes may include 4, 8, 32, 64, 128, 256, 512, 1024, 2048.

[0094] During embedding model training, the second embedding vector is cropped using each cropping step in the group of cropping steps, so that the second sub-vector cropped by each cropping step participates in the model training; When the vector knowledge base is established, the third embedded vector is trimmed using each trimming step in the set of trimming steps, so that the third sub-vectors of different dimensions can be completely stored in the vector knowledge base, so as to facilitate defining a retrieval range for the first sub-vector of any dimension from the third sub-vector of the same dimension; When searching the vector knowledge base, one or more cutting steps may be selected from the group of cutting steps to cut the first embedded vector as needed, and the cut first sub-vectors of different dimensions may be used for searching in the vector knowledge base.

[0095] On the basis of the above embodiment, there is a multiple relationship between different cropping step sizes, so that the number of dimensions of the cropped sub-vectors can change regularly, which is convenient for subsequent processing.

[0096] The artificial intelligence-based vector knowledge base retrieval method proposed in this application generates an embedding vector of smaller size (i.e., number of dimensions) when initializing the establishment of a vector knowledge base, which reduces the hardware resources required for data processing and storage while reducing costs. This advantage is suitable for resource-constrained application scenarios, such as mobile devices or cloud computing environments, making large-scale data processing more economical and efficient. Using sub-vectors with smaller dimensions in the preliminary retrieval stage can quickly locate retrieval results, and then using the complete embedding vector for detailed processing. This hierarchical processing method improves the overall query speed and data processing capabilities. This not only improves the user experience, but also enhances the system's ability to handle a large number of requests.

[0097] Secondly, the total number of dimensions and complexity of the embedded vector can be adjusted according to the actual application requirements and application load. This flexibility enables the system to effectively cope with different data processing and query scenarios. Whether it is large-scale data or specific query tasks, this method can be used to find suitable processing strategies and resource allocations. In addition, this adaptability also facilitates future upgrades and expansions.

[0098] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0099] In one embodiment, a vector knowledge base retrieval device based on artificial intelligence is provided, and the vector knowledge base retrieval device based on artificial intelligence corresponds one-to-one to the vector knowledge base retrieval method based on artificial intelligence in the above-mentioned embodiment.

[0100] like Figure 7 As shown, the vector knowledge base retrieval device 700 based on artificial intelligence includes a vector embedding module 701, an acquisition module 702, a vector cutting module 703 and a retrieval module 704. The functional modules are described in detail as follows: A vector embedding module 701 is used to input the search terms into a trained embedding model so that the embedding model maps information of different importance levels corresponding to the search terms to different dimensions of a vector space to obtain a first embedding vector; An acquisition module 702 is used to acquire a search granularity for a vector knowledge base; A vector clipping module 703, configured to clip the first embedded vector according to the search granularity to obtain a clipped first sub-vector; A retrieval module 704 is used to select one of the first sub-vectors from small to large in order according to the number of dimensions of the first sub-vector to participate in a single retrieval task of the vector knowledge base, so as to retrieve a candidate vector matching the first sub-vector from the vector knowledge base; Wherein, the search range of the next search task in the vector knowledge base is determined according to the candidate vectors retrieved by the previous search task; The candidate vector retrieved by the last retrieval task is used as the target vector matching the retrieval item.

[0101] Based on the above embodiment, the AI-based vector detection model is also used to train the embedding model in the following manner: Inputting the training sample into the embedding model to obtain a second embedding vector; respectively calculating loss functions between second sub-vectors with different dimensions corresponding to the second embedding vector and the labels of the training samples; Importance scaling is performed on each of the loss functions, and all loss functions after importance scaling are aggregated to obtain a target loss function corresponding to the training sample; The embedding model is trained based on the target loss function until a loss value of the trained embedding model is less than a preset loss threshold.

[0102] On the basis of the above embodiment, the vector knowledge base retrieval device based on artificial intelligence is also used to: multiply each loss function by the importance weight corresponding to the loss function to complete the importance scaling of the loss function; wherein, the smaller the number of dimensions of the second sub-vector corresponding to the loss function, the greater the importance weight corresponding to the loss function.

[0103] Based on the above embodiment, the vector knowledge base retrieval device based on artificial intelligence is also used to obtain a second sub-vector with different dimensions in the following manner: the second embedded vector is subjected to multiple vector cropping starting from the head element with different cropping steps to obtain the second sub-vector including the head element.

[0104] Based on the above embodiment, the vector cropping module is used to: obtain the maximum cropping step corresponding to the retrieval granularity; perform vector cropping on the second embedded vector multiple times starting from the head element with different cropping steps less than or equal to the maximum cropping step to obtain the first sub-vector including the head element.

[0105] On the basis of the above embodiments, the vector knowledge base retrieval device based on artificial intelligence is also used to construct the vector knowledge base in the following manner: using a trained embedding model to obtain a third embedding vector required for constructing the vector knowledge base; performing vector cropping on the third embedding vector multiple times starting from the head element with different cropping steps to obtain the third sub-vector including the head element; storing the third sub-vectors with different dimensions in different storage spaces to construct the vector knowledge base.

[0106] Based on the above embodiment, there is a multiple relationship between different cutting step lengths.

[0107] The present invention provides a vector knowledge base retrieval device based on artificial intelligence. The retrieval items that need to be referenced when searching the vector knowledge base are input into a trained embedding model. The embedding model is used to map the information of different importance levels of the retrieval items to different dimensions of the vector space to obtain the first embedding vector of the retrieval item. The first embedding vector is then cut into first sub-vectors according to the retrieval granularity. The first sub-vector with a small dimension is first selected to participate in the retrieval of the vector knowledge base. After the candidate vector is retrieved, the retrieval range of the next retrieval task is narrowed according to the candidate vector. In this way, when the first sub-vector with a larger dimension is used to search the vector knowledge base, the retrieval can be carried out within a smaller retrieval range. In this way, the similarity calculation tasks involving the vector with a larger dimension are reduced, the similarity calculation amount of the high-dimensional vector is reduced, and the retrieval efficiency is improved.

[0108] For the specific definition of the vector knowledge base retrieval device based on artificial intelligence, please refer to the definition of the vector knowledge base retrieval method based on artificial intelligence in the above text, which will not be repeated here. Each module in the above-mentioned vector knowledge base retrieval device based on artificial intelligence can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0109] In one embodiment, Figure 8 As shown, a computer device 800 is provided, including a memory 801, a processor 802, and a computer program stored in the memory 801 and executable on the processor 802. When the processor executes the computer program, the steps of the vector knowledge base retrieval method based on artificial intelligence described in any of the above embodiments are implemented.

[0110] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the vector knowledge base retrieval method based on artificial intelligence described in any of the above embodiments are implemented.

[0111] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0112] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0113] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0114] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A vector knowledge base retrieval method based on artificial intelligence, characterized in that: The method comprises: Inputting the search terms into the trained embedding model, so that the embedding model maps information of different importance levels corresponding to the search terms to different dimensions of the vector space, to obtain a first embedding vector; Obtaining the retrieval granularity of the vector knowledge base; According to the retrieval granularity, the first embedding vector is trimmed to obtain a trimmed first sub-vector; According to the number of dimensions of the first sub-vector, one of the first sub-vectors is selected in order from small to large to participate in a single search task of the vector knowledge base, so as to retrieve a candidate vector matching the first sub-vector from the vector knowledge base; Wherein, the search range of the next search task in the vector knowledge base is determined according to the candidate vectors retrieved by the previous search task; The candidate vector retrieved by the last retrieval task is used as the target vector matching the retrieval item.

2. The method according to claim 1, characterized in that The embedding model is trained as follows: Inputting the training sample into the embedding model to obtain a second embedding vector; respectively calculating loss functions between second sub-vectors with different dimensions corresponding to the second embedding vector and the labels of the training samples; Importance scaling is performed on each of the loss functions, and all the loss functions after importance scaling are aggregated to obtain a target loss function corresponding to the training sample; The embedding model is trained based on the target loss function until a loss value of the trained embedding model is less than a preset loss threshold.

3. The method according to claim 2, characterized in that The importance scaling of each loss function comprises: Multiply each of the loss functions by the importance weight corresponding to the loss function to complete the importance scaling of the loss function; Among them, the smaller the dimension of the second sub-vector corresponding to the loss function, the greater the importance weight corresponding to the loss function.

4. The method according to claim 2, characterized in that The second sub-vector with different dimensions is obtained by: The second embedded vector is subjected to multiple vector cropping starting from the head element with different cropping step sizes to obtain the second sub-vector including the head element.

5. The method according to claim 1, characterized in that The step of trimming the first embedding vector according to the search granularity to obtain a trimmed first sub-vector includes: Obtaining the maximum cropping step corresponding to the retrieval granularity; Perform vector clipping starting from the head element on the second embedded vector multiple times with different clipping steps that are less than or equal to the maximum clipping step to obtain the first sub-vector including the head element.

6. The method according to claim 1, characterized in that The vector knowledge base is constructed in the following way: Using the trained embedding model, obtaining a third embedding vector required for constructing the vector knowledge base; Performing vector clipping starting from the head element on the third embedded vector multiple times with different clipping step sizes to obtain the third sub-vector including the head element; The third sub-vectors with different dimensions are stored in different storage spaces to construct the vector knowledge base.

7. The method according to any one of claims 4 to 6, characterized in that: There is a multiple relationship between the different cutting step lengths.

8. A vector knowledge base retrieval device based on artificial intelligence, characterized in that: The device comprises: A vector embedding module, used for inputting a search term into a trained embedding model, so that the embedding model maps information of different importance levels corresponding to the search term to different dimensions of a vector space, thereby obtaining a first embedding vector; An acquisition module, used for acquiring the retrieval granularity of the vector knowledge base; A vector clipping module, used for clipping the first embedded vector according to the retrieval granularity to obtain a clipped first sub-vector; A retrieval module, configured to select one of the first sub-vectors from small to large in order according to the number of dimensions of the first sub-vector to participate in a single retrieval task of the vector knowledge base, so as to retrieve a candidate vector matching the first sub-vector from the vector knowledge base; Wherein, the search range of the next search task in the vector knowledge base is determined according to the candidate vectors retrieved by the previous search task; The candidate vector retrieved by the last retrieval task is used as the target vector matching the retrieval item.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the vector knowledge base retrieval method based on artificial intelligence as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the vector knowledge base retrieval method based on artificial intelligence as claimed in any one of claims 1 to 7 are implemented.

Images