Artificial intelligence-based vector knowledge base retrieval method, device, equipment and medium

Abstract

The application discloses a vector knowledge base retrieval method and device based on artificial intelligence, equipment and medium, relates to the technical field of artificial intelligence and financial technology, the method inputs the retrieval item needed to be referred to when retrieving the vector knowledge base into the trained embedding model, the embedding model is used for mapping the information of different importance degrees of the retrieval item to different dimensions of the vector space to obtain the first embedding vector of the retrieval item, then the first embedding vector is cut into the first subvector according to the retrieval granularity, the first subvector with small dimension number is selected to participate in the retrieval of the vector knowledge base, and after the candidate vector is retrieved, the retrieval range of the next retrieval task is reduced according to the candidate vector, so that the first subvector with large dimension number is used for the vector knowledge base retrieval, the retrieval can be carried out in a smaller retrieval range, the similarity calculation task of the vector with large dimension number is reduced, the similarity calculation amount of the high-dimensional vector is reduced, and the retrieval efficiency is improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and financial technology, and in particular to a vector knowledge base retrieval method, apparatus, device, and medium based on artificial intelligence. Background Technology

[0002] The development of vector knowledge bases provides a feasible and effective approach for enhanced generation of large-scale model retrieval. Vector knowledge bases are applicable to various fields, including economic analysis and fintech. For example, they can be used in the research and application of large-scale economic models to achieve economic data question answering, analysis, and report writing. In some financial business scenarios, users can use question-answering models built on vector knowledge bases to query information about insurance products, loans, stocks, and funds.

[0003] Currently, the construction of existing vector knowledge bases in the industry generally follows the process of feature extraction, pre-embedding, and large-scale storage, and on this basis, similarity retrieval is realized. Some optimized similarity retrieval can be calculated on a graphics computing card, and the retrieval results can be obtained relatively quickly.

[0004] However, despite the existence of some optimized retrieval techniques that allow for fast computation on graphics cards, similarity retrieval still requires calculating and comparing a large number of vectors in a high-dimensional space when dealing with large datasets. The retrieval process remains relatively slow and inefficient. Summary of the Invention

[0005] This invention provides a vector knowledge base retrieval method, apparatus, device, and medium based on artificial intelligence to solve the problem of low retrieval efficiency.

[0006] Firstly, an artificial intelligence-based vector knowledge base retrieval method is provided. The method includes: inputting a retrieval term into a trained embedding model, so that the embedding model maps information of different importance levels corresponding to the retrieval term to different dimensions of a vector space, obtaining a first embedding vector; obtaining the retrieval granularity of the vector knowledge base; pruning the first embedding vector according to the retrieval granularity to obtain a pruned first sub-vector; selecting one first sub-vector in ascending order of its dimension to participate in a single retrieval task of the vector knowledge base, so as to retrieve candidate vectors matching the first sub-vectors from the vector knowledge base; wherein the retrieval scope of the next retrieval task in the vector knowledge base is determined based on the candidate vectors retrieved in the previous retrieval task; and the candidate vectors retrieved in the last retrieval task are used as target vectors matching the retrieval term.

[0007] Secondly, an artificial intelligence-based vector knowledge base retrieval device is provided. The device includes: a vector embedding module for inputting a retrieval item into a trained embedding model, so that the embedding model maps information of different importance levels corresponding to the retrieval item to different dimensions of a vector space, obtaining a first embedded vector; an acquisition module for acquiring the retrieval granularity of the vector knowledge base; a vector pruning module for pruning the first embedded vector according to the retrieval granularity, obtaining a pruned first sub-vector; and a retrieval module for selecting one of the first sub-vectors in ascending order of its dimension to participate in a single retrieval task of the vector knowledge base, thereby retrieving candidate vectors matching the first sub-vectors from the vector knowledge base; wherein the retrieval range of the next retrieval task in the vector knowledge base is determined based on the candidate vectors retrieved in the previous retrieval task; and the candidate vectors retrieved in the last retrieval task serve as the target vectors matching the retrieval item.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described vector knowledge base retrieval method based on artificial intelligence.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the aforementioned vector knowledge base retrieval method based on artificial intelligence.

[0010] In the aforementioned scheme implemented by the AI-based vector knowledge base retrieval method, apparatus, device, and medium, the retrieval items to be referenced when retrieving the vector knowledge base are input into a trained embedding model. This embedding model is used to map information of different importance levels of the retrieval items to different dimensions of the vector space to obtain the first embedding vector of the retrieval item. Then, according to the retrieval granularity, the first embedding vector is truncated into a first sub-vector. The first sub-vector with a smaller dimension is selected first to participate in the retrieval of the vector knowledge base. After retrieving the candidate vector, the retrieval scope of the next retrieval task is narrowed according to the candidate vector. In this way, when using the first sub-vector with a larger dimension to retrieve the vector knowledge base, the retrieval can be carried out within a smaller retrieval scope. This reduces the similarity calculation task involving vectors with a larger dimension, reduces the similarity calculation amount of high-dimensional vectors, and improves retrieval efficiency. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an application environment for a vector knowledge base retrieval method based on artificial intelligence, according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating a vector knowledge base retrieval method based on artificial intelligence in one embodiment of the present invention; Figure 3 This is a flowchart illustrating a vector knowledge base retrieval method based on artificial intelligence in another embodiment of the present invention; Figure 4 This is a schematic diagram of clipping the second embedding vector in another embodiment of the present invention; Figure 5 This is a flowchart illustrating a vector knowledge base retrieval method based on artificial intelligence in another embodiment of the present invention; Figure 6 This is a flowchart illustrating a vector knowledge base retrieval method based on artificial intelligence in another embodiment of the present invention; Figure 7 This is a schematic diagram of a vector knowledge base retrieval device based on artificial intelligence in one embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] The vector knowledge base retrieval method based on artificial intelligence provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can obtain the search terms entered by the user on the client's interactive interface. The server uses a trained embedding model to obtain the first embedding vector corresponding to the search term, and then trims the first embedding vector according to the retrieval granularity to obtain first sub-vectors of different dimensions. These first sub-vectors are used to conduct multiple rounds of retrieval of the vector knowledge base. As the dimension of the first sub-vectors involved in the retrieval increases, the retrieval range will continuously shrink. When using a first sub-vector with a large dimension to retrieve the vector knowledge base, it is not necessary to search the entire vector knowledge base; it is only necessary to search within the range defined by the candidate vectors retrieved by the previous first sub-vector. Furthermore, the retrieved target vector can be returned to the client so that the client can generate response information based on the target vector. The computational cost and complexity of similarity matching between high-dimensional vectors are much higher than those of low-dimensional vectors. This application uses sub-vectors of different dimensions to conduct multiple rounds of retrieval of the vector knowledge base, gradually narrowing the retrieval range of high-dimensional sub-vectors, reducing the computational cost of similarity between high-dimensional vectors, improving overall computational efficiency, and improving retrieval efficiency.

[0015] The client can include, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0016] Please see Figure 2 As shown, Figure 2 A flowchart illustrating an artificial intelligence-based vector knowledge base retrieval method provided in an embodiment of the present invention includes the following steps: S201, input the retrieval item into the trained embedding model so that the embedding model maps the information of different importance levels corresponding to the retrieval item to different dimensions of the vector space to obtain the first embedding vector.

[0017] Vector knowledge base retrieval can be understood as calculating the similarity between a given embedding vector A and the embedding vectors stored in the vector knowledge base, thereby selecting the embedding vector with the highest similarity to A from the embedding vectors stored in the vector knowledge base.

[0018] The retrieval item can be understood as the text, image, or other data that needs to be referenced when searching the vector knowledge base. The trained embedding model generates an embedding vector for the retrieval item, namely the above-mentioned embedding vector A. For ease of distinction, this application refers to it as the first embedding vector.

[0019] For example, when applying a vector knowledge base to a retrieval system, users can input text information related to their query needs, allowing the system to find relevant knowledge entries in the vector knowledge base. In this scenario, a search term can be understood as any type of information entered into the retrieval system, whether it's text, keywords, phrases, more complex queries, or images, etc.

[0020] For example, a search term could be "Please recommend insurance products suitable for people aged 20-30".

[0021] The trained embedding model processes the retrieval item and finally outputs the first embedding vector of the retrieval item.

[0022] The embedding model is used to map information of different importance levels corresponding to the search terms to different dimensions of the vector space to obtain the first embedding vector.

[0023] For example, in the search term "Please recommend insurance products suitable for people aged 20-30", the following ranking can be obtained based on the importance of the words: "insurance products" > "20-30 years old" > "recommend" > "suitable" > "people" > "of" > "please". Therefore, information corresponding to words with higher importance can be mapped to lower dimensions, and as importance decreases, information is gradually mapped to higher dimensions.

[0024] If the dimension of the vector output by the fixed embedding model is 2048, then the embedding model can map the text information such as "insurance products", "20-30 years old" and "recommendation" to the first 521 dimensions, and the text information such as "suitable" and "target group" to the first 522-1024 dimensions, and so on, so that the search term "please recommend insurance products suitable for people aged 20-30" can be completely mapped into the vector space, resulting in the first embedding vector of 2048 dimensions.

[0025] In other words, for the first 2048-dimensional embedding vector, information such as "insurance product", "20-30 years old", and "recommendation" can be obtained from the first 512-dimensional vector elements; information such as "insurance product", "20-30 years old", "recommendation", "suitable", and "target group" can be obtained from the first 1024-dimensional vector elements.

[0026] It should be noted that the "first" in "first 512 dimensions", "first 1024 dimensions", "first 512 dimensions", etc., described in this embodiment can be understood as follows: In the two vector elements x1 and x2 in the first embedding vector, the vector element closer to the head element of the first embedding vector (i.e., the first vector element of the first embedding vector) is the first element, and the vector element farther away from the head element is the second element.

[0027] This embodiment of the application utilizes elements at different positions in the first embedding vector to represent information of varying importance. Preferably, information of high importance is represented first, and then information of decreasing importance is continuously added to the first embedding vector as the dimension increases. This allows the first embedding vector to fully represent the content of the search term. In this way, the key / general content of the search term can be obtained through the low-dimensional data of the first embedding vector. In summary, the lower the dimension of the vector element in the first embedding vector, the higher the importance of the information it represents.

[0028] Embedded models map information of varying importance onto different dimensions of a vector space, a capability that can be acquired through model training.

[0029] S202, obtain the granularity of retrieval for the vector knowledge base.

[0030] The retrieval granularity of the vector knowledge base can be preset and flexibly switched according to scenario requirements or application needs.

[0031] For example, retrieval granularity can be broadly categorized into coarse-grained and fine-grained.

[0032] In some embodiments, the retrieval granularity can be quantified, dividing it into multiple retrieval granularities between coarse-grained and fine-grained granularities.

[0033] Coarse-grained: This usually refers to a classification or division in which each category or division unit covers a large range with less detail.

[0034] Fine-grained: Conversely, it refers to a classification or division in which each category or division unit covers a smaller range and has more details.

[0035] "Coarse-grained" is more "rough" in terms of granularity because it covers a wide range and has few details. Conversely, "fine-grained" is more "refined" in terms of granularity because it covers a narrow range and has many details.

[0036] Based on the above analysis, it is clear that "coarse-grained" granularity is greater than "fine-grained." Here, "coarse" refers to a broader scope and fewer details.

[0037] For example, coarse-grained retrieval can be performed when a general search is needed to find statements in a vector knowledge base that have a similar meaning to a given text. Fine-grained retrieval can be performed when a precise search is needed to find response information that matches a given question and answer.

[0038] S203, according to the retrieval granularity, the first embedding vector is truncated to obtain the truncated first sub-vector.

[0039] When the retrieval granularity is large, the maximum dimension of the first sub-vector can be relatively reduced; and / or, the number of the first sub-vectors can also be relatively reduced. When the retrieval granularity is small, the maximum dimension of the first sub-vector can be increased relatively; and / or, the number of sub-vectors can also be increased relatively.

[0040] For example, when performing coarse-grained retrieval, the 2048-dimensional first embedding vector is pruned once to obtain a 128-dimensional first sub-vector; when performing fine-grained retrieval, the 2048-dimensional second embedding vector is pruned four times to obtain a 64-dimensional, a 128-dimensional, a 512-dimensional, and a 2048-dimensional first sub-vector, respectively.

[0041] S204, according to the dimension number of the first sub-vector, select one of the first sub-vectors in order of increasing dimension to participate in a single retrieval task of the vector knowledge base, so as to retrieve candidate vectors that match the first sub-vectors from the vector knowledge base; wherein, the retrieval range of the next retrieval task in the vector knowledge base is determined based on the candidate vectors retrieved in the previous retrieval task; the candidate vectors retrieved in the last retrieval task are used as target vectors that match the retrieval item.

[0042] Based on the order of the dimension of the first sub-vector, one first sub-vector is selected in ascending order to participate in a single retrieval task of the vector knowledge base.

[0043] The number of vector dimensions can be understood as the size of the vector and the number of vector elements it includes. For example, a 64-dimensional first subvector can include 64 vector elements, while a 128-dimensional first subvector can include 128 vector elements. Comparing a 64-dimensional first subvector and a 128-dimensional first subvector, the 64-dimensional first subvector has a smaller number of dimensions, while the 128-dimensional first subvector has a larger number of dimensions.

[0044] For example, if the first sub-vectors cut out in step S203 include 64-dimensional, 128-dimensional, 512-dimensional, and 2048-dimensional first sub-vectors, then vector knowledge base retrieval can be achieved through the following process: First, a first retrieval task is performed in the vector knowledge base using the 64-dimensional first sub-vector. Candidate vectors that match the 64-dimensional first sub-vector are retrieved from the vector knowledge base, which are the candidate vectors corresponding to this retrieval task.

[0045] Then, based on the candidate vectors that match the 64-dimensional first sub-vector, the retrieval range of the second retrieval task in the vector knowledge base is determined. Then, the second retrieval task is continued within this retrieval range using the 128-dimensional first sub-vector. For example, the similarity between the 128-dimensional first sub-vector and the vectors in the vector knowledge base that belong to this retrieval range is calculated, and candidate vectors that match the 128-dimensional first sub-vector are selected as candidate vectors corresponding to the second retrieval task.

[0046] Based on the candidate vectors that match the 128-dimensional first sub-vector, the search scope of the third search task in the vector knowledge base is determined. Then, the third search task is continued within this search scope using the 512-dimensional first sub-vector to filter out candidate vectors that match the 512-dimensional first sub-vector.

[0047] Repeat the above process, and use the first sub-vector of 2048 dimensions to carry out the fourth search task within the search range determined by the candidate vectors retrieved in the third search task, to obtain the candidate vector that matches the first sub-vector of 2048 dimensions, which is then used as the candidate vector retrieved in the fourth search task.

[0048] In this example, the fourth retrieval task is the last retrieval task, so the candidate vector retrieved by the fourth retrieval task is used as the target vector to match the retrieval item.

[0049] In some embodiments, if only N target vectors matching the search term are required, the candidate vectors retrieved by the fourth search task are ranked according to their similarity to the first sub-vector of the 2048-dimensional dimension, and the top N candidate vectors are used as target vectors.

[0050] In summary, the vector knowledge base retrieval method based on artificial intelligence proposed in this application inputs the retrieval items needed for retrieval of the vector knowledge base into a trained embedding model. This embedding model maps information of different importance levels of the retrieval items to different dimensions of the vector space to obtain the first embedding vector of the retrieval item. Then, according to the retrieval granularity, the first embedding vector is truncated into a first sub-vector. First, the first sub-vector with a smaller dimension is selected to participate in the retrieval of the vector knowledge base. After retrieving the candidate vector, the retrieval scope of the next retrieval task is narrowed down according to the candidate vector. In this way, when using the first sub-vector with a larger dimension to retrieve the vector knowledge base, the retrieval can be carried out within a smaller retrieval scope. This reduces the similarity calculation task involving vectors with a larger dimension, reduces the similarity calculation amount of high-dimensional vectors, and improves retrieval efficiency.

[0051] Based on the above embodiments, such as Figure 3 As shown, the embedding model is trained in the following manner: S301, Input the training samples into the embedding model to obtain the second embedding vector.

[0052] S302, calculate the loss function between the second sub-vectors with different dimensions corresponding to the second embedding vector and the labels of the training samples.

[0053] S303, perform importance scaling on each loss function, and aggregate all the importance-scaled loss functions to obtain the target loss function corresponding to the training sample.

[0054] S304, Train the embedding model based on the target loss function until the loss value of the trained embedding model is less than a preset loss threshold.

[0055] The embedding model can be a word embedding model, a recurrent neural network, a convolutional neural network, a Transformer model, etc., and this application does not impose any restrictions.

[0056] In training the embedding model, a set of labeled training samples can be selected, and the training samples can be input into the embedding model to generate the embedding vector corresponding to the training sample. For ease of distinction, this application refers to it as the second embedding vector.

[0057] For each training sample, the second embedding vector is split to obtain second sub-vectors with different dimensions. Then, the loss function between each second sub-vector and the label of the training sample is calculated. Then, the importance scaling of all loss functions corresponding to the training sample is performed, for example, by multiplying each loss function by the corresponding importance weight. Then, all loss functions after importance scaling are aggregated to obtain the loss function corresponding to the training sample.

[0058] Using the embedding model Generate a second embedding vector Z for a training sample. This second embedding vector Z is a d-dimensional vector, denoted as... Then, the elements of the first m dimensions of the second embedding vector are taken as the second sub-vector, denoted as... m is any value in set M, such as set M. .

[0059] Then, the loss function between each second subvector and the label of the training sample is calculated, and the loss is optimized by standard empirical minimization.

[0060] For example, let's select a labeled dataset. Embedding model F, multi-class loss function The trained embedding model F is obtained by training to obtain the minimum value shown in the following formula: in, The embedded model F consists of learnable weights. Parameterization This represents the i-th training sample in dataset D. Indicates training samples The tag, These are the parameters of a linear classifier. The total number of the second subvector. This represents the importance weight corresponding to the m-th second sub-vector.

[0061] In this embodiment, during model training, the loss between the information represented by elements in different dimensions of the second embedding vector and the labels of the training samples is considered, and different attention levels are assigned to elements in different dimensions (i.e., importance scaling is applied to the loss function corresponding to different second sub-vectors). This allows the trained model to map information of different importance levels of the retrieval items to different dimensions of the vector space. For example, by training the embedding model, it learns to map key information in the retrieval items to low dimensions and gradually map the remaining information to high dimensions. This ensures that the final embedding vector preferentially represents important information in low-dimensional elements. Thus, when performing coarse-grained retrieval of the vector knowledge base, only the first 1 / 3 of the first embedding vector needs to be used. Searching can be performed using sub-vectors with lower dimensionality, such as vector elements in two segments or the first third of the vector elements, to improve search efficiency. In fine-grained search, candidate vectors that match the key information of the search term can be selected by prioritizing low-dimensional vector elements in the first embedding vector. The search scope can then be narrowed based on these candidate vectors. Then, some high-dimensional vector elements can be superimposed on these vector elements to further search within the search scope to obtain candidate vectors that highly match most of the information of the search term. This avoids performing a large number of similarity calculations in the vector knowledge base using high-dimensional first embedding vectors, reduces the similarity calculation task of high-dimensional vectors, improves the overall computational efficiency, and thus improves search efficiency.

[0062] Based on the above embodiments, the importance scaling of each loss function in step S303 may include the following steps: Multiply each loss function by the importance weight corresponding to that loss function to perform importance scaling of the loss function; The smaller the dimension of the second sub-vector corresponding to the loss function, the greater the importance weight corresponding to the loss function.

[0063] In this embodiment of the application, more attention is paid to the second sub-vector with low dimensionality during model training. The loss of the second sub-vector with low dimensionality is reduced through training, so that the model learns to map the important information of the input data to elements of fixed dimension in the first embedding vector. The number of these elements is small. By using the second sub-vector containing these elements to perform vector knowledge base retrieval, vectors that match the important information of the input data can be retrieved from the vector knowledge base through fast similarity matching.

[0064] Based on the above embodiments, the second sub-vectors with different dimensions in step S302 can be obtained in the following way: The second embedding vector is truncated multiple times starting from the head element using different truncating steps to obtain the second sub-vector including the head element.

[0065] like Figure 4 As shown, for a second embedding vector If the second embedding vector has a dimension of d, meaning it contains d vector elements, then a pruning operation is performed starting from the head element with a step size of d / 16 to obtain the second sub-vector. The vector elements of the second subvector are all vector elements from the head element to the d / 16th vector element in the second embedded vector.

[0066] We can perform another trimming step of d / 8 to obtain the second sub-vector. ; We can perform another trimming step of d / 4 to obtain the second sub-vector. ; We can perform another trimming step with d / 2 as the trimming step size to obtain the second sub-vector. ; We can perform another trimming operation with d as the trimming step size to obtain the second sub-vector. .

[0067] Thus, we can obtain the following: Figure 4 The four second sub-vectors are shown.

[0068] In this embodiment of the application, when pruning the second embedded vector, the step size is calculated starting from the head element of the second embedded vector each time. In this way, in the obtained second sub-vector, as the dimension of the second sub-vector increases, the information represented by the second sub-vector continues to accumulate. The second sub-vector with a high dimension will contain all the vector elements of the second sub-vector with a low dimension, and the information that can be represented becomes more and more complete.

[0069] Based on the above embodiments, such as Figure 5As shown, the step S203 above, "clipping the first embedding vector according to the retrieval granularity to obtain the clipped first sub-vector," may include the following steps: S501, obtain the maximum trimming step size corresponding to the retrieval granularity.

[0070] In this embodiment, the dimension of the first sub-vector required for the last retrieval task can be preset for different retrieval granularities, and this dimension serves as the maximum retrieval step size corresponding to that retrieval granularity.

[0071] For example, when performing coarse-grained retrieval, if the dimension of the first sub-vector required for the last retrieval task is 512, then the maximum pruning step size when pruning the first embedding vector is 512. The vector elements 1 to 512 in the first embedding vector can be used as the 512-dimensional first sub-vector.

[0072] S502, the second embedded vector is truncated multiple times starting from the head element with different truncating steps less than or equal to the maximum truncating step size to obtain the first sub-vector including the head element.

[0073] After determining the maximum pruning step size corresponding to the retrieval granularity, multiple pruning step sizes can be selected without exceeding the maximum pruning step size, and these pruning step sizes can be used to perform multiple vector pruning on the second embedding vector.

[0074] In some embodiments, the total number of clipping steps used during vector clipping can also be pre-configured for the retrieval granularity. For example, the maximum clipping step size corresponding to different retrieval granularities and the total number of clipping steps can be pre-stored; or all clipping steps corresponding to different retrieval granularities can be directly stored.

[0075] For example, one configuration is as follows: the retrieval granularity is 50%, the maximum cropping step size is 512, and the total number of cropping steps is 4. Thus, when performing clipping of the 2048-dimensional first embedded vector, it is possible to clip out a 512-dimensional first sub-vector, a 256-dimensional first sub-vector, a 64-dimensional first sub-vector, a 32-dimensional first sub-vector; or to clip out a 512-dimensional first sub-vector, a 64-dimensional first sub-vector, a 32-dimensional first sub-vector, an 8-dimensional first sub-vector, and so on, etc.

[0076] Another configuration is as follows: the retrieval granularity is 50%, and all clipping step sizes include: 512, 256, 64, and 32. In this method, a first sub-vector with a fixed combination of dimensions can be clipped.

[0077] With the trained embedding model prioritizing mapping of highly important information to the lower dimension of the vector space, each pruning of the first embedding vector starts from the head element of the first embedding vector, so that each pruned second sub-vector can represent the important information of the retrieval item. As the dimension of the first sub-vector increases, other information of the retrieval item is continuously expanded on the basis of this important information.

[0078] In this embodiment of the application, the amount of information required for vector knowledge base retrieval varies depending on the retrieval granularity. For example, less information is required at the coarse-grained level, while more information is required at the fine-grained level. Therefore, the dimension of the first sub-vector used in the last retrieval task when performing vector knowledge base retrieval at the coarse-grained level is less than the dimension of the first sub-vector used in the last retrieval task when performing vector knowledge base retrieval at the fine-grained level.

[0079] In other embodiments, there may be one first sub-vector, and the retrieval task performed using this first sub-vector is both the first and last retrieval task. The dimension of this first sub-vector is equal to the maximum pruning step size corresponding to the retrieval granularity. This first sub-vector can represent the important information of the retrieval item, and, based on the requirements of the retrieval granularity, expands upon the important information to a certain extent. The candidate vectors retrieved from the vector knowledge base using this first sub-vector can serve as target vectors that meet the retrieval granularity requirements and can provide response information for the retrieval item.

[0080] In this embodiment, after determining the maximum clipping step size, the first embedded vector can be clipped multiple times without exceeding the maximum clipping step size, resulting in multiple first sub-vectors with increasing dimensionality. When using these first sub-vectors for vector knowledge base retrieval, a large-scale retrieval can be performed first using the first sub-vectors with smaller dimensionality. Then, the retrieval range for the next retrieval task is determined based on the candidate vectors retrieved in the previous retrieval task. As the dimensionality of the first sub-vectors participating in the retrieval task increases, the retrieval range will continuously shrink. When the first sub-vector with the largest dimensionality is used for retrieval, the retrieval range will be much smaller than that of the case with a single first sub-vector. Thus, the similarity calculation for the first sub-vector with the largest dimensionality will also be reduced, improving retrieval efficiency.

[0081] Based on the above embodiments, such as Figure 6 As shown, the above vector knowledge base is constructed in the following way: S601, using the trained embedding model, obtain the third embedding vector required to construct the vector knowledge base; S602, the third embedded vector is truncated multiple times starting from the head element with different truncating step sizes to obtain the third sub-vector including the head element; S603, the third sub-vectors with different dimensions are stored in different storage spaces to construct the vector knowledge base.

[0082] In constructing a vector knowledge base, data in the form of text, images, etc., that need to be stored in the vector knowledge base are obtained. This data is then input into a trained embedding model to obtain the corresponding embedding vector. For ease of distinction, this embodiment of the application refers to it as the third embedding vector.

[0083] For example, when building an intelligent response system for fund products, the basic information of each fund product, the latest market analysis, the target audience, etc., can be constructed into text for input embedding models.

[0084] The embedding model generates a third embedding vector for each piece of text. This model prioritizes mapping high-importance information in the text to lower dimensions of the vector space, and then expands to higher dimensions as importance decreases. This ensures that the lower-dimensional vector elements in the resulting third embedding vector represent the key information of the text.

[0085] When storing the third embedding vector, the third embedding vector can be clipped first. Each clipping step is selected, and clipping starts from the head element of the third embedding vector. Each clipping results in a third sub-vector.

[0086] For example, for a 2048-dimensional third embedding vector, first use an 8-dimensional clipping step to clip it, and obtain a third sub-vector that includes the 1st to 8th vector elements in the third embedding vector; then use a 16-dimensional clipping step to clip it, and obtain a third sub-vector that includes the 1st to 16th vector elements in the third embedding vector, until the clipping step is increased to 2048 dimensions. The remaining clipping process repeats the above steps, which will not be elaborated here.

[0087] All the third sub-vectors obtained after clipping are stored separately: third sub-vectors with the same dimension are stored in the same location, while third sub-vectors with different dimensions are stored separately.

[0088] It should be noted that different storage locations can be different storage spaces partitioned from the same storage device, such as storing third subvectors of different dimensions in different directories; or different storage locations can also be different storage devices. Different methods can be chosen to implement separate storage as needed, and this application does not impose any restrictions.

[0089] For example, in the final vector knowledge base, 8-dimensional embedding vectors are stored in the same space, 16-dimensional embedding vectors are stored in the same space, and so on up to 2048-dimensional embedding vectors are stored in the same space.

[0090] When performing vector knowledge base retrieval, the first step is to use the 8-dimensional first sub-vector to search the space storing 8-dimensional embedding vectors. The retrieved candidate vectors are then sorted by similarity, and the top 50% of candidate vectors are selected. The 16-dimensional embedding vectors corresponding to the top 50% of candidate vectors are then selected from the storage space storing 16-dimensional embedding vectors. This 16-dimensional first sub-vector is used as the retrieval range. The similarity between the 16-dimensional first sub-vector and the selected 16-dimensional embedding vectors is then calculated, and another batch of candidate vectors is retrieved. The top 50% of candidate vectors with the highest similarity are then selected from this batch of candidate vectors. The above steps are repeated to determine the retrieval range corresponding to the 32-dimensional first sub-vector, and so on, until the retrieval task is completed using all the first sub-vectors.

[0091] It should be noted that the above similarity ranking refers to ranking the candidate vectors according to the similarity between the candidate vector and the first sub-vector. The ranking percentage for candidate vector screening can be set as needed, and the percentage used each time can be different. This application does not impose any restrictions.

[0092] In some embodiments, a set of clipping steps can be preset, and the first embedding vector, the second embedding vector, and the third embedding vector are clipped based on this set of clipping steps during model training, vector knowledge base establishment, and vector knowledge base retrieval.

[0093] For example, a set of cutting steps may include 4, 8, 32, 64, 128, 256, 512, 1024, 2048.

[0094] During the training of the embedding model, the second embedding vector is truncated using each of the truncating steps in the set, so that the second sub-vector truncated by each truncating step participates in the model training. When establishing the vector knowledge base, the third embedded vector is truncated using each truncating step in the set of truncating steps, so that the vector knowledge base can completely store the third sub-vectors with different dimensions, so as to facilitate the definition of the retrieval range from the third sub-vectors with the same dimension for any first sub-vector with any dimension. When searching the vector knowledge base, one or more clipping steps can be selected from the set of clipping steps to clip the first embedded vector as needed. The first sub-vectors with different dimensions can be used to search the vector knowledge base.

[0095] Based on the above embodiments, there is a multiple relationship between different clipping step sizes, which allows the dimension of the clipped sub-vectors to change regularly, facilitating subsequent processing.

[0096] The proposed AI-based vector knowledge base retrieval method generates smaller-sized (i.e., fewer-dimensional) embedding vectors during the initial setup of the vector knowledge base. This reduces the hardware resources required for data processing and storage, thereby lowering costs. This advantage is suitable for resource-constrained applications, such as mobile devices or cloud computing environments, making large-scale data processing more economical and efficient. Using smaller-dimensional sub-vectors in the initial retrieval stage allows for rapid location of search results, followed by detailed processing using the complete embedding vectors. This hierarchical processing method improves overall query speed and data processing capabilities. This not only enhances the user experience but also strengthens the system's ability to handle large volumes of requests.

[0097] Secondly, the total dimension and complexity of the embedded vectors can be adjusted according to actual application needs and application load. This flexibility enables the system to effectively handle different data processing and query scenarios. Whether it's large-scale data or specific query tasks, this method can find suitable processing strategies and resource configurations. Furthermore, this adaptability also facilitates future upgrades and expansions.

[0098] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0099] In one embodiment, an AI-based vector knowledge base retrieval device is provided, which corresponds one-to-one with the AI-based vector knowledge base retrieval method described in the above embodiments.

[0100] like Figure 7 As shown, the AI-based vector knowledge base retrieval device 700 includes a vector embedding module 701, an acquisition module 702, a vector clipping module 703, and a retrieval module 704. Detailed descriptions of each functional module are as follows: The vector embedding module 701 is used to input the retrieval item into the trained embedding model so that the embedding model maps the information of different importance levels corresponding to the retrieval item to different dimensions of the vector space to obtain the first embedding vector. The acquisition module 702 is used to acquire the retrieval granularity of the vector knowledge base; The vector trimming module 703 is used to trim the first embedded vector according to the retrieval granularity to obtain the trimmed first sub-vector; The retrieval module 704 is used to select one of the first sub-vectors in order of increasing dimension from the first sub-vector to participate in a single retrieval task of the vector knowledge base, so as to retrieve candidate vectors that match the first sub-vector from the vector knowledge base. The search scope of the next search task in the vector knowledge base is determined based on the candidate vectors retrieved in the previous search task. The candidate vector retrieved in the last retrieval task is used as the target vector that matches the retrieval item.

[0101] Based on the above embodiments, the AI-based vector detection model is also used to train the embedding model in the following manner: The training samples are input into the embedding model to obtain the second embedding vector; Calculate the loss function between the second sub-vectors with different dimensions corresponding to the second embedding vector and the labels of the training samples respectively; Each loss function is weighted by importance scaling, and all weighted loss functions are aggregated to obtain the target loss function corresponding to the training sample. The embedding model is trained based on the target loss function until the loss value of the trained embedding model is less than a preset loss threshold.

[0102] Based on the above embodiments, the AI-based vector knowledge base retrieval device is further configured to: multiply each loss function by the importance weight corresponding to the loss function to complete the importance scaling of the loss function; wherein, the smaller the dimension of the second sub-vector corresponding to the loss function, the greater the importance weight corresponding to the loss function.

[0103] Based on the above embodiments, the AI-based vector knowledge base retrieval device is further used to obtain second sub-vectors with different numbers of dimensions by performing multiple vector clipping operations on the second embedded vector starting from the head element with different clipping step sizes, thereby obtaining the second sub-vector including the head element.

[0104] Based on the above embodiments, the vector clipping module is used to: obtain the maximum clipping step size corresponding to the retrieval granularity; and perform multiple vector clippings on the second embedded vector starting from the head element with different clipping step sizes less than or equal to the maximum clipping step size to obtain the first sub-vector including the head element.

[0105] Based on the above embodiments, the AI-based vector knowledge base retrieval device is further configured to construct the vector knowledge base in the following manner: using a trained embedding model to obtain a third embedding vector required for constructing the vector knowledge base; performing multiple vector pruning operations on the third embedding vector starting from the head element with different pruning step sizes to obtain the third sub-vector including the head element; storing the third sub-vectors with different dimensions in different storage spaces to construct the vector knowledge base.

[0106] Based on the above embodiments, there is a multiple relationship between the different cutting step lengths.

[0107] This invention provides an artificial intelligence-based vector knowledge base retrieval device. The device inputs the retrieval items needed for searching the vector knowledge base into a trained embedding model. This model maps information of varying importance of the retrieval items to different dimensions of the vector space, obtaining a first embedding vector for the retrieval item. Then, based on the retrieval granularity, this first embedding vector is truncated into first sub-vectors. First, the first sub-vector with the smaller dimension is selected to participate in the vector knowledge base retrieval. After retrieving candidate vectors, the retrieval scope for the next retrieval task is narrowed down based on these candidate vectors. Thus, when using the first sub-vector with the larger dimension for vector knowledge base retrieval, the retrieval can be carried out within a smaller retrieval scope. This reduces the similarity calculation task involving vectors with larger dimensions, lowers the similarity calculation workload of high-dimensional vectors, and improves retrieval efficiency.

[0108] Specific limitations regarding the AI-based vector knowledge base retrieval device can be found in the limitations of the AI-based vector knowledge base retrieval method described above, and will not be repeated here. Each module in the aforementioned AI-based vector knowledge base retrieval device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0109] In one embodiment, such as Figure 8 As shown, a computer device 800 is provided, including a memory 801, a processor 802, and a computer program stored on the memory 801 and executable on the processor 802. When the processor executes the computer program, it implements the steps of the vector knowledge base retrieval method based on artificial intelligence described in any of the above embodiments.

[0110] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the AI-based vector knowledge base retrieval method described in any of the above embodiments.

[0111] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0112] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0113] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0114] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A vector knowledge base retrieval method based on artificial intelligence, characterized in that, The method includes: The search terms are input into the trained embedding model so that the embedding model maps the information of different importance levels corresponding to the search terms to different dimensions of the vector space to obtain the first embedding vector. The lower the dimension of the vector element in the first embedding vector, the higher the importance of the information it represents. Obtain the granularity of retrieval from the vector knowledge base; Based on the retrieval granularity, the first embedding vector is truncated to obtain the truncated first sub-vector; Based on the dimension number of the first sub-vector, one of the first sub-vectors is selected in ascending order to participate in a single retrieval task of the vector knowledge base, so as to retrieve candidate vectors that match the first sub-vector from the vector knowledge base; The search scope of the next search task in the vector knowledge base is determined based on the candidate vectors retrieved in the previous search task. The candidate vector retrieved in the last retrieval task is used as the target vector to match the retrieval item; The step of pruning the first embedded vector according to the retrieval granularity to obtain the pruned first sub-vector includes: Obtain the maximum trimming step size corresponding to the retrieval granularity; The first embedded vector is truncated multiple times starting from the head element using different truncating steps that are less than or equal to the maximum truncating step size, to obtain the first sub-vector that includes the head element.

2. The method as described in claim 1, characterized in that, The embedding model is trained in the following manner: The training samples are input into the embedding model to obtain the second embedding vector; Calculate the loss function between the second sub-vectors with different dimensions corresponding to the second embedding vector and the labels of the training samples respectively; Each of the loss functions is weighted by importance scaling, and all the weighted loss functions are aggregated to obtain the target loss function corresponding to the training sample. The embedding model is trained based on the target loss function until the loss value of the trained embedding model is less than a preset loss threshold.

3. The method as described in claim 2, characterized in that, The importance scaling of each loss function includes: Each loss function is multiplied by its corresponding importance weight to perform importance scaling on the loss function. The smaller the dimension of the second sub-vector corresponding to the loss function, the greater the importance weight corresponding to the loss function.

4. The method as described in claim 2, characterized in that, The second subvector with different dimensions is obtained in the following way: The second embedding vector is truncated multiple times starting from the head element using different truncating steps to obtain the second sub-vector including the head element.

5. The method as described in claim 1, characterized in that, The vector knowledge base is constructed in the following manner: Using the trained embedding model, obtain the third embedding vector required to construct the vector knowledge base; The third embedding vector is truncated multiple times starting from the head element using different truncating step sizes to obtain a third sub-vector including the head element. The third sub-vectors with different dimensions are stored in different storage spaces to construct the vector knowledge base.

6. The method according to any one of claims 4-5, characterized in that, There is a multiple relationship between the different cutting step sizes.

7. A vector knowledge base retrieval device based on artificial intelligence, characterized in that, The device includes: The vector embedding module is used to input the retrieval item into the trained embedding model, so that the embedding model maps the information of different importance levels corresponding to the retrieval item to different dimensions of the vector space to obtain the first embedding vector. The lower the dimension of the vector element in the first embedding vector, the higher the importance of the information it represents. The acquisition module is used to obtain the granularity of retrieval from the vector knowledge base; The vector pruning module is used to prune the first embedded vector according to the retrieval granularity to obtain the pruned first sub-vector; The retrieval module is used to select one of the first sub-vectors in order of increasing dimension to participate in a single retrieval task of the vector knowledge base, so as to retrieve candidate vectors that match the first sub-vector from the vector knowledge base. The search scope of the next search task in the vector knowledge base is determined based on the candidate vectors retrieved in the previous search task. The candidate vector retrieved in the last retrieval task is used as the target vector to match the retrieval item; The step of pruning the first embedded vector according to the retrieval granularity to obtain the pruned first sub-vector includes: Obtain the maximum trimming step size corresponding to the retrieval granularity; The first embedded vector is truncated multiple times starting from the head element using different truncating steps that are less than or equal to the maximum truncating step size, to obtain the first sub-vector that includes the head element.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the vector knowledge base retrieval method based on artificial intelligence as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the vector knowledge base retrieval method based on artificial intelligence as described in any one of claims 1 to 6.

Images