Content retrieval method and device, computer equipment and readable storage medium

Abstract

The invention relates to a content retrieval method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a content retrieval request, and respectively constructing high-dimensional vector representation and sparse vector representation based on the content retrieval request; performing retrieval in a preset high-dimensional vector library based on an approximate retrieval mode according to high-dimensional vector representation to obtain a first retrieval result; the high-dimensional vector library comprises high-dimensional vector representations corresponding to the candidate contents; retrieving in a preset sparse vector library according to the sparse vector representation to obtain a second retrieval result; the sparse vector library comprises sparse vector representations corresponding to the candidate contents; and obtaining a content retrieval result corresponding to the content retrieval request based on the first retrieval result and the second retrieval result. By adopting the method, the accuracy of content retrieval can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a content retrieval method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of information technology, knowledge base systems are increasingly used in information management and knowledge acquisition. As the main carriers of information, images and documents are growing explosively in number. Whether in scientific research, education, medical care or business, it is crucial to efficiently retrieve and utilize the information in these images and documents. However, when faced with millions or even tens of millions of images and documents, traditional content retrieval methods have low accuracy. Summary of the invention

[0003] Based on this, it is necessary to provide a content retrieval method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the accuracy of content retrieval in response to the above technical problems.

[0004] In a first aspect, the present application provides a content retrieval method, comprising:

[0005] Obtain a content retrieval request, and construct a high-dimensional vector representation and a sparse vector representation based on the content retrieval request;

[0006] According to the high-dimensional vector representation, searching is performed in a preset high-dimensional vector library based on an approximate search method to obtain a first search result; the high-dimensional vector library includes a high-dimensional vector representation corresponding to the candidate content;

[0007] Searching in a preset sparse vector library according to the sparse vector representation to obtain a second search result; the sparse vector library includes the sparse vector representation corresponding to the candidate content;

[0008] Based on the first search result and the second search result, a content search result corresponding to the content search request is obtained.

[0009] In a second aspect, the present application further provides a content retrieval device, comprising:

[0010] A retrieval request acquisition module, used to acquire a content retrieval request, and construct a high-dimensional vector representation and a sparse vector representation based on the content retrieval request;

[0011] A high-dimensional vector retrieval module, used to search in a preset high-dimensional vector library based on an approximate search method according to the high-dimensional vector representation to obtain a first search result; the high-dimensional vector library includes the high-dimensional vector representation corresponding to the candidate content;

[0012] A sparse vector search module, used to search in a preset sparse vector library according to the sparse vector representation to obtain a second search result; the sparse vector library includes the sparse vector representation corresponding to the candidate content;

[0013] The retrieval result obtaining module is used to obtain the content retrieval result corresponding to the content retrieval request based on the first retrieval result and the second retrieval result.

[0014] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above content retrieval method are implemented.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above content retrieval method are implemented.

[0016] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above content retrieval method are implemented.

[0017] The above-mentioned content retrieval method, apparatus, computer device, computer-readable storage medium and computer program product respectively construct a high-dimensional vector representation and a sparse vector representation based on a content retrieval request, retrieve a first retrieval result from a preset high-dimensional vector library based on an approximate retrieval method according to the high-dimensional vector representation, retrieve a second retrieval result from a preset sparse vector library according to the sparse vector representation, and obtain a content retrieval result based on the first retrieval result and the second retrieval result. During the content retrieval process, constructing a high-dimensional vector representation and a sparse vector representation for the content retrieval request and performing retrieval respectively can realize multi-dimensional content retrieval of high-dimensional vectors and coefficient vectors, thereby improving the accuracy of content retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 is an application environment diagram of a content retrieval method in an embodiment;

[0020] Figure 2 is a flow chart of a content retrieval method in one embodiment;

[0021] Figure 3 A schematic diagram of a process for determining content retrieval results in one embodiment;

[0022] Figure 4 A schematic diagram of a process of storing documents in another embodiment;

[0023] Figure 5 A schematic diagram of a process of searching multiple document repositories in one embodiment;

[0024] Figure 6 is a flowchart of a content retrieval method in yet another embodiment;

[0025] Figure 7 is a structural block diagram of a content retrieval device in one embodiment;

[0026] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] The content retrieval method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0029] The user can send a content search request to the server 104 through the terminal 102 to request the server 104 to perform content search. After the server 104 obtains the content search request sent by the terminal 102, the server 104 constructs a high-dimensional vector representation and a sparse vector representation based on the content search request, and searches the preset high-dimensional vector library based on the approximate search method according to the high-dimensional vector representation to obtain a first search result, and searches the preset sparse vector library based on the sparse vector representation to obtain a second search result. The server 104 obtains the content search result based on the first search result and the second search result. The server 104 can also send the obtained content search result to the terminal 102 to display the corresponding content search result in the terminal 102.

[0030] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0031] In an exemplary embodiment, Figure 2 As shown, a content retrieval method is provided, which is applied to Figure 1 The server in the example is used to illustrate, including the following steps 202 to 208. Among them:

[0032] Step 202: Obtain a content retrieval request, and construct a high-dimensional vector representation and a sparse vector representation based on the content retrieval request.

[0033] Among them, the content retrieval request is a message for requesting content retrieval, which can be sent by the terminal to the server. The content retrieval request can carry a search statement to retrieve the corresponding required content. For example, the content retrieval request can include document keywords to retrieve documents that match the document keywords. The high-dimensional vector representation is a vector representation including multiple dimensional elements, each dimensional element can represent a feature, and the high-dimensional vector representation can represent multiple features of the content retrieval request from multiple dimensions. The sparse vector representation is a vector representation in which most elements are zero and only a few elements are non-zero. The sparse vector representation can greatly reduce the storage space and calculation time by storing only non-zero elements and their position information, and can represent as much knowledge as possible with as few resources as possible, thereby improving the efficiency and accuracy of data processing.

[0034] Exemplarily, the server may obtain a content retrieval request sent by a user through a terminal, and the server may construct a corresponding vector representation based on the content retrieval request, which may specifically include a high-dimensional vector representation and a sparse vector representation. In some embodiments, the server may perform vector mapping for the content retrieval request to map the content retrieval request into a high-dimensional vector representation and a sparse vector representation, respectively.

[0035] Step 204: search in a preset high-dimensional vector library based on an approximate search method according to the high-dimensional vector representation to obtain a first search result; the high-dimensional vector library includes the high-dimensional vector representation corresponding to the candidate content.

[0036] Among them, the approximate retrieval method is a method of finding a vector similar to the target vector in the vector space without traversing the entire data set, such as including but not limited to at least one of a tree-based search method, a hash-based method, a graph-based method, and a quantization-based method. The high-dimensional vector library is a vector database pre-constructed for candidate content. The high-dimensional vector library may include high-dimensional vector representations corresponding to the candidate content. The candidate content is for retrievable content, such as content stored in a knowledge base. Users can retrieve the corresponding required target content from the candidate content through a content retrieval request. The first retrieval result is the result retrieved from the high-dimensional vector library, which may specifically include a high-dimensional vector representation corresponding to the candidate content that matches the high-dimensional vector representation of the content retrieval request. The corresponding candidate content can be determined based on the retrieved high-dimensional vector representation, thereby obtaining a retrieval result corresponding to the content retrieval request.

[0037] Exemplarily, the server may obtain a preset high-dimensional vector library, in which the high-dimensional vector representations corresponding to each candidate content are stored. In some embodiments, the server may pre-map each candidate content separately to obtain the high-dimensional vector representation corresponding to the candidate content, and the server may construct a high-dimensional vector library based on the high-dimensional vector representation of each candidate content. In some embodiments, the server may also record the mapping relationship between each candidate content and the corresponding high-dimensional vector representation, so that the corresponding candidate content can be determined based on the corresponding high-dimensional vector representation. The server may search in the high-dimensional vector library based on the high-dimensional vector representation of the content retrieval request, and specifically may search in the high-dimensional vector library based on an approximate retrieval method, such as searching in the high-dimensional vector library based on an approximate nearest neighbor search (ANNS) algorithm to obtain a first search result.

[0038] Step 206, searching in a preset sparse vector library according to the sparse vector representation to obtain a second search result; the sparse vector library includes the sparse vector representation corresponding to the candidate content.

[0039] The sparse vector library is a vector database pre-built for the candidate content, and the sparse vector library may include sparse vector representations corresponding to the candidate content. The second search result is a result retrieved from the sparse vector library, and may specifically include sparse vector representations corresponding to the candidate content that matches the sparse vector representation of the content search request.

[0040] Exemplarily, the server may query a pre-built sparse vector library and search in the sparse vector library according to the sparse vector representation to obtain a second search result. In some embodiments, the server may map each candidate content in advance to obtain a sparse vector representation corresponding to the candidate content, and the server may construct a sparse vector library based on the sparse vector representation of each candidate content. The server may match the sparse vector representation of the content search request with the sparse vector representation corresponding to the candidate content in the sparse vector library, and obtain a second search result based on the matching result, such as representing the sparse vector representation corresponding to the candidate content that matches the matching result as the second search result.

[0041] Step 208: Obtain a content search result corresponding to the content search request based on the first search result and the second search result.

[0042] Among them, the content retrieval result is the required target content retrieved from each candidate content based on the content retrieval request. Optionally, the server can determine the corresponding retrieved content based on the first retrieval result and the second retrieval result, and obtain the content retrieval result corresponding to the content retrieval request for the retrieved content. In some embodiments, the server can determine the respective retrieval content based on the first retrieval result and the second retrieval result, and merge, remove duplicates, sort, etc. for the respective retrieval content, so as to obtain the content retrieval result corresponding to the content retrieval request, and the content retrieval result may include the target content retrieved from each candidate content and matching the content retrieval request.

[0043] In some embodiments, different retrieval weights may be set for the first retrieval result and the second retrieval result, and the first retrieval result and the second retrieval result may be weighted and combined according to their respective weights to obtain a content retrieval result corresponding to the content retrieval request. In some embodiments, the respective weights of the first retrieval result and the second retrieval result may be configured based on the content retrieval request, that is, different weights may be set for different retrieval requirements, so that the first retrieval result and the second retrieval result may be combined according to the weights corresponding to the content retrieval request, which may further ensure the accuracy of the content retrieval result.

[0044] In the above content retrieval method, a high-dimensional vector representation and a sparse vector representation are respectively constructed based on the content retrieval request, a first retrieval result is obtained by searching in a preset high-dimensional vector library based on the high-dimensional vector representation based on an approximate retrieval method, a second retrieval result is obtained by searching in a preset sparse vector library based on the sparse vector representation, and a content retrieval result is obtained based on the first retrieval result and the second retrieval result. During the content retrieval process, a high-dimensional vector representation and a sparse vector representation are constructed for the content retrieval request and searched separately, which can realize multi-dimensional content retrieval of high-dimensional vectors and coefficient vectors, thereby improving the accuracy of content retrieval.

[0045] In an exemplary embodiment, the high-dimensional vector library includes at least two content vector libraries; according to the high-dimensional vector representation, based on the approximate retrieval method, a search is performed in a preset high-dimensional vector library to obtain a first search result, including: according to the database scale of each of the at least two content vector libraries, respectively determining the approximate retrieval algorithm of each of the at least two content vector libraries; according to the high-dimensional vector representation, based on the approximate retrieval algorithm of each of the at least two content vector libraries, respectively searching to obtain the search results of each of the at least two content vector libraries; and obtaining the first search result based on the search results of each of the at least two content vector libraries.

[0046] The high-dimensional vector library includes at least two content vector libraries, each of which may have different scales, that is, the content vector library includes high-dimensional vector representations corresponding to different numbers of candidate contents. Content vector libraries of different scales may be searched according to different approximate search algorithms.

[0047] Exemplarily, the server may determine the approximate retrieval algorithm of each content vector library according to the database scale of each content vector library. For example, for a content vector library with a small data volume, the corresponding approximate retrieval algorithm may include a FLAT index algorithm, and the retrieval may be performed in the corresponding content vector library based on the FLAT index algorithm. For another example, for a content vector library with a large data volume, the corresponding approximate retrieval algorithm may include at least one of a SCANN (Scalable Nearest Neighbor) index algorithm or an HNSW (Hierarchical Navigable Small World) index algorithm, that is, the retrieval may be performed in the corresponding content vector library based on the SCANN index algorithm or the HNSW index algorithm. In some embodiments, after determining the approximate retrieval algorithm of each content vector library, the server may perform retrieval according to the high-dimensional vector representation based on the respective approximate retrieval algorithm to obtain the retrieval results of each content vector library, and the server may obtain the first retrieval result based on the retrieval results of each content vector library. For example, the server may merge, remove duplicates, and other processing on the retrieval results of each content vector library to obtain the first retrieval result.

[0048] In this embodiment, when the high-dimensional vector library includes at least two content vector libraries, the server performs searches respectively based on the database size of each content vector library through a corresponding approximate search algorithm, which can effectively balance the search accuracy and processing efficiency.

[0049] In an exemplary embodiment, according to the high-dimensional vector representation, searches are performed based on the respective approximate search algorithms of at least two content vector libraries to obtain the respective search results of the at least two content vector libraries, including: for a target content vector library in the at least two content vector libraries, according to the approximate search algorithm corresponding to the target content vector library, respectively calculating the first vector matching parameters between the high-dimensional vector representation and the high-dimensional vector representation included in the target content vector library; according to the first vector matching parameters, determining the first matching vector representation that matches the high-dimensional vector representation from the target content vector library; according to the first matching vector representations respectively determined from the at least two content vector libraries, obtaining the respective search results of the at least two content vector libraries.

[0050] The target content vector library is the content vector library currently being searched, and the server can traverse each content vector library, that is, the server can search each content vector library as the target content vector library. The first vector matching parameter is used to characterize the degree of match between the high-dimensional vector representation of the content search request and the high-dimensional vector representation included in the target content vector library, and may specifically include but is not limited to various metric parameters such as similarity and vector distance. The first matching vector representation is a vector representation determined from the target content vector library that matches the high-dimensional vector representation of the content search request.

[0051] Optionally, the server may traverse each content vector library to perform retrieval processing respectively. For the target content vector library currently being processed for retrieval, the server may match the high-dimensional vector representation of the content retrieval request with the high-dimensional vector representation included in the target content vector library according to the corresponding approximate retrieval algorithm to obtain the corresponding first vector matching parameter. The server may determine the vector representation that matches the high-dimensional vector representation from the target content vector library based on the first vector matching parameter, thereby obtaining the first matching vector representation. For example, the first vector matching parameter may include a vector distance, and the server may determine the vector representation corresponding to the vector distance less than a preset distance threshold in the target content vector library as the first matching vector representation that matches the high-dimensional vector representation of the content retrieval request. After traversing each content vector library, the server may obtain the first matching vector representation of each content vector library, and the server may obtain the retrieval results of each content vector library based on each first matching vector representation. For example, the server may directly use the first matching vector as the retrieval result of the corresponding content vector library. In some embodiments, the server may also perform further screening on the first matching vector representation, such as combining, sorting, etc. on the first matching vector representation, thereby obtaining the first retrieval result.

[0052] In this embodiment, the server calculates the first vector matching parameters to determine the first matching vector representation that matches the high-dimensional vector representation of the content retrieval request from the content vector library, and can achieve accurate matching of the content vector library based on the first vector matching parameters, thereby improving the accuracy of the retrieval results.

[0053] In an exemplary embodiment, a search is performed in a preset sparse vector library according to the sparse vector representation to obtain a second search result, including: respectively calculating second vector matching parameters between the sparse vector representation and the sparse vector representation included in the preset sparse vector library; based on the second vector matching parameters, determining a second matching vector representation that matches the sparse vector representation from the sparse vector library; and obtaining a second search result according to the second matching vector representation.

[0054] The second vector matching parameter is used to characterize the degree of matching between the sparse vector representation of the content retrieval request and the sparse vector representation included in the sparse vector library, and may specifically include but not be limited to various metric parameters including similarity, vector distance, etc. The second matching vector representation is a vector representation determined from the sparse vector library that matches the sparse vector representation of the content retrieval request.

[0055] Exemplarily, the server may match the sparse vector representation of the content retrieval request with the sparse vector representations included in the sparse vector library, respectively, and specifically may calculate the second vector matching parameter between the sparse vector representation of the content retrieval request and each sparse vector representation in the sparse vector library, and the second vector matching parameter may specifically include at least one of vector similarity and vector distance. The server may determine the vector representation that matches the sparse vector representation of the content retrieval request from the sparse vector library based on the second vector matching parameter, thereby obtaining the second matching vector representation. For example, the second vector matching parameter may include vector similarity, and the server may determine the sparse vector representation corresponding to the vector similarity less than the similarity threshold in the sparse vector library as the second matching vector representation that matches the sparse vector representation of the content retrieval request. The server may obtain the second search result based on the determined second matching vector representation, such as the server may directly use the second matching vector representation as the second search result. In some embodiments, the server may also perform further screening on the second matching vector representation, such as combining, sorting, etc. on the second matching vector representation, thereby obtaining the second search result.

[0056] In this embodiment, the server calculates the second vector matching parameters to determine a second matching vector representation that matches the sparse vector representation of the content retrieval request from the sparse vector library, and can achieve accurate matching of the sparse vector library based on the second vector matching parameters, thereby improving the accuracy of the retrieval results.

[0057] In an exemplary embodiment, Figure 3 As shown, the process of determining the content search result, that is, obtaining the content search result corresponding to the content search request based on the first search result and the second search result, includes steps 302 to 310. Among them:

[0058] Step 302: obtaining first search content based on the first search result and the first vector mapping relationship, where the first vector mapping relationship is a mapping relationship between candidate content included in the high-dimensional vector library and the corresponding high-dimensional vector representation.

[0059] The high-dimensional vector library includes high-dimensional vector representations corresponding to the candidate content, and a first vector mapping relationship can be established between the candidate content and the corresponding high-dimensional vector representation, so that the corresponding candidate content can be determined through the first vector mapping relationship and the high-dimensional vector representation. The first search content is the candidate content obtained by the corresponding search based on the high-dimensional vector representation in the first search result.

[0060] Optionally, the server may obtain a first vector mapping relationship between candidate content and corresponding high-dimensional vector representations included in the high-dimensional vector library, and the first vector mapping relationship may be generated when the high-dimensional vector library is constructed. The server may determine the corresponding first search content based on the first search result and the first vector mapping relationship, such as the server may perform a candidate content query based on the high-dimensional vector representation in the first search result and the first vector mapping relationship, thereby determining the corresponding first search content.

[0061] Step 304: obtaining second search content based on the second search result and the second vector mapping relationship, where the second vector mapping relationship is a mapping relationship between candidate content included in the sparse vector library and corresponding sparse vector representations.

[0062] The sparse vector library includes sparse vector representations corresponding to the candidate content, and a second vector mapping relationship can be established between the candidate content and the corresponding sparse vector representation, so that the corresponding candidate content can be determined through the second vector mapping relationship and the sparse vector representation. The second search content is the corresponding searched candidate content determined based on the sparse vector representation in the second search result.

[0063] Optionally, the server may obtain a second vector mapping relationship between the candidate content and the corresponding sparse vector representation included in the sparse vector library, and the second vector mapping relationship may be generated when the sparse vector library is constructed. The server may determine the corresponding second search content based on the second search result and the second vector mapping relationship, such as the server may perform a candidate content query based on the sparse vector representation in the second search result and the second vector mapping relationship, thereby determining the corresponding second search content.

[0064] Step 306: Screening is performed based on the first search content and the second search content to obtain preliminary screened content.

[0065] Exemplarily, the server may perform screening based on the first search content and the second search content, such as the server performs screening processing such as combining and removing duplicates on the first search content and the second search content, thereby obtaining preliminary screening content.

[0066] Step 308: determining a correlation parameter between the preliminary screening content and the content retrieval request, and determining target content from the preliminary screening content based on the correlation parameter.

[0067] The relevance parameter is used to characterize the relevance between the preliminary screening content and the content search request, so that based on the relevance parameter, it can be reflected whether the retrieved preliminary screening content meets the search requirements of the content search request. The target content is the content obtained by further screening from the preliminary screening content based on the relevance parameter.

[0068] Optionally, the preliminary screening content may include at least one, and the server may traverse each preliminary screening content, and match each preliminary screening content with the content retrieval request respectively to determine the correlation parameter between the preliminary screening content and the content retrieval request. For example, the server may pre-construct a correlation analysis model, and the server may input the preliminary screening content and the content retrieval request into the correlation analysis model for correlation analysis, so that the correlation analysis model outputs the corresponding correlation parameter. The server may further screen each preliminary screening content based on the correlation parameter corresponding to each preliminary screening content, so as to determine the target content. For example, a correlation threshold may be set in advance, and the server may determine the preliminary screening content whose correlation parameter is greater than the correlation threshold as the retrieved target content.

[0069] Step 310: Perform description conversion on the target content through the large language model to obtain a content search result corresponding to the content search request.

[0070] Among them, Large Language Models (LLM) is an important technology in the field of Natural Language Processing (NLP). It is based on deep learning technology, especially the Transformer architecture, and realizes the understanding and generation of natural language through training of large-scale text data. It is widely used in various tasks such as text generation, machine translation, sentiment analysis, and question-answering systems. Through large language models, description conversion can be performed on the target content, such as converting the target content into a description in natural language form, so as to obtain the content retrieval results corresponding to the content retrieval request.

[0071] Exemplarily, the server can query the large language model and perform description conversion on the target content through the large language model. For example, the server can input the target content into the large language model, and the large language model can perform description conversion on the target content. Specifically, the large language model can perform description conversion on the target content in a required format and output the content retrieval result corresponding to the content retrieval request.

[0072] In this embodiment, the server obtains the first search content and the second search content respectively according to the respective mapping relationships based on the first search result and the second search result, determines the target content according to the relevance parameter after screening based on the first search content and the second search content, and performs description conversion on the target content through the large language model to obtain the content retrieval result corresponding to the content retrieval request, thereby optimizing the content retrieval result and improving the accuracy of the content retrieval result.

[0073] In an exemplary embodiment, the high-dimensional vector representation corresponding to the candidate content included in the high-dimensional vector library is obtained by mapping the candidate content through a high-dimensional vector mapping method; the sparse vector representation corresponding to the candidate content included in the sparse vector library is obtained by mapping the candidate content through a sparse vector mapping method; based on the content retrieval request, the high-dimensional vector representation and the sparse vector representation are respectively constructed, including: mapping the content retrieval request according to the high-dimensional vector mapping method to obtain the high-dimensional vector representation corresponding to the content retrieval request; mapping the content retrieval request according to the sparse vector mapping method to obtain the sparse vector representation corresponding to the content retrieval request.

[0074] Among them, the high-dimensional vector mapping method is a mapping method that maps the target object into a corresponding high-dimensional vector representation, which can be implemented through a mapping algorithm, such as an embedding embedding mapping algorithm. The sparse vector mapping method is a mapping method that maps the target object into a corresponding sparse vector representation, which can be implemented through a mapping algorithm, such as an embedding embedding mapping algorithm.

[0075] For example, for the high-dimensional vector representation corresponding to the candidate content in the high-dimensional vector library, the server can map the candidate content in advance according to the high-dimensional vector mapping method for each candidate content; for the sparse vector representation corresponding to the candidate content in the sparse vector library, the server can map the candidate content in advance according to the sparse vector mapping method for each candidate content. For content retrieval requests, the server can map according to the high-dimensional vector mapping method and the sparse vector mapping method, respectively, to obtain the high-dimensional vector representation and sparse vector representation corresponding to the content retrieval request.

[0076] In this embodiment, the server maps the candidate content and the content search request respectively through a high-dimensional vector mapping method and a sparse vector mapping method, so that a vector representation can be constructed based on the same mapping method for matching search, thereby ensuring the accuracy of content search.

[0077] The present application also provides an application scenario, and the application scenario applies the above-mentioned content retrieval method. Specifically, the application of the content retrieval method in the application scenario is as follows:

[0078] This application scenario involves the technical fields of information retrieval and natural language processing, and specifically involves a knowledge base retrieval enhancement method in a retrieval enhancement generation system for a large model. It combines information retrieval and document library design to achieve efficient management and efficient retrieval of massive documents, and supports processing and providing efficient, accurate and highly relevant content returns based on tens of millions of documents.

[0079] With the rapid development of big data and artificial intelligence technologies, knowledge base systems are increasingly used in information management and knowledge acquisition. Traditional knowledge base retrieval systems have efficiency and accuracy issues when processing large-scale documents. Especially when faced with millions or even tens of millions of documents, retrieval systems often face performance bottlenecks, resulting in long response times and inaccurate results. In addition, although natural language generation models based on deep learning, such as GPT (Generative Pre-trained Transformer) and BERT (Bidirectional Encoder Representations from Transformers), have demonstrated powerful capabilities in generating natural language text, the content generated by these models often lacks relevance and accuracy without contextual support.

[0080] In order to make up for the shortcomings of traditional retrieval systems and generation models, the content retrieval method provided in this application supports knowledge base retrieval enhancement of tens of millions of documents. Knowledge base retrieval enhancement is to improve the retrieval efficiency, accuracy and relevance in the knowledge base system by improving and optimizing information retrieval algorithms and strategies, especially when faced with massive documents, it can quickly locate and extract the most relevant information. The content retrieval method provided in this application can quickly locate relevant information in a large-scale knowledge base by combining efficient information retrieval, and generate high-quality natural language answers based on this. The content retrieval method provided in this application not only improves the retrieval efficiency and generation quality of the system, but also expands the application scope of the knowledge base system, and can meet users' needs for efficient, accurate and intelligent information services.

[0081] The content retrieval method provided in this application can support enhanced knowledge base retrieval of tens of millions of documents, and is used to solve the performance bottlenecks and inaccurate retrieval results that exist in the prior art when processing large-scale documents, while improving the relevance and accuracy of natural language generation. Through the content retrieval method provided in this application, users can obtain more efficient, accurate and intelligent knowledge retrieval services when processing large-scale documents at the level of tens of millions, significantly improving user experience and application effects. Through the information retrieved from the knowledge base, the generative large model system is supported to use external information sources to generate more accurate and more contextual answers, so as to help LLM generate more accurate and more contextual results while reducing hallucinations.

[0082] In the content retrieval method provided by this application, Figure 4 As shown in the figure, in the document storage process, users can upload documents, and the uploaded documents can be stored in an object storage manner. The server can receive and parse documents uploaded by users in the object storage, support multiple file formats (such as PDF, Word, TXT, etc.), and perform optical character recognition (OCR) on scanned documents to achieve file parsing, specifically converting image content into searchable text, and finally returning the Json (JavaScript Object Notation) text type. After completing the creation of the database and vector library data tables, the text content is segmented, and long documents are segmented into multiple text segments; and the acquired text segmentation data is embedded and mapped to the document segments using the pre-trained Embedding model to achieve vectorization processing, and each text segment is converted into a high-dimensional vector representation or a sparse vector representation. Furthermore, the text vector data is inserted into the vector table in the vector library in batches, and the text data is inserted into the database table. The segmented data, the original information of the document and its corresponding vector ID are stored in the database, and the correspondence between the document and the vector is established to achieve efficient data management and complete the warehousing of the document.

[0083] Furthermore, for ANNS retrieval, the vector table will use different indexing strategies to store data of different magnitudes. Among them, the ANNS retrieval algorithm is an algorithm for quickly finding the nearest point to the query point in a large-scale data set. Since performing an exact nearest neighbor search in high-dimensional data is very time-consuming and computationally complex, the approximate nearest neighbor search significantly improves the search speed by sacrificing a certain degree of accuracy. Figure 5 As shown in the figure, the document library can be divided into self-owned document library and customer-built document library. Self-owned document library is a billion-level document library, which provides a large amount of document content for user retrieval; customer-built document library is divided into billion-level and ten thousand-level document library. For customer-built billion-level document library, considering that users will not store a large amount of data at one time, two actual vector libraries are used, one is billion-level data vector library and the other is ten thousand-level vector library. The ten thousand-level vector library is used for incremental uploaded documents. When the document library size reaches 10,000, the incremental documents will be moved to the billion-level data vector library.

[0084] In the document retrieval process, after the user enters a query, the Embedding model is used again to vectorize the question to obtain a high-dimensional vector representation and a sparse vector representation of the question. Among them, a sparse vector is a vector with most elements being zero. Sparse vectors are very common in fields such as computer science, machine learning, and data processing because they can efficiently represent and process large-scale, sparse data. ANNS search is performed on high-dimensional vectors, BM25 (information retrieval algorithm) or sparse matrix search is performed on sparse vectors, and then the original document information is obtained from the database for the searched ID. In the vector search process, the information retrieved by multiple strategies is recalled in multiple ways, that is, the semantic representation provided by the vector search and the precise query results of the sparse vector for the full-text search are combined to achieve hybrid search, and the search results are preliminarily screened and recalled to ensure the high relevance of the results. The search results are then re-ranked by ReRanker (re-ranking algorithm) to improve the accuracy of the results and user experience. Finally, the re-ranking results are finally de-duplicated and ranked, and the retrieved highly relevant document fragments can be sent to the LLM to complete the natural language answer of the large model.

[0085] Specifically, Figure 5 As shown in the figure, vector retrieval uses an approximate nearest neighbor search algorithm. Compared with the usually very time-consuming precise search, the core idea of ​​ANNS is no longer limited to returning the most accurate results, but only searching for the neighbors of the target. The ANNS algorithm quickly finds the approximate nearest neighbor vectors of the query point by building an index structure. For the similarity between vectors, the inner product (IP) is used to calculate the distance between two vectors, which can be specifically expressed as follows:

[0086]

[0087] in, is the distance between two vectors, , , is the dimension of the vector.

[0088] For the index of ANNS, select the FLAT index and the SCANN or HNSW index. The FLAT index linearly scans the data points to find the nearest neighbor of the query point, which can ensure a 100% recall rate; the SCANN index divides the large-scale data set into multiple subsets that are easy to process through a hierarchical index structure and an efficient search strategy. It greatly improves the search efficiency while sacrificing less precision and can maintain a good recall rate, making it suitable for big data scenarios; the HNSW index can provide high accuracy while maintaining high efficiency through a hierarchical structure and the characteristics of a small-world network, making it suitable for big data scenarios. Therefore, the vector library can be divided into the following categories:

[0089] The self-owned document library supports the storage and retrieval of data at the hundred-million level, is maintained by the administrator, and provides users with a prefabricated large amount of document data. It adopts the SCANN or HNSW index to support efficient and accurate similar vector retrieval;

[0090] The customer self-built document library is divided into a customer self-built document library at the hundred-million level and a customer self-built document library at the ten-thousand level;

[0091] For the customer self-built document library at the hundred-million level, considering that users will not store a large amount of data at one time, two actual vector libraries are divided. One is the hundred-million data vector library, and the other is the ten-thousand level vector library. For the hundred-million data vector library, the SCANN or HNSW index is adopted to support the storage and retrieval of all data and improve the retrieval efficiency; for the ten-thousand level vector library, the FLAT index is adopted to ensure a 100% recall rate, store the latest documents uploaded by users, ensure that the content of the latest documents is accurately retrieved 100%, provide accurate content for customers, and at the same time design the size of this vector library to be 10,000 to ensure efficient retrieval. When this vector library reaches 10,000 data, the previous data will be moved to the hundred-million level vector library to complete the incremental update. Note that when retrieving, the ten-thousand level vector library and the hundred-million level vector library are retrieved simultaneously, and then the results are merged, deduplicated, and sorted.

[0092] For the customer self-built document library at the ten-thousand level, in the case of a small amount of customer data, a ten-thousand level document library is provided, and the FLAT index is adopted to facilitate customer upload and retrieval and provide a 100% recall rate.

[0093] Through the retrieval of the above vector library, the vector ID information is obtained, and then the original document data content corresponding to the vector is returned through the database.

[0094] Furthermore, the content retrieval method provided by this application supports the multi-way recall method of ANNS, BM25 / sparse matrix search, then merges and deduplicates and performs rough ranking to obtain the candidate results of TopN (the first N), and then performs ReRanker re-ranking to obtain the final results of TopK (the first K, K < N), which can improve the recall rate of the retrieval results and ensure accuracy. Such as Figure 6 As shown in the figure, when the user triggers an input question, embedding mapping is performed on the input question, and approximate retrieval and sparse retrieval are performed based on the database and vector library respectively to obtain respective database query results, thereby obtaining multi-way recall results. The server can re-rank the multi-way recall results (ReRanker) and perform de-duplication sorting to obtain the final content retrieval results.

[0095] Specifically: ANNS is mainly used for approximate nearest neighbor search of high-dimensional vectors. It can quickly find the vector closest to the query vector in a large vector space. Assuming there is a query vector q and a document vector set D, the ANNS score is calculated as:

[0096]

[0097] in, is the vector q and the vector in the document vector set D ANNS scores between; is the L2 norm. ANNS processes high-dimensional, dense vectors and can capture deep, nonlinear feature relationships.

[0098] Sparse vectors use vector embedding to represent words or phrases, where most elements are zero and only one non-zero element indicates the presence of a specific word. Sparse vectors are superior to dense vectors in knowledge search, keyword awareness, and interpretability. The score calculation for sparse vectors can be as follows:

[0099]

[0100] in, is the vector q and the vector in the document vector set D The fraction of sparse vectors between It is a term In the document vector The number of occurrences in is the document vector Length, is the average length of the document collection, and is the adjustment parameter, It is the calculated inverse document frequency (IDF).

[0101] Sparse vector search usually uses sparse representation methods such as term frequency and inverse document frequency (TF-IDF) to measure the relevance of documents by calculating the degree of match between query terms and document terms.

[0102] By combining ANNS and BM25 / sparse matrix search, we can take full advantage of their respective advantages to achieve more efficient and accurate retrieval. Then merge the candidate information obtained in multiple ways, remove duplicates, and then sort the merged candidate documents according to the predefined weights.

[0103] The main function of ReRanker is to finely sort the candidate results obtained in the preliminary retrieval stage to improve the quality of the final retrieval results. It can use more complex features and models to capture more detailed correlations between questions and documents. Therefore, after completing the rough sorting, the TopN candidate data results are obtained, and then the TopN results and the question Query (query statement) are sent to the ReRanker deep learning model. The model captures the complex relationship between the question and the candidate results, and calculates the correlation score to obtain the TopK (top K) results. Finally, the TopK results are subjected to the final de-duplication and sorting post-processing, and the retrieval results are input into the LLM large language model to obtain the final answer to the question.

[0104] The content retrieval method provided in this embodiment supports enhanced knowledge base retrieval of tens of millions of documents. Combined with information retrieval and document library design, it can achieve efficient management and efficient retrieval of massive documents. The design of specific different document libraries combined with multiple indexing strategies can support efficient management of tens of millions of documents. Moreover, the multi-way recall and screening processing in the content retrieval method provided in this embodiment provides a high recall rate of retrieval results by retrieving dense vectors and sparse vectors and performing rough sorting and reordering. The content retrieval method provided in this embodiment supports document management at the billion level, and achieves rapid warehousing and efficient retrieval of massive documents through efficient indexing and retrieval technology. Moreover, according to the scale of the document library and the application scenario, the combination of different indexes is adopted to ensure both retrieval efficiency and high recall rate. In addition, a variety of search algorithms and rough sorting and reordering technologies are used to ensure high relevance and accuracy of retrieval results.

[0105] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0106] Based on the same inventive concept, the embodiment of the present application also provides a content retrieval device for implementing the content retrieval method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more content retrieval device embodiments provided below can refer to the limitations on the content retrieval method above, and will not be repeated here.

[0107] In an exemplary embodiment, Figure 7 As shown, a content retrieval device 700 is provided, comprising: a retrieval request acquisition module 702, a high-dimensional vector retrieval module 704, a sparse vector retrieval module 706 and a retrieval result acquisition module 708, wherein:

[0108] A search request acquisition module 702, used to acquire a content search request, and construct a high-dimensional vector representation and a sparse vector representation based on the content search request;

[0109] A high-dimensional vector search module 704 is used to search in a preset high-dimensional vector library based on an approximate search method according to the high-dimensional vector representation to obtain a first search result; the high-dimensional vector library includes the high-dimensional vector representation corresponding to the candidate content;

[0110] A sparse vector search module 706 is used to search in a preset sparse vector library according to the sparse vector representation to obtain a second search result; the sparse vector library includes the sparse vector representation corresponding to the candidate content;

[0111] The search result obtaining module 708 is used to obtain the content search result corresponding to the content search request based on the first search result and the second search result.

[0112] In an exemplary embodiment, the high-dimensional vector library includes at least two content vector libraries; the high-dimensional vector retrieval module 704 is further used to determine the approximate retrieval algorithms of the at least two content vector libraries respectively according to the database scale of each of the at least two content vector libraries; perform retrievals based on the approximate retrieval algorithms of the at least two content vector libraries respectively according to the high-dimensional vector representation to obtain the retrieval results of the at least two content vector libraries respectively; obtain the first retrieval result according to the retrieval results of the at least two content vector libraries respectively.

[0113] In an exemplary embodiment, the high-dimensional vector retrieval module 704 is further used to calculate, for a target content vector library in the at least two content vector libraries, first vector matching parameters between the high-dimensional vector representation and the high-dimensional vector representation included in the target content vector library according to the approximate retrieval algorithm corresponding to the target content vector library; determine, from the target content vector library, a first matching vector representation that matches the high-dimensional vector representation according to the first vector matching parameters; and obtain respective retrieval results for the at least two content vector libraries according to the first matching vector representations respectively determined from the at least two content vector libraries.

[0114] In an exemplary embodiment, the sparse vector retrieval module 706 is also used to respectively calculate the second vector matching parameters between the sparse vector representation and the sparse vector representation included in a preset sparse vector library; based on the second vector matching parameters, determine a second matching vector representation that matches the sparse vector representation from the sparse vector library; and obtain a second retrieval result based on the second matching vector representation.

[0115] In an exemplary embodiment, the retrieval result obtaining module 708 is also used to obtain a first search content based on a first search result and a first vector mapping relationship, where the first vector mapping relationship is a mapping relationship between candidate content included in a high-dimensional vector library and corresponding high-dimensional vector representations; obtain a second search content based on a second search result and a second vector mapping relationship, where the second vector mapping relationship is a mapping relationship between candidate content included in a sparse vector library and corresponding sparse vector representations; perform screening based on the first search content and the second search content to obtain preliminary screening content; determine a correlation parameter between the preliminary screening content and the content retrieval request, and determine the target content from the preliminary screening content based on the correlation parameter; perform description conversion on the target content through a large language model to obtain a content retrieval result corresponding to the content retrieval request.

[0116] In an exemplary embodiment, the high-dimensional vector representation corresponding to the candidate content included in the high-dimensional vector library is obtained by mapping the candidate content through a high-dimensional vector mapping method; the sparse vector representation corresponding to the candidate content included in the sparse vector library is obtained by mapping the candidate content through a sparse vector mapping method; the retrieval request acquisition module 702 is also used to map the content retrieval request according to the high-dimensional vector mapping method to obtain the high-dimensional vector representation corresponding to the content retrieval request; map the content retrieval request according to the sparse vector mapping method to obtain the sparse vector representation corresponding to the content retrieval request.

[0117] Each module in the above content retrieval device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0118] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the content retrieval method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a content retrieval method is implemented.

[0119] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0121] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0122] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0124] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0125] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0126] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A content retrieval method, characterized in that: The method comprises: Obtaining a content retrieval request, and constructing a high-dimensional vector representation and a sparse vector representation based on the content retrieval request; According to the high-dimensional vector representation, searching is performed in a preset high-dimensional vector library based on an approximate search method to obtain a first search result; the high-dimensional vector library includes high-dimensional vector representations corresponding to the candidate content; Searching in a preset sparse vector library according to the sparse vector representation to obtain a second search result; the sparse vector library includes the sparse vector representation corresponding to the candidate content; Based on the first search result and the second search result, a content search result corresponding to the content search request is obtained.

2. The method according to claim 1, characterized in that The high-dimensional vector library includes at least two content vector libraries; the step of searching in a preset high-dimensional vector library based on an approximate search method according to the high-dimensional vector representation to obtain a first search result includes: Determining the approximate retrieval algorithms of the at least two content vector libraries respectively according to the database sizes of the at least two content vector libraries respectively; According to the high-dimensional vector representation, searching is performed based on the respective approximate search algorithms of the at least two content vector libraries to obtain respective search results of the at least two content vector libraries; A first search result is obtained according to the search results of each of the at least two content vector libraries.

3. The method according to claim 2, characterized in that The step of performing searches based on the high-dimensional vector representation and the approximate search algorithms of the at least two content vector libraries to obtain search results of the at least two content vector libraries includes: For a target content vector library among the at least two content vector libraries, respectively calculating first vector matching parameters between the high-dimensional vector representation and the high-dimensional vector representation included in the target content vector library according to an approximate search algorithm corresponding to the target content vector library; Determining, according to the first vector matching parameter, a first matching vector representation that matches the high-dimensional vector representation from the target content vector library; Retrieval results of the at least two content vector libraries are obtained according to the first matching vector representations respectively determined from the at least two content vector libraries.

4. The method according to claim 1, characterized in that: The step of searching in a preset sparse vector library according to the sparse vector representation to obtain a second search result includes: respectively calculating second vector matching parameters between the sparse vector representation and the sparse vector representations included in a preset sparse vector library; Determining, from the sparse vector library, a second matching vector representation that matches the sparse vector representation based on the second vector matching parameter; A second search result is obtained according to the second matching vector representation.

5. The method according to claim 1, characterized in that The obtaining, based on the first search result and the second search result, a content search result corresponding to the content search request includes: Obtaining first search content based on the first search result and a first vector mapping relationship, wherein the first vector mapping relationship is a mapping relationship between candidate content included in the high-dimensional vector library and corresponding high-dimensional vector representations; Obtaining second search content based on the second search result and a second vector mapping relationship, wherein the second vector mapping relationship is a mapping relationship between the candidate content included in the sparse vector library and the corresponding sparse vector representation; Perform screening based on the first search content and the second search content to obtain preliminary screening content; Determining a correlation parameter between the preliminary screening content and the content retrieval request, and determining target content from the preliminary screening content based on the correlation parameter; The target content is described and converted by a large language model to obtain a content search result corresponding to the content search request.

6. The method according to any one of claims 1 to 5, characterized in that: The high-dimensional vector representation corresponding to the candidate content included in the high-dimensional vector library is obtained by mapping the candidate content in a high-dimensional vector mapping manner; the sparse vector representation corresponding to the candidate content included in the sparse vector library is obtained by mapping the candidate content in a sparse vector mapping manner; The constructing a high-dimensional vector representation and a sparse vector representation based on the content retrieval request respectively includes: Mapping the content retrieval request according to the high-dimensional vector mapping method to obtain a high-dimensional vector representation corresponding to the content retrieval request; Mapping is performed on the content retrieval request according to the sparse vector mapping method to obtain a sparse vector representation corresponding to the content retrieval request.

7. A content retrieval device, characterized in that: The device comprises: A search request acquisition module, used to acquire a content search request, and construct a high-dimensional vector representation and a sparse vector representation based on the content search request; A high-dimensional vector retrieval module, configured to search in a preset high-dimensional vector library based on an approximate search method according to the high-dimensional vector representation to obtain a first search result; the high-dimensional vector library includes high-dimensional vector representations corresponding to the candidate content; A sparse vector search module, configured to search in a preset sparse vector library according to the sparse vector representation to obtain a second search result; the sparse vector library includes the sparse vector representation corresponding to the candidate content; The retrieval result obtaining module is used to obtain the content retrieval result corresponding to the content retrieval request based on the first retrieval result and the second retrieval result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images