Text search method and device based on vector retrieval service

Abstract

The invention relates to the technical field of information retrieval, and particularly provides a text search method and device based on vector retrieval service, and the method comprises the following steps: S1, uploading and analyzing a file; s2, vectorizing the text; s3, carrying out vectorization storage; s4, carrying out vector retrieval; and S5, displaying a retrieval effect. Compared with the prior art, the method has the advantages that required information can be quickly and accurately retrieved from massive text information, and the method has high fault tolerance and error correction capability.

Description

Technical Field

[0001] The present invention relates to the technical field of information retrieval, and specifically provides a text search method and device based on vector retrieval service. Background Art

[0002] With the explosive growth of the Internet and digital information, it has become a challenge to efficiently and accurately retrieve the required information from massive text data. Traditional keyword retrieval methods have limitations in dealing with problems such as complex semantics and polysemous words. They are often inefficient when processing massive data and have difficulty accurately capturing semantic information, which cannot meet the needs of modern information retrieval.

[0003] In recent years, natural language processing technology based on deep learning, especially the development of word embedding and vector space model, has provided new ideas for improving text retrieval. Therefore, the present invention proposes a text search method based on vector retrieval service to improve the efficiency and accuracy of retrieval. Summary of the invention

[0004] The present invention aims at solving the above-mentioned deficiencies of the prior art and provides a text search method based on vector retrieval service with strong practicality.

[0005] A further technical task of the present invention is to provide a text search device based on vector retrieval service that is reasonably designed, safe and applicable.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A text search method based on vector retrieval service has the following steps:

[0008] S1, file upload and parsing;

[0009] S2, text vectorization;

[0010] S3, vectorized storage;

[0011] S4, vector retrieval;

[0012] S5. Display of search results.

[0013] Furthermore, in step S1, the content in different formats is converted into plain text data that can be understood by the large language model with the help of a document loader, and the text content is extracted and stored in a vector database after vectorization.

[0014] Furthermore, in step S2, document slicing is performed to divide the lengthy document into a plurality of small text blocks;

[0015] First, identify and perform preliminary sentence-level segmentation according to the sentence boundaries in the document. In the segmentation process, some overlapping content is deliberately introduced to ensure the integrity of the sentence content.

[0016] The segmented document fragments will be vectorized and properly stored to provide data support for subsequent retrieval and generation tasks.

[0017] Furthermore, in step S3, a suitable vectorized storage and retrieval engine is selected based on specific application requirements and data characteristics, wherein the application requirements and data characteristics include data size, query frequency, search speed, scalability, and deployment difficulty.

[0018] Furthermore, in step S4, the user's query question is converted into a vector and matched in a vector database to optimize the query expression and retrieval parameter settings;

[0019] First, a large language model is used in combination with historical conversation information to rewrite or expand the user's original query to reflect the user's true intentions and needs.

[0020] Furthermore, during the retrieval process, a reasonable number of retrieval results, topK, is set to cover multiple aspects of the query while avoiding information overload;

[0021] Select a similarity measurement method suitable for the current task to accurately calculate the semantic similarity between vectors.

[0022] Furthermore, the display page of the text search effect presents the search results and performance to the user in an intuitive way. The user can quickly search for documents by entering the query words and view the titles and brief descriptions of related documents;

[0023] The page also displays the search accuracy through a simple progress bar or star rating, and provides sorting options to meet different needs;

[0024] In addition, users can quickly provide satisfaction ratings and suggestions on search results through feedback forms to help optimize system performance.

[0025] A text search device based on vector retrieval service, comprising: at least one memory and at least one processor;

[0026] The at least one memory is used to store a machine-readable program;

[0027] The at least one processor is used to call the machine-readable program to execute a text search method based on a vector retrieval service.

[0028] Compared with the prior art, the text search method and device based on vector retrieval service of the present invention have the following outstanding beneficial effects:

[0029] The present invention can quickly and accurately retrieve required information from massive text information and has strong fault tolerance and error correction capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Attached Figure 1 It is a flowchart of a text search method based on vector retrieval service. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] A best embodiment is given below:

[0034] like Figure 1 As shown, a text search method based on a vector search service in this embodiment has the following steps:

[0035] S1, file upload and parsing;

[0036] In real-world scenarios, there are a variety of file information sources. We need to deal with file types including Word documents, TXT files, CSV data tables, Excel tables, and PDF. In order to convert these different formats of content into plain text data that can be easily understood by large language models, we need to use a dedicated document loader. By using loaders of different file types, we can accurately extract the text content and store it in a vector database after vectorization.

[0037] S2, text vectorization;

[0038] Document slicing is one of the crucial steps, which divides the long document into multiple small text chunks.

[0039] This approach not only helps to reduce the computational burden of large language models and improve processing efficiency, but also improves accuracy during retrieval, allowing the system to more accurately locate the information users need.

[0040] We adopted a document segmentation strategy based on token length.

[0041] First, we identify and perform preliminary sentence-level segmentation according to the sentence boundaries in the document. In order to improve the coverage and accuracy of retrieval, we deliberately introduce some overlapping content in the slicing process. This approach not only ensures the integrity of the sentence content, but also significantly reduces the computational burden of the model, thereby improving the overall information retrieval efficiency and accuracy. These carefully segmented document fragments will be vectorized and properly stored to provide strong data support for subsequent retrieval and generation tasks.

[0042] S3, vectorized storage;

[0043] Choose the appropriate vectorized storage and retrieval engine based on specific application requirements and data characteristics, including data size, query frequency, search speed, scalability, and deployment difficulty. For example, for large-scale text data processing, the efficient algorithms of Faiss and Milvus are more suitable, while on edge devices, Vearch's low memory usage and high-speed search are preferred.

[0044] S4, vector retrieval;

[0045] In order to provide users with more accurate and comprehensive search results, it is necessary not only to convert user query questions into vectors and match them in the vector database, but also to adopt a series of strategies to optimize the query expression and retrieval parameter settings.

[0046] First, a large language model is used in combination with historical conversation information to rewrite or expand the user's original query to more accurately reflect the user's true intentions and needs.

[0047] Secondly, during the retrieval process, by setting a reasonable number of retrieval results (topK), we can cover multiple aspects of the query and avoid information overload. We can accurately calculate the semantic similarity between vectors by selecting a similarity measurement method suitable for the current task. The combination of these strategies and technologies can significantly improve the efficiency and accuracy of text search, providing a more accurate, comprehensive and personalized search experience.

[0048] S5, retrieval effect display;

[0049] The display page of the text search effect presents the search results and performance to the user in a concise and intuitive way. Users can quickly search for documents by entering query words and view the titles and brief descriptions of related documents.

[0050] The page also displays the search accuracy through a simple progress bar or star rating, and provides sorting options to meet different needs. In addition, users can quickly provide satisfaction evaluation and suggestions on the search results through the feedback form, thereby helping to optimize the performance of the entire system.

[0051] Based on the above method, a text search device based on vector retrieval service in this embodiment includes: at least one memory and at least one processor;

[0052] The at least one memory is used to store a machine-readable program;

[0053] The at least one processor is used to call the machine-readable program to execute a text search method based on a vector retrieval service.

[0054] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0055] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A text search method based on vector retrieval service, characterized in that: The steps are as follows: S1, file upload and parsing; S2, text vectorization; S3, vectorized storage; S4, vector retrieval; S5. Display of search results.

2. A text search method based on vector retrieval service according to claim 1, characterized in that: In step S1, the document loader converts the contents in different formats into plain text data that can be understood by the large language model, extracts the text content, and stores it in the vector database after vectorization.

3. A text search method based on vector retrieval service according to claim 2, characterized in that: In step S2, document slicing is performed to divide the lengthy document into multiple small text blocks; First, identify and perform preliminary sentence-level segmentation according to the sentence boundaries in the document. In the segmentation process, some overlapping content is deliberately introduced to ensure the integrity of the sentence content. The segmented document fragments will be vectorized and properly stored to provide data support for subsequent retrieval and generation tasks.

4. A text search method based on vector retrieval service according to claim 3, characterized in that: In step S3, a suitable vectorized storage and retrieval engine is selected based on specific application requirements and data characteristics, wherein the application requirements and data characteristics include data scale, query frequency, search speed, scalability, and deployment difficulty.

5. A text search method based on vector retrieval service according to claim 4, characterized in that: In step S4, the user's query question is converted into a vector and matched in the vector database to optimize the query expression and retrieval parameter settings; First, a large language model is used in combination with historical conversation information to rewrite or expand the user's original query to reflect the user's true intentions and needs.

6. A text search method based on vector retrieval service according to claim 5, characterized in that: During the retrieval process, we set a reasonable number of topK search results to cover multiple aspects of the query while avoiding information overload; Select a similarity measurement method suitable for the current task to accurately calculate the semantic similarity between vectors.

7. A text search method based on vector retrieval service according to claim 6, characterized in that: The display page of text search results presents the search results and performance to users in an intuitive way. Users can quickly search for documents by entering query words and view the titles and brief descriptions of related documents. The page also displays the search accuracy through a simple progress bar or star rating, and provides sorting options to meet different needs; In addition, users can quickly provide satisfaction ratings and suggestions on search results through feedback forms to help optimize system performance.

8. A text search device based on vector retrieval service, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 7.

Images