Method and device for optimizing text search quality through sentence fragments, equipment and medium

Abstract

The invention relates to the technical field of text search, and discloses a method and device for optimizing text search quality through sentence fragments, equipment and a medium, and the method comprises the steps that the sentence fragments of conjunctions in a corpus are obtained, and the sentence fragments serve as extraction type abstract elements of a document; training the sentence segments and storing the sentence segments in a vector database; and in the vector database, searching and obtaining related files by taking the sentence fragments as retrieval indexes. According to the method, the advantages of distributed representation are fully utilized, and the search content is processed in a vectorization mode, so that more accurate, more efficient and more diversified document search is realized. By combining vector processing of a word2vec word vector model and a BERT pre-training model, semantic search can be performed on different levels, a more comprehensive and superior solution is provided for natural language processing and information retrieval tasks, and more comprehensive, accurate and diversified search results can be provided by using a method of participating in search by sentence fragments.

Description

Technical Field

[0001] The present invention relates to the field of text search technology, and in particular to a method, device, equipment and medium for optimizing text search quality through sentence fragments. Background Art

[0002] A document search system is a software tool or platform used to find and retrieve document content. It can help users quickly find the information they need in a large number of documents, improving the efficiency and accuracy of information search. These systems usually use search engine technology and text analysis algorithms to process and index documents, allowing users to locate the required documents or specific information sentence fragments through keywords, phrases or other search conditions. Its application scenarios include but are not limited to academic research, web search, archive management, enterprise knowledge management, etc., improving the efficiency and accuracy of information retrieval and providing convenience and support for users.

[0003] However, traditional document search systems have some obvious shortcomings and defects, including: the quality of retrieval results decreases. When the search input involves complex sentences or multiple keywords, the accuracy of the results of traditional systems decreases, and users often need to spend more time to screen the correct documents. Summary of the invention

[0004] The present invention provides a method, device, computer equipment and medium for optimizing text search quality through sentence fragments to solve the technical problem that the quality of retrieval results of traditional document search systems decreases and the accuracy of results of traditional systems decreases when the search input involves complex sentences or multiple keywords.

[0005] In a first aspect, a method for optimizing text search quality by using sentence fragments is provided, comprising:

[0006] Obtain sentence fragments of connecting words in a corpus, and use the sentence fragments as extractive summary elements of the document;

[0007] Training the sentence fragments and storing them in a vector database;

[0008] In the vector database, sentence fragments are used as retrieval indexes to search and obtain related documents.

[0009] In a second aspect, a device for optimizing text search quality by sentence fragments is provided, comprising:

[0010] An acquisition module, used for acquiring sentence fragments of connecting words in a corpus, and using the sentence fragments as extractive summary elements of a document;

[0011] A training storage module, used for training the sentence fragments and storing them in a vector database;

[0012] The search module is used to search and obtain related documents in the vector database by using sentence fragments as retrieval indexes.

[0013] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for optimizing text search quality through sentence fragments when executing the computer program.

[0014] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for optimizing text search quality through sentence fragments are implemented.

[0015] In the above-mentioned method, device, computer equipment and storage medium for optimizing text search quality through sentence fragments, the sentence fragments of connecting words in the corpus can be obtained, and the sentence fragments can be used as extractive summary elements of the document; the sentence fragments are trained and stored in the vector database; in the vector database, the sentence fragments are used as retrieval indexes to search and obtain related files. The advantages of distributed representation are fully utilized, and the search content is processed in a vectorized manner, thereby achieving more accurate, more efficient and more diverse document searches. By combining the vector processing of the word2vec word vector model and the BERT pre-trained model, semantic search can be performed at different levels, providing a more comprehensive and superior solution for natural language processing and information retrieval tasks. The method of using sentence fragments to participate in the search can provide more comprehensive, accurate and diverse search results, process complex search content, provide extractive summaries, and support real-time search, thereby improving the quality and efficiency of document search and enhancing user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0017] Figure 1 is a schematic diagram of an application environment of a method for optimizing text search quality through sentence fragments in one embodiment of the present invention;

[0018] Figure 2 is a flow chart of a method for optimizing text search quality through sentence fragments in one embodiment of the present invention;

[0019] Figure 3 is a structural schematic diagram of a device for optimizing text search quality by using sentence fragments in one embodiment of the present invention;

[0020] Figure 4 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0021] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The method for optimizing text search quality by using sentence fragments provided in the embodiment of the present invention can be applied in the following aspects: Figure 1 In an application environment, the client obtains sentence fragments of connecting words in a corpus and uses the sentence fragments as extractive summary elements of documents; trains and stores the sentence fragments in a vector database; in the vector database, uses the sentence fragments as retrieval indexes to search and obtain related files. The advantages of distributed representation are fully utilized, and the search content is processed in a vectorized manner, thereby achieving more accurate, more efficient and more diverse document searches. It is suitable for large-scale document searches and also has potential application value in real-time search use cases. By combining the vector processing of the word2vec word vector model and the BERT pre-trained model, semantic searches can be performed at different levels, providing a more comprehensive and superior solution for natural language processing and information retrieval tasks. The method of using sentence fragments to participate in the search can provide more comprehensive, accurate and diverse search results, process complex search content, provide extractive summaries, and support real-time searches, thereby improving the quality and efficiency of document searches and enhancing user experience. The present invention is described in detail below through specific embodiments.

[0024] See also Figure 2 As shown, Figure 2 A flow chart of a method for optimizing text search quality by using sentence fragments provided in an embodiment of the present invention includes the following steps:

[0025] S10: Obtain sentence fragments of connectives in the corpus, and use the sentence fragments as extractive summary elements of the document;

[0026] Specifically, connecting words are obtained from the corpus, and phrases and sentence fragments of the connecting words are obtained based on the combination of the part-of-speech tagger and the chunker; text vectors are created for the phrases and sentence fragments by the word2vec word vector model and the BERT pre-trained model, respectively.

[0027] A corpus is a resource containing a large amount of text or language samples that can be used to study language structure and train natural language processing systems. The steps to create a corpus are roughly divided into:

[0028] Collect text data: Taking the project of building a macroeconomic knowledge base as an example, the relevant text data can come from various sources, including economics journals, academic papers, government reports, news articles, brokerage research reports, economic data releases, etc. These text data can be obtained through online databases, open data sources or data crawling.

[0029] Cleaning and processing data: After obtaining the raw text data, data cleaning and processing are required to ensure the quality and accuracy of the knowledge base. Data cleaning steps may include removing non-text content, standardizing text, etc.

[0030] Regular updating and maintenance: Knowledge in the economic field is constantly developing and updating, so building a macroeconomic knowledge base is not a one-time task, but a process that requires regular updating and maintenance. The frequency of updates can be determined based on the update frequency and importance of the data source.

[0031] It should be noted that the steps of obtaining connectives from a corpus and obtaining phrases and sentence fragments of connectives based on a combination of a part-of-speech tagger and a chunker include: obtaining connectives from a corpus and detecting the language type of the connectives; when the connectives are in English, performing part-of-speech tagging on each word or phrase in the connectives by a part-of-speech tagger, and combining continuous word sequences into phrase blocks or phrases by a chunker; when the connectives are in Chinese, performing word segmentation on the Chinese connectives using the Jieba library, performing part-of-speech tagging on each word or phrase in the segmented connectives by a part-of-speech tagger, and combining continuous word sequences into phrase blocks or phrases by a chunker.

[0032] Text processing (POS tagger and chunker combination): POS tagger (part-of-speech tag) processing: Tag each word or phrase in the natural language text with part-of-speech, that is, classify each word into different part-of-speech categories such as noun, verb, adjective, adverb, etc.

[0033] Chunker processing: It is used to group consecutive sequences of words in natural language text into larger phrase blocks or phrases, called "chunks". These chunks can be noun phrases, verb phrases, adjective phrases, etc.

[0034] This embodiment uses the NLTK (Natural Language Toolkit) library in Python to implement part-of-speech tagging and chunking. However, the default part-of-speech tagging and chunking of NLTK are for English, which is different from the grammar and rules of Chinese. To address this problem, this embodiment will first use the jieba library (jieba word segmentation, a commonly used Chinese word segmentation library) to segment Chinese text, and then use the part-of-speech tagging and chunking of NLTK.

[0035] Simple example: Input (economic corpus): "Economic growth has an impact on the inflation rate."

[0036] Output before processing (using jieba participle): ['economy', 'growth', 'on', 'inflation rate', 'has', 'influence', '. ']

[0037] Processed output (using NLTK part-of-speech tags): [('economy','NN'),('growth','NN'),('for','P'),('inflation rate','NN'),('have','V'),('influence','NN'),('.','x')]

[0038] Among them, the steps of creating text vectors for phrases and sentence fragments respectively by the word2vec word vector model and the BERT pre-trained model include: using the word2vec word vector model to generate word-level word vectors; inputting the entire sentence into the BERT pre-trained model for training and obtaining sentence-level sentence vectors; and arranging the word vectors obtained after processing by the word2vec word vector model and the sentence vectors obtained after processing by the BERT pre-trained model into text vectors, in which each row represents a vector.

[0039] Embedding vector processing: A technique that maps discrete symbols or objects (such as words, phrases, or sentences) into real number vectors. It represents each word as a vector in a high-dimensional space, so that words with similar semantics are closer in the vector space.

[0040] This example uses two popular methods, word2vec word vector model and BERT pre-training model, to perform embedding processing:

[0041] Application of word2vec word vector model in macroeconomic knowledge corpus: Assume that there is a macroeconomic text corpus, which contains various economic academic papers and news articles. Use word2vec word vector model to generate word-level word embeddings to learn the distributed representation of each word in the corpus. After training, each word is represented as a real number vector, so that words with similar meanings in the vector space are close to each other. For example, after training, the word2vec word vector model may represent the two words "inflation" and "price increases" as close vectors because they have similar meanings in the macroeconomic context.

[0042] Application of BERT pre-trained model in macroeconomic knowledge corpus: Assume that there is a corpus containing economics texts and macroeconomic forecasts. Since the BERT pre-trained model is suitable for processing complex sentence-level tasks, it can be used to generate higher-level word embeddings to better understand the sentence semantics in the text.

[0043] Unlike the word2vec word vector model, the BERT pre-trained model is a pre-trained model based on the Transformer architecture, which can consider contextual information in a bidirectional manner. We can input the entire sentence into the BERT pre-trained model and obtain the sentence-level embedding representation.

[0044] For example, if you input an economic prediction sentence, such as "GDP is expected to grow by 5% next year", the BERT pre-trained model will learn the representation of this sentence and map it into a real number vector. In the vector space, sentences with similar semantics will be closer, so sentences with similar economic predictions may be clustered in close locations.

[0045] S20: training the sentence fragment and storing it in a vector database;

[0046] The step of training and storing the sentence fragments in a vector database includes: indexing the created text vectors using the "index" class of the vector database; and adding the sorted text vectors to the index of the vector database.

[0047] The document index is a structured data used to accelerate information retrieval. The document index built by the indexer can quickly locate documents containing specific words. This embodiment uses the Faiss vector database to build an index to accelerate vector similarity search. The Faiss vector database uses efficient algorithms and data structures to quickly find similar vectors or nearest neighbor vectors in large-scale vector data.

[0048] The following is a brief explanation of the steps involved in index building:

[0049] Prepare vector data: First, organize the word vectors or sentence vectors obtained after processing by the word2vec word vector model or the BERT pre-trained model into vector data, where each row represents a vector.

[0050] Constructing the Faiss vector database index: Use the "Index indicator" class of the Faiss vector database to construct the index. Taking the Chinese economic knowledge corpus as the background, since it contains a large amount of vector data, this patent uses the IVF (Inverted File) hierarchical index type that can efficiently process large-scale vector data.

[0051] In the Faiss vector database, the Index indicator class is used to build indexes. Common index types include Flat index, IVF index (Inverted File index), PCA index, etc. Indexes can be built in memory or on disk. Specifically, some random 128-dimensional vector data were first generated, then an IndexFlatL2 index object was created, and the vector data was added to the index. Finally, we used the index object to perform a search operation to find the nearest neighbors of these vectors in the original data set. This process shows how to use the "Index" class and its subclasses in the Faiss vector database library to build indexes and manage vector data.

[0052] Add vector: Add the vector data processed by the word2vec word vector model or the BERT pre-trained model to the Faiss vector database index. Use the add method of the Faiss vector database.Index indicator class to insert the vector into the index.

[0053] d) Vector search: Use the Faiss vector database. The search method of the Index class can search the index for the vector that is most similar to the given vector. You can specify the number of nearest neighbors to return.

[0054] e) Clustering: Use the Faiss vector database. The Clustering class can cluster vectors. You can choose to use the k-means or average clustering algorithm.

[0055] f) Vector transformation: Use the Faiss vector database. The PCAMatrix class can reduce the dimension of the vector to reduce the time of index construction and search.

[0056] j) Multi-GPU support: Faiss vector database also provides support for multiple GPUs, which can accelerate the index building and search process.

[0057] The Faiss vector database is a powerful and efficient similarity search library, which is often used in similarity search tasks in fields such as images and texts. Using the Faiss vector database can greatly improve search efficiency and can handle large-scale vector data sets.

[0058] S30: In the vector database, the sentence fragment is used as a retrieval index to search and obtain related documents.

[0059] It should be noted that the step of using sentence fragments as retrieval indexes to search and obtain relevant files in the vector database includes: when a query vector is received, searching through the index of the vector database, the vector database finds the vector that is most similar or closest to the query vector, and returns the similarity score or the nearest neighbor index. Among them, the step of the vector database finding the vector that is most similar or closest to the query vector includes: a breadth-first search based on the word2vec word vector model embedding to process vocabulary and phrases, quickly and in large quantities to obtain documents related to the input keywords; a depth-first search based on the BERT pre-trained model embedding to process sentence fragments, so as to accurately and optimize the most matching document results.

[0060] Specifically, search input processing: After using Sentence Transformer to process the sentence fragments or words input by the user, two types of vector representations are obtained: the word2vec word vector model vector represents the semantics of each word, and the BERT pre-trained model vector represents the sentence-level semantics of the entire input sentence fragment or word. These vectors will be used for further vector searches.

[0061] Vector search is a search technique based on vector similarity, which aims to find other vectors in the vector space that are most similar or closest to a given query vector. Similarity is usually measured using metrics such as cosine similarity, Euclidean distance, or inner product.

[0062] In terms of search form, this patent uses a distributed representation that is more advantageous than the traditional pure symbolic search method - the distributed representation of document elements is effective for performing breadth-first search (BFS) and depth-first search (DFS), and different embedding choices show unique and complementary properties when searching. Common distributed representation vector search methods include the use of efficient vector index structures, such as the Faiss vector database and Annoy.

[0063] This embodiment uses the Faiss vector database for vector search, and the steps are the same as the document index construction and search input vectorization processing described above: Combine two embedding options for vector search: When there is a query vector, search through the Faiss vector database index. The Faiss vector database will find the vector that is most similar or closest to the query vector, and return the similarity score or the index of the nearest neighbor. Combining the characteristics and advantages of the two embedding options of the word2vec word vector model and the BERT pre-trained model, first use the word2vec word vector model embedding to process the breadth-first search of words and phrases, quickly and in large quantities to obtain documents related to the input keywords, and then use the BERT pre-trained model embedding to process the depth-first search of sentence fragments, so as to accurately and optimize the most matching document results.

[0064] This article will use a simple example to show how to use the word2vec word embedding model and the BERT pre-training model two-stage search method to improve and accurately search for documents related to "the impact and correlation of inflation on economic growth" from the macroeconomic knowledge corpus:

[0065] 1. Breadth-first search (word2vec word vector model): First, the keywords "inflation" and "economic growth" are converted into word vectors using the word2vec word vector model. In the macroeconomic knowledge corpus, the word2vec word vector model is used to find other phrases or words with high similarity to these two word vectors. These phrases may include topics and concepts related to "inflation" and "economic growth".

[0066] Example breadth search results:

[0067] - Similar phrase 1: "inflation rate"

[0068] - Similar phrase 2: "economic growth forecast"

[0069] - Similar phrase 3: "the impact of inflation on GDP"

[0070] -...(More similar phrases to "inflation" and "economic growth")

[0071] 2. Depth-first search (BERT pre-trained model): Next, the input sentence "The impact and correlation of inflation on economic growth" is converted into a sentence vector using the BERT pre-trained model. The BERT pre-trained model is used to calculate the similarity between the input sentence vector and the sentence vector in each document in the corpus to find the sentence most relevant to the input sentence.

[0072] Example deep search results:

[0073] - Sentence 1: "Studies on the impact of inflation on economic growth show that there is a large correlation." (From Document A)

[0074] - Sentence 2: "Economists have analyzed the relationship between inflation and economic growth." (from Document B)

[0075] - Sentence 3: "A rise in inflation could pose a challenge to economic growth." (from Document C)

[0076] -...(more sentences that are most relevant to the input sentence)

[0077] 3. Final search results: Finally, through depth-first search, we can find the sentences most relevant to "the impact and correlation of inflation on economic growth". These sentences can provide deeper and more accurate information about the relationship between inflation and economic growth.

[0078] Example end result:

[0079] - Sentence 1: "Studies on the impact of inflation on economic growth show that there is a large correlation." (From Document A)

[0080] In summary, this embodiment retains sentence fragments of these connecting words as extractive summary elements of the document, thereby realizing extractive summary. Such sentence fragments are embedded into an appropriate embedding space, such as the BERT pre-trained model, and the sentence fragments are used as retrieval input to obtain and find related files. Specifically, a part-of-speech tagger and a chunker are combined to obtain sentence fragments from a corpus, and a word2vec word vector model (which converts words into vectors) and a BERT pre-trained model are used to create text vectors for phrases and sentence fragments, respectively. The created text vectors are stored and indexed using the vector database Faiss vector database for user search.

[0081] This embodiment has the following advantages:

[0082] More comprehensive search: Sentence fragment embedding can capture the semantic information of sentences, so it can search for relevant documents more comprehensively. Compared with traditional keyword matching, sentence fragments can better understand the search intent, thus providing more accurate and comprehensive search results.

[0083] Handling complex search content: Sentence fragment embedding can handle long sentences and search content containing connecting words, effectively solving the limitations of traditional search systems in handling complex search content. This makes search results more accurate and users can more easily find documents related to their search intent.

[0084] Provide extractive summaries: Sentence fragment embedding can convert sentence fragments into extractive summaries, and present sentence fragments related to the search content in the document as summaries to users. This helps users obtain key information of the document more quickly and improves user experience.

[0085] Diversified search: Sentence fragment embedding can generate multiple embedding representations, thereby achieving diversified search. This makes the search results richer and more diverse, and can meet the different needs of users in different scenarios.

[0086] Support for real-time search: The sentence fragment embedding method can be applied in real-time search use cases to provide fast response and efficient search. This allows users to obtain relevant documents immediately when needed, enhancing the practicality of the search system.

[0087] The present invention provides a method for optimizing text search quality through sentence fragments, firstly obtaining sentence fragments of connecting words in a corpus, and using the sentence fragments as extractive summary elements of documents; training the sentence fragments and storing them in a vector database; in the vector database, using the sentence fragments as retrieval indexes to search and obtain related files. The advantages of distributed representation are fully utilized, and the search content is processed in a vectorized manner, thereby achieving more accurate, more efficient and more diverse document searches. It is suitable for large-scale document searches and also has potential application value in real-time search use cases. By combining the vector processing of the word2vec word vector model and the BERT pre-trained model, semantic searches can be performed at different levels, providing a more comprehensive and superior solution for natural language processing and information retrieval tasks. The method of using sentence fragments to participate in the search can provide more comprehensive, accurate and diverse search results, process complex search content, provide extractive summaries, and support real-time search, thereby improving the quality and efficiency of document searches and enhancing user experience.

[0088] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0089] In one embodiment, a device for optimizing text search quality by sentence fragments is provided, and the device for optimizing text search quality by sentence fragments corresponds to the method for optimizing text search quality by sentence fragments in the above embodiment. Figure 3 As shown, the device for optimizing text search quality by sentence fragments includes an acquisition module 101, a training storage module 102, and a search module 103. The detailed description of each functional module is as follows:

[0090] An acquisition module 101 is used to acquire sentence fragments of connectives in a corpus and use the sentence fragments as extractive summary elements of a document;

[0091] A training storage module 102, used for training the sentence fragments and storing them in a vector database;

[0092] The search module 103 is used to search and obtain related documents in the vector database by using the sentence fragments as retrieval indexes.

[0093] In one embodiment, the acquisition module 101 is specifically configured to:

[0094] The acquisition submodule is used to obtain the connecting words from the corpus, and obtain the connecting word phrases and sentence fragments based on the combined processing of the part-of-speech tagger and the chunker;

[0095] The vector creation submodule is used to create text vectors for phrases and sentence fragments using the word2vec word vector model and the BERT pre-trained model respectively.

[0096] In one embodiment, the submodule is obtained, specifically for:

[0097] An acquisition unit, used for acquiring connectives from a corpus and detecting the language type of the connectives;

[0098] English phrase unit, which is used to tag each word or phrase in the conjunction with a part-of-speech tagger when the conjunction is in English, and to combine a continuous word sequence into a phrase block or phrase by a chunker;

[0099] Chinese phrase unit, when the conjunction is in Chinese, uses the Jieba library to segment the Chinese conjunction, uses the part-of-speech tagger to tag each word or phrase in the conjunction after segmentation, and uses the chunker to combine continuous word sequences into phrase blocks or phrases.

[0100] In one embodiment, the vector creation submodule is specifically used for:

[0101] The word vector generation unit is used to generate word-level word vectors using the word2vec word vector model;

[0102] The sentence vector generation unit is used to input the entire sentence into the BERT pre-training model for training and obtain sentence-level sentence vectors;

[0103] The text vector arrangement unit is used to arrange the word vectors obtained after processing by the word2vec word vector model and the sentence vectors obtained after processing by the BERT pre-training model into text vectors, where each row represents a vector.

[0104] In one embodiment, the training storage module 102 is further configured to:

[0105] The index building submodule is used to build an index for the created text vector using the "Indicator" class of the vector database;

[0106] The text vector adding submodule is used to add the sorted text vector to the index of the vector database.

[0107] In one embodiment, the search module 103 is specifically configured to:

[0108] The query submodule is used to search through the index of the vector database when a query vector is received. The vector database finds the vector that is most similar to or closest to the query vector and returns the similarity score or the index of the nearest neighbor.

[0109] In one embodiment, the query submodule is specifically used to:

[0110] The word vector embedding unit is used to embed the word2vec word vector model to process the breadth-first search of words and phrases, and quickly obtain documents related to the input keywords in large quantities;

[0111] The sentence vector embedding unit is used to embed the sentence fragments based on the BERT pre-trained model to perform a depth-first search, so as to accurately and optimize the best matching document results.

[0112] The present invention provides a device for optimizing text search quality through sentence fragments, which first obtains sentence fragments of connecting words in a corpus and uses the sentence fragments as extractive summary elements of documents; trains the sentence fragments and stores them in a vector database; in the vector database, uses the sentence fragments as retrieval indexes to search and obtain related files. The advantages of distributed representation are fully utilized, and the search content is processed in a vectorized manner, thereby achieving more accurate, more efficient, and more diverse document searches. By combining the vector processing of the word2vec word vector model and the BERT pre-trained model, semantic searches can be performed at different levels, providing a more comprehensive and superior solution for natural language processing and information retrieval tasks. The method of using sentence fragments to participate in the search can provide more comprehensive, accurate, and diverse search results, process complex search content, provide extractive summaries, and support real-time search, thereby improving the quality and efficiency of document searches and enhancing user experience.

[0113] For the specific definition of the device for optimizing text search quality through sentence fragments, please refer to the definition of the method for optimizing text search quality through sentence fragments above, which will not be repeated here. Each module in the above-mentioned device for optimizing text search quality through sentence fragments can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0114] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements a function or step on the server side of a method for optimizing text search quality through sentence fragments.

[0115] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client side of a method for optimizing text search quality through sentence fragments

[0116] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0117] Obtain sentence fragments of connecting words in a corpus, and use the sentence fragments as extractive summary elements of the document;

[0118] Training the sentence fragments and storing them in a vector database;

[0119] In the vector database, sentence fragments are used as retrieval indexes to search and obtain related documents.

[0120] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0121] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0122] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0123] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for optimizing text search quality by sentence fragments, characterized in that: include: Obtain sentence fragments of connecting words in a corpus, and use the sentence fragments as extractive summary elements of the document; Training the sentence fragments and storing them in a vector database; In the vector database, sentence fragments are used as retrieval indexes to search and obtain related documents.

2. The method for optimizing text search quality by sentence fragments according to claim 1, characterized in that: The step of obtaining sentence fragments of connecting words in the corpus and using the sentence fragments as extractive summary elements of the document includes: Get the connectives from the corpus, and obtain the phrases and sentence fragments of the connectives based on the combination of the part-of-speech tagger and the chunker; The word2vec word vector model and the BERT pre-trained model create text vectors for phrases and sentence fragments respectively.

3. The method for optimizing text search quality by sentence fragments as claimed in claim 2, characterized in that: The step of obtaining a conjunction from a corpus and obtaining phrases and sentence fragments of the conjunction based on a combination of a part-of-speech tagger and a chunker comprises: Get connectives from the corpus and detect the language type of the connectives; When the conjunction is in English, each word or phrase in the conjunction is tagged with a part-of-speech tagger, and a continuous word sequence is combined into phrase blocks or phrases by a chunker; When the connectives are in Chinese, the Jieba library is used to segment the Chinese connectives, the part-of-speech tagger is used to tag each word or phrase in the segmented connectives, and the chunker is used to combine continuous word sequences into phrase blocks or phrases.

4. The method for optimizing text search quality by sentence fragments as claimed in claim 2, characterized in that: The steps of creating text vectors for phrases and sentence fragments respectively by using the word2vec word vector model and the BERT pre-trained model include: Use the word2vec word vector model to generate word-level word vectors; Input the entire sentence into the BERT pre-trained model for training and obtain sentence-level sentence vectors; The word vectors obtained by processing the word2vec word vector model and the sentence vectors obtained by processing the BERT pre-trained model are organized into text vectors, where each row represents a vector.

5. The method for optimizing text search quality by sentence fragments as claimed in claim 4, characterized in that: The step of training the sentence fragments and storing them in a vector database comprises: Use the "Indicator" class of the vector database to build an index for the created text vector; Add the sorted text vector to the index of the vector database.

6. The method for optimizing text search quality by sentence fragments according to claim 5, characterized in that: The step of using the sentence fragment as a retrieval index to search and obtain related documents in the vector database includes: When a query vector is received, a search is performed through the index of the vector database, which finds the vector that is most similar or closest to the query vector and returns a similarity score or the index of the nearest neighbor.

7. The method for optimizing text search quality by sentence fragments according to claim 6, characterized in that: The step of finding the vector that is most similar to or closest to the query vector in the vector database comprises: Based on the word2vec word vector model, the breadth-first search of words and phrases can be embedded to quickly obtain documents related to the input keywords in large quantities; The BERT pre-trained model embeds a depth-first search to process sentence fragments, thereby accurately and optimally obtaining the most matching document results.

8. A device for optimizing text search quality by sentence fragments, characterized in that: include: An acquisition module, used for acquiring sentence fragments of connectives in a corpus, and using the sentence fragments as extractive summary elements of a document; A training storage module, used for training the sentence fragments and storing them in a vector database; The search module is used to search and obtain related documents in the vector database by using sentence fragments as retrieval indexes.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for optimizing text search quality by using sentence fragments as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for optimizing text search quality by sentence fragments as claimed in any one of claims 1 to 7 are implemented.

Images