Solr-based full-text retrieval method, system, device and medium

Abstract

The invention discloses a full-text search method, system, device and medium based on Solr, and belongs to the technical field of information retrieval. The technical problem to be solved by the invention is how to realize the query operation of fast positioning of system data and search keyword-related information, optimize the service query speed, and improve the work efficiency of users. The technical scheme is as follows: service deployment: adopting distributed deployment of Solr service, reading data in data source, and using built-in text analyzer to decompose text information; index creation: converting text information into index; data retrieval: user submits search request, Solr service retrieves index according to request data, loads data by reverse loading and reverse loading of inverted table, and returns the result to user; highlight display and summary generation: by adding corresponding parameters, marking the part matching the search keyword in the search result, and providing corresponding summary information; index update.

Description

Technical Field

[0001] The present invention relates to the field of information retrieval technology, and in particular to a Solr-based full-text retrieval method, system, device and medium. Background Art

[0002] With the rapid development of the internet, information data is growing and its data types are becoming increasingly diverse. The rapid and accurate retrieval of this massive amount of data has become a pressing challenge. The application of new-generation information technology, represented by computer technology, has enabled full-text retrieval to evolve from its initial string matching and simple Boolean logic retrieval techniques to a complex technology capable of comprehensively managing large amounts of unstructured data, including text, audio, images, and moving images.

[0003] Traditional search engines still have the following technical problems that need to be improved and solved when processing large-scale data: ① Inefficiency: In document search scenarios, the operating system searches for document content by first loading it into memory and then searching from memory, resulting in low search efficiency; ②Does not support searching of massive data; ③Does not support searching for highly concurrent data.

[0004] Therefore, how to quickly locate system data and search for keyword-related information, optimize service query speed, and improve user efficiency is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The technical task of the present invention is to provide a full-text retrieval method, system, device and medium based on Solr to solve the problem of how to quickly locate system data and search for keyword-related information, optimize service query speed, and improve user efficiency.

[0006] The technical task of the present invention is achieved in the following manner: a full-text retrieval method based on Solr, the method is as follows: Service deployment: Use distributed deployment of Solr services to read data from the data source and use built-in text analyzers (such as IKAnalyzer) to decompose text information; Index creation: converting text information into an index for fast search and receiving data retrieval requests; Data retrieval: When a user submits a search request, the Solr service retrieves the index based on the requested data, loads the data using reverse loading and inverted table reverse loading, and returns the results to the user. Highlighting and summary generation: By adding corresponding parameters, the parts matching the search keywords are marked in the search results and corresponding summary information is provided; Index update: Regularly update and maintain the Solr service index based on application requirements to keep the data up to date.

[0007] Preferably, the Solr service includes a client, a ZooKeeper cluster, a Solr cluster (SolrCloud), and several disks; the Solr cluster includes several SolrServers; The client communicates with the SolrServer in the Solr cluster through HTTP or HTTPS protocols and performs distributed indexing and distributed search operations; SolrServer provides index creation and full-text search services and is the data calculation and processing unit in the Solr cluster; The ZooKeeper cluster provides distributed collaboration services for each process in the Solr cluster; Each SolrServer registers its own information in the ZooKeeper cluster. The Client perceives the health status of each SolrServer based on its registration information and then decides on the distribution of indexing and search requests. Each SolrServer reads and writes index files from the disk based on the received indexing and search requests.

[0008] Better yet, the Solr service uses a reverse indexing approach to map keywords to documents. This allows for quick location of documents during keyword searches by building relationships between keywords and documents.

[0009] More optimally, create the index as follows: Index document tokenization: The original document content to be indexed is sent to the word segmentation component, which decomposes the document into vocabulary units Token through the Tokenize process; The lexical unit Token is processed through language analysis: the corresponding language characteristics analysis is performed according to different languages ​​and then processed to obtain the word Term; Inverted index builds an index library: The index component establishes an index relationship between the term and the document to form an index library; the index component creates a dictionary for all terms, sorts them, and merges identical terms to ensure the uniqueness of the search terms.

[0010] Preferably, the Tokenize process is as follows: ① Divide the document into words; ②Remove punctuation marks from the document; ③Remove stop words; The contents of the index library include the term, document frequency, and the document ID and frequency in the document; among them, the term is the keyword to be retrieved after building the index library, and the corresponding document frequency indicates the number of times the corresponding term appears in the document; the document ID in the document is used to link with the term, so as to index the document according to the term, and the corresponding frequency indicates the number of times the searched term appears in the corresponding document.

[0011] As a preference, data retrieval is specifically as follows: Calculate weights: Place all searched document vectors and query vectors into an N-dimensional space, where each term is a dimension. The dimension of the vector space is the union of the term sets of the two. Calculate the relevance score. This is done by sorting the keyword documents and the scores of each document in the search result set in descending order, thereby obtaining a relevance-ranked result list after keyword search. Search index: Randomly query the syntax tree after lexical analysis, grammatical analysis, and language processing in the constructed index library to obtain a preliminary set of all documents that match the syntax tree; Result sorting: Sort by the correlation between keywords and documents, and evaluate the correlation between the index of documents in the document result set and the keyword index using the space vector model algorithm.

[0012] More optimally, the process of obtaining the document vector and query vector is as follows: The full assist of the word Term in the document is formed into a vector as follows: Document = {term1, term2, …… ,term N}; Document Vector = {weight1, weight2, ……,weight N}; The keywords are grouped into a document and the corresponding document is represented by a vector, as follows: Query = {term1, term 2, ……, term N}; Query Vector = {weight1, weight2, ……,weight N}.

[0013] A Solr-based full-text retrieval system is provided, which is used to implement the Solr-based full-text retrieval method described above; the system comprises: The service deployment module is used to deploy Solr services in a distributed manner, read data from the data source, and use built-in text analyzers (such as IKAnalyzer) to decompose text information; An index creation module, used to convert text information into an index for fast search and receiving data retrieval requests; The data retrieval module is used by users to submit search requests. The Solr service retrieves the index based on the requested data, loads the data using reverse loading and inverted table reverse loading, and returns the results to the user. A highlighting and summary generation module is used to mark the parts matching the search keywords in the search results by adding corresponding parameters and provide corresponding summary information; The index update module is used to regularly update and maintain the Solr service index according to application requirements to keep the data up to date.

[0014] An electronic device comprising: a memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the Solr-based full-text retrieval method as described above.

[0015] A computer-readable storage medium stores a computer program, wherein the computer program can be executed by a processor to implement the above-mentioned Solr-based full-text retrieval method.

[0016] The Solr-based full-text search method, system, device, and medium of the present invention have the following advantages: (1) The present invention optimizes service query speed and improves user efficiency by enabling rapid location of system data and query operations for keyword-related information. (2) Fast and efficient: By indexing and optimizing text data, the present invention's full-text search engine can quickly perform search operations on large-scale data sets; (3) Weighted sorting: The present invention can sort the results according to the frequency and importance of the search keywords in the text to provide more accurate search results; (4) Word segmentation and semantic analysis: The present invention can segment text into appropriate words and perform semantic analysis to better understand the user's search intent; (5) Scalability: The present invention provides a wealth of configuration options and expansion interfaces to meet the needs of different application scenarios; (6) The present invention performs word segmentation and inverted indexing on search keywords, allowing for fast and efficient querying through completed indexed documents. It supports highly concurrent data retrieval and efficient indexing, supports horizontal scalability, and can distribute indexing and search requests across multiple servers, improving search performance while providing high availability and fault tolerance. (7) The present invention can process various types of text content, demonstrate excellent word segmentation, retrieval, sorting and filtering capabilities, support fast full-text search in large-scale data sets, including fuzzy search, wildcard search, range search, etc., and support full-text search in multiple languages. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention will be further described below with reference to the accompanying drawings.

[0018] Attachment Figure 1 Schematic diagram of Solr service deployment; Attachment Figure 2 This is a schematic diagram of the reverse index; Attachment Figure 3 Schematic diagram created for the index. DETAILED DESCRIPTION

[0019] The Solr-based full-text search method, system, device, and medium of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Example 1: This example provides a full-text search method based on Solr, which is as follows: S1. Service deployment: Use a distributed deployment of Solr services to read data from the data source and use built-in text analyzers (such as IKAnalyzer) to decompose text information. S2. Index creation: Convert text information into an index to facilitate fast search and receive data retrieval requests; S3, Data Retrieval: The user submits a search request, and the Solr service retrieves the index based on the requested data, loads the data using reverse loading and inverted table reverse loading, and returns the results to the user; S4. Highlighting and summary generation: By adding corresponding parameters, the parts matching the search keywords are marked in the search results and corresponding summary information is provided; S5. Index update: Regularly update and maintain the Solr service index based on application requirements to keep the data up to date.

[0021] As attached Figure 1As shown, the Solr service in step S1 of this embodiment includes a Client, a ZooKeeper cluster, a Solr cluster (SolrCloud), and several disks; the Solr cluster includes several SolrServers; The client communicates with the SolrServer in the Solr cluster through HTTP or HTTPS protocols and performs distributed indexing and distributed search operations; SolrServer provides index creation and full-text search services and is the data calculation and processing unit in the Solr cluster; The ZooKeeper cluster provides distributed collaboration services for each process in the Solr cluster; Each SolrServer registers its own information in the ZooKeeper cluster. The Client perceives the health status of each SolrServer based on its registration information and then decides on the distribution of indexing and search requests. Each SolrServer reads and writes index files from the disk based on the received indexing and search requests.

[0022] The Solr service in step S1 of this embodiment uses a reverse indexing method to map keywords to documents, that is, by building a relationship between keywords and documents, it can quickly locate the document position during the keyword query process. Figure 2 As shown, the left side stores a sequence of strings, and the right side contains a list of document numbers for the strings, called an inverted index. Together, the list of field strings and the list of document numbers form a dictionary. If we search for "lucene," the index directly tells us that the documents containing "lucene" are: 2, 3, 10, 35, 92, without having to search the entire document library one by one. If we search for documents containing both "lucene" and "solr," we can simply intersect the two inverted indexes to obtain: 3, 10, 35, 92.

[0023] As attached Figure 3 As shown, the creation of the index in step S2 of this embodiment is specifically as follows: S201, index document tokenization: the original document content to be indexed is sent to the word segmentation component, and the word segmentation component decomposes the document into vocabulary units Token through the Tokenize process; S202, reprocessing the vocabulary unit token through language analysis: performing language characteristic analysis and reprocessing according to different languages ​​to obtain word terms; for example, operations such as lowercase, stemming, and lemmatization may be performed on English; S203. Build an index library for the inverted index: Establish an index relationship between the term and the document through the index component to form an index library. Among them, the index component creates a dictionary, sorts, and merges the same terms for all terms to ensure the uniqueness of the search terms.

[0024] The Tokenize process in step S201 of this embodiment is specifically as follows: ① Divide the document into words; ② Remove punctuation marks from the document; ③ Remove stop words (Stop word, that is, auxiliary words, predicates, etc. in the language that have no actual meaning. For example, "a", "the" in English, and "啊", "唉", "之", "乎", "者", "也" in Chinese, etc.); The content of the index library includes the term, document frequency (Document Frequency), the document has a document ID (Document ID) and frequency (Frequency) in the document. Among them, the term is the keyword to be retrieved after building the index library, and the corresponding document frequency represents the number of articles in which the corresponding term appears in the document; the document ID in the document is used to link with the term, so that the document can be indexed according to the term, and the corresponding frequency represents the number of times the searched term appears in the corresponding document.

[0025] Data retrieval is the main operation for the user interface, and the processing of retrieval is an important step in whether the search technology can achieve the expected effect.

[0026] When building an index, we divide the document into terms. After using the operation of building an index to tokenize the keyword and build an index, by searching and matching the indexes in the index database through these indexes, we can find the corresponding results.

[0027] This process is actually a process of finding which terms are more important for the relationship between two documents, which are the most important, and the process of the frequency of the terms in the main document appearing in the document to be matched. This process of judging the importance of the terms of the keyword to the terms of the document is called the process of calculating the term weight (Term Weight).

[0028] There are two parameters for calculating the term weight: one is the term, and the other is the document. The term weight represents the importance of this term in the document. The more important the term, the greater the weight, because it will play a greater role in calculating the relevance between documents.

[0029] The vector space model algorithm can be used to determine the relationship between words, thereby quantifying the correlation between documents.

[0030] The data retrieval in step S3 of this embodiment is specifically as follows: S301. Calculate weight: There are two main factors that affect the importance of a term in a document: 1) Term Frequency (tf): This indicates how many times a term appears in the document. A larger tf indicates a more important term.

[0031] 2) Document Frequency (df): This is how many documents contain the term. A larger df indicates a less important term.

[0032] The more often a term appears in a document, the larger its tf is, indicating that the documents primarily introduce or reference this concept. Consequently, the correlation between them increases, resulting in a higher weight. However, when many documents contain the same term, its ability to distinguish documents becomes less effective. In other words, a term with a very large df is less likely to distinguish the correlation between documents, resulting in a less important weight.

[0033] Put all searched document vectors and query vectors into an N-dimensional space, where each term is a dimension. The dimension of the vector space is the union of the term sets of the two. Calculate the relevance score. That is, sort the keyword documents and the scores of each document in the search result set in descending order to obtain a relevance-ranked result list after the keyword search. S302, search index: the syntax tree after lexical analysis, grammatical analysis and language processing is randomly searched in the constructed index library to initially obtain a set of all documents that match the syntax tree; S303, result sorting: sorting based on the correlation between the keyword and each document, and evaluating the correlation between the index of the document in the document result set and the keyword index using a space vector model algorithm.

[0034] Correlation itself also has some limitations. When sorting correlation, other mathematical techniques such as singular value decomposition or vocabulary database can be used to compensate for the algorithm limitations.

[0035] The process of obtaining the document vector and query vector in step S301 of this embodiment is as follows: The full assist of the word Term in the document is formed into a vector as follows: Document = {term1, term2, …… ,term N}; Document Vector = {weight1, weight2, ……,weight N}; The keywords are grouped into a document and the corresponding document is represented by a vector, as follows: Query = {term1, term 2, ……, term N}; Query Vector = {weight1, weight2, ……,weight N}.

[0036] This embodiment mainly implements full-text search based on Solr to improve the efficiency of user data retrieval. The following takes the user's search for "a provident fund" as an example: ①The user enters the search term "a certain provident fund" in the client; ② Obtain a series of words by grammatical analysis and language analysis of the search words; ③ Obtain a query tree through syntax analysis; ④ Read the index into memory through index storage; ⑤ Use the query tree to search the index to obtain the document list for each term, perform intersection and difference on the document list, and obtain the result document; ⑥ Sort the search result documents by their relevance to the query; ⑦Return the query results to the user.

[0037] Example 2: This example provides a Solr-based full-text search system, which is used to implement the Solr-based full-text search method in Example 1; the system includes: The service deployment module is used to deploy Solr services in a distributed manner, read data from the data source, and use built-in text analyzers (such as IKAnalyzer) to decompose text information; An index creation module, used to convert text information into an index for fast search and receiving data retrieval requests; The data retrieval module is used by users to submit search requests. The Solr service retrieves the index based on the requested data, loads the data using reverse loading and inverted table reverse loading, and returns the results to the user. A highlighting and summary generation module is used to mark the parts matching the search keywords in the search results by adding corresponding parameters and provide corresponding summary information; The index update module is used to regularly update and maintain the Solr service index according to application requirements to keep the data up to date.

[0038] Embodiment 3: This embodiment further provides an electronic device, including: a memory and at least one processor; wherein the memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the Solr-based full-text retrieval method in any embodiment of the present invention.

[0039] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0040] The memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory and accessing the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system and at least one application required for the function; the data storage area can store data generated based on the use of the terminal. In addition, the memory can also include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0041] Embodiment 4: This embodiment also provides a computer-readable storage medium storing a plurality of instructions. The instructions are loaded by a processor, causing the processor to execute the Solr-based full-text search method according to any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided. The storage medium stores software program code that implements the functions of any of the above embodiments, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.

[0042] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0043] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RWs, DVD+RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer via a communications network.

[0044] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0045] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A full-text retrieval method based on Solr, characterized in that: The method is as follows: Service deployment: Distributed deployment of Solr service is adopted to read data from data sources and use built-in text analyzer to decompose text information; Index creation: converting text information into indexes for quick search and receiving data retrieval requests; Data retrieval: When a user submits a search request, the Solr service retrieves the index based on the requested data, loads the data using reverse loading and inverted table reverse loading, and returns the results to the user; Highlighting and summary generation: By adding corresponding parameters, the parts matching the search keywords are marked in the search results and corresponding summary information is provided; Index update: Regularly update and maintain the Solr service index based on application requirements to keep the data current.

2. The full-text retrieval method based on Solr according to claim 1, characterized in that: The Solr service includes Client, ZooKeeper cluster, Solr cluster and several disks; the Solr cluster includes several SolrServers; The client communicates with the SolrServer in the Solr cluster through HTTP or HTTPS protocol and performs distributed indexing and distributed search operations; SolrServer provides index creation and full-text search services and is the data computing and processing unit in the Solr cluster; The ZooKeeper cluster provides distributed collaboration services for each process in the Solr cluster; Each SolrServer registers its own information in the ZooKeeper cluster. The Client perceives the health status of each SolrServer based on its registration information and then decides on the distribution of index and search requests. Each SolrServer reads and writes index files from the disk based on the received index and search requests.

3. The full-text retrieval method based on Solr according to claim 1 or 2, characterized in that: The Solr service uses a reverse index to map keywords to documents, that is, by building a relationship between keywords and documents, it can quickly locate the document during keyword queries.

4. The full-text retrieval method based on Solr according to claim 3, characterized in that: To create an index, follow these steps: Index document tokenization: The original document content to be indexed is sent to the word segmentation component, which decomposes the document into vocabulary units Token through the Tokenize process; The vocabulary unit Token is processed through language analysis: the corresponding language characteristics analysis is performed according to different languages ​​and then processed to obtain the word Term; Inverted index builds index library: The index component establishes an index relationship between the term Term and the document Document to form an index library; the index component creates a dictionary for all the terms, sorts them, and merges the same terms to ensure the uniqueness of the search terms.

5. The full-text retrieval method based on Solr according to claim 4, characterized in that: The tokenize process is as follows: ① Divide the document into words; ②Remove punctuation marks from the document; ③Remove stop words; The contents of the index library include term, document frequency, and document ID and frequency in the document; term is the keyword to be retrieved after building the index library, and the corresponding document frequency indicates the number of times the corresponding term appears in the document; the document ID in the document Document is used to link with the term, so as to index the document Document according to the term, and the corresponding frequency indicates the number of times the searched term appears in the corresponding document Document.

6. The full-text retrieval method based on Solr according to claim 1, characterized in that: The data retrieval is as follows: Calculate weights: Put all searched document vectors and query vectors into an N-dimensional space, where each term is a dimension. The dimension of the vector space is the union of the term sets of the two, and calculate the relevance score. That is, sort the keyword documents and the scores of each document in the search result set in descending order, and then obtain a result list with relevance sorting after keyword search; Search index: Randomly query the syntax tree after lexical analysis, grammatical analysis and language processing in the constructed index library to initially obtain a collection of all documents that match the syntax tree; Result sorting: Sort by the correlation between keywords and each document, and evaluate the correlation between the index of the document in the document result set and the keyword index through the space vector model algorithm.

7. The full-text retrieval method based on Solr according to claim 6, characterized in that: The process of obtaining document vectors and query vectors is as follows: The full assists of the word Term in the document are formed into a vector, as follows: Document = {term1, term2, …… ,term N}; Document Vector = {weight1, weight2, ……,weight N}; The keywords are grouped into a document and the corresponding documents are represented by vectors, as follows: Query = {term1, term 2, ……, term N}; Query Vector = {weight1, weight2, ……,weight N}.

8. A full-text retrieval system based on Solr, characterized in that: The system is used to implement the Solr-based full-text retrieval method according to any one of claims 1 to 7; the system comprises: The service deployment module is used to deploy the Solr service in a distributed manner, read the data in the data source, and use the built-in text analyzer to decompose the text information; An index creation module, used to convert text information into an index for quick search and receiving data retrieval requests; The data retrieval module is used for users to submit search requests. The Solr service retrieves the index according to the requested data, loads the data by reverse loading and reverse loading of the inverted table, and returns the results to the user. A highlighting and summary generation module is used to mark the part matching the search keyword in the search results by adding corresponding parameters and provide corresponding summary information; The index update module is used to regularly update and maintain the index of the Solr service according to application requirements to keep the data up to date.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the Solr-based full-text retrieval method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the Solr-based full-text retrieval method according to any one of claims 1 to 7.

Images