A library platform semantic retrieval method based on fusion topic model

Abstract

The application relates to the technical field of information retrieval, in particular to a library platform semantic retrieval method fusing a topic model, which comprises the following steps: data preprocessing is conducted on all the retrieved documents to obtain qualified retrieved documents; a user's retrieval words are matched with the qualified retrieved documents by using a Boolean model, the documents are classified according to the matching results, and different weight values are given to the documents; the qualified retrieved documents are calculated by using a topic model to obtain the topic distribution of each document and the distribution of topic words; the frequency distribution of the topic words is converted into the word topic distribution; the similarity score between the word topic distribution of the retrieval words and the retrieved documents is calculated; the connection score of the retrieval words and each retrieved document is calculated by using the document weight and the similarity score, and the query results are returned to the user after the connection score is sorted. The application can greatly improve the semantic retrieval efficiency of the library platform and effectively meet the demand of the user for knowledge retrieval service.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, specifically a semantic retrieval method for a library platform that integrates a topic model. Background Technology

[0002] With the formation of the digital environment and the rapid development of information technology, library information resources are constantly increasing. Faced with various information resources in the digital age, users expect to be able to accurately search, conveniently access, and remotely utilize massive amounts of information resources. In order to better meet users' needs, the library community is actively working to improve and perfect its information resource retrieval platform.

[0003] Current library platform information resource retrieval methods primarily employ keyword retrieval technology, which obtains search results by strictly matching query terms in documents and counting the frequency of those terms. However, when using traditional keyword retrieval techniques on library platforms, the fixed keywords used for matching ignore their extended meanings, leading to diverse search results for the same search intent due to different users using different keywords. Furthermore, the polysemy and polyphic nature of words in natural language further impact the quality of user searches.

[0004] Keyword matching technology has limitations in revealing the semantics and pragmatics of information, failing to meet users' increasingly sophisticated retrieval needs. This has created conditions for the development of semantic retrieval. Semantic retrieval aims to ensure that when users input their desired information as search terms, the search results contain related or more information. With the emergence of the Semantic Web and the development of information technology, semantic-based retrieval methods have been gradually applied to various retrieval systems, and the application of semantic retrieval in libraries has become a focus of research for scholars both domestically and internationally.

[0005] In recent years, semantic retrieval has primarily relied on topic models and deep learning methods. Topic models are statistical models used to discover abstract topics within a series of documents, enabling semantic retrieval from the perspective of topic-related words. Deep learning models extract deep semantic features from text, thereby achieving higher-quality retrieval of scientific datasets. While deep learning can improve retrieval performance, it is computationally intensive, requires high-end hardware, has complex model design, and suffers from poor interpretability. Summary of the Invention

[0006] The purpose of this invention is to provide a semantic retrieval method for a library platform that integrates topic models, in order to solve the problems mentioned in the background art.

[0007] The technical solution of this invention is: a semantic retrieval method for a library platform that integrates topic models, comprising the following steps:

[0008] S1. Perform data preprocessing on all retrieved documents, including word segmentation, cleaning, and standardization, to obtain qualified retrieved documents.

[0009] S2. Use the Boolean model to match the user's search terms with the qualified search documents obtained in S1. Based on the matching results, all documents are divided into two categories: relevant and irrelevant, and assigned different weight values.

[0010] S3. Calculate the topic distribution and topic word distribution of each document by using the topic model to process the retrieved documents in S1.

[0011] S4. Convert the frequency distribution of the topic words obtained in S3 into a topic word distribution;

[0012] S5. Calculate the similarity score between the word topic distribution of the search terms and the retrieved document;

[0013] S6. Calculate the connection score between the search terms and each retrieved document using the document weights in S2 and the similarity scores in S5, sort the results according to the connection scores, and return the query results to the user.

[0014] Preferably, in step S1, data preprocessing includes: performing word segmentation on the document set D, and processing document d... n After word segmentation, we get d. n = (w1, w2, ...); Next, the segmented document is cleaned, removing useless tags, special symbols, and stop words; Finally, the text content is standardized, converting the same word in some parts of the text with different forms into the same form.

[0015] Preferably, in step S2, in the Boolean-based matching, the retrieved document D can be represented as: D = (d1, d2, d3, ..., d...). n The user's retrieval Q can be represented as: Q = (q1, q2, q3, ...), where d n Let q1 represent the nth retrieved document and q2 represent the first search term. The Boolean model has three main logical operators: AND, OR, and NOT. Assuming a user query contains two search terms q1 and q2, considering the conditions for each term under different logical operators, there are four conditions: (1) q1 and q2; (2) q1 or q2; (3) q1 not q2; (4) q2 not q1. The formula for assigning weights to different documents can be expressed as:

[0016]

[0017] in, This represents the weight of document dn; therefore, the retrieved documents have different weight coefficients.

[0018] Preferably, in step S3, the generation process of the topic model consists of four steps:

[0019] S31. For topic z, obtain a word multinomial distribution vector φ on topic z according to the Dirichlet distribution Dir(β);

[0020] S32. Obtain the number of words N in the text based on the Poisson distribution P;

[0021] S33. Obtain a topic distribution probability vector θ for the text based on the Dirichlet distribution Dir(α);

[0022] S34. For each word W in the M words of this text m First, a topic z is randomly selected from the multinomial distribution of θ (Multinomial(θ)); then, a word is selected as W from the multinomial conditional probability distribution of topic z (Multinomial(φ)). m ;

[0023] Using the Gibbs sampling method to calculate the above generation process, the joint probability distribution function of the topic model is:

[0024]

[0025] The Gibbs sampling algorithm avoids estimating the actual parameters by integrating the data, instead sampling the topic of each word. Once the topic of each word is determined, the parameters can be calculated by statistically analyzing the frequencies. Therefore, the parameter estimation problem becomes calculating the conditional probability of the topic sequence under the word sequence, as shown in the following formula:

[0026]

[0027] Among them, z nm This represents the topic variable corresponding to the m-th word in document n; -nm indicates that the m-th item is not included. β represents the number of times word v appears in topic k; v It is the Dirichlet prior of the word v; α represents the number of times document n appears in topic k; k It is the Dirichlet prior of topic k; the basic idea of ​​Gibbs sampling is to fix a certain dimension z. nm Then through other dimensions z -nmThe value of z is used to sample the value of that dimension. The Markov chain can converge to a stable probability distribution through the transition probability matrix. When the Markov chain eliminates the influence of the initial parameters during the iteration phase and the algorithm converges, the distribution θ of the document on the topic and the distribution φ of the topic on the word are calculated based on the current distribution of z. The formula is:

[0028]

[0029]

[0030] Where, φ k，v θ represents the probability of word v in topic k. n，k This represents the probability of topic k in document n.

[0031] Preferably, in step S4, the word-topic frequency distribution is represented as follows:

[0032]

[0033] Preferably, in step S5, when the user inputs the search term v, the distribution is obtained. and To calculate the relationship between search terms and the retrieved documents, similarity is calculated using the following formula:

[0034]

[0035] Preferably, in step S6, the returned search results are represented as follows:

[0036]

[0037] according to The values ​​are sorted by size, and the results of the retrieved documents are returned to the user. The larger the value, the earlier the returned result appears; The smaller the value, the later the result is returned.

[0038] This invention provides an improved semantic retrieval method for library platforms that integrates a topic model. Compared with existing technologies, it has the following improvements and advantages:

[0039] This invention utilizes a Boolean model to distinguish the importance of documents and assign importance weights to each document, alleviating the problem of Boolean retrieval that only considers document matching based on keywords. It organically integrates document importance weights with topic-based document and search term similarity, taking into account both keyword retrieval and semantic retrieval, which can greatly improve the semantic retrieval efficiency of library platforms and effectively meet users' needs for retrieving knowledge services. Attached Figure Description

[0040] The present invention will be further explained below with reference to the accompanying drawings and embodiments:

[0041] Figure 1 This is a flowchart of the present invention;

[0042] Figure 2 The graph showing the variation of the average coherence index value of this invention under different numbers of topics;

[0043] Figure 3 This is a graph showing the change of the coherence index value of the present invention under different iterations when the number of topics is 10.

[0044] Figure 4 This is a graph showing the change in precision values ​​for the search results of the present invention when the search term is "art".

[0045] Figure 5 This is a graph showing the change in recall values ​​for the search results of the present invention when the search term is "art".

[0046] Figure 6 This is a graph showing the change in the F-value of the search results for the search term "art" according to the present invention. Detailed Implementation

[0047] The present invention will now be described in detail, and the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] This invention provides an improved semantic retrieval method for a library platform that integrates a topic model. The technical solution of this invention is as follows:

[0049] like Figure 1 As shown, a semantic retrieval method for a library platform that integrates topic models includes the following steps:

[0050] S1. Data preprocessing operations for the N sets of retrieved documents D: First, perform word segmentation on the document set D. Word segmentation divides the documents into a series of words or phrases. Jieba is selected for word segmentation. n After word segmentation, we get d. n = (w1, w2, ...); Then, the segmented document is cleaned, removing useless tags, special characters, and stop words; finally, the text content is standardized, converting the same word in some parts of the text with different forms into the same form;

[0051] S2. Use a Boolean model to match the search terms with the preprocessed document set D. In Boolean model-based matching, the retrieved document D can be represented as: D = (d1, d2, d3, ..., d...). n The user's retrieval Q can be represented as: Q = (q1, q2, q3, ...), where d n Let q1 represent the nth retrieved document and q2 represent the first search term. The Boolean model has three main logical operators: AND, OR, and NOT. Assume a user query contains two search terms, q1 and q2, and consider the conditions for each term under different logical operators. There are four conditions: (1) q1 and q2; (2) q1 or q2; (3) q1 not q2; (4) q2 not q1. The formula for assigning weights to different documents can be expressed as:

[0052]

[0053] in, This represents the weight of document dn. Therefore, the retrieved documents have different weight coefficients;

[0054] S3. Calculate the topic model for the preprocessed document set D, which contains N documents. The topic model generation process consists of four steps: First, for topic z, obtain a word multinomial distribution vector φ based on the Dirichlet distribution Dir(β); second, obtain the number of words N in the text based on the Poisson distribution P; third, obtain a topic distribution probability vector θ for the text based on the Dirichlet distribution Dir(α); fourth, for each word W in the M words of the text... m First, a topic z is randomly selected from the multinomial distribution of θ (Multinomial(θ)); then, a word is selected as W from the multinomial conditional probability distribution of topic z (Multinomial(φ)). m Using the Gibbs sampling method to calculate the above generation process, the joint probability distribution function of the topic model is:

[0055]

[0056] The Gibbs sampling algorithm avoids the actual parameters to be estimated by integrating, instead sampling the topic of each word. Once the topic of each word is determined, the parameters can be calculated by counting frequencies. Therefore, the parameter estimation problem becomes calculating the conditional probability of the topic sequence under the word sequence, as shown in the following formula:

[0057]

[0058] Among them, Znm This represents the topic variable corresponding to the m-th word in document n; -nm indicates that the m-th item is not included. β represents the number of times word v appears in topic k; v It is the Dirichlet prior of the word v; α represents the number of times document n appears in topic k; k It is the Dirichlet prior of topic k. The basic idea of ​​Gibbs sampling is to fix a certain dimension Z. nm Then through other dimensions z -nm The value of z is used to sample the value of that dimension. The Markov chain can converge to a stable probability distribution through the transition probability matrix. When the Markov chain eliminates the influence of initial parameters during the iteration phase and reaches algorithm convergence, the distribution θ of documents on topics and the distribution φ of topics on words are calculated based on the current distribution of z. The formula is:

[0059]

[0060]

[0061] in, θ represents the probability of word v in topic k. n，k This represents the probability of topic k in document n;

[0062] S4. Through the calculation in S3, the distributions of θ and φ can be obtained. The intermediate results in the calculation process are as follows: and The word-topic frequency distribution is represented as follows:

[0063]

[0064] S5. Word-topic frequency distribution calculated based on S4 The document-topic distribution θ obtained from S3 n，k Calculate topic-based similarity between two distributions. When a user inputs a search term v, the distribution is obtained. and To calculate the relationship between search terms and the retrieved documents, similarity is calculated using the following formula:

[0065]

[0066] It should be further noted that, to verify the retrieval performance of the proposed semantic retrieval method for library platforms integrating topic models, experiments were conducted on the Chinese corpus provided by Fudan University. To ensure the balance of the experiment, 9800 documents were selected from the Chinese corpus as the experimental dataset, encompassing 20 categories. Each document includes information such as document number, original source, original journal issue number, title, and body text. All documents underwent text preprocessing, including word segmentation, cleaning, and standardization, resulting in experimental datasets with different vocabulary. During the cleaning process, useless tags, special symbols, and stop words were removed.

[0067] To evaluate the performance of the proposed algorithm, this invention compares the retrieval performance under different importance coefficient values ​​(a). The retrieval performance evaluation metrics are precision, recall, and F-score, and the relevant calculations are performed using the following formula.

[0068]

[0069]

[0070]

[0071] In the three formulas, Num total Num represents the total number of documents retrieved. actual To determine the number of documents related to the retrieved document for the search terms obtained by the proposed method, Num correct To ensure the correct number of documents retrieved.

[0072] Experimental results are as follows Figures 4 to 6 As shown in the figure, the horizontal axis represents the number of search results displayed, and the vertical axis represents precision, recall, and F-score, respectively. Different curves represent evaluation index values ​​under different importance coefficients α. It can be seen that this invention achieves optimal results by utilizing keyword matching and semantic retrieval, meaning that this invention effectively solves the retrieval problem of library platforms.

[0073] S6. Based on the weight scores of different documents obtained in S2 and the similarity results obtained in S5, the relationship between different documents and search terms is obtained, the searched documents are sorted according to relevance, and finally the search results are displayed to the user.

[0074] The returned search results are represented as follows:

[0075]

[0076] according to The values ​​are sorted by size, and the results of the retrieved documents are returned to the user. The larger the value, the earlier the returned result appears; The smaller the value, the later the result is returned.

[0077] This invention designs experiments to compare retrieval performance under different importance coefficients 'a'. Figure 2 The graph shows the changes in the coherence index value under different numbers of topics. Figure 3 The graph shows the changes in the coherence index value under different iteration numbers. According to... Figure 2 and Figure 3 The results determine the setting of the number of topics and the number of iterations in the topic model during the experiment. Figure 4 , 5 Figure 6 shows the changes in precision, recall, and F-value for the search results when the search term is "art". Because the number of search results displayed varies depending on the settings of different platforms, the horizontal axis represents different numbers of search results displayed, from 1 to 20. Figures 4 to 6 In the diagram, different indicator values ​​represent different importance coefficients 'a'. It can be seen that, considering both keyword matching and semantic retrieval, the retrieval performance using irrelevant documents is lower than that using relevant documents; therefore, considering both keyword matching and semantic retrieval simultaneously achieves the best semantic retrieval results.

[0078] The foregoing description enables those skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A semantic retrieval method for a library platform that integrates topic models, characterized in that: Includes the following steps: S1. Perform data preprocessing on all retrieved documents, including word segmentation, cleaning, and standardization, to obtain qualified retrieved documents. S2. Use the Boolean model to match the user's search terms with the qualified search documents obtained in S1. Based on the matching results, all documents are divided into two categories: relevant and irrelevant, and assigned different weight values. S3. Calculate the topic distribution and topic word distribution for each document by using a topic model on the retrieved documents processed in S1. The topic model generation process consists of four steps: S31. For topic z, obtain a word multinomial distribution vector φ on topic z according to the Dirichlet distribution Dir(β); S32. Obtain the number of words N in the text based on the Poisson distribution P; S33. Obtain a topic distribution probability vector θ of the text based on the Dirichlet distribution Dir(α); S34. For each word in the M words of the text First, a topic z is randomly selected from the multinomial distribution (Multinomial(θ)) of topic θ; then, a word is selected from the multinomial conditional probability distribution (Multinomial(φ)) of topic z. ; Using the Gibbs sampling method to calculate the above generation process, the joint probability distribution function of the topic model is: ； The Gibbs sampling algorithm avoids estimating the actual parameters by integrating the data, instead sampling the topic of each word. Once the topic of each word is determined, the parameters can be calculated by statistically analyzing the frequencies. Therefore, the parameter estimation problem becomes calculating the conditional probability of the topic sequence under the word sequence, as shown in the following formula: ； in, This represents the topic variable corresponding to the m-th word in document n; -nm indicates that the m-th item is not included. This indicates the number of times word v appears in topic k; It is the Dirichlet prior of the word v; This indicates the number of times topic k appears in document n; It is the Dirichlet prior of topic k; the basic idea of ​​Gibbs sampling is to fix a certain dimension. Then through other dimensions The value of z is used to sample the value of that dimension. The Markov chain can converge to a stable probability distribution through the transition probability matrix. When the Markov chain eliminates the influence of the initial parameters during the iteration phase and the algorithm reaches convergence, the distribution θ of the document on the topic and the distribution of the topic on the word are calculated based on the current distribution of z. The formula is: ； ； in, This represents the probability of word v in topic k. This represents the probability of topic k in document n; S4. Convert the frequency distribution of the topic words obtained in S3 into a topic word distribution; S5. Calculate the similarity score between the word topic distribution of the search terms and the retrieved document; S6. Calculate the connection score between the search terms and each retrieved document using the document weights in S2 and the similarity scores in S5, sort the results according to the connection scores, and return the query results to the user.

2. The semantic retrieval method for a library platform integrating topic models according to claim 1, characterized in that: In step S1, data preprocessing specifically includes: performing word segmentation on the document set D, and processing the documents... After word segmentation, we get Next, the segmented documents are cleaned, removing useless tags, special symbols, and stop words. Finally, the text content is standardized, converting different forms of the same word in some texts into the same form.

3. The semantic retrieval method for a library platform integrating topic models according to claim 1, characterized in that: In S2, during Boolean-based matching, the retrieved document It can be represented as: User search It can be represented as: ,in This represents the nth document being retrieved. This represents the first search term; the Boolean model has three main logical operators: AND, OR, and NOT. Assume the user's query contains two search terms. and Consider the conditions of words under different logical operators. There are four conditions: (1) (2) (3) (4) The formula for assigning weights to different documents can be expressed as: ； in, Document The weights are determined by the number of documents being retrieved; therefore, the retrieved documents have different weight coefficients.

4. The semantic retrieval method for a library platform integrating topic models according to claim 3, characterized in that: In S4, the word-topic frequency distribution is represented as follows: 。 5. The semantic retrieval method for a library platform integrating topic models according to claim 4, characterized in that: In step S5, when the user inputs the search term v, the distribution is obtained. and To calculate the relationship between search terms and the retrieved documents, similarity is calculated using the following formula: 。 6. The semantic retrieval method for a library platform integrating topic models according to claim 5, characterized in that: In step S6, the returned search results are represented as follows: ； according to The values ​​are sorted by size, and the results of the retrieved documents are returned to the user. The larger the value, the earlier the returned result appears; The smaller the value, the later the result is returned.

Images