A WIKI knowledge base search word analysis method and device based on Word2Vec and TFIDF

Abstract

The application provides a WIKI knowledge base search word analysis method and device based on Word2Vec and TFIDF, and belongs to the technical field of artificial intelligence. The method of the application adopts Word2Vec to construct a semantic model for the text in the knowledge base, and extracts search keywords in combination with an improved TF-IDF. First, useless records are filtered from the search text of each user, and word segmentation and stop word filtering are performed. Second, a semantic model is constructed for the text in the knowledge base by using Word2Vec, and the similarity between search words is obtained. Finally, the keywords are calculated according to the improved TF-IDF method, the keywords can be effectively obtained from the search words of the user, the search trend is understood, the lack of knowledge is found, and the management of the knowledge base is facilitated.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for analyzing WIKI knowledge base search terms based on Word2Vec and TFIDF. Background Technology

[0002] With the rapid development of online information, more and more companies are building their own WIKI knowledge bases to record company projects, platform tool usage, onboarding guides, and other content, enabling employees to quickly access the information they need. Extracting search keywords not only helps track search trends but also allows managers to promptly supplement missing content, thus facilitating knowledge base management. However, search history is often hampered by AJAX functionality, which records the input of each word. For example, searching for "onboarding guide" might result in the pinyin outputs like "ru," "ruzhi," and "ruzhizhinan." Furthermore, corporate WIKI knowledge bases record numerous abbreviations and synonyms related to work, projects, and tools. Therefore, extracting user search keywords from this complex and varied search text to build a comprehensive knowledge base management system is of great significance.

[0003] Currently, scholars from various countries have achieved good results in natural language processing using machine learning models for keyword extraction, such as the commonly used TextRank and TFIDF. However, these methods cannot effectively extract keywords in scenarios where there is a lot of noise and synonyms in the text. Therefore, there is an urgent need to provide a WIKI knowledge base search term analysis method based on Word2Vec and TFIDF to effectively extract keywords in scenarios where there is a lot of noise and synonyms in the text. Summary of the Invention

[0004] This application provides a method and apparatus for analyzing search terms in a WIKI knowledge base based on Word2Vec and TF-IDF. It uses Word2Vec to construct a semantic model of the text in the knowledge base and combines it with an improved TF-IDF to extract search keywords, thereby achieving effective extraction of keywords in scenarios where there is a lot of noise and synonyms in the text.

[0005] Firstly, this application provides a WIKI knowledge base search term analysis method based on Word2Vec and TFIDF, the WIKI knowledge base search term analysis method including:

[0006] The search text for each user is filtered, and a preprocessed search text set is obtained by filtering through word segmentation and stop words;

[0007] A semantic model for the knowledge base is constructed using Word2Vec, and the similarity between search terms is calculated using the cosine similarity algorithm to obtain a similarity matrix.

[0008] An improved TF-IDF method is used to extract keywords from search terms and obtain keyword weights;

[0009] Based on the keyword weights, the corresponding search keywords are displayed on the front end using a word cloud approach.

[0010] Optionally, the filtering of each user's search text, and the obtaining of a preprocessed search text set through word segmentation and stop word filtering, includes:

[0011] In the search data, the Chinese search text is converted into the corresponding pinyin text to obtain the first search dataset;

[0012] In the first search dataset, the search data is sorted in ascending order of search time. For each user's search data, if the search data meets the preset rules, it is considered as the same input and merged to obtain the second search dataset.

[0013] Filter the content of the same input in the second search dataset and retain the last data that is closest to the current time node to obtain the filtered third dataset;

[0014] The original Chinese data in the third dataset was restored, and a preprocessed search text set was obtained by word segmentation and stop word filtering.

[0015] Optionally, the step of constructing a knowledge base semantic model using Word2Vec and calculating the similarity between search terms using a cosine similarity algorithm to obtain a similarity matrix specifically includes:

[0016] For the content in the knowledge base, a semantic model of the knowledge base is constructed using Word2Vec to obtain the word vectors of each word. Then, the cosine similarity method is used to calculate the similarity between each search term, resulting in a similarity matrix M.

[0017] Optionally, the improved TF-IDF method is used to extract keywords from search terms and obtain keyword weights, including:

[0018] After deduplication of each search term in the preprocessed search text set, a search term set is obtained. Then, based on the improved TF-IDF method, the word weight of each search term is calculated, and the word weight of search term i is determined. The calculation formula is:

[0019]

[0020]

[0021] Where γ∈(0,1] is the penalty coefficient, ε∈(0,1] is the similarity threshold, count(T) is the total number of words in the preprocessed search text set, and N is the total number of users in the preprocessed search text set. To find the number of users whose search terms contain the word "i". Let be the similarity between word j and i.

[0022] Secondly, this application provides a WIKI knowledge base search term analysis device based on Word2Vec and TFIDF, the WIKI knowledge base search term analysis device comprising:

[0023] The preprocessing module is used to filter the search text for each user and obtain a preprocessed search text set through word segmentation and stop word filtering;

[0024] The calculation module is used to build a semantic model of the knowledge base using Word2Vec and to calculate the similarity between search terms using the cosine similarity algorithm to obtain a similarity matrix.

[0025] The extraction module is used to extract keywords from search terms and obtain keyword weights using an improved TF-IDF method.

[0026] The display module is used to display the corresponding search keywords on the front end in the form of a word cloud based on the keyword weights.

[0027] Optionally, the preprocessing module is specifically used for:

[0028] In the search data, the Chinese search text is converted into the corresponding pinyin text to obtain the first search dataset;

[0029] In the first search dataset, the search data is sorted in ascending order of search time. For each user's search data, if the search data meets the preset rules, it is considered as the same input and merged to obtain the second search dataset.

[0030] Filter the content of the same input in the second search dataset and retain the last data that is closest to the current time node to obtain the filtered third dataset;

[0031] The original Chinese data in the third dataset was restored, and a preprocessed search text set was obtained by word segmentation and stop word filtering.

[0032] Optionally, the extraction module is specifically used for:

[0033] After deduplication of each search term in the preprocessed search text set, a search term set is obtained. Then, based on the improved TF-IDF method, the word weight of each search term is calculated, and the word weight of search term i is determined. The calculation formula is:

[0034]

[0035]

[0036] Where γ∈(0,1] is the penalty coefficient, ε∈(0,1] is the similarity threshold, count(T) is the total number of words in the preprocessed search text set, and N is the total number of users in the preprocessed search text set. To find the number of users whose search terms contain the word "i". Let be the similarity between word j and i.

[0037] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0038] The memory stores computer-executed instructions;

[0039] The processor executes computer execution instructions stored in the memory to implement the method described in the first aspect above.

[0040] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect above.

[0041] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0042] This application provides a method and apparatus for analyzing search terms in a WIKI knowledge base based on Word2Vec and TF-IDF. It uses Word2Vec to construct a semantic model of the text in the knowledge base and combines this with an improved TF-IDF method to extract search keywords. First, it filters out useless records from each user's search text, performs word segmentation, and filters stop words. Second, it constructs a semantic model of the text in the knowledge base using Word2Vec to obtain the similarity between search terms. Finally, it calculates keywords based on the improved TF-IDF method. This method effectively extracts keywords from users' search terms, thereby understanding search trends and identifying knowledge gaps, facilitating knowledge base management. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] Figure 1A flowchart illustrating a WIKI knowledge base search term analysis method based on Word2Vec and TFIDF, provided as an exemplary embodiment of this application;

[0045] Figure 2 Example of search text source data provided for an exemplary embodiment of this application;

[0046] Figure 3 A schematic diagram illustrating the merging of simultaneous input search data provided in an exemplary embodiment of this application;

[0047] Figure 4 A schematic diagram of a WIKI knowledge base search term analysis device based on Word2Vec and TFIDF, provided as another exemplary embodiment of this application;

[0048] Figure 5 This is a schematic diagram of the structure of an electronic device provided for another exemplary embodiment of this application.

[0049] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0050] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0051] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0052] Figure 1 This document provides a flowchart of a WIKI knowledge base search term analysis method based on Word2Vec and TFIDF, as an exemplary embodiment of this application. Figure 1 As shown in the figure, the specific steps of the WIKI knowledge base search term analysis method based on Word2Vec and TFIDF provided by this embodiment of the invention are as follows:

[0053] Step S101: Filter the search text for each user and obtain a preprocessed search text set by word segmentation and stop word filtering.

[0054] Specifically, in the search data, the Chinese search text is converted into the corresponding pinyin text to obtain the first search dataset; then, in the first search dataset, the search data is sorted in ascending order according to search time, and the search data of each user that meets the preset rules is regarded as the same input and merged to obtain the second search dataset; then, the content of the same input in the second search dataset is filtered, and the last data closest to the current time node is retained to obtain the filtered third dataset; finally, the original Chinese data in the third dataset is restored, and the preprocessed search text set is obtained by word segmentation and stop word filtering.

[0055] refer to Figure 1 and Figure 2 As shown, for the search text source data (such as...) Figure 2 As shown), firstly, the Chinese content is converted into Pinyin text and sorted in ascending order by search time. For each user's search data, those that meet the rules are considered as input from the same time and are merged (e.g., ...). Figure 3 As shown in the figure, the pinyin records during the search are identified. Then, the merged content is filtered, and only the last record in terms of time is retained. The merging rules are as follows:

[0056]

[0057] Finally, for the filtered dataset, if the original text data was in Chinese, the pinyin needs to be restored to the original Chinese, and then word segmentation and stop word filtering preprocessing are performed to obtain the preprocessed search text set T.

[0058] Step S102: Use Word2Vec to construct a semantic model of the knowledge base, and use the cosine similarity algorithm to calculate the similarity between search terms to obtain a similarity matrix.

[0059] Specifically, for the content in the knowledge base, a semantic model of the knowledge base is constructed using Word2Vec to obtain the word vectors of each word. Then, the cosine similarity method is combined to calculate the similarity between each search term, resulting in a similarity matrix M.

[0060] Word2vec is a group of related models used to generate word vectors. These models are shallow, two-layer neural networks trained to reconstruct linguistic word text. The network represents words and needs to guess the input words in adjacent positions. Under the bag-of-words assumption in word2vec, the order of words is unimportant. After training, the word2vec model can be used to map each word to a vector, which can be used to represent the relationship between words. This vector is the hidden layer of the neural network.

[0061] Word vectors possess strong semantic properties and are a common way to represent word features. Each dimension of a word vector represents a feature with specific semantic and grammatical interpretations. Therefore, each dimension of a word vector can be called a word feature. Word vectors have various forms, one of which is distributed representation. A distributed representation is a dense, low-dimensional real-valued vector. Each dimension of a distributed representation represents a latent feature of a word, capturing useful syntactic and semantic characteristics. Thus, the word "distributed" in distributed representation reflects this characteristic of word vectors: distributing the different syntactic and semantic features of a word across its various dimensions.

[0062] Step S103: Use the improved TF-IDF method to extract keywords from the search terms and obtain keyword weights.

[0063] Specifically, after deduplication of each search term in the preprocessed search text set T, a search term set K is obtained. Then, based on the improved TF-IDF method, the word weight of each search term is calculated, and the word weight of search term i is... The calculation formula is:

[0064]

[0065]

[0066] Where γ∈(0,1] is the penalty coefficient, ε∈(0,1] is the similarity threshold, count(T) is the total number of words in the preprocessed search text set, and N is the total number of users in the preprocessed search text set. To find the number of users whose search terms contain the word "i". Let be the similarity between word j and i.

[0067] TF-IDF (term frequency–inverse document frequency) is a commonly used weighting technique for information retrieval and data mining. TF stands for Term Frequency, and IDF stands for Inverse Document Frequency.

[0068] TF-IDF is a statistical method used to assess the importance of a word to a document within a set of documents or a corpus. A word's importance increases proportionally to its frequency in a document, but decreases inversely proportionally to its frequency in the corpus. Various weighted forms of TF-IDF are commonly used by search engines as a measure or ranking of the relevance between documents and user queries.

[0069] If a word or phrase has a high frequency (TF) in one article but rarely appears in other articles, it is considered to have good class distinguishing ability and is suitable for classification. TF-IDF is actually TF*IDF, where TF is Term Frequency and IDF is Inverse Document Frequency. TF represents the frequency of a term in document d. The main idea of ​​IDF is: the fewer documents containing term t (i.e., the smaller n is), the larger the IDF, indicating that term t has good class distinguishing ability. If the number of documents containing term t in a certain class C is m, and the total number of documents containing t in other classes is k, then obviously the total number of documents containing t is n = m + k. When m is large, n is also large, and the IDF value obtained according to the IDF formula will be small, indicating that term t has weak class distinguishing ability. However, in reality, if a term appears frequently in documents of a certain class, it means that the term can well represent the characteristics of the text of that class. Such terms should be given higher weights and selected as feature words of that class of text to distinguish it from documents of other classes. This is where IDF falls short. In a given document, term frequency (TF) refers to how often a given word appears in that document. This number is normalized from the term count to prevent it from being biased towards long documents.

[0070] Step S104: Based on the keyword weights, display the corresponding search keywords on the front end using a word cloud method.

[0071] Finally, based on the keyword weight W, a word cloud is displayed to the front end, thus providing management support for administrators.

[0072] This application provides a method and apparatus for analyzing search terms in a WIKI knowledge base based on Word2Vec and TF-IDF. It uses Word2Vec to construct a semantic model of the text in the knowledge base and combines this with an improved TF-IDF method to extract search keywords. First, it filters out useless records from each user's search text, performs word segmentation, and filters stop words. Second, it constructs a semantic model of the text in the knowledge base using Word2Vec to obtain the similarity between search terms. Finally, it calculates keywords based on the improved TF-IDF method. This method effectively extracts keywords from users' search terms, thereby understanding search trends and identifying knowledge gaps, facilitating knowledge base management.

[0073] Figure 4 This is a schematic diagram illustrating the structure of a WIKI knowledge base search term analysis device based on Word2Vec and TFIDF, provided as an exemplary embodiment of this application. The WIKI knowledge base search term analysis device based on Word2Vec and TFIDF provided in this embodiment can execute the processing flow provided in an embodiment of a WIKI knowledge base search term analysis method based on Word2Vec and TFIDF. For example... Figure 4 As shown, the WIKI knowledge base search term analysis device 20 based on Word2Vec and TFIDF provided in this application includes:

[0074] The preprocessing module 201 is used to filter the search text for each user and obtain a preprocessed search text set through word segmentation and stop word filtering.

[0075] The calculation module 202 is used to construct a knowledge base semantic model using Word2Vec and to calculate the similarity between search terms using the cosine similarity algorithm to obtain a similarity matrix;

[0076] Extraction module 203 is used to extract keywords from search terms using an improved TF-IDF method to obtain keyword weights;

[0077] The display module 204 is used to display the corresponding search keywords on the front end in the form of a word cloud based on the keyword weights.

[0078] Furthermore, the preprocessing module 201 is specifically used for:

[0079] In the search data, the Chinese search text is converted into the corresponding pinyin text to obtain the first search dataset;

[0080] In the first search dataset, the search data is sorted in ascending order of search time. For each user's search data, if the search data meets the preset rules, it is considered as the same input and merged to obtain the second search dataset.

[0081] Filter the content of the same input in the second search dataset and retain the last data that is closest to the current time node to obtain the filtered third dataset;

[0082] The original Chinese data in the third dataset was restored, and a preprocessed search text set was obtained by word segmentation and stop word filtering.

[0083] Furthermore, the extraction module 203 is specifically used for:

[0084] After deduplication of each search term in the preprocessed search text set, a search term set is obtained. Then, based on the improved TF-IDF method, the word weight of each search term is calculated, and the word weight of search term i is determined. The calculation formula is:

[0085]

[0086]

[0087] Where γ∈(0,1] is the penalty coefficient, ε∈(0,1] is the similarity threshold, count(T) is the total number of words in the preprocessed search text set, and N is the total number of users in the preprocessed search text set. To find the number of users whose search terms contain the word "i". Let be the similarity between word j and i.

[0088] The apparatus provided in this application embodiment can be specifically used to perform the above-described... Figure 1 The specific functions and technical effects of the solutions provided in the corresponding method embodiments will not be elaborated here.

[0089] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an example embodiment of this application. Figure 5 As shown, the electronic device 30 includes a processor 301 and a memory 302 communicatively connected to the processor 301, the memory 302 storing computer execution instructions.

[0090] The processor executes computer execution instructions stored in the memory to implement the solution provided in any of the above method embodiments; the specific functions and technical effects achieved are not elaborated here. The electronic device can be the server mentioned above.

[0091] This application also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, the computer-executable instructions are used to implement the solution provided in any of the above method embodiments. The specific functions and technical effects to be achieved are not described here.

[0092] This application also provides a computer program product, which includes a computer program stored in a readable storage medium. At least one processor of the electronic device can read the computer program from the readable storage medium. The at least one processor executes the computer program to cause the electronic device to perform the solution provided in any of the above method embodiments. The specific functions and technical effects that can be achieved are not described here.

[0093] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0094] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for analyzing search terms in a WIKI knowledge base based on Word2Vec and TFIDF, characterized in that, The WIKI knowledge base search term analysis method includes: The search text for each user is filtered, and a preprocessed search text set is obtained by filtering through word segmentation and stop words; A semantic model for the knowledge base is constructed using Word2Vec, and the similarity between search terms is calculated using the cosine similarity algorithm to obtain a similarity matrix. An improved TF-IDF method is used to extract keywords from search terms and obtain keyword weights. The TF-IDF method is used to evaluate the importance of a word to a document in a document set or a corpus. Based on the keyword weights, the corresponding search keywords are displayed on the front end using a word cloud approach; The process involves filtering the search text for each user, and obtaining a preprocessed search text set through word segmentation and stop word filtering. In the search data, the Chinese search text is converted into the corresponding pinyin text to obtain the first search dataset; In the first search dataset, search data is sorted in ascending order of search time. Search data from each user that meets a preset rule is considered as the same input and merged to obtain the second search dataset. The preset rule is: ,in, , These are different thresholds; Filter the content of the same input in the second search dataset and retain the last data that is closest to the current time node to obtain the filtered third dataset; The original Chinese data in the third dataset was restored, and a preprocessed search text set was obtained by word segmentation and stop word filtering; The improved TF-IDF method is used to extract keywords from search terms and obtain keyword weights, including: After deduplication of each search term in the preprocessed search text set, a search term set is obtained. Then, based on the improved TF-IDF method, the word weight of each search term is calculated, and the word weight of search term i is determined. The calculation formula is: Where γ∈(0,1] is the penalty coefficient, ε∈(0,1] is the similarity threshold, count(T) is the total number of words in the preprocessed search text set, and N is the total number of users in the preprocessed search text set. To find the number of users whose search terms contain the word "i". Let be the similarity between word j and i.

2. The WIKI knowledge base search term analysis method according to claim 1, characterized in that, The process involves constructing a knowledge base semantic model using Word2Vec and calculating the similarity between search terms using a cosine similarity algorithm to obtain a similarity matrix. Specifically, this includes: For the content in the knowledge base, a semantic model of the knowledge base is constructed using Word2Vec to obtain the word vectors of each word. Then, the cosine similarity method is used to calculate the similarity between each search term, resulting in a similarity matrix M.

3. A WIKI knowledge base search term analysis device based on Word2Vec and TFIDF, characterized in that, The WIKI knowledge base search term analysis device includes: The preprocessing module is used to filter the search text for each user and obtain a preprocessed search text set through word segmentation and stop word filtering; The calculation module is used to build a semantic model of the knowledge base using Word2Vec and to calculate the similarity between search terms using the cosine similarity algorithm to obtain a similarity matrix. The extraction module is used to extract keywords from search terms using an improved TF-IDF method to obtain keyword weights. The TF-IDF method is used to evaluate the importance of a word to a document in a document set or a corpus. The display module is used to display the corresponding search keywords on the front end in the form of a word cloud based on the keyword weights; Specifically, the preprocessing module is used for: In the search data, the Chinese search text is converted into the corresponding pinyin text to obtain the first search dataset; In the first search dataset, search data is sorted in ascending order of search time. Search data from each user that meets a preset rule is considered as the same input and merged to obtain the second search dataset. The preset rule is: ,in, , These are different thresholds; Filter the content of the same input in the second search dataset and retain the last data that is closest to the current time node to obtain the filtered third dataset; The original Chinese data in the third dataset was restored, and a preprocessed search text set was obtained by word segmentation and stop word filtering; Specifically, the extraction module is used for: After deduplication of each search term in the preprocessed search text set, a search term set is obtained. Then, based on the improved TF-IDF method, the word weight of each search term is calculated, and the word weight of search term i is determined. The calculation formula is: Where γ∈(0,1] is the penalty coefficient, ε∈(0,1] is the similarity threshold, count(T) is the total number of words in the preprocessed search text set, and N is the total number of users in the preprocessed search text set. To find the number of users whose search terms contain the word "i". Let be the similarity between word j and i.

4. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-2.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-2.

6. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-2.

Images