Search engine construction method based on knowledge graph in certain field

Abstract

The invention relates to the field of search engine construction, in particular to a search engine construction method based on a knowledge graph. The invention discloses a search engine construction method based on a knowledge graph of a certain field, a search engine for deep interaction between the field and the knowledge graph is constructed through the constructed knowledge graph of the field and a question and answer model of the field, and the search engine provided by the method comprises three modules, namely a crawler module, an index module and an interaction module. The invention discloses a PageRank algorithm DATPR (Dynamic Authority-Topic PageRank) which comprehensively considers the timeliness, the authority and the theme correlation, and the three parameters are integrated into the sorting of the webpage, so that the sorting effect of the webpage is improved.

Description

Technical Field

[0001] This invention relates to the field of designing search engine construction methods, and more particularly to a method for designing a knowledge graph-based search engine construction method. Background Technology

[0002] Currently, traditional keyword-based search engines often struggle to handle specialized queries and accurately understand users' search intent to provide highly relevant information. In this context, applying Knowledge Graph (KG) to search engine optimization across the entire kiwifruit industry chain is particularly important and urgent.

[0003] While directly segmenting search queries into words and extracting keywords can yield results, it's prone to providing irrelevant information. Applying knowledge graphs to search engines helps them better understand user semantics and provides richer background information for search results. Meanwhile, the limitations of the PageRank algorithm significantly impact the final result ranking. Therefore, researching search engines that deeply interact with knowledge graphs and developing improved PageRank algorithms to provide users with more accurate search results is crucial. Summary of the Invention

[0004] The problem this invention aims to solve is to construct a search engine that deeply interacts with the domain knowledge graph and the question-answering model of the domain, and to study an improved PageRank algorithm to improve the ranking effect of web pages.

[0005] To achieve the above-mentioned objectives, the present invention employs the following technical method: a method for constructing a search engine based on a knowledge graph in a certain domain, characterized by comprising the following steps: S1: Web crawler module construction. Based on requirements, collect authoritative links in the field and construct a domain terminology set. Build the web crawler module and crawl relevant web pages in the field based on authoritative links, using the domain terminology set to limit the scope of the web pages.

[0006] S2: Index module construction. Based on requirements, build an index module to process the data collected by the crawler module and transform it into structured data.

[0007] S3: Interactive module construction. Based on requirements, construct interactive modules to provide users with interfaces and display results.

[0008] Furthermore, step S1 includes the following steps: S101. Construct a thesaurus based on the domain knowledge graph; S102. Collect authoritative links in the field and construct a domain terminology based on the domain's knowledge graph; S103. Web page crawling using the Clawler4j framework. S104. Analyze the content of the webpage, extract the data required by the system from the page content, and transform it into a structured data structure including {page number, page title, page link, parent link, last modification time, keywords} for storage. Keyword extraction uses the Aho-Corasick algorithm (AC algorithm) combined with TF-IDF analysis to confirm the keywords and their importance in each document.

[0009] Furthermore, step S2 includes the following sub-steps: S201. Determine the data type and the corresponding mapping structure; S202. In ElasticSearch, these mappings are actually created to index the web page data crawled by the crawler module, laying the foundation based on the provided keywords.

[0010] Furthermore, step S3 includes the following sub-steps: S301. Obtain the extended word library ext.dic through the existing knowledge graph of this domain, construct the stop word library stopword.dic, and configure the IK_Analyer word segmenter; S302. Deploy the existing question-answering model to the Flask service; S303. Configuration module, which enables ordinary search to obtain keywords through word segmentation as described in S301, and precise search to obtain keywords through the question-answering model deployed in S302; S304. Configure the webpage ranking algorithm of the module to rank the acquired webpages according to timeliness, authority and topic relevance.

[0011] Furthermore, a PageRank algorithm called DATPR (Dynamic Authority-Topic PageRank) that comprehensively considers timeliness, authority, and topic relevance is proposed, and web pages are ranked based on this algorithm.

[0012] The beneficial effects of this invention are: it proposes a method for constructing a search engine based on a domain-specific knowledge graph. Using the domain-specific knowledge graph as the data foundation and a domain-specific question-answering model as the core of accurate search, it proposes a PageRank algorithm, DATPR (Dynamic Authority-TopicPageRank), that comprehensively considers timeliness, authority, and topic relevance. This improves the ranking effect of the search engine, realizes the construction of a domain-specific search engine, and provides information support for practitioners in this field. Compared to the traditional PageRank algorithm, the DATPR algorithm proposed in this invention can accurately rank web pages and improve the ranking effect of search results. Attached Figure Description

[0013] Figure 1 Overall architecture diagram of the search engine Figure 2 A diagram of search engine crawling strategies Figure 3 Flowchart for web crawling Figure 4 Example of a Trie tree Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention and to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to embodiments and accompanying drawings. The illustrative embodiments and descriptions used in this invention are for illustrative purposes only and do not constitute a further limitation of the invention.

[0015] like Figure 1 As shown, the search engine system architecture design of the present invention is as follows: Figure 1 As shown, it is mainly divided into three modules: crawler module, index module, and interaction module.

[0016] This invention provides a method for constructing a search engine based on a knowledge graph in a specific domain. Taking the entire kiwifruit industry chain as an example, the method includes the following steps: S1: Crawler module construction: S101. Construct a thesaurus for the kiwifruit domain, and obtain relevant entities from the knowledge graph of the entire kiwifruit industry chain to form the thesaurus.

[0017] S102. Using mainstream search engines such as Baidu and Sogou as important seed link sources, a series of authoritative websites covering various aspects such as kiwi fruit cultivation, research, nutritional value and market dynamics were selected. S103. Use the Crawler4j framework to crawl web pages. Define a subclass "WebCrawler" that inherits from Crawler4j, obtain seed links from the "seed.txt" file, set the crawler parallelism to 8, the crawl depth limit to 5, and the crawl interval to 0.5s, simulating Chrome browser behavior, and then perform the crawl. After crawling is complete, set access restrictions in the shouldVisit() function to only visit web pages containing keywords from the keyword database, and process the crawled web pages in the visit() function. Figure 2-3 As shown.

[0018] S104. Analyze the content of the webpages, extract the data required by the system from the page content, and transform it into a structured data structure including {page number, page title, page link, parent link, last modification time, keywords} for storage. Keyword extraction uses the Aho-Corasick algorithm (AC algorithm) combined with TF-IDF analysis to confirm the keywords and their importance in each document.

[0019] like Figure 4 As shown, a Trie tree is presented using {kiwifruit leaf spot, hardy kiwifruit, kiwifruit canker} as an example. After constructing the Trie tree, the number of keywords corresponding to each page is obtained through the AC algorithm, and then the TF-IDF algorithm is applied to calculate the five most closely related keywords for each page.

[0020] S2: Index module construction: S201. Determine the data types and corresponding mapping structures. Stored data fields include page ID, page title, page link, parent link, last modification time, and keywords. The page ID, as a unique identifier for a document, is typically used to track and retrieve specific page records. In Elasticsearch, the page ID is defined as an integer type to optimize storage and query efficiency. The page title provides the page's title information, which is crucial for users to understand the page content. Since the title may contain keywords, this field is defined as a text type and equipped with ik_smart (intelligent word segmentation) to support full-text search. The page link and parent link store the URLs of the current page and its parent pages. These two fields are very useful for building a website's structure diagram and tracking the link relationships between pages. Since URLs have a fixed structure and are usually queried as a whole, they are defined as keyword types for exact matching. The last modification time records the timestamp of the page's last update, which is crucial for determining the page's age and the timeliness of its content. This field is set to date type, allowing for time-based range queries. Keywords contain a list of keywords and their importance on the page. Since each webpage may contain multiple keywords and their relative importance, this field is designed as a nested type. Each nested object contains two properties: key (a keyword type, used for exact keyword matching) and weight (a float type, recording the keyword's TF-IDF value). Based on the above data model, the mapping for the Elasticsearch index is further defined. The mapping is a set of rules in Elasticsearch used to define how documents and their contained fields are stored and indexed.

[0021] S202. In ElasticSearch, these mappings are actually created to index the web page data crawled by the crawler module, laying the foundation based on the provided keywords.

[0022] S3: Interactive Module Construction: S301. Obtain the extended word library ext.dic through the existing knowledge graph of this domain, construct the stop word library stopword.dic, and configure the IK_Analyer word segmenter; S302. Deploy the existing question-answering model to the Flask service; S303. Configuration module, which enables ordinary search to obtain keywords through word segmentation as described in S301, and precise search to obtain keywords through the question-answering model deployed in S302; S304. Configure the webpage ranking algorithm of the module to rank the acquired webpages according to timeliness, authority and topic relevance.

[0023] To address the shortcomings of the PageRank algorithm, this invention proposes a new PageRank algorithm, DATPR (Dynamic Authority-Topic PageRank), which comprehensively considers timeliness, authority, and topic relevance, incorporating these three parameters into the ranking of web pages.

[0024] (1) Authority. The selected seed links are all authoritative websites in the kiwi fruit field, and the data is more reliable, so the authority value is high. The authority value of the websites that can be crawled directly from the seed links should also be relatively high. Therefore, this invention defines the authority value of the webpage as the authority value SA (Static Authority) passed down from the seed link, and the formula is as follows:

[0025] In the formula, P is the webpage to be evaluated, S is the set of seed links, and d(i,P) is the shortest number of jumps from seed link i to webpage P. To ensure that the SA of the seed link is greater than that of other webpages, the d(i,P) of the seed link is set to 0. In this way, the SA value of the seed link is the sum of the number of seed links, and other webpages must be less than this value.

[0026] (2) Freshness. Freshness considers the timeliness of a webpage. A webpage that has been published for a shorter period of time means that it provides the most up-to-date information and is of greater value to users. The formula for calculating freshness is as follows: In the formula This refers to the current time. This refers to the time when the webpage was last updated. It is a decay factor used to adjust the degree to which time differences affect freshness in this system. Use 0.5.

[0027] (3) Topic Relevance. The stored webpage information includes the five keywords with the highest TF-IDF values ​​for the webpage and their corresponding TF-IDF values. When keyword information is received from the receiving module, the topic relevance of the webpage can be considered by examining whether the webpage contains keywords and how many keywords it contains. The more keywords it contains and the higher the TF-IDF values ​​of the keywords, the stronger the relevance of the webpage to the current search. The formula for calculating relevance is as follows: In the formula Representative webpage, This represents the inquiry. This represents the set of keywords contained in webpage P. Identify the set of keywords corresponding to this query.

[0028] After deriving the calculation methods for these three aspects, the calculation formula of the DATPR algorithm proposed in this invention is a weighted sum of several parts, as follows:

[0029] In the formula , , , The sum is 1, by default. , , , The values ​​are 0.5, 0.2, 0.1, and 0.2 respectively. Users can adjust the sorting scheme as needed to obtain three different results: the original PageRank algorithm, authority priority, freshness priority, and topic relevance priority.

Claims

1. This invention proposes a method for constructing a search engine based on a domain-specific knowledge graph. Taking the search engine as the research object, the construction of the search engine is divided into three modules: a crawler module, an indexing module, and an interaction module. The crawler module's function is to crawl and store data from the Internet. It traverses network links using predetermined algorithms and strategies, downloads network content, and filters web pages related to the domain using an existing domain knowledge graph, storing the data in a database. The indexing module is responsible for transforming the web page data crawled by the crawler into a structure that the search engine can quickly query. It transforms the raw web page data into a structured data format, builds an index, and returns documents through keywords. The interaction module provides a user interface, displays search results, accepts user queries, and takes different actions based on the query type. Precise searches use existing question-answering models, while ordinary searches are performed using Chinese word segmentation. After obtaining the results, they are sorted and displayed to the user.

2. The method for constructing a search engine based on a domain knowledge graph according to claim 1, characterized in that, The steps of this constructor are as follows: S1: Web crawler module construction. Based on requirements, collect authoritative links in the field and construct a domain terminology set. Build the web crawler module and crawl relevant web pages in the field based on authoritative links, using the domain terminology set to limit the scope of the web pages; S2: Index module construction. Based on requirements, an index module is built to process the data collected by the crawler module and transform it into structured data. S3: Interactive module construction. Based on requirements, construct interactive modules to provide users with interfaces and display results.

3. The method for constructing a search engine based on a domain knowledge graph according to claim 1, characterized in that, Step S1 includes the following sub-steps: S101. Construct a thesaurus based on the domain knowledge graph; S102. Collect authoritative links in the field and construct a domain terminology based on the domain's knowledge graph; S103. Web page crawling using the Clawler4j framework. S104. Analyze the content of the webpage, extract the data required by the system from the page content, and transform it into a structured data structure including {page number, page title, page link, parent link, last modification time, keywords} for storage. Keyword extraction uses the Aho-Corasick algorithm (AC algorithm) combined with TF-IDF analysis to confirm the keywords and their importance in each document.

4. The method for constructing a search engine based on a domain knowledge graph according to claim 1, characterized in that, Step S2 includes the following sub-steps: S201. Determine the data type and the corresponding mapping structure; S202. In ElasticSearch, these mappings are actually created to index the web page data crawled by the crawler module, laying the foundation based on the provided keywords.

5. The method for constructing a search engine based on a domain knowledge graph according to claim 1, characterized in that, Step S3 includes the following sub-steps: S301. Obtain the extended word library ext.dic through the existing knowledge graph of this domain, construct the stop word library stopword.dic, and configure the IK_Analyer word segmenter; S302. Deploy the existing question-answering model to the Flask service; S303. Configuration module, which enables ordinary search to obtain keywords through word segmentation as described in S301, and precise search to obtain keywords through the question-answering model deployed in S302; S304. Configure the webpage ranking algorithm of the module to rank the acquired webpages according to timeliness, authority and topic relevance.

6. The method for constructing a search engine based on a domain knowledge graph according to claim 1, characterized in that, A PageRank algorithm, DATPR (Dynamic Authority-Topic PageRank), is proposed that comprehensively considers timeliness, authority, and topic relevance. Web pages are ranked based on this algorithm.

Images