Theme crawler method and device based on candidate link theme edge text

Abstract

The invention discloses a topic crawler method and device based on candidate connection topic edge texts, and relates to the technical field of information retrieval. According to the method, anchor text keywords and keywords, similar to the anchor text keywords in terms of meaning, in a webpage text are extracted through a Dunwei decimal classification method, the keywords are called candidate link theme edge texts, when a naive Bayes classifier is used for conducting correlation judgment, feature attributes with different occurrence frequencies are weighted, and the candidate link theme edge texts are obtained. And storing the obtained candidate links into a queue according to the size sequence of the judgment result to wait for the next round of access.

Description

Technical Field

[0001] The invention discloses a topic crawler method and a device based on candidate link topic edge text, and relates to the technical field of information retrieval. Background Art

[0002] The vigorous development of Internet technology has made network resources more diversified and heterogeneous. The current amount of web data far exceeds the coverage of search engines. When providing retrieval services to users, search engines cannot meet the professional, accurate and in-depth query needs in professional fields. As the core technology of search engines, the main function of subject crawlers is to crawl web pages related to the topic and filter out web pages unrelated to the topic. The quality of its performance directly determines the query experience of front-end users. How to improve the efficiency of subject crawlers is an urgent problem to be solved.

[0003] In order to improve the acquisition rate of relevant web pages, researchers have proposed a variety of topic crawler methods, such as the method of guiding topic crawlers to retrieve relevant pages based on web page segmentation [Improved Focused Crawling Approach for Retrieving Relevant Pages Based on Block Partitoning [C]. International Conference on Education Technology and Computer. 2010, pp269-217.], which divides web pages into blocks and calculates the relevance of each block with the topic. The overall relevance of the web page is equal to the sum of the relevance of each block. For example, the topic crawler algorithm based on tunnel traversal technology [Peng Tao, Meng Yu, Zuo Wanli. Tunnel traversal technology in topic crawling [J]. Computer Research and Development, 2010, 47 (4): 628-637] divides the tunnels in topic crawling into gray tunnels and black tunnels, and different tunnels use their own methods.

[0004] The research has improved the performance of the topic crawler to a certain extent, but there are still some issues worth pondering. First, the feature attributes involve a large number of keywords that are irrelevant to the topic. Second, the impact of feature attributes with different frequencies on the relevance determination is not considered. If the above two issues can be solved, there will be a great breakthrough in improving the performance of the topic crawler. Summary of the invention

[0005] In view of the problems of the prior art, the present invention provides a topic crawler method and device based on the marginal text of candidate link topics, and the technical solution adopted is:

[0006] In a first aspect, a topic crawler method based on candidate link topic edge text comprises:

[0007] S1, extract the initial seed connection according to the crawling queue;

[0008] S2, according to the seed connection, sending a corresponding page request to the network through a subject crawler, and obtaining corresponding web page information;

[0009] S3, the subject crawler temporarily stores the acquired web pages in the response queue and waits for processing;

[0010] S4, obtaining the candidate link topic edge text according to the web page information in the response queue;

[0011] S5, judging whether the candidate link topic edge text is related to the feature topic by constructing a feature weighted naive Bayesian; if relevant, saving it to the background database; otherwise, discarding it;

[0012] S6, storing the web pages that match the specific main image into a page database according to the backend database;

[0013] S7, storing the candidate links in the relevant web pages into a crawling queue, and sorting them in the queue according to the size of the CNB.

[0014] In some implementations, S4 specifically includes:

[0015] S41, performing word segmentation processing on the webpage body and anchor text through ICTCLAS according to the webpage information, and marking the Dewey Decimal Classification number of the keyword;

[0016] S42, drawing a corresponding coordinate system in a two-dimensional space according to the Dewey Decimal Classification number;

[0017] S43, representing the candidate link topic edge texts through two-dimensional space lines according to the keyword distribution trend of the coordinate system.

[0018] In some implementations, S42, wherein the length of the classification number is represented by the X-axis, the detailed classification number of the keyword is represented by the Y-axis, and the keyword is represented by a dot.

[0019] In some implementations, in S5, the naive Bayes text classification formula is:

[0020]

[0021] Among them, the target weight depends on P(c j ) and P(x i |c j ), so the naive Bayes text classifier needs to estimate different P(c j ) and P(x i |c j) items, that is, their frequencies in the training set. j ) and P(x i |c j ) is calculated as follows:

[0022]

[0023] Where C is the text category of the training set, N(C=c j ) indicates that the training set belongs to category c j The number of texts in the training set, N represents the total number of texts in the training set, N(X=x i ,C=c j ) means that in the training set, category c j Contains the attribute x i The number of texts in the candidate link topic edge text is represented by M.

[0024] In some implementations, in S5, the attributes of the candidate link topic edge text have different effects on the weights output by the classifier, and the formula is as follows:

[0025]

[0026] Among them, N(x i ) represents the training set x i The number of N(x i =(aΛc j )) is category c j The number of instances with the ith feature number a in N(x i =a) is the number of instances with feature a in the training set; belongs to category c j The more instances of the characteristic attribute a, the greater the iaj The larger the weight, the greater the target relevance.

[0027] In a second aspect, an embodiment of the present invention provides a topic crawler device based on candidate link topic edge text, comprising:

[0028] The link crawling module is used to extract the initial seed link according to the crawling queue;

[0029] A request processing module, used to send a corresponding page request to the network through a subject crawler according to the seed link, and obtain corresponding web page information;

[0030] A page acquisition module is used to acquire the web page information through a subject crawler and temporarily store it in a response queue for waiting processing;

[0031] An edge drawing module, used for obtaining the edge text of the candidate link topic according to the web page information in the response queue;

[0032] The topic determination module determines whether the candidate link topic edge text is related to the feature topic by constructing a feature weighted naive Bayesian; if relevant, it is saved in the background database; otherwise, it is discarded;

[0033] A page storage module, used to store web pages that meet a specific theme into a page database according to the backend database;

[0034] The link acquisition module is used to store candidate links in relevant web pages into the crawling queue and NB The sizes are sorted in the queue.

[0035] In some implementations, the edge drawing module specifically includes:

[0036] A word segmentation unit is used to perform word segmentation processing on the web page body and anchor text through ICTCLAS according to the web page information, and mark the Dewey Decimal Classification number of the keyword;

[0037] A drawing unit, for drawing a corresponding coordinate system in a two-dimensional space according to the Dewey Decimal Classification number;

[0038] The connection unit is used to represent the candidate link topic edge text through two-dimensional space connection lines according to the keyword distribution trend of the coordinate system.

[0039] In some implementations, in the drawing unit, the length of the classification number is represented by the X-axis, the detailed classification number of the keyword is represented by the Y-axis, and the keyword is represented by a dot.

[0040] In some implementations, in the edge drawing module, the naive Bayes text classification formula is:

[0041]

[0042] Among them, the target weight depends on P(c j ) and P(x i |c j ), so the naive Bayes text classifier needs to estimate different P(c j ) and P(x i |c j ) items, that is, their frequencies in the training set. j ) and P(x i |c j ) is calculated as follows:

[0043]

[0044] Where C is the text category of the training set, N(C=c j ) indicates that the training set belongs to category c j The number of texts in the training set, N represents the total number of texts in the training set, N(X=x i ,C=c j ) means that in the training set, category c j Contains the attribute x i The number of keywords in the edge text of the candidate link topic is represented by M.

[0045] In some implementations, in the edge drawing module, the attributes of the candidate link topic edge text have different effects on the weights output by the classifier, and the formula is as follows:

[0046]

[0047] Among them, N(x i ) represents the training set x i The number of N(x i =(aΛc j )) is category c j The number of instances with the ith feature number a in N(x i =a) is the number of instances with feature a in the training set. j The more instances of the characteristic attribute a, the greater the iaj The larger the weight, the greater the target relevance.

[0048] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein when the one or more computer instructions are executed by the processor, the method described in the first aspect above is implemented.

[0049] In a fourth aspect, an embodiment of the present invention provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the method described in the first aspect.

[0050] One or more embodiments of the present invention can bring at least the following beneficial effects:

[0051] The method of the present invention uses the Dewey Decimal Classification to extract anchor text keywords and keywords in the body of a web page that are similar in meaning to the anchor text keywords, which are called candidate link topic edge texts. When using a naive Bayes classifier to determine the relevance, feature attributes with different frequencies of occurrence are weighted, and the obtained candidate links are stored in a queue in the order of the size of the determination results and wait for the next round of access. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 is a flow chart of a topic crawler method based on candidate connection topic edge texts provided by an embodiment of the present invention;

[0054] Figure 2 It is a block diagram of a topic crawler device based on candidate connection topic edge text provided by an embodiment of the present invention;

[0055] Figure 3 is a schematic diagram of extracting edge text of candidate link topics provided by an embodiment of the present invention;

[0056] Figure 4 This is a flow chart of the classification of edge texts of candidate link main graphs provided by an embodiment of the present invention;

[0057] Figure 5 It is a schematic diagram of the subject crawler architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0059] Embodiment 1:

[0060] Figure 1 A flow chart of a topic crawler method based on candidate connection topic edge text is shown, such as Figure 1 As shown, the topic crawler method based on candidate connection topic edge text provided in this embodiment includes:

[0061] S1, extract the initial seed connection according to the crawling queue;

[0062] S2, according to the seed connection, sending a corresponding page request to the network through a subject crawler, and obtaining corresponding web page information;

[0063] S3, according to the web page information, the web page information is obtained by the subject crawler and temporarily stored in the response queue to wait for processing;

[0064] S4, obtaining the candidate link topic edge text according to the web page information in the response queue;

[0065] S5, judging whether the candidate link topic edge text is related to the feature topic by constructing a feature weighted naive Bayesian; if relevant, saving it to the background database; otherwise, discarding it;

[0066] S6, storing the web pages that match the specific main image into a page database according to the backend database;

[0067] S7, storing the candidate links in the relevant web pages into a crawling queue, and sorting them in the queue according to the size of the CNB.

[0068] First, according to S1, the initial seed link is extracted from the crawling queue;

[0069] Next, according to S2, the subject crawler sends a corresponding page request to the network, and then obtains the corresponding web page information from the network;

[0070] In the next step, S3, the subject crawler temporarily stores the acquired web pages in the response queue, awaiting further processing.

[0071] Next, S4, use ICTCLAS to segment the web page text and anchor text, and annotate the Dewey Decimal Classification numbers of keywords, such as Figure 3 As shown, S4 specifically includes:

[0072] S41, performing word segmentation processing on the webpage body and anchor text through ICTCLAS according to the webpage information, and marking the Dewey Decimal Classification number of the keyword;

[0073] S42, drawing a corresponding coordinate system in a two-dimensional space according to the Dewey Decimal Classification number;

[0074] S43, representing the candidate link topic edge texts through two-dimensional space lines according to the keyword distribution trend of the coordinate system.

[0075] According to S41, use ICTCLAS to segment the webpage text and anchor text, and mark the Dewey Decimal Classification number of the keyword; according to S42, draw a two-dimensional coordinate, the X axis of the coordinate is the length of the classification number, and the Y axis is the detailed classification number corresponding to the keyword. Use dots to represent keywords, and draw dots at the corresponding positions in the two-dimensional space; according to S43, analyze the distribution trend of the dots in the two-dimensional coordinate axis, such as Figure 5As shown in the figure, in the two-dimensional space, the gray dots represent anchor text keywords, and the black dots represent web page body keywords. There are multiple black dots tightly gathered around the gray dots, and the keywords corresponding to these dots have similar meanings or themes, that is, candidate link theme edge texts.

[0076] Next, according to S5, a feature-weighted naive Bayesian is constructed to determine whether the candidate link topic edge text is related to a specific topic.

[0077] Figure 4 The detailed process of candidate link topic edge text classification is shown. The candidate link topic edge text is regarded as a "bag" of keywords, represented by X = {x1, x2, ..., xn}. The probability formula of candidate link topic edge text X belonging to category cj is as follows:

[0078]

[0079] CNB is the weight of the classifier output. P(cj) is called the prior probability, that is, the initial frequency of category cj in the corpus. P(x1,x2,…,xn|cj) is called the joint probability, which is the probability of text X appearing under the condition that category cj is known. P(cj|x1,x2,…,xn) is called the posterior probability, which is the confidence that category cj is established after the data text X appears. P(x1,x2,…,xn) is a constant, so the solution to formula (1) is:

[0080]

[0081] Bayesian theory believes that the attributes x1, x2, ..., xn of the text feature vector are conditionally independent of each other, and the probability of the joint x1, x2, ..., xn is equal to the product of the probability distribution of each individual attribute, that is:

[0082]

[0083] Furthermore, in S5, formula (C) is substituted into formula (B) to obtain the naive Bayes text classification formula, which is:

[0084]

[0085] Among them, the target weight depends on P(cj) and P(xi|cj), so the naive Bayes text classifier needs to estimate different P(cj) and P(xi|cj) items, that is, their frequencies in the training set. The probability values ​​of P(cj) and P(xi|cj) are calculated as follows:

[0086]

[0087] Where C is the text category of the training set, N(C=cj) represents the number of texts belonging to category cj in the training set, N represents the total number of texts in the training set, and N(X=xi,C=cj) represents the number of texts in category cj that contain attribute xi in the training set. The number of keywords in the edge text of the candidate link topic is represented by M.

[0088] In some implementations, in S5, the attributes of the candidate link topic edge text have different effects on the weights output by the classifier, and the formula is as follows:

[0089]

[0090] Each posterior probability is weighted, and the value of wiaj is (5), where N(xi) represents the number of xi in the training set, N(xi=(aΛcj)) is the number of instances with the ith feature number a in category cj, and N(xi=a) is the number of instances with the ith feature number a in the training set. From (5), we can see that the more instances of feature attribute a belonging to category cj, the greater the weight of wiaj and the greater the target relevance.

[0091]

[0092] Among them, N(xi) represents the number of xi in the training set, N(xi=(aΛcj)) is the number of instances with the ith feature number a in category cj, and N(xi=a) is the number of instances with the ith feature number a in the training set. The more instances of feature attribute a belonging to category cj, the greater the weight of wiaj and the greater the target relevance.

[0093] Embodiment 2:

[0094] Figure 2 A block diagram of a topic crawler device based on candidate connection topic edge text is shown, such as Figure 2 As shown, the subject crawler device based on candidate connection subject edge text provided in this embodiment includes:

[0095] The link crawling module is used to extract the initial seed connection according to the crawling queue;

[0096] A request processing module, used to send a corresponding page request to the network through a subject crawler according to the seed connection, and obtain corresponding web page information;

[0097] A page acquisition module is used to acquire the web page information through a subject crawler and temporarily store it in a response queue for waiting processing;

[0098] An edge drawing module, used for obtaining the edge text of the candidate link topic according to the web page information in the response queue;

[0099] The topic determination module determines whether the candidate link topic edge text is related to the feature topic by constructing a feature weighted naive Bayesian; if relevant, it is saved in the background database; otherwise, it is discarded;

[0100] A page storage module, used to store web pages that match a specific main image into a page database according to the backend database;

[0101] The link acquisition module is used to store candidate links in the relevant web pages into the crawling queue and sort them in the queue according to the size of the CNB.

[0102] In some implementations, the edge drawing module specifically includes:

[0103] A word segmentation unit is used to perform word segmentation processing on the web page body and anchor text through ICTCLAS according to the web page information, and mark the Dewey Decimal Classification number of the keyword;

[0104] A drawing unit, for drawing a corresponding coordinate system in a two-dimensional space according to the Dewey Decimal Classification number;

[0105] The connection unit is used to represent the candidate link topic edge text through two-dimensional space connection lines according to the keyword distribution trend of the coordinate system.

[0106] In some implementations, in the drawing unit, the length of the classification number is represented by the X-axis, the detailed classification number of the keyword is represented by the Y-axis, and the keyword is represented by a dot.

[0107] In some implementations, in the edge drawing module, the naive Bayes text classification formula is:

[0108]

[0109] Among them, the target weight depends on P(cj) and P(xi|cj), so the naive Bayes text classifier needs to estimate different P(cj) and P(xi|cj) items, that is, their frequencies in the training set. The probability values ​​of P(cj) and P(xi|cj) are calculated as follows:

[0110]

[0111] Where C is the text category of the training set, N(C=cj) represents the number of texts belonging to category cj in the training set, N represents the total number of texts in the training set, and N(X=xi,C=cj) represents the number of texts in category cj that contain attribute xi in the training set. The number of keywords in the edge text of the candidate link topic is represented by M.

[0112] In some implementations, in the edge drawing module, the attributes of the candidate link topic edge text have different effects on the weights output by the classifier, and the formula is as follows:

[0113]

[0114] Among them, N(xi) represents the number of xi in the training set, N(xi=(aΛcj)) is the number of instances with the ith feature number a in category cj, and N(xi=a) is the number of instances with the ith feature number a in the training set. The more instances of feature attribute a belonging to category cj, the greater the weight of wiaj and the greater the target relevance.

[0115] Embodiment three:

[0116] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions implement the method of embodiment 1 when executed by the processor;

[0117] In practical applications, the processor can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller unit (MCU), a microprocessor or other electronic components to execute the methods in the above embodiments.

[0118] The method implemented in this embodiment is as shown in the first embodiment.

[0119] Embodiment 4:

[0120] This embodiment further provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method of the first embodiment is implemented;

[0121] Among them, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0122] The method implemented in this embodiment is as shown in the first embodiment.

[0123] In the several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed system and method can also be implemented in other ways. The system and method embodiments described above are merely illustrative.

[0124] It should be noted that, in this article, the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. The terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0125] Although the embodiments disclosed in the present invention are as above, the above contents are only embodiments adopted for facilitating the understanding of the present invention and are not intended to limit the present invention. Any technician in the technical field to which the present invention belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present invention, but the patent protection scope of the present invention shall still be subject to the scope defined in the attached claims.

Claims

1. A topic crawler method based on candidate link topic edge text, characterized in that: include: S1, extract the initial seed link according to the crawling queue; S2, according to the seed link, sending a corresponding page request to the network through a subject crawler, and obtaining corresponding web page information; S3, the subject crawler temporarily stores the acquired web pages in the response queue and waits for processing; S4, obtaining the candidate link topic edge text according to the web page information in the response queue; S5, judging whether the candidate link topic edge text is related to the feature topic by constructing a feature weighted naive Bayesian; if relevant, saving it to the background database; otherwise, discarding it; S6, storing web pages that meet the specific theme into a page database according to the backend database; S7, storing candidate links in relevant web pages into the crawling queue, and NB The sizes are sorted in the queue.

2. The method according to claim 1, characterized in that S4 specifically includes: S41, performing word segmentation processing on the webpage body and anchor text through ICTCLAS according to the webpage information, and marking the Dewey Decimal Classification number of the keyword; S42, drawing a corresponding coordinate system in a two-dimensional space according to the Dewey Decimal Classification number; S43, representing the candidate link topic edge texts through two-dimensional space lines according to the keyword distribution trend of the coordinate system.

3. The method according to claim 2, characterized in that S42, wherein the length of the classification number is represented by the X-axis, the detailed classification number of the keyword is represented by the Y-axis, and the keyword is represented by dots.

4. The method according to claim 3, characterized in that In S5, the Naive Bayes text classification formula is: Among them, the target weight depends on P(cj) and P(xi|cj), so the naive Bayes text classifier needs to estimate different P(cj) and P(xi|cj) items, that is, their frequencies in the training set; the probability values ​​of P(cj) and P(xi|cj) are calculated as follows: Where C is the text category of the training set, N(C=c j ) indicates that the training set belongs to category c j The number of texts in the training set, N represents the total number of texts in the training set, N(X=x i ,C=c j ) means that in the training set, category c j Contains the attribute x i The number of texts in the candidate link topic edge text is represented by M.

5. The method according to claim 4, characterized in that In S5, the attributes of the candidate link topic edge text have different effects on the weights output by the classifier, and the formula is as follows: Among them, N(x i ) represents the training set x i The number of N(x i =(aΛc j )) is category c j The number of instances with the ith feature number a in N(x i =a) is the number of instances with feature a in the training set; belongs to category c j The more instances of the characteristic attribute a, the greater the iaj The larger the weight, the greater the target relevance.

6. A topic crawler device based on candidate connection topic edge text, characterized in that: include: The link crawling module is used to extract the initial seed link according to the crawling queue; A request processing module, used to send a corresponding page request to the network through a subject crawler according to the seed link, and obtain corresponding web page information; A page acquisition module is used to acquire the web page information through a subject crawler and temporarily store it in a response queue for waiting processing; An edge drawing module, used for obtaining the edge text of the candidate link topic according to the web page information in the response queue; The topic determination module determines whether the candidate link topic edge text is related to the feature topic by constructing a feature weighted naive Bayesian; if relevant, it is saved in the background database; otherwise, it is discarded; A page storage module, used to store web pages that match a specific main image into a page database according to the backend database; The link acquisition module is used to store candidate links in the relevant web pages into the crawling queue and sort them in the queue according to the size of the CNB.

7. The device according to claim 6, characterized in that The edge drawing module specifically includes: A word segmentation unit is used to perform word segmentation processing on the web page body and anchor text through ICTCLAS according to the web page information, and mark the Dewey Decimal Classification number of the keyword; A drawing unit, for drawing a corresponding coordinate system in a two-dimensional space according to the Dewey Decimal Classification number; The connection unit is used to represent the candidate link topic edge text through two-dimensional space connection lines according to the keyword distribution trend of the coordinate system.

8. The device according to claim 7, characterized in that In the drawing unit, the length of the classification number is represented by the X-axis, the detailed classification number of the keyword is represented by the Y-axis, and the keyword is represented by a dot.

9. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the method as described in any one of claims 1 to 5.

Images