News link extraction method and device, storage medium and electronic equipment

Abstract

The disclosure provides a news link extraction method and device, electronic equipment and storage medium, and relates to the technical field of computers. The method comprises the following steps: obtaining a homepage link of a news website homepage; performing link data extraction according to the homepage link to obtain a homepage news link extraction result, a homepage non-news link extraction result and a homepage news link update speed of the news website homepage; judging whether a news board page exists in the news website homepage according to the homepage non-news link extraction result and the homepage news link update speed; in the case where the news board page exists, extracting a board page news link extraction result in the news board page; and determining a website news link extraction result of the news website homepage according to the homepage news link extraction result and the board page news link extraction result. The method can continuously and efficiently output all website news link extraction results in the news website homepage.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a news link extraction method and device, a storage medium and an electronic device. BACKGROUND

[0002] With the development of computer technology and Internet technology, browsing information through a news website has become an important way to obtain news.

[0003] In the related art, a news link in a news website can be extracted by analyzing a webpage structure in combination with a manual annotation. Since different websites have completely different website organization structures, position information manually annotated on a news website cannot be migrated to other news websites, so the reusability of this method is poor. Moreover, the process of manual annotation is time-consuming and laborious, and has low accuracy.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The purpose of the present disclosure is to provide a news link extraction method and device, an electronic device and a storage medium, so as to continuously and efficiently output all website news link extraction results in a news website homepage.

[0006] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a news link extraction method is provided, comprising: obtaining a homepage link of a news website homepage; performing link data extraction according to the homepage link to obtain a homepage news link extraction result, a homepage non-news link extraction result and a homepage news link update speed of the news website homepage; judging whether there is a news panel page in the news website homepage according to the homepage non-news link extraction result and the homepage news link update speed; in the case that there is a news panel page, extracting a panel page news link extraction result in the news panel page; and determining a website news link extraction result of the news website homepage according to the homepage news link extraction result and the panel page news link extraction result.

[0008] In one embodiment of the present disclosure, whether a news board page exists in a homepage of a news website is determined according to a homepage non-news link extraction result and a homepage news link update speed, including: link data extraction is performed on each homepage non-news link in the homepage non-news link extraction result to obtain a sub-page news update speed of a sub-page pointed to by each homepage non-news link; if a target sub-page exists in the sub-pages and a ratio of the sub-page news update speed of the target sub-page to the homepage news link update speed is greater than an update threshold, it is determined that the news board page exists in the homepage of the news website, and the target sub-page is determined as the news board page.

[0009] In one embodiment of the present disclosure, link data extraction is performed on homepage links to obtain a homepage news link extraction result of a homepage of a news website, including: a plurality of extraction time points are determined based on a preset extraction frequency, and a same-domain link set of the homepage of the news website at each extraction time point is obtained according to the homepage links; at a current extraction time point, a first preset number of same-domain link sets are obtained by using a preset news extraction sliding window; the first preset number of same-domain link sets correspond to the current extraction time point and extraction time points before the current extraction time point; new homepage news links at the current extraction time point are obtained according to the first preset number of same-domain link sets; and the new homepage news links at the current extraction time point are output as the homepage news link extraction result of the homepage of the news website.

[0010] In one embodiment of the present disclosure, the first preset number is 2; the first preset number of same-domain link sets include a current same-domain link set at the current extraction time point and a previous same-domain link set at a previous time point of the current extraction time point; and the new homepage news links at the current extraction time point are obtained according to the first preset number of same-domain link sets, including: a difference set between the current same-domain link set and the previous same-domain link set is determined; and the new homepage news links at the current extraction time point are determined according to links in the difference set.

[0011] In one embodiment of the present disclosure, link data extraction is performed on homepage links to obtain a homepage non-news link extraction result of a homepage of a news website, including: at a current extraction time point, a second preset number of same-domain link sets are obtained by using a preset non-news extraction sliding window; the second preset number of same-domain link sets correspond to the current extraction time point and extraction time points before the current extraction time point; homepage non-news links at the current extraction time point are obtained according to the second preset number of same-domain link sets; and the homepage non-news links at the current extraction time point are output as the homepage non-news link extraction result of the homepage of the news website.

[0012] In one embodiment of the present disclosure, the second preset quantity is greater than the first preset quantity; and the homepage non-news link at the current extraction time is obtained according to the second preset quantity of the same domain name link sets, including: determining the intersection of the second preset quantity of the same domain name link sets; and determining the homepage non-news link at the current extraction time according to the link in the intersection.

[0013] In one embodiment of the present disclosure, the homepage news link update speed of the homepage of the news website is obtained by performing link data extraction according to the homepage link, including: obtaining a preset time period containing the current extraction time, and determining the total extraction times of the extraction time in the preset time period; determining the total amount of the newly added homepage news link according to the newly added homepage news link at all extraction times in the preset time period; and determining the homepage news link update speed of the homepage of the news website in the preset time period according to the total amount of the newly added homepage news link and the total extraction times.

[0014] According to another aspect of the present disclosure, there is provided an extraction device of news link, including: an acquisition module, configured to acquire the homepage link of the homepage of the news website; a homepage link data extraction module, configured to perform link data extraction according to the homepage link, and obtain the homepage news link extraction result, the homepage non-news link extraction result and the homepage news link update speed of the homepage of the news website; a judgment module, configured to judge whether there is a news board page in the homepage of the news website according to the homepage non-news link extraction result and the homepage news link update speed; a board link extraction module, configured to extract the board page news link extraction result in the news board page in the case that there is a news board page; and a determination module, configured to determine the website news link extraction result of the homepage of the news website according to the homepage news link extraction result and the board page news link extraction result.

[0015] According to still another aspect of the present disclosure, there is provided a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the above-mentioned extraction method of news link.

[0016] According to yet another aspect of the present disclosure, there is provided an electronic device, including: a processor; and a memory, configured to store executable instructions of the processor; and the processor is configured to execute the above-mentioned extraction method of news link by executing the executable instructions.

[0017] The news link extraction method provided by the embodiment of the present disclosure can first extract link data from a news website homepage to obtain homepage news link extraction results, homepage non-news link extraction results and homepage news link update speed, then further judge whether there is a news board page in the news website homepage according to the homepage non-news link extraction results and the homepage news link update speed, and extract board page news link extraction results in the news board page in the case of existence, and then take the homepage news link extraction results and the board page news link extraction results together as the final website news link extraction results of the news website homepage for output. The method can automatically and efficiently obtain newly added homepage news links at each extraction time, quickly judge whether there is a news board page in the news website homepage, and then automatically and efficiently obtain board page news link extraction results in the news board page. It can be seen that the method can not only quickly obtain news links in the news website homepage, but also mine news board pages in the news website homepage and quickly obtain news links in the news board pages, so as to continuously and efficiently and comprehensively output all website news link extraction results in the news website homepage.

[0018] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained according to these drawings without creative labor for those skilled in the art.

[0020] Figure 1 A schematic diagram of an exemplary system architecture to which the news link extraction method of the embodiment of the present disclosure can be applied is shown;

[0021] Figure 2 A flowchart of the news link extraction method of one embodiment of the present disclosure is shown;

[0022] Figure 3 A flowchart of obtaining homepage news link extraction results in the news link extraction method of one embodiment of the present disclosure is shown;

[0023] Figure 4 A flowchart of obtaining homepage non-news link extraction results in the news link extraction method of one embodiment of the present disclosure is shown;

[0024] Figure 5A flow chart showing the obtaining of the update speed of the homepage news link in the news link extraction method of one embodiment of the present disclosure is shown;

[0025] Figure 6 A flow chart showing the judging of whether there is a news board page in the news website homepage in the news link extraction method of one embodiment of the present disclosure is shown;

[0026] Figure 7 A schematic diagram of the news link extraction method of one embodiment of the present disclosure is shown;

[0027] Figure 8 A schematic diagram of the news link extractor in the news link extraction method of one embodiment of the present disclosure is shown;

[0028] Figure 9 A schematic diagram of the board link extractor in the news link extraction method of one embodiment of the present disclosure is shown;

[0029] Figure 10 A block diagram of the news link extraction device of one embodiment of the present disclosure is shown; and

[0030] Figure 11 A structural block diagram of a news link extraction computer device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the inventive concept to those skilled in the art. Like reference numerals refer to like elements throughout the description.

[0032] Moreover, the drawings are not necessarily to scale. Like reference numerals designate like parts throughout the specification. Some of the block components, which have the same function, are denoted by the same reference numerals, and thus the repeated description thereof will be omitted. Some of the block components shown in the drawings are functional components, and thus can not necessarily correspond to physical or logical components. The functional components can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0033] In addition, the terms "first", "second", etc. are used only for descriptive purposes and are not to be construed as indicating or implying relative importance or an indicated number of technical features. Thus, features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present disclosure, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0034] Figure 1 A schematic diagram of an exemplary system architecture of the news link extraction method to which embodiments of the present disclosure can be applied is shown.

[0035] As Figure 1 shown, the system architecture can include a server 101, a network 102 and a client 103. The network 102 is a medium for providing a communication link between the client 103 and the server 101. The network 102 can include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0036] In an exemplary embodiment, the client 103 that transmits data with the server 101 can include, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, AR (Augmented Reality) devices, VR (Virtual Reality) devices, smart wearable devices, etc. Optionally, the operating system running on the electronic device can include, but is not limited to, Android system, IOS system, Linux system, Windows system, etc.

[0037] The server 101 can be a standalone physical server, or a server cluster or distributed system formed by multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDNs (Content Delivery Networks), and basic cloud computing services such as big data and artificial intelligence platforms. In some practical applications, the server 101 can also be a server of a network platform, such as a transaction platform, a live streaming platform, a social platform, or a music platform, etc., and the present disclosure does not limit the same. The server can be a single server or a cluster formed by multiple servers, and the present disclosure does not limit the specific architecture of the server.

[0038] In the example embodiment, an input interface for receiving a homepage link of a homepage of a news website can be installed in the client 103, and a system capable of implementing the news link extraction method of the embodiment of the present disclosure can be deployed in the server 101; after the client 103 sends the received homepage link input by the user to the server 101, the server 101 can start processing the homepage link and implementing the news link extraction method of the embodiment of the present disclosure.

[0039] In the example embodiment, the process of the server 101 for implementing the news link extraction method can be: the server 101 acquires a homepage link of a homepage of a news website; the server 101 performs link data extraction according to the homepage link to obtain a homepage news link extraction result, a homepage non-news link extraction result and a homepage news link update speed of the homepage of the news website; the server 101 determines whether there is a news board page in the homepage of the news website according to the homepage non-news link extraction result and the homepage news link update speed; the server 101 extracts a board page news link extraction result in the news board page in the case that there is a news board page; and the server 101 determines a website news link extraction result of the homepage of the news website according to the homepage news link extraction result and the board page news link extraction result.

[0040] In addition, it should be noted that, Figure 1 The server 101, the network 102 and the client 103 shown are only an application environment of the news link extraction method provided by the present disclosure. Figure 1 The number of the server 101, the network 102 and the client 103 shown is only illustrative, and any number of clients, networks and servers can be provided according to actual needs.

[0041] In order to make the ordinary person skilled in the art better understand the technical solutions of the present disclosure, the steps of the news link extraction method in the example embodiment of the present disclosure will be described in more detail below in combination with the drawings and examples.

[0042] Figure 2 A flowchart of the news link extraction method of one embodiment of the present disclosure is shown. The method provided by the embodiment of the present disclosure can be executed by the server 101 or the client 103 as shown, but the present disclosure is not limited thereto. Figure 1

[0043] In the following example, the server 101 is taken as an example to illustrate.

[0044] As shown in Figure 2 The news link extraction method provided by the embodiment of the present disclosure can include the following steps.

[0045] Step S201, acquiring a homepage link of a homepage of a news website. ​

[0046] The homepage of the news website can be a homepage page of the news website, and the homepage link can be a hyperlink that can jump to the homepage of the news website after being clicked, i.e., the page pointed to by the homepage link is the homepage of the news website.

[0047] In step S203, link data extraction is performed according to the homepage link to obtain the homepage news link extraction result, the homepage non-news link extraction result, and the homepage news link update speed of the homepage of the news website.

[0048] The homepage of the news website can be accessed based on the homepage link to obtain homepage data, and then relevant link data can be extracted from the homepage data to obtain the homepage news link extraction result, the homepage non-news link extraction result, and the homepage news link update speed.

[0049] Figure 3 A flowchart of obtaining the homepage news link extraction result in the method for extracting news links according to an embodiment of the present disclosure is shown. As shown in the flowchart, Figure 3 In some embodiments, the step S203 of "extracting link data according to the homepage link to obtain the homepage news link extraction result of the homepage of the news website" can include the following steps.

[0050] In step S301, a plurality of extraction time points are determined based on a preset extraction frequency, and the same-domain-name link set of the homepage of the news website at each extraction time point is obtained according to the homepage link.

[0051] In this step, the preset extraction frequency can be set based on actual conditions, for example, it can be extracted once every 1 hour, once every 6 hours, etc. Assuming that it is extracted once every 1 hour, then in a day, the plurality of extraction time points can be 0 o'clock, 1 o'clock, 2 o'clock, …, 22 o'clock, and 23 o'clock; assuming that it is extracted once every 6 hours, then in a day, the plurality of extraction time points can be 0 o'clock, 6 o'clock, 12 o'clock, and 18 o'clock.

[0052] In this step, the homepage link can be processed by a crawler technology to obtain homepage data in the homepage of the news website; then the link data in the homepage data can be determined, and the same-domain-name links can be obtained by screening the link data based on the domain name of the homepage link. That is, the same-domain-name links in the link data as the domain name of the homepage link can be determined as the same-domain-name links, and then the same-domain-name link set can be obtained.

[0053] In some practical applications, there are some "friend links" in a website which are different from the domain name of the website, and the "friend links" are not same-domain links; for example, at the end of the homepage of China News Network (chinanews.com), there are a series of links with other domain names which are different from "chinanews.com", and after excluding these links with other domain names, a same-domain link set of the homepage of the news website can be obtained.

[0054] In step S303, a first preset number of same-domain link sets are obtained by using the preset news extraction sliding window at the current extraction time; wherein the first preset number of same-domain link sets correspond to the current extraction time and extraction times before the current extraction time.

[0055] The first preset number can be the number of elements that can be processed by the news extraction sliding window at one time, that is, the first preset number can be the window size (window size) of the news extraction sliding window. In addition, assuming that the current time point is exactly an extraction time, the current extraction time can be understood as the current time.

[0056] For example, if the first preset number is 2, then two same-domain link sets of the current time and the time one hour before the current time can be obtained by using the news extraction sliding window at the current time.

[0057] In the processing of multiple continuous extraction times, the news extraction sliding window can be used to slide the multiple continuous same-domain link sets with a sliding step of 1; for example, assuming that the link data extraction is performed every 1 hour, then at 15:00 on June 25 (i.e., the current extraction time is 15:00 on June 25), the same-domain link set extracted at 14:00 on June 25 and the same-domain link set extracted at 15:00 on June 25 can be put into the news extraction sliding window; at the next extraction time, i.e., 16:00 on June 25 (i.e., the new current extraction time is 16:00 on June 25), the same-domain link set extracted at 15:00 on June 25 and the same-domain link set extracted at 16:00 on June 25 can be put into the news extraction sliding window, and so on. At each current extraction time, a total of two same-domain link sets of the current extraction time and the time one hour before the current extraction time can be obtained for subsequent processing.

[0058] In step S305, the new homepage news link of the current extraction time is obtained according to the first preset number of same-domain link sets.

[0059] In some embodiments, the first preset number can be 2; the first preset number of the same domain name link sets can include a current same domain name link set at a current extraction time and a previous same domain name link set at a previous time of the current extraction time. Based on this, step S305 can further include: determining a difference set between the current same domain name link set and the previous same domain name link set; and determining the newly added homepage news link at the extraction time according to the links in the difference set.

[0060] In this step, the difference set between the two adjacent same domain name link sets can be calculated, and the newly added homepage news link can be determined according to the difference set and output. Wherein, the calculation of the difference set can be understood as follows: the difference set of set A and set B represents the set of elements in set A and not in set B. In this embodiment, the difference set between the current same domain name link set and the previous same domain name link set can be the links in the current same domain name link set and not in the previous same domain name link set. For example, the links in the previous same domain name link set are (a, b, c), and the links in the current same domain name link set are (a, c, d, e). Then, the link d and the link e are the links that exist in the current same domain name link set and do not exist in the previous same domain name link set. Therefore, it can be obtained that the link d and the link e are the newly added homepage news links at the current extraction time.

[0061] The difference set between the current same domain name link set and the previous same domain name link set can be considered as the links newly added in the news website homepage between the previous extraction time (the previous time of the above extraction time) and the current extraction time (the above extraction time). In the news website, the newly added same domain name link can be considered as the real-time newly added news link that can jump to the news page. Therefore, the newly added homepage news link at the extraction time can be determined according to the difference set.

[0062] Step S307 outputs the newly added homepage news link at the current extraction time as the homepage news link extraction result of the news website homepage.

[0063] The newly added homepage news link at the extraction time can be calculated and output at each extraction time, that is, the homepage news link extraction result of the news website homepage can be continuously output as the link data extraction according to the homepage link, and the homepage news link extraction result corresponds to each link data extraction, which is the newly added homepage news link obtained after each link data extraction.

[0064] In some actual applications, a time period for statistics can also be pre-set, and all the newly added homepage news links at the extraction time in the time period are collectively output as the homepage news link extraction result of the news website homepage.

[0065] For example, assuming that the extraction is performed every 6 hours, the first preset number is 2, and the time period for statistics is 7 days, then there are 28 extraction time points in the 7 days, and a same-domain link set is extracted at each of the 28 extraction time points; assuming that the first preset number is 2, then 27 difference sets can be calculated for the 28 extraction time points, that is, except for the first extraction time point in the 7 days, there is no new homepage news link, and each of the remaining 27 extraction time points corresponds to a calculated result of the new homepage news link, and then the union of the 27 calculated results of the new homepage news link can obtain the homepage news link extraction result of the homepage of the news website in the 7 days and output.

[0066] It can be seen that, by the method in the embodiment of the disclosure, the new homepage news link of each extraction time point can be automatically and efficiently obtained, so that the homepage news link extraction result of the homepage of the news website can be continuously output.

[0067] In some embodiments, after obtaining the same-domain link set of each extraction time point, the homepage non-news link extraction result of the homepage of the news website can be further obtained according to the same-domain link set of each extraction time point.

[0068] Figure 4 A flowchart of obtaining the homepage non-news link extraction result in the news link extraction method of one embodiment of the disclosure is shown. As shown in Figure 4 In some embodiments, the step S203 of "performing link extraction according to the homepage link to obtain the homepage non-news link extraction result of the homepage of the news website" can include the following steps.

[0069] Step S401, determine a plurality of extraction time points based on a preset extraction frequency, and obtain a same-domain link set of the homepage of the news website at each extraction time point according to the homepage link.

[0070] The execution mode of this step is similar to step S301; or, this step and step S301 can be the same step, that is, this step can reuse the result obtained in step S301 (i.e., the same-domain link set of the homepage of the news website at each extraction time point) to perform the subsequent steps.

[0071] Step S403, at the current extraction time point, a second preset number of same-domain link sets are obtained by using a preset non-news extraction sliding window; wherein the second preset number of same-domain link sets correspond to the current extraction time point and the extraction time points before the current extraction time point.

[0072] The second preset number can be a window size of a non-news extraction sliding window. In some embodiments, the second preset number can be greater than the first preset number. For example, the first preset number is assumed to be 2, and the second preset number can be 32, 36, 40, or the like. Taking the example of extracting the same-domain link set every 6 hours, the second preset number of 36 means that all same-domain link sets at extraction time points (36 extraction time points) in the past 9 days are processed once.

[0073] In step S405, the homepage non-news link at the current extraction time point is obtained according to the second preset number of same-domain link sets.

[0074] In some embodiments, step S403 can further include: determining the intersection of the second preset number of same-domain link sets; and determining the homepage non-news link at the current extraction time point according to the link in the intersection.

[0075] The link obtained by taking the intersection of the second preset number (for example, 36) of same-domain link sets can be considered as a same-domain link that has been continuously present on the homepage of the news website in the time period covered by the second preset number of extraction time points. Obviously, the update speed of such a same-domain link that has not been removed is relatively low and does not have the real-time nature of news, and therefore it can be considered as a homepage non-news link in the homepage of the news website.

[0076] In step S407, the homepage non-news link at the current extraction time point is output as the homepage non-news link extraction result of the homepage of the news website.

[0077] The homepage non-news link at each extraction time point can be calculated and output. That is, as the link data is extracted according to the homepage link, the homepage non-news link extraction result of the homepage of the news website can be continuously output, and the homepage non-news link extraction result corresponds to each link data extraction and is the homepage non-news link obtained after each link data extraction.

[0078] As can be seen, through the method in the embodiments of the present disclosure, the homepage non-news link determined at each extraction time point can be automatically and efficiently obtained, so that the homepage non-news link extraction result of the homepage of the news website can be continuously output.

[0079] Figure 5 A flowchart of obtaining the homepage news link update speed in the news link extraction method of one embodiment of the present disclosure is shown. As shown in Figure 5 In some embodiments, step S203 of "extracting link data according to the homepage link to obtain the homepage news link update speed of the homepage of the news website" can include the following steps.

[0080] Step S501, a preset time period is obtained, and a total number of extraction times of extraction time points in the preset time period is determined.

[0081] The preset time period can be configured based on actual needs or can be a fixed configuration value, and the present disclosure does not limit this. For example, the preset time period can be the total running time of the system configured with the method, or can be a fixed 180 days, 60 days, etc.

[0082] For example, in the case of extraction every 6 hours, assuming that the system has been running for 180 days, the total number of extractions is 720; and as the system runs, when the time reaches the next extraction time, new link data extraction is performed, so the total number of extractions can be constantly increasing over time.

[0083] Step S503, determining a total number of new homepage news links based on new homepage news links of all extraction time points in the preset time period.

[0084] Step S505, determining a homepage news link update speed of the homepage of the news website in the preset time period based on the total number of new homepage news links and the total number of extractions.

[0085] After the preset time period is determined, the average number of new homepage news links in this period can be calculated and output as the homepage news link update speed. That is, as the link data extraction based on the homepage link is performed, the homepage news link update speed of the homepage of the news website can be continuously output.

[0086] Step S205, determining whether a news panel page exists in the homepage of the news website based on the homepage non-news link extraction result and the homepage news link update speed.

[0087] The links in the homepage non-news link extraction result can be considered as infrequently updated links. In actual applications, they can be news panel links with frequently updated page data, or other links (such as announcement page links) with infrequently updated page data. Therefore, the homepage non-news link extraction result can be determined to determine whether a news panel page exists in the homepage of the news website.

[0088] In some actual applications, a news link extractor can be set to perform link data extraction based on the homepage link to obtain the homepage news link extraction result, the homepage non-news link extraction result, and the homepage news link update speed of the homepage of the news website.

[0089] Figure 6 A flowchart of determining whether a news panel page exists in the homepage of the news website in the news link extraction method of one embodiment of the present disclosure is shown. As shown in FIG. 5, the method comprises the following steps. Figure 6As shown, in some embodiments, step S205 can include the following steps.

[0090] Step S601, according to each homepage non-news link in the homepage non-news link extraction result, link data extraction is carried out, and the subpage news update speed of each homepage non-news link pointed to subpage is obtained.

[0091] Among them, the way to "obtain the subpage news update speed of each homepage non-news link pointed to subpage" can be similar to the way to "obtain the homepage news link update speed"; or the news link extractor can output the subpage news update speed of each subpage, which will not be repeated here.

[0092] Step S603, if there is a target subpage in the subpage news update speed and the ratio of the homepage news link update speed is greater than the update threshold, it is determined that there is a news board page in the homepage of the news website, and the target subpage is determined as the news board page.

[0093] Among them, the update threshold can be set based on actual situation or actual demand, or can be obtained based on machine learning labeling, or can be obtained based on known homepage news link update speed and news board page news link update speed statistical calculation, and the present disclosure does not limit it. For example, first, according to a plurality of news websites, the speed ratio of the subpage and the homepage news link update speed of each news website is calculated, and then it is labeled which of these subpages are news board pages and which are not, and then the limit value is determined as the update threshold.

[0094] The meaning of the update threshold can be: if it is higher than this value, it is considered that the update speed of the news link in the corresponding subpage has reached the level of the news link update speed, and the subpage should be considered as a news board page.

[0095] In some actual applications, the update threshold can be 10%, which can be the result of the comprehensive consideration of recall rate and precision rate in actual application, wherein the recall rate reflects whether the board is found completely or not, and the precision rate reflects whether the proportion of the actual news board page in the link determined as the news board page is high enough.

[0096] Step S207, in the case where there is a news board page, the board page news link extraction result in the news board page is extracted.

[0097] Among them, the way to "extract the board page news link extraction result in the news board page" can be similar to the way to "obtain the homepage news link extraction result"; or the news link extractor can output the board page news link extraction result in the news board page, which will not be repeated here.

[0098] In some practical applications, the block link extractor can be configured to determine whether there is a news block page in the news website homepage, and automatically extract the block page news link extraction result if there is. Further, the link of the news block page can be automatically input into the news link extractor to output the block page news link extraction result in the news block page if there is.

[0099] In step S209, the website news link extraction result of the news website homepage is determined according to the homepage news link extraction result and the block page news link extraction result.

[0100] The homepage news link extraction result and the block page news link extraction result obtained can be automatically output as the final news link extraction result.

[0101] The news link extraction method provided by the present disclosure can first extract link data from the news website homepage to obtain the homepage news link extraction result, the homepage non-news link extraction result and the homepage news link update speed of the homepage, then further determine whether there is a news block page in the news website homepage according to the homepage non-news link extraction result and the homepage news link update speed, and extract the block page news link extraction result in the news block page if there is, and then output the homepage news link extraction result and the block page news link extraction result as the final website news link extraction result of the news website homepage. This method can automatically and efficiently obtain the newly added homepage news link at each extraction time, quickly determine whether there is a news block page in the news website homepage, and then automatically and efficiently obtain the block page news link extraction result in the news block page. It can be seen that this method can not only quickly obtain the news link in the news website homepage, but also can mine the news block page in the news website homepage and quickly obtain the news link in the news block page, so as to continuously and efficiently and comprehensively output the final website news link extraction result in the news website homepage.

[0102] In some practical applications, the news link extraction method provided by the present disclosure can be configured with two components, "news link extractor" and "block link extractor", and the two components can be used to realize the news link extraction method.

[0103] Figure 7 A schematic diagram of the news link extraction method of one embodiment of the present disclosure is shown. As shown in Figure 7 The news link extraction method of one embodiment of the present disclosure is shown. As shown in

[0104] The input of the news link extractor 701 can be a page link 700 (e.g., a homepage link, a block link) containing news links. The output of the news link extractor 701 can be news links (i.e., homepage news link extraction results), non-news links (i.e., homepage non-news link extraction results), and the update speed of news links (i.e., homepage news link update speed).

[0105] The input of the block link extractor 702 can be non-news links and the homepage news link update speed. The output of the block link extractor 702 can be the determined block link (i.e., the link of the news block page).

[0106] In the initial state, the homepage link of the news website (i.e., the homepage link of the homepage of the news website) can be obtained as the starting input of the entire news link extraction method process. The homepage link can be input to the news link extractor 701 first, and the homepage news link 703, the non-news link 704, and the news link update speed 705 on the homepage can be output.

[0107] The non-news link 704 and the news link update speed 705 can be input to the block link extractor 702. After the block link 706 is extracted, the block link can be input to the news link extractor 701 again, and the block page news link 707 in the page pointed to by the block link can be output from the news link extractor.

[0108] Finally, the homepage news link 703 and the block page news link 707 can be used as the final news website website news link extraction result 708.

[0109] Figure 8 A schematic diagram of a news link extractor in a news link extraction method of an embodiment of the present disclosure is shown. As shown in the figure, Figure 8 The news link extractor 800 can have one input parameter and three output parameters. The input parameter can be a page link 801 to be collected, such as a homepage link or a block page link, which points to a page containing a large number of news links. There are three output parameters. The first output parameter is a news link 802, which can be a component of the final website news link extraction result. The second output parameter is a non-news link 803. The third output parameter is a news link update speed 804.

[0110] The design of the news link extractor 800 requires two sliding windows, one is the news extraction sliding window 805, the sliding window size of the news extraction sliding window is 2, which can be used to extract news links, and the other is the non-news extraction sliding window 806, the sliding window size of the non-news extraction sliding window is a larger value β (such as 36), which can be used to extract non-news links. For the input page link 801, all links on the page can be collected every fixed time (such as 6 hours), and other domain name links are filtered out, and this result forms a sequence element (i.e. a same domain name link set).

[0111] As shown in Figure 8 , where M is the current time. The news extraction sliding window 805 is used to continuously slide backward with time, and the latest two sequence elements containing the current extraction time are processed. Specifically, the processing operation is to calculate the difference set between the two sequence elements, and the difference set is output as the newly added news link 802.

[0112] The non-news extraction sliding window 806 is also used to continuously slide backward with time, and the latest β sequence elements containing the current extraction time are processed. The intersection of all link sets in the non-news extraction sliding window 806 can be taken, and then the intersection is output as the non-news link 803.

[0113] In this embodiment, two variables are also maintained in the news link extractor 800, which are used to record the total number of news links collected by each page in a preset period (i.e. the total amount of newly added homepage news links) and the collection round number (i.e. the total extraction times). The total collection number divided by the collection round number can obtain the news link update speed 804 of the page to be output.

[0114] Figure 9 A schematic diagram of the block link extractor in the news link extraction method of one embodiment of the present disclosure is shown. As shown in Figure 9 , the block link extractor 900 can have two input parameters and one output parameter. Among them, the input parameters can be the non-news links of the homepage 901 and the news link update speed of the homepage 902, and the output parameter can be the block link 903. The running process is as follows:

[0115] All non-news links 901 are input into the news link extractor, and the page news link update speed 904 (i.e. the sub-page news update speed) of the page pointed to by each non-news link 901 is output by the news link extractor; the page news link update speed 904 is compared with the input news link update speed 902 of the homepage, and if the ratio result exceeds the update threshold (such as 10%), the page can be regarded as a block page, and the link of the page is output as the block link 903.

[0116] It should be noted that the above-described figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended for limiting purposes. It is easy to understand that the processes shown in the above-described figures do not indicate or limit the time sequence of the processes. In addition, it is also easy to understand that the processes can be executed synchronously or asynchronously, for example, in a plurality of modules.

[0117] Figure 10 A block diagram of the news link extraction device 1000 according to one embodiment of the present disclosure is shown; as shown, comprising: an acquisition module 1001 configured to acquire a homepage link of a homepage of a news website; a homepage link data extraction module 1002 configured to perform link data extraction according to the homepage link to obtain a homepage news link extraction result, a homepage non-news link extraction result, and a homepage news link update speed of the homepage of the news website; a judgment module 1003 configured to judge whether there is a news board page in the homepage of the news website according to the homepage non-news link extraction result and the homepage news link update speed; a board link extraction module 1004 configured to extract a board page news link extraction result in the news board page in the case where there is a news board page; and a determination module 1005 configured to determine a website news link extraction result of the homepage of the news website according to the homepage news link extraction result and the board page news link extraction result. Figure 10 The news link extraction device provided by the present disclosure can first perform link data extraction on the homepage of the news website to obtain a homepage news link extraction result, a homepage non-news link extraction result, and a homepage news link update speed of the homepage, then further judge whether there is a news board page in the homepage of the news website according to the homepage non-news link extraction result and the homepage news link update speed, and further extract a board page news link extraction result in the news board page in the case where there is a news board page, and then output the homepage news link extraction result and the board page news link extraction result as the final website news link extraction result of the homepage of the news website. The present method can automatically and efficiently obtain the newly added homepage news link at each extraction time, quickly judge whether there is a news board page in the homepage of the news website, and then automatically and efficiently obtain the board page news link extraction result in the news board page. It can be seen that the present method can not only quickly obtain the news link in the homepage of the news website, but also can mine the news board page in the homepage of the news website and quickly obtain the news link in the news board page, so as to continuously and efficiently and comprehensively output the website news link extraction result in the homepage of the news website.

[0118]

[0119] ​In some embodiments, the determining module 1003 determines whether the news website homepage contains a news block page according to the homepage non-news link extraction result and the homepage news link update speed, including: performing link data extraction on each homepage non-news link in the homepage non-news link extraction result to obtain a sub-page news update speed of a sub-page pointed to by each homepage non-news link; if there is a target sub-page in the sub-page whose sub-page news update speed and the homepage news link update speed have a ratio greater than an update threshold, it is determined that the news website homepage contains a news block page, and the target sub-page is determined as the news block page.

[0120] In some embodiments, the homepage link data extraction module 1002 performs link data extraction on the homepage link to obtain a homepage news link extraction result of the news website homepage, including: determining a plurality of extraction time points based on a preset extraction frequency, and obtaining a same-domain link set of the news website homepage at each extraction time point according to the homepage link; at the current extraction time point, a first preset number of same-domain link sets are obtained using a preset news extraction sliding window; wherein the first preset number of same-domain link sets correspond to the current extraction time point and the extraction time points before it; the newly added homepage news link at the current extraction time point is obtained according to the first preset number of same-domain link sets; and the newly added homepage news link at the current extraction time point is output as the homepage news link extraction result of the news website homepage.

[0121] In some embodiments, the first preset number is 2; the first preset number of same-domain link sets includes a current same-domain link set at the current extraction time point and a previous same-domain link set at the previous time point of the current extraction time point; wherein the homepage link data extraction module 1002 obtains the newly added homepage news link at the current extraction time point according to the first preset number of same-domain link sets, including: determining the difference set between the current same-domain link set and the previous same-domain link set; and determining the newly added homepage news link at the current extraction time point according to the links in the difference set.

[0122] In some embodiments, the homepage link data extraction module 1002 performs link extraction on the homepage link to obtain a homepage non-news link extraction result of the news website homepage, including: at the current extraction time point, a second preset number of same-domain link sets are obtained using a preset non-news extraction sliding window; wherein the second preset number of same-domain link sets correspond to the current extraction time point and the extraction time points before it; the homepage non-news link at the current extraction time point is obtained according to the second preset number of same-domain link sets; and the homepage non-news link at the current extraction time point is output as the homepage non-news link extraction result of the news website homepage.

[0123] In some embodiments, the second preset quantity is greater than the first preset quantity; and the homepage link data extraction module 1002 extracts the homepage non-news link at the current extraction time according to the second preset quantity of the same domain name link sets, including: determining the intersection of the second preset quantity of the same domain name link sets; and determining the homepage non-news link at the current extraction time according to the link in the intersection.

[0124] In some embodiments, the homepage link data extraction module 1002 extracts the link data according to the homepage link, and obtains the homepage news link update speed of the news website homepage, including: obtaining a preset time period containing the current extraction time, and determining the total extraction times of the extraction time in the preset time period; determining the total amount of the newly added homepage news link according to the newly added homepage news link at all extraction times in the preset time period; and determining the homepage news link update speed of the news website homepage in the preset time period according to the total amount of the newly added homepage news link and the total extraction times.

[0125] Figure 10 Other contents of the embodiments can refer to the above-mentioned other embodiments.

[0126] Those skilled in the art can understand that each aspect of the present application can be implemented as a system, a method or a program product. Therefore, each aspect of the present application can be specifically implemented as follows: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, which can be collectively referred to as "circuit", "module" or "system" here.

[0127] Figure 11 A structural block diagram of a news link extraction computer device in the embodiment of the present disclosure is shown. It should be noted that the electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0128] The electronic device 1100 according to this embodiment of the present application will be described below with reference to Figure 11 Figure 11 The electronic device 1100 shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0129] As Figure 11 shown, the electronic device 1100 is in the form of a general computing device. The components of the electronic device 1100 can include but are not limited to: the above-mentioned at least one processing unit 1110, the above-mentioned at least one storage unit 1120, and a bus 1130 connecting different system components (including the storage unit 1120 and the processing unit 1110).

[0130] ​The storage unit stores program codes which can be executed by the processing unit 1110, so that the processing unit 1110 performs the steps described in the above "Exemplary Methods" section according to various exemplary embodiments of the present application. For example, the processing unit 1110 can perform the method as shown in the above Figure 2

[0131] The storage unit 1120 can include a readable medium in the form of volatile storage such as a random access memory (RAM) 11201 and / or cache memory 11202, and also can include a non-volatile storage such as read-only memory (ROM) 11203.

[0132] The storage unit 1120 can further include program / utility 11204 having a set of programs / modules 11205, including an operating system, one or more application programs, other programs, and programmatic data, each or some combination thereof, which can include implementation of a network environment.

[0133] The bus 1130 can represent one or more of several types of bus structures, including a storage bus or bus controller, a peripheral bus, a graphics bus, a processor or local bus using any of a variety of bus architectures.

[0134] The electronic device 1100 can also communicate with one or more external devices 1200 such as a keyboard or pointing device, using one or more communication ports 1150. Communication can occur over the I / O interface 1150. Still yet, the electronic device 1100 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the Internet, through a network adapter 1160. As depicted, the network adapter 1160 communicates with the other components of the electronic device 1100 via the bus 1130. It should be understood that although not shown, other hardware and / or software components could be used in conjunction with the electronic device 1100. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0135] ​In the exemplary embodiments of the present disclosure, a computer readable storage medium is also provided, on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes program codes for causing a terminal device to perform the steps described in the above-mentioned "Exemplary Method" section of the present specification according to various exemplary embodiments of the present application when the program product is run on the terminal device.

[0136] The program product for implementing the above-mentioned method according to the embodiments of the present application can take a portable compact disc read-only memory (CD-ROM) and include program codes, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited to this, and in the present document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in conjunction with an instruction execution system, apparatus or device.

[0137] The program product can take any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0138] The computer readable signal medium can include a data signal propagated in a baseband or as a carrier wave in a propagated data signal, in which readable program codes are carried. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The readable signal medium can also be any readable medium other than the readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device.

[0139] The program codes contained on the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0140] The program code may be implemented in any of various ways, including procedure-based, object-oriented approaches, Java, C++, C# or other programming languages, etc. In an embodiment, the program code can be implemented in subsets of one or more programming languages. Additionally or alternatively, one or more separate programs may

[0141] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, such a division is not mandatory. Indeed, according to an embodiment of the disclosure, the features and functionalities of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functionalities of one module or unit described above can be further divided into embodied by multiple modules or units.

[0142] Furthermore, although the various steps of the methods of the disclosure are described in a particular order in the figures, this is not required or implied as to the order of execution of the steps, nor is it required that all of the steps be executed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, one step can be broken into multiple steps, etc.

[0143] From the above description of embodiments, those skilled in the art will readily perceive that the example embodiments described herein can be implemented by software and / or by hardware coupled with software. Accordingly, the technical solutions according to the embodiments of the disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.) or network, and includes a number of instructions to make a computing device (which can be a personal computer, server, mobile terminal, or network device, etc.) execute the methods according to the embodiments of the disclosure.

[0144] According to an aspect of the present disclosure, a computer program product or computer program including computer instructions stored in a computer-readable storage medium is provided. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method provided in various optional implementations of the above-described embodiments.

[0145] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure including those variations that are now deemed to fall within the general principles of the present disclosure and including those variations that are deemed to fall within the patentably distinct field of this technology. The specification and examples are to be considered exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A method of extracting news links, characterized by, The method comprises the following steps: obtaining a homepage link of a homepage of a news website; performing link data extraction according to the homepage link to obtain a homepage news link extraction result, a homepage non-news link extraction result and a homepage news link update speed of the homepage of the news website; judging whether a news board page exists in the homepage of the news website according to the homepage non-news link extraction result and the homepage news link update speed; in the case where the news board page exists, extracting a board page news link extraction result in the news board page; determining a website news link extraction result of the homepage of the news website according to the homepage news link extraction result and the board page news link extraction result; wherein, judging whether a news board page exists in the homepage of the news website according to the homepage non-news link extraction result and the homepage news link update speed comprises: performing link data extraction according to each homepage non-news link in the homepage non-news link extraction result to obtain a sub-page news update speed of a sub-page pointed to by each homepage non-news link; if a target sub-page exists in the sub-page, and a ratio of the sub-page news update speed of the target sub-page to the homepage news link update speed is greater than an update threshold, it is determined that the news board page exists in the homepage of the news website, and the target sub-page is determined as the news board page.

2. The method of claim 1, wherein, performing link data extraction according to the homepage link to obtain a homepage news link extraction result of the homepage of the news website, comprising: determining a plurality of extraction time points based on a preset extraction frequency, and obtaining a same-domain link set of the homepage of the news website at each extraction time point according to the homepage link; at a current extraction time point, a first preset number of same-domain link sets are obtained by using a preset news extraction sliding window; wherein the first preset number of same-domain link sets correspond to the current extraction time point and extraction time points before the current extraction time point; new homepage news links at the current extraction time point are obtained according to the first preset number of same-domain link sets; the new homepage news links at the current extraction time point are output as the homepage news link extraction result of the homepage of the news website.

3. The method of claim 2, wherein, the first preset number is 2; the first preset number of same-domain link sets comprises a current same-domain link set at the current extraction time point and a previous same-domain link set at a previous time point of the current extraction time point; wherein, obtaining the new homepage news links at the current extraction time point according to the first preset number of same-domain link sets comprises: determining a difference set between the current same-domain link set and the previous same-domain link set; determining the new homepage news links at the current extraction time point according to links in the difference set.

4. The method of claim 2, wherein, performing link extraction according to the homepage link to obtain a homepage non-news link extraction result of the homepage of the news website, comprising: at a current extraction time point, a second preset number of same-domain link sets are obtained by using a preset non-news extraction sliding window; wherein the second preset number of same-domain link sets correspond to the current extraction time point and extraction time points before the current extraction time point; obtaining the homepage non-news link at the current extraction time according to the second preset number of the same domain name link sets; outputting the homepage non-news link at the current extraction time as the homepage non-news link extraction result of the homepage of the news website.

5. The method of claim 4, wherein, The second preset number is greater than the first preset number. The method further includes: determining the intersection of the second preset number of the same domain name link sets; determining the homepage non-news link at the current extraction time according to the link in the intersection.

6. The method of claim 2, wherein, The method further includes: obtaining the homepage news link update speed of the homepage of the news website according to the homepage link, including: acquiring a preset time period containing the current extraction time and determining the total extraction times of extraction time within the preset time period; determining the total amount of newly added homepage news link according to the newly added homepage news link of all extraction time within the preset time period; 7. A news link extraction apparatus characterized by comprising: determining the homepage news link update speed of the homepage of the news website in the preset time period according to the total amount of newly added homepage news link and the total extraction times. The method further includes: an acquisition module, configured to acquire the homepage link of the homepage of the news website; a homepage link data extraction module, configured to extract link data according to the homepage link to obtain the homepage news link extraction result, the homepage non-news link extraction result and the homepage news link update speed of the homepage of the news website; a judgment module, configured to determine whether there is a news board page in the homepage of the news website according to the homepage non-news link extraction result and the homepage news link update speed; a board link extraction module, configured to extract the board page news link extraction result in the news board page if there is a news board page; a determination module, configured to determine the website news link extraction result of the homepage of the news website according to the homepage news link extraction result and the board page news link extraction result; The method further includes: extracting link data according to each homepage non-news link in the homepage non-news link extraction result to obtain the sub-page news update speed of the sub-page pointed by each homepage non-news link; if there is a target sub-page in the sub-page with a sub-page news update speed and the ratio of the sub-page news update speed to the homepage news link update speed is greater than an update threshold, it is determined that there is a news board page in the homepage of the news website, and the target sub-page is determined as the news board page.

9. An electronic device, comprising:

8. A computer readable storage medium having stored thereon a computer program, the program, when executed by a processor, implementing the method for extracting news link according to any one of claims 1 to 6. The method further includes: one or more processors; a storage device, configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method for extracting news link according to any one of claims 1 to 6.

Images