News information collection and analysis method and system, electronic equipment and storage medium

Abstract

The invention discloses a news information collection and analysis method and system, electronic equipment and a storage medium. The method comprises the following steps: downloading a page source code by utilizing a playwright framework; analyzing the page source code into a dom tree by utilizing lxml; extracting contents of an article title and an article url from the dom tree by using xpath; filtering the content of the article title and the article url to obtain a news title and a corresponding url list; asynchronously and concurrently loading the url list by using a browser cluster to obtain a corresponding page source code; and obtaining news detail data based on the page source code. According to the method, the node distance from the title is introduced to assist in judgment, and the text closer to the title is higher in confidence coefficient; and the farther the text is, the lower the confidence coefficient is.

Description

Technical Field

[0001] The present invention relates to the technical field of information collection and analysis, and in particular to a news information collection and analysis method, system, electronic equipment and storage medium. Background Art

[0002] In the field of public opinion detection, data collection from hundreds or even thousands of news websites is usually involved. The traditional data collection solution is that each website writes a set of rules to match the article URLs of the news list page and the various fields of the article details page. This solution usually requires developers to devote a lot of time and effort. To this end, it is necessary to design a highly versatile data collection solution to achieve the collection of a large number of news websites in a short period of time. The difficulty and core of the design of a general news crawler framework mainly lie in two points: extraction of the list of articles on the list page; extraction of the content of the article on the news details page. In the existing technical solutions, for the extraction of the URL of the article list page, the URL is mainly screened by keyword matching through a preset positive keyword list and a negative keyword list. For the extraction of the content of the article details page, common web page tags (such as , , <article>etc.) and their attributes (such as id, class) to determine whether the content contained in these tags is the main text. It uses some heuristic rules and keywords (such as "article", "content", "body", etc.) to identify tags that may contain the main text. The defects of the existing technology are: the news URL extraction of the news list page is not comprehensive enough. For the extraction of the list of articles on the list page, only the URLs with these keywords can be matched through keyword matching, but when the article URLs of some websites do not contain these keywords, there will be no match. And through comparative observation, there are differences between the article URL generation rules of foreign websites and domestic websites, so the matching rules cannot be universal; the news body content extraction is not comprehensive enough. If the naming of the body tag attributes of some websites is unconventional, the correct body position may not be located; the acquisition of the content field of the news details page is not comprehensive enough. In the current collection scheme for parsing news, the website logo field is not extracted. In public opinion monitoring, the website logo has a positive effect on displaying the data source. The URL of the current collection solution is usually the original URL of the list page (when some news websites display the content of other news websites, the list page only displays the URL of the current site). When it is necessary to collect the news URL after the jump, it cannot meet the needs; the collection speed is slow, and the background technology usually uses a synchronous interface to parse page data. In the face of large-scale collection, it cannot meet the timeliness requirements. Summary of the invention

[0003] In view of the shortcomings existing in the above problems, the present invention provides a news information collection and analysis method, system, electronic device and storage medium.

[0004] To achieve the above object, the present invention provides a news information collection and analysis method, comprising:

[0005] Use the playwright framework to download the page source code;

[0006] Use lxml to parse the page source code into a DOM tree;

[0007] Extract the content of article title and article URL from the DOM tree using XPath;

[0008] Filter the content of the article title and article URL to obtain a list of news titles and corresponding URLs;

[0009] Use the browser cluster to asynchronously and concurrently load the URL list to obtain the corresponding page source code;

[0010] The news detail data is obtained based on the page source code.

[0011] Preferably, filtering the content of the article title and article URL includes:

[0012] Filter by limiting the minimum title length;

[0013] Filter by URL keywords;

[0014] Match the title by connecting it with a url dash;

[0015] Match by URL fixed length article ID.

[0016] Preferably, the news details data includes news website logo, news redirection URL, news title, news release time, news pictures and videos, news website name and website domain name and news text.

[0017] Preferably, the news text obtaining method comprises:

[0018] Calculate the unit distance between each text and the title text;

[0019] Calculate unit text length;

[0020] The news text is obtained based on the unit distance between each text and the title text and the unit text length.

[0021] Preferably, the unit distance between each text and the title text is calculated using the formula:

[0022]

[0023] In the formula: Index is the text index; MaxLength is the list length;

[0024] Wherein, the list length is obtained in the following steps:

[0025] Find the content of the article title;

[0026] Using the lxml library, extract the text content of the title tag, h1 tag, p tag, and div tag from the article title content;

[0027] Eliminate the empty text content of the title tag, h1 tag, p tag, and div tag, sort out the text list in page order, and split the text list into a new list after the title text, and the length of the new list is MaxLength.

[0028] Preferably, the formula for calculating the unit text length is:

[0029]

[0030] Wherein: MaxStringLength is the longest text length in the new list; Length is the length of the text obtained by traversing the new list.

[0031] Preferably, the news text is obtained based on a formula for evaluating text relevance, which is:

[0032] Score=UnitLength×0.7+UnitDistance×0.3.

[0033] The present invention also provides a news information collection and analysis system, comprising:

[0034] Download module, used to download page source code using the playwright framework;

[0035] A parsing module, used to parse the page source code into a DOM tree using lxml;

[0036] An extraction module is used to extract the content of the article title and article URL from the DOM tree using XPath;

[0037] A filtering module is used to filter the content of the article title and article URL to obtain a list of news titles and corresponding URLs;

[0038] A loading module is used to use a browser cluster to asynchronously and concurrently load the URL list to obtain the corresponding page source code;

[0039] The acquisition module is used to obtain news detail data based on the page source code.

[0040] The present invention also provides an electronic device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0041] The present invention also provides a storage medium storing a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device executes the above method.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention summarizes the page rules of news details pages of domestic and foreign news websites, and selects two dimensions, text length and distance from the title, to determine whether the text belongs to the main content. Under normal circumstances, in the news details page, the longest text is the main content of the data article, but for articles with shorter lengths, it may be disturbed by various website descriptions at the bottom of the page, and the text with a longer length below is used as the main content of the article. In the face of this problem, the present invention introduces the node distance from the title to assist in the judgment. The closer the text is to the title, the higher the confidence; the farther the text is, the lower the confidence. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the news information collection and analysis method of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Reference Figure 1 The present invention provides a news information collection and analysis method, comprising:

[0047] Use the playwright framework to download the page source code;

[0048] Use lxml to parse the page source code into a DOM tree;

[0049] Use xpath to extract the content of article title and article url from dom tree;

[0050] Filter the content of article titles and article URLs to obtain news titles and corresponding URL lists;

[0051] Use the browser cluster to asynchronously and concurrently load the URL list to obtain the corresponding page source code;

[0052] Get news details data based on page source code.

[0053] In this embodiment, the playwright automated browser technology is used to load the web page and obtain the complete list page source code. The article title and article URL are filtered out through multiple rule matching schemes such as preset keyword matching, horizontal bar link word matching, fixed length article ID matching (core), and article title length rule.

[0054] Specifically, through the input list page URL, this project first downloads the page source code through the playwright framework. Compared with the requests library requesting a single API download, this method avoids the situation where data such as article URLs are returned through another Ajax interface request, resulting in missing data in the downloaded web page source code. After obtaining the complete HTML page source code, this project uses the lxml library to parse HTML into a DOM tree, making it convenient to use xpath to extract the content containing article titles and article URLs. By limiting the minimum length of the article title, a batch of possible article URLs can be obtained, and then filtered using the following method:

[0055] Title filtering rules

[0056] Parse the HTML page source code into a DOM structure through lxml, then match all the a tags of the article, obtain their href attributes and text content, and treat them as candidate news titles and news URLs respectively. By limiting the minimum length of the title and the negative keyword list, the unqualified titles and corresponding news URLs can be filtered out.

[0057] URL keyword rule matching

[0058] By summarizing the article URL routing rules of news websites, a group of positive keywords are preset, such as " / article / ", " / read / ", etc. If the URL contains positive keywords, it will be filtered through this part.

[0059] URL bar connection title rule matching

[0060] For some foreign news websites, the routing naming rule of linking words with a horizontal bar is often used, for example

[0061] https: / / www.sample.com / culture / article / 20240820-why-people-are-upset-abou t-the-surprise-cameo-in-alien-romulus (the website domain has been anonymized). This type of link usually has more than two hyphens at the end. The hyphen link word rule will match this type of link.

[0062] URL fixed length article id matching

[0063] In most websites, an id generation algorithm is usually used to generate an article id, and then the generated id is concatenated to the URL, and usually the length of the generated id is fixed. In this case, the fixed-length article id rule can be used for matching.

[0064] First, extract all the href attributes of the a tag and their text content on the list page. Then extract the path attribute of the URL through the urllib library, split each URL with "\", get the last part of the split result, remove the content after '.com' and '.cn', and mark it as the ID string to be verified.

[0065] At this time, there will be many single words like index in the id string, as well as strings connected by hyphens like atricle-109092561. Split the hyphens using the spilt method to further get the id string list. The Python library spellchecker can be used to filter out words such as index and article. Such normally spelled words are generally not used as article ids. Filter out the spelled parts of the words, leaving the strings with abnormal word spellings.

[0066] Most of the strings that pass the filter are regular article IDs. At this time, the length of each ID string is calculated to count the length of the ID string that appears the most times. The length that appears the most times is used as the length of the last filtered ID, and then the URL corresponding to this characteristic ID is found and listed as the news article URL.

[0067] In this embodiment, the news details data includes news website logo, news redirection URL, news title, news release time, news pictures and videos, news website name and website domain name and news text.

[0068] Specifically, the present invention combines asyncio with the asynchronous interface async_playwright of the playwright library, inserts the news URL into a queue, uses a browser cluster to asynchronously and concurrently load news web pages, download page data, and perform data parsing, thereby improving the news data parsing speed.

[0069] Get news redirect url

[0070] By automatically loading a web page, you can get the page URL after the page is loaded. If the page is redirected, you will get the URL after the redirection.

[0071] Get news website logo

[0072] The present invention does not adopt the conventional practice of extracting the logo from the page source code (the website logo link in the ico format is specified in the HTML of some news websites), but saves the website logo in the ico format by intercepting the network when the automatic browser loads the news URL.

[0073] Get news headlines

[0074] By setting regular expression rules, extract article titles from meta tags, title tags, and h1 tags according to priority. The title tag may contain redundant information such as website name and news classification. If the title tag text contains h1 text, h1 can be used to remove it and retain the correct title information.

[0075] Get news release time

[0076] Firstly extract from the web page metadata, match the tags containing keywords such as datetime, pubdate, publishdate, etc. in the meta tag, and obtain the tag text content. Then use the datetime module to convert the irregular time into the regular time of "%Y-%m-%d%H:%M:%S".

[0077] Get news content

[0078] Based on the text length and node distance, we designed a scoring algorithm based on unit text length and unit node distance, and followed the principle that the probability of the selected text being the main content is positively correlated with the text length and negatively correlated with the distance from the title node, to screen out the text that is most likely to be the main content. Then, we search for the previous level div node through this text, obtain all the text under this node, and concatenate them as the main content.

[0079] Get news photos and videos

[0080] After locating the node of the news content, you can use lxml's xpath method to get all the nodes under this node. Tags, and all <video>Tag, get the src attribute of the tag as the picture link and video link of the news. When the link lacks the network protocol and domain name, you need to use the urljoin method of urllib to splice the news URL to the media link.

[0081] Get the name of the news website and its domain name

[0082] For website names, priority is given to <meta> The tag content with the tag matching attribute value of og:site_name; if no match is found, the text content of the itle tag after the title text is searched, and the text is cut using identifiers such as "_", "|", and "-" to filter out the website name. For the website domain name, the urllib library can be used to parse the website URL to obtain its hostname attribute. Some websites do not define the URL of the web page logo in the HTML source code, so the logo cannot be obtained by conventionally parsing the HTML page source code. Therefore, the present invention adopts the method of intercepting network requests to obtain the URL of the logo. The logo field displayed on the web page in the browser tab bar is usually a URL with the suffix '.ico'. The website logo can be downloaded by obtaining this URL. This solution uses Playwright to automate the route method of the page object of the browser, and defines a callback function to intercept the network request corresponding to the URL ending with ".ico" in the initial stage of loading the news URL. During the loading process of the website, the URL of this website logo is obtained.

[0083] In this embodiment, the paragraphs of the article are usually contained in div or p tags. The main idea of ​​this method is to first locate the tag where the longest paragraph is located, and then find the tag of the previous level corresponding to this tag. However, sometimes there are long texts that are not part of the main content on the page, such as the website statement at the bottom of the website. At this time, the node unit distance is needed to assist in verification.

[0084] Unit node distance calculation

[0085] Find the article title content

[0086] First, search for the h1 tag content and determine whether the h1 tag text content exists in the title tag (the title often contains the complete title content, but will also include the website name, news category, etc.). If the title tag content does not contain the h1 tag content, the title tag content is used as the title content.

[0087] Extract an ordered list of page text

[0088] Using the lxml library, extract all the text contents of title, h1, p, div and other tags, filter out the content with empty text information, and sort out the text list in page order.

[0089] Calculate the unit distance between each text and the title text

[0090] Traverse the list by the index of the list, and if the text of the title is matched, split the list after the title text into a new list (excluding the title) as the candidate list (PreliminaryList). Get the length of the new list, record it as MaxLength, traverse the new list, get the text index Index, then the unit node distance is

[0091] Unit text length calculation

[0092] Get the longest text length in the list to be selected, and record it as MaxStringLength. Traverse the list to be selected and get the text length. The unit text length is

[0093]

[0094] For positioning text content, it is positively correlated with the unit text length (UnitLength) and negatively correlated with the unit node distance (UnitDistance). Based on experience, the former is set to 0.7 and the latter is set to 0.3, thus deriving the formula for evaluating text relevance (Score):

[0095] Score=UnitLength×0.7+UnitDistance×0.3

[0096] The score obtained based on the formula is compared with the threshold to determine whether it is the main text.

[0097] In addition to the conventional use of keywords to filter news URLs, the present invention summarizes the news URLs of domestic and foreign news websites, adds a news title link rule matching method, and matches the news URLs generated by the horizontal bar link title. In addition, for the URLs with obvious ID features in the news URLs, the fixed length ID matching method is used, which has a good matching effect. In addition to matching the URL, the news URL is also assisted by the corresponding article title. Through a variety of matching rules, it has the advantages of good versatility and high accuracy; the present invention summarizes the page rules of the news details page of domestic and foreign news websites, and selects two dimensions, text length and distance from the title, to determine whether the text belongs to the body content. Under normal circumstances, in the news details page, the longest text is the body content of the data article, but for articles with shorter lengths, it may be interfered by various website descriptions at the bottom of the page, and the text with a longer text length below is used as the body content of the article. In the face of this problem, the present invention introduces the node distance from the title to assist in judgment. The closer the text is to the title, the higher the confidence; the farther the text is, the lower the confidence; the present invention obtains news information beyond the scope of simple page parsing. In the process of loading the complete news page source code from the automated browser, the characteristics of the playwright library are used to obtain the article URL after redirection, and the website icon data is collected by network interception; the present invention implements browser clustering through the asynchronous interface async_playwright of playwright, and concurrently downloads the page source code. In combination with the asynchronous framework asyncio, concurrent parsing of news data is realized, shortening the collection time.

[0098] Example 1

[0099] List page collection of a foreign mainstream news website

[0100] By entering the list page URL (https: / / www.sample1.com / , the website domain name has been anonymized), the following 52 news data were collected. Table 1 shows some of the data:

[0101] Table 1

[0102]

[0103] Example 2

[0104] Collection of details of a foreign mainstream news page

[0105] Collect news from the news list page

[0106] url(https: / / www.sample1.com / news / articles / cd9dj4vdx2jo), get the sample data in Table 2:

[0107] Table 2

[0108]

[0109]

[0110] Example 3

[0111] List page collection of a domestic mainstream website

[0112] Enter the list page URL (https: / / www.sample2.com / , the website domain name has been anonymized), and collect 135 news data. Table 3 shows some of the data:

[0113] Table 3

[0114]

[0115] Example 4

[0116] Collection of details pages of a domestic mainstream news website

[0117] Collect news from the news list page

[0118] url(https: / / www.sample2.com.cn / gj / 2024 / 08-21 / 10272614.shtml), get the sample data in Table 4:

[0119] Table 4

[0120]

[0121]

[0122] The present invention also provides a news information collection and analysis system, comprising:

[0123] Download module, used to download page source code using the playwright framework;

[0124] Parsing module, used to parse the page source code into a DOM tree using lxml;

[0125] Extraction module, used to extract the content of article title and article URL from DOM tree using XPath;

[0126] The filtering module is used to filter the content of article titles and article URLs to obtain news titles and corresponding URL lists;

[0127] The loading module is used to use the browser cluster to asynchronously and concurrently load the URL list and obtain the corresponding page source code;

[0128] The acquisition module is used to obtain news detail data based on the page source code.

[0129] The present invention also provides an electronic device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0130] The present invention also provides a storage medium storing a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the above method.

[0131] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.< / video> < / article>

Claims

1. A news information collection and analysis method, characterized in that: include: Use the playwright framework to download the page source code; Use lxml to parse the page source code into a DOM tree; Extract the content of article title and article URL from the DOM tree using XPath; Filter the content of the article title and article URL to obtain a list of news titles and corresponding URLs; Use the browser cluster to asynchronously and concurrently load the URL list to obtain the corresponding page source code; The news detail data is obtained based on the page source code.

2. The news information collection and analysis method according to claim 1 is characterized in that: Filtering the content of the article title and article URL includes: Filter by limiting the minimum title length; Filter by URL keywords; Match the title by connecting it with a url dash; Match by URL fixed length article ID.

3. The news information collection and analysis method according to claim 2 is characterized in that: The news details data includes news website logo, news redirection URL, news title, news release time, news pictures and videos, news website name and website domain name and news text.

4. The news information collection and analysis method according to claim 3 is characterized in that: The news text obtaining method comprises: Calculate the unit distance between each text and the title text; Calculate unit text length; The news text is obtained based on the unit distance between each text and the title text and the unit text length.

5. The news information collection and analysis method according to claim 4 is characterized in that: Calculate the unit distance between each text and the title text. The formula is: In the formula: Index is the text index; MaxLength is the list length; Wherein, the list length is obtained in the following steps: Find the content of the article title; Using the lxml library, extract the text content of the title tag, h1 tag, p tag, and div tag from the article title content; Eliminate the empty text content of the title tag, h1 tag, p tag, and div tag, sort out the text list in page order, and split the text list into a new list after the title text, and the length of the new list is MaxLength.

6. The news information collection and analysis method according to claim 5 is characterized in that: The formula for calculating the unit text length is: Wherein: MaxStringLength is the longest text length in the new list; Length is the length of the text obtained by traversing the new list.

7. The news information collection and analysis method according to claim 6 is characterized in that: The news body is obtained based on the formula for evaluating the body relevance, which is: Score=UnitLength×0.7+UnitDistance×0.

3.

8. A news information collection and analysis system, characterized in that: include: Download module, used to download page source code using the playwright framework; A parsing module, used to parse the page source code into a DOM tree using lxml; An extraction module is used to extract the content of the article title and article URL from the DOM tree using XPath; A filtering module is used to filter the content of the article title and article URL to obtain a list of news titles and corresponding URLs; A loading module is used to use a browser cluster to asynchronously and concurrently load the URL list to obtain the corresponding page source code; The acquisition module is used to obtain news detail data based on the page source code.

9. An electronic device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: It stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the method according to any one of claims 1 to 7.

Images