News site analysis method and device
By analyzing the link structure of the news site and the html page source code, news data is extracted, and news data is solved, and news data is obtained in the existing technology is achieved.
Patent Information
- Application Number
- CN202411045093.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-31
AI Technical Summary
The prior art has low accuracy when extracting news data and cannot quickly and effectively mine news data in the list of news sites.
By obtaining the site list of news websites, analyzing the link structure, extracting news links, and extracting news data, including the news body, release time and title, based on the html page source code and attribute information of the html node.
It improves the analysis accuracy of news data, can effectively extract news links and corresponding news data, and enhances the efficiency of news data acquisition.
Smart Images

Figure CN118916564A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of Internet technology, and in particular to a method and device for parsing a news site. Background Art
[0002] In the related art, when extracting news data, the news details page is usually parsed by using calibration templates, regular parsing, html tag screening, etc., and the news data in the news details page is extracted according to the parsing results, such as the news text, news title and news release time. However, in actual applications, the accuracy of the news data extracted by the above parsing methods is low, and it is not possible to quickly and effectively mine news data for the news site list. Summary of the invention
[0003] The present application provides a method and device for parsing news sites, which are used to solve the problem of low accuracy when extracting news data from news detail pages in related technologies, and the inability to quickly and effectively mine news data for news site lists.
[0004] To solve the above technical problems, this application is implemented as follows:
[0005] In a first aspect, a method for parsing a news site is provided, comprising:
[0006] Obtain a site list of a news website, wherein the site list includes a plurality of links;
[0007] Extracting news links from the multiple links according to the link structure of the multiple links;
[0008] According to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, the news data of the news link is extracted, and the news data includes at least one of the news text, news release time and news title.
[0009] In a second aspect, a news site parsing device is provided, comprising:
[0010] An acquisition module is used to acquire a site list of a news website, wherein the site list includes a plurality of links;
[0011] A link extraction module, extracting news links from the multiple links according to the link structure of the multiple links;
[0012] The data extraction module extracts the news data of the news link according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, wherein the news data includes at least one of the news text, news release time and news title.
[0013] According to a third aspect, an electronic device is provided, including:
[0014] processor;
[0015] a memory for storing instructions executable by the processor;
[0016] The processor is configured to execute the instructions to implement the method as described in the first aspect.
[0017] According to a fourth aspect, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method described in the first aspect.
[0018] According to a fifth aspect, a computer program product is provided, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the method described in the first aspect.
[0019] When parsing a news site, the embodiment of the present application can extract news links from multiple links in the news site according to the link structure of these links, and then extract news data of the news link, such as news text, news release time, news title, etc., by analyzing the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code. In this way, since the link structure of the news link is different from the link structure of other links, the news link can be effectively extracted from the news site by analyzing the link structure of multiple links in the news site; after the news link is extracted, since the news detail page can be analyzed according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, the news data corresponding to the news link can be effectively extracted, thereby improving the parsing accuracy of the news data. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0021] Figure 1 It is a flowchart of a method for parsing a news site in one embodiment of the present application;
[0022] Figure 2 It is a flowchart of a method for parsing a news site according to another embodiment of the present application;
[0023] Figure 3 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;
[0024] Figure 4 It is a structural diagram of a parsing device for a news site according to an embodiment of the present application. DETAILED DESCRIPTION
[0025] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in this application will be described clearly and completely below in conjunction with the drawings in one or more embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.
[0026] The terms "first", "second", etc. in the present application and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the data used in this way can be interchanged where appropriate so that the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in the present application and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.
[0027] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0028] Figure 1 It is a flowchart of a method for parsing a news site according to an embodiment of the present application. Figure 1 The parsing method of the shown news site is described below.
[0029] S102: Acquire a site list of a news website, where the site list includes multiple links.
[0030] When parsing news sites, you can get a site list of news sites. For example, a website contains a financial channel, a sports channel, etc. When getting the site list of the website, you can get a site list of the financial channel or a site list of the sports channel. The site list may include multiple links, which usually include news links and non-news links.
[0031] S104: extracting news links from the multiple links according to the link structures of the multiple links.
[0032] The multiple links in the site list usually include news links and non-news links. When parsing the news site, it is necessary to extract the news link from the multiple links in the site list. The embodiment of the present application finds that the link structure of the news link is usually different from the link structure of the non-news link by summarizing the rules of the news site. Therefore, when extracting the news link from the multiple links in the site list, the news link can be extracted according to the link structure of the multiple links.
[0033] In some implementations, extracting news links from multiple links according to the link structure of multiple links may include the following steps:
[0034] Extract multiple a-label links from multiple links;
[0035] Determine structural features of each of the a-label links according to the link structures of the a-label links, wherein the structural features include module length and module structured score;
[0036] According to the structural features of the multiple a-label links, news links are extracted from the multiple a-label links.
[0037] The a-label link is a hyperlink. Since news links are generally a-label links, when extracting news links from multiple links, the a-label link can be extracted first. When extracting the a-label link, all a-label links in the multiple links can be extracted to obtain multiple a-label links. Optionally, in order to facilitate the extraction of a-label links from multiple links, the html can be structured first, and then the a-label link can be extracted from the structured multiple links.
[0038] After obtaining multiple a-tag links, the structural features of each of the multiple a-tag links can be determined based on the link structure of the multiple a-tag links, so as to further extract news links from the multiple a-tag links based on the structural features. Optionally, before determining the structural features of each of the multiple a-tag links, the multiple a-tag links can be normalized to ensure the overall structural unity of the multiple a-link tags. For example, for a-tag links that do not start with https: / / , they can be automatically spliced into links with https: / / . After normalizing the multiple a-tag links, the structural features of each of the normalized a-tag links can be determined based on the link structure of the normalized a-tag links.
[0039] For each a-tag link, the structural features of the a-tag link may include but are not limited to the module length and module structured score of the a-tag link. The module length may represent the length of the a-tag link, and the module structured score may represent the length of each module (or each part) constituting the a-tag link. In some embodiments, according to the link structure of multiple a-tag links, determining the structural features of each of the multiple a-tag links may include the following steps:
[0040] For each a-label link, segment the a-label link according to the link structure of the a-label link and the specified rule to obtain a segmentation result, which includes the number of modules obtained by segmentation and the number of characters contained in each module;
[0041] Determine the module length of the a-label link according to the number of modules in the segmentation result;
[0042] The module structured score of the a-label link is determined according to the number of characters contained in each module in the segmentation result.
[0043] The a-label link is segmented according to the specified rules, and the a-label link can be segmented according to the specified symbol in the a-label link, and the specified symbol can be the symbol " / ". After the a-label link is segmented according to the specified rules, multiple modules and the number of characters contained in each module can be obtained. When determining the module length of the a-label link, the module length of the a-label link can be determined according to the number of modules obtained by segmentation. For example, the number of modules can be determined as the module length. When determining the module structured score of the a-label link, the module structured score of the a-label link can be determined according to the number of characters contained in each module obtained by segmentation. For example, the number of characters included in each module can be spliced to obtain the module structured score.
[0044] For example, taking the a-tag link https: / / www.123.com / as an example, when splitting the a-tag link, it can be split according to the symbol " / " to obtain 4 modules, namely https:, empty character, www.123.com and empty character. The number of characters contained in each module is 6, 0, 11 and 0 respectively. Therefore, it can be determined that the module length of the a-tag link https: / / www.123.com / is 4, and the module structured score is 60110.
[0045] It should be noted that the above-mentioned segmentation of the a-label link according to the symbol " / ", determining the number of modules obtained by the segmentation as the module length, and concatenating the number of characters included in each module to obtain the module structured score is only for exemplary purposes. In other possible implementations, the a-label link can also be segmented according to other symbols, and the module length and module structured score of the a-label link can be determined according to other methods, which will not be explained one by one here.
[0046] After obtaining the structural features (including module length and module structuring score) of each of the multiple a-tag links according to the above method, news links can be further extracted from the multiple a-tags according to the structural features of each of the multiple a-tag links.
[0047] In some implementations, extracting news links from a plurality of a-label links according to the structural features of each of the a-label links may include the following steps:
[0048] Determine a plurality of first links from the plurality of a-label links according to respective module lengths of the plurality of a-label links, wherein the module length of each first link is greater than a preset length and is inconsistent with the module length of the primary domain name;
[0049] According to the respective module structured scores of the plurality of first links, a plurality of news links are extracted from the plurality of first links, wherein the plurality of news links are links having the same module structured score and having the highest proportion among the plurality of first links.
[0050] Since the module length of a news link is generally not less than a preset length, and the module length of a news link is usually inconsistent with that of the main domain name, which is at least one level more than the main domain name, when extracting news links from multiple a-label links, you can first extract multiple links whose module lengths are greater than the preset length and inconsistent with the module length of the main domain name (greater than the module length of the main domain name) from multiple a-label links based on the module lengths of each of the multiple a-label links. Here, for ease of distinction, the extracted multiple links can be represented as multiple first links. Among them, the preset length can be obtained by summarizing the structural rules of news links. Optionally, the preset length can be set to 4.
[0051] For example, taking the preset length as 4, assuming that a tag link is https: / / www.123.com / , then, since its module length is equal to 4, it can be determined that the a tag link is not a news link and needs to be removed. Assuming that a tag link is https: / / www.123.com / xxxx / xxxxxxxx.html, then, since its module length is greater than 4 and is inconsistent with the length of the main domain name, the a tag link can be extracted as the first link.
[0052] After extracting multiple first links from multiple a-label links, news links can be further extracted from the multiple first links according to the module structured scores of the multiple first links. Since a news list page normally has the most news links, when extracting news links from multiple first links, the link with the same module structured score and the highest proportion among the multiple first links can be determined as the news link.
[0053] For example, assuming that the number of first links is 50, 30 of them have a module structured score of A, 10 have a module structured score of B, 5 have a module structured score of C, and 5 have a module structured score of D, then the 30 links with the largest proportion of module structured scores of A can be determined as news links.
[0054] S106: extracting news data of the news link according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, where the news data includes at least one of the news text, news release time and news title.
[0055] After extracting the news link, the news link can be downloaded to obtain the HTML page source code of the news link, which includes multiple levels of tags, each level of tags can correspond to an HTML node, and the HTML page source code can include multiple HTML nodes. After obtaining the HTML page source code of the news link, the news data of the news link can be extracted according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, and the news data includes but is not limited to at least one of the news text, news release time and news title. Among them, the attribute information of each HTML node can include at least one of the number of a tags, the number of words carried by a tags, the total number of node words, the number of node tags, and the attribute information of child nodes.
[0056] In some implementations, when the news data includes the news text, extracting the news data of the news link according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code may include the following steps:
[0057] Determine the label score of each HTML node according to the attribute information of each HTML node in the HTML page source code;
[0058] The text content of the node with the highest label score among all HTML nodes is determined as the news text.
[0059] The label score of the html node can represent the possibility that the html node belongs to the core area node of the news page, or the possibility that the html node belongs to the text node of the news. In this embodiment, the higher the label score of the html node, the greater the possibility that the html node belongs to the core area node of the news page (or the text node of the news). Among them, when determining the corresponding label score according to the attribute information of the html node, the implementation method may include multiple, which is not specifically limited here. Optionally, in some implementation methods, the attribute information of the html node can be calculated according to a specific formula to obtain the corresponding label score. The specific formula here can be a recursive formula for calculating text density. The higher the calculated text density, the higher the final label score.
[0060] After obtaining the label score of each HTML node in the HTML page source code, the HTML node with the highest label score can be determined as the core area node of the news page, that is, the text node of the news, and the text content of the node can be extracted as the news text.
[0061] In some implementations, when the news data includes the news release time, extracting the news data of the news link according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code may include the following steps:
[0062] Match the HTML page source code with multiple pre-set time tags;
[0063] If the match is successful, extract the news release time from the HTML page source code according to the matched time tag;
[0064] In case of a match failure, the page source code of the parent node of the text node corresponding to the news text is matched with multiple preset regular expressions, and the news release time is extracted from the HTML page source code according to the matched regular expressions.
[0065] The pre-set multiple time tags may be time tags corresponding to a regular or standard news page, and may be obtained by summarizing the time tags of the news page. When matching the HTML page source code with the pre-set multiple time tags, the multiple time tags may be matched in sequence according to a certain order (such as tag order).
[0066] After matching the HTML page source code with multiple time tags, if the match is successful, the corresponding time can be extracted from the HTML page source code according to the matched time tag, and the time can be determined as the news release time. If the match fails, that is, the HTML page source code does not match multiple time tags, it can be said that the news release time cannot be obtained through the time tag. At this time, in order to obtain the news release time, a regular matching method can be used.
[0067] When using regular matching to obtain news release time, since the html node corresponding to the news release time is usually the parent node of the news body node, the page source code of the parent node of the news body node can be matched with multiple preset regular expressions. For example, the page source code within the 6 parent nodes above the news body node can be matched with multiple preset regular expressions. If the match is successful, the news release time can be extracted from the html page source code according to the matched regular expression.
[0068] In this way, since the news release time can be extracted from the HTML page source code by using the time tag matching method first, and if the extraction fails, the news release time can be extracted from the HTML page source code by using the regular matching method, the news release time can be effectively extracted and the accuracy of the news release time can be improved. In addition, since the regular expression can be matched with the page source code corresponding to the parent node of the text node when performing regular matching, instead of matching with the entire page source code, the matching time can be shortened and the matching efficiency can be improved.
[0069] In some implementations, when the news data includes a news title, extracting the news data of the news link according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code may include the following steps:
[0070] Get multiple HTML titles according to the attribute information of each HTML node in the HTML page source code;
[0071] Determine the similarity between each of multiple HTML titles and the news body;
[0072] The HTML title with the highest similarity is determined as the news title.
[0073] When obtaining the HTML title according to the attribute information of each HTML node in the HTML page source code, considering that the text description of the news title is usually relatively brief, therefore, by analyzing the attribute information of each HTML node in the HTML page source code, the text content corresponding to the node with less text can be determined as the HTML title, thereby obtaining multiple HTML titles. Optionally, in some embodiments, considering that the HTML node corresponding to the news title is usually the parent node of the text node of the news, therefore, when obtaining the HTML title according to the attribute information of each HTML node in the HTML page source code, the HTML title can be obtained according to the page source code of the parent node of the text node of the news, for example, the HTML title can be obtained according to the page source code of the 6 parent nodes above the text node of the news, thereby shortening the time to obtain the HTML title and improving efficiency.
[0074] After obtaining multiple HTML titles, the news title can be determined based on the multiple HTML titles. When determining the news title based on the multiple HTML titles, specifically, one title can be selected from the multiple HTML titles as the news title. Among them, since the content similarity between the news title and the news body is high, therefore, when selecting a title from the multiple HTML titles as the news title, the content similarity between each HTML title and the news body (determined according to the method provided in the embodiment of the present application) can be calculated, and the HTML title with the highest similarity can be determined as the news title.
[0075] By parsing the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, news data such as the news text, news release time and news title corresponding to the news link can be finally obtained.
[0076] When parsing a news site, the embodiment of the present application can extract news links from multiple links in the news site according to the link structure of these links, and then extract news data of the news link, such as news text, news release time, news title, etc., by analyzing the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code. In this way, since the link structure of the news link is different from the link structure of other links, the news link can be effectively extracted from the news site by analyzing the link structure of multiple links in the news site; after the news link is extracted, since the news detail page can be analyzed according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, the news data corresponding to the news link can be effectively extracted, thereby improving the parsing accuracy of the news data.
[0077] In order to facilitate understanding of the news site parsing method provided in the embodiment of the present application, a more specific implementation method will be used as an example for explanation below. Figure 2 , Figure 2 The parsing method of the news site shown includes the following steps.
[0078] Input: Collect a site list, which includes multiple links.
[0079] Step 1: a tag link extraction.
[0080] You can structure the HTML and then extract all the a tag links.
[0081] Step 2: Link normalization.
[0082] After extracting multiple a-label links, the multiple a-label links can be normalized to ensure the overall structure of the a-label links is unified. For example, links that do not begin with https: / / can be automatically spliced into links with https: / / .
[0083] Step 3: Link structure calculation.
[0084] For the normalized a-label link, the a-label link can be split according to the symbol " / ", and then the module length and module structure score of each a-label link can be recorded. For example, for the a-label link https: / / www.123.com / , the module length can be recorded as 4 (the corresponding modules obtained by segmentation are https:, empty characters, www.123.com, empty characters, and the number of modules is 4), and the module structure score is 60110 (the number of characters contained in each module is 6, 0, 11, 0).
[0085] Step 4: Remove interfering links.
[0086] Remove a-label links with too short link structured module lengths. For example, remove a-label links with module lengths less than or equal to 4, because a link with at least 4 modules must be a news content link. For example, the a-label link https: / / www.123.com / mentioned above is no longer in the news content link format because its module length is 4, so it needs to be removed. In addition, it is also necessary to remove a-label links with the same module length as the main domain name. These are not news links either, because common news links have at least one more level than the main domain name. For example, the news link https: / / www.123.com / xxxx / xxxxxxxx.html has 2 more levels than the main domain name.
[0087] Step 5: Count the structural links with the highest scores.
[0088] Since a news list page normally has the most news links, the a-label links with the highest structural score ratio can be determined as news links. For example, after executing step 4, a total of 50 links are obtained, of which 30 links have a module structural score of A, 10 links have a module structural score of B, 5 links have a module structural score of C, and 5 links have a module structural score of D. Then, the 30 links with a module structural score of A can be determined as news links.
[0089] The above steps 1 to 5 are the automatic parsing process of the news list page.
[0090] Step 6: Parse several detail page links (news pages).
[0091] After extracting multiple news links according to step 5, the extracted news links can be downloaded to obtain the HTML page source code of each news article.
[0092] Step 7: Count the attribute information of each HTML node.
[0093] For example, you can count the number of a tags in each HTML node in the HTML page source code, the number of words in the a tag, the total number of node words, the number of node tags, and the attributes of child nodes.
[0094] Step 8: Calculate the score based on the attribute information.
[0095] According to the attribute information of each HTML node in step 7, the score of each HTML node can be calculated according to a specific formula.
[0096] Step 9: Get the node with the most appropriate score as the core node.
[0097] The higher the score of the html node, the higher the possibility that the html node is a node in the core area of the page, that is, the higher the possibility that the html node is a text node. Therefore, the node with the highest score can be determined as a node in the core area of the page, that is, a text node.
[0098] Step 10: Extract the content of the core node (news content).
[0099] After determining the core area node (text node) of the page according to step 9, the content text of the node can be extracted, and the content text is the news text.
[0100] Step 11: Extract information such as release time according to preset tags and regular expressions.
[0101] You can first match the HTML page source code with a preset time tag, and obtain news release time information based on the preset time tag. The preset time tag can be as follows:
[0102] You can match the acquisition time in order of tags:
[0103]
[0104]
[0105] If the time information cannot be obtained according to the above tags, regular matching can be used, specifically matching the regular expression with the page source code within the 6 parent nodes above the body node. The preset regular expression for matching time is as follows:
[0106] You can match the acquisition time in regular order:
[0107]
[0108]
[0109] Step 12: Get the html title.
[0110] Multiple HTML titles can be obtained based on the attribute information of each HTML node in the HTML page source code, and then the similarity between each of the multiple HTML titles and the news text (the news text determined in step 10) is determined, and the HTML title with the highest similarity is determined as the news title.
[0111] Step 13: Combine output data.
[0112] The extracted news title, news text, and news release time are combined into a JSON-structured news data and pushed to the downstream for processing.
[0113] The above steps 7 to 13 are the automatic parsing process of the news details page.
[0114] Output: News data.
[0115] The parsing method of news sites provided in the embodiment of the present application effectively extracts regular news web page links by summarizing the rules of news sites, and extracts important content of news data through the accumulated commonly used positioning tags and HTML text density algorithm (tags with a relatively large proportion of positioning text), so as to achieve more efficient data collection for regular Internet news sites and extract news data with higher accuracy. The embodiment of the present application can automatically extract news data links from news websites in multiple languages, and extract important fields from news pages corresponding to the data links. The extraction of time fields in multiple countries has been deeply upgraded and optimized to improve the parsing accuracy of time fields.
[0116] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0117] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.
[0118] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0119] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.
[0120] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a parsing device for the news site at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:
[0121] Obtain a site list of a news website, wherein the site list includes a plurality of links;
[0122] Extracting news links from the multiple links according to the link structure of the multiple links;
[0123] According to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, the news data of the news link is extracted, and the news data includes at least one of the news text, news release time and news title.
[0124] The above application Figure 3The method performed by the parsing device of the news site disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in this application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in this application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0125] The electronic device may also perform Figure 1 and Figure 2 Method, and realize the parsing device of the news site in Figure 1 and Figure 2 The functions in the illustrated embodiments will not be described in detail in this application.
[0126] Of course, in addition to software implementation, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0127] The present application also proposes a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 1 and Figure 2 The method of the embodiment shown is specifically used to perform the following operations:
[0128] Obtain a site list of a news website, wherein the site list includes a plurality of links;
[0129] Extracting news links from the multiple links according to the link structure of the multiple links;
[0130] According to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, the news data of the news link is extracted, and the news data includes at least one of the news text, news release time and news title.
[0131] Figure 4 This is a schematic diagram of the structure of a news site analysis device 40 according to an embodiment of the present application. Figure 4 In a software implementation, the news site parsing device 40 may include: an acquisition module 41, a link extraction module 42 and a data extraction module 43, wherein:
[0132] An acquisition module 41 acquires a site list of a news website, wherein the site list includes a plurality of links;
[0133] A link extraction module 42 extracts news links from the multiple links according to the link structure of the multiple links;
[0134] The data extraction module 43 extracts the news data of the news link according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, and the news data includes at least one of the news text, news release time and news title.
[0135] In some implementations, the link extraction module 42 extracts news links from the multiple links according to the link structure of the multiple links, including:
[0136] Extract multiple a-label links from the multiple links;
[0137] Determining structural features of each of the plurality of a-tag links according to the link structures of the plurality of a-tag links, wherein the structural features include module length and module structured score;
[0138] According to the structural features of each of the multiple a-label links, news links are extracted from the multiple a-label links.
[0139] In some implementations, the link extraction module 42 determines the structural features of each of the a-label links according to the link structures of the multiple a-label links, including:
[0140] For each a-label link, segment the a-label link according to the link structure of the a-label link and a specified rule to obtain a segmentation result, wherein the segmentation result includes the number of modules obtained by segmentation and the number of characters contained in each module;
[0141] Determine the module length of the a-tag link according to the number of modules in the segmentation result;
[0142] The module structured score of the a-label link is determined according to the number of characters contained in each module in the segmentation result.
[0143] In some implementations, the link extraction module 42 extracts news links from the multiple a-label links according to the structural features of the multiple a-label links, including:
[0144] Determine a plurality of first links from the plurality of a-label links according to respective module lengths of the plurality of a-label links, wherein the module length of each of the first links is greater than a preset length and is inconsistent with the module length of the primary domain name;
[0145] According to the respective module structured scores of the plurality of first links, a plurality of news links are extracted from the plurality of first links, wherein the plurality of news links are links having the same module structured score and having the highest proportion among the plurality of first links.
[0146] In some implementations, when the news data includes the news text, the data extraction module 43 extracts the news data of the news link according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, including:
[0147] Determine the tag score of each HTML node according to the attribute information of each HTML node in the HTML page source code;
[0148] The text content of the node with the highest tag score among the HTML nodes is determined as the news text.
[0149] In some implementations, when the news data includes the news release time, the data extraction module 43 extracts the news data of the news link according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, including:
[0150] Matching the HTML page source code with a plurality of preset time tags;
[0151] In case of successful matching, extracting the news release time from the HTML page source code according to the matched time tag;
[0152] In case of a match failure, the page source code of the parent node of the text node corresponding to the news text is matched with a plurality of preset regular expressions, and the news release time is extracted from the HTML page source code according to the matched regular expressions.
[0153] In some implementations, when the news data includes the news title, the data extraction module 43 extracts the news data of the news link according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, including:
[0154] Obtain multiple HTML titles according to the attribute information of each HTML node in the HTML page source code;
[0155] Determine the similarity between each of the plurality of HTML titles and the news body;
[0156] The HTML title with the highest similarity is determined as the news title.
[0157] The news site parsing device 40 provided in the embodiment of the present application can also execute Figure 1 and Figure 2 The method and the parsing device 40 of the news site are implemented in Figure 1 and Figure 2 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.
[0158] The present application also proposes a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute part or all of the steps in the above-mentioned news site parsing method embodiment.
[0159] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
[0160] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0161] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0162] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0163] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
Claims
1. A method for parsing a news site, characterized in that: include: Obtain a site list of a news website, wherein the site list includes a plurality of links; Extracting news links from the multiple links according to the link structure of the multiple links; According to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, the news data of the news link is extracted, and the news data includes at least one of the news text, news release time and news title.
2. The method according to claim 1, characterized in that Extracting news links from the multiple links according to the link structure of the multiple links includes: Extract multiple a-label links from the multiple links; Determining structural features of each of the plurality of a-tag links according to the link structures of the plurality of a-tag links, wherein the structural features include module length and module structured score; According to the structural features of each of the multiple a-label links, news links are extracted from the multiple a-label links.
3. The method according to claim 2, characterized in that Determining the structural features of each of the a-label links according to the link structures of the plurality of a-label links includes: For each a-label link, segment the a-label link according to the link structure of the a-label link and a specified rule to obtain a segmentation result, wherein the segmentation result includes the number of modules obtained by segmentation and the number of characters contained in each module; Determine the module length of the a-tag link according to the number of modules in the segmentation result; The module structured score of the a-label link is determined according to the number of characters contained in each module in the segmentation result.
4. The method according to claim 2, characterized in that The step of extracting news links from the plurality of a-label links according to the structural features of the plurality of a-label links respectively comprises: Determine a plurality of first links from the plurality of a-label links according to respective module lengths of the plurality of a-label links, wherein the module length of each of the first links is greater than a preset length and is inconsistent with the module length of the primary domain name; According to the respective module structured scores of the plurality of first links, a plurality of news links are extracted from the plurality of first links, wherein the plurality of news links are links having the same module structured score and having the highest proportion among the plurality of first links.
5. The method according to claim 1, characterized in that In the case where the news data includes the news text, extracting the news data of the news link according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code includes: Determine the tag score of each HTML node according to the attribute information of each HTML node in the HTML page source code; The text content of the node with the highest tag score among the HTML nodes is determined as the news text.
6. The method according to claim 1, characterized in that In the case where the news data includes the news release time, extracting the news data of the news link according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code includes: Matching the HTML page source code with a plurality of preset time tags; In case of successful matching, extracting the news release time from the HTML page source code according to the matched time tag; In case of a match failure, the page source code of the parent node of the text node corresponding to the news text is matched with a plurality of preset regular expressions, and the news release time is extracted from the HTML page source code according to the matched regular expressions.
7. The method according to claim 1, characterized in that In the case where the news data includes the news title, extracting the news data of the news link according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code includes: Obtain multiple HTML titles according to the attribute information of each HTML node in the HTML page source code; Determine the similarity between each of the plurality of HTML titles and the news body; The HTML title with the highest similarity is determined as the news title.
8. A news site parsing device, characterized in that: include: An acquisition module is used to acquire a site list of a news website, wherein the site list includes a plurality of links; A link extraction module, extracting news links from the multiple links according to the link structure of the multiple links; The data extraction module extracts the news data of the news link according to the HTML page source code of the news link and the attribute information of each HTML node in the HTML page source code, wherein the news data includes at least one of the news text, news release time and news title.
9. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
System and method for improved searching on the internet or similar networks and especially improved metanews and / or improved automatically generated newspapers
CA2444774A1
Method and system for extracting news record metadata from news list page
CN102467501A
News data crawling method and device, computer equipment and storage medium
CN108388614A
News webpage key information automatic extraction method based on labels and block features
CN109857956A
Web page classification based on noise removal
US20180025012A1
