Webpage content extraction method and electronic equipment

Abstract

The invention provides a webpage content extraction method and electronic device.The method comprises the steps that paragraph extraction is conducted on a webpage to be extracted, at least one page paragraph of the webpage to be extracted is obtained, whether non-label content in each page paragraph is text content or not is determined according to label content of each page paragraph, and if yes, the non-label content in each page paragraph is extracted. And if the page paragraph does not exist, extracting the text content, taking the page paragraph as a target text paragraph, and performing merging processing on the at least one target text paragraph according to the page layout information of the to-be-extracted webpage to obtain a text extraction result of the to-be-extracted webpage. By analyzing the webpage content and identifying the text content of the webpage based on the label in the webpage content, automatic text content extraction can be realized, and the method and the device do not need to delete image notes, links and the like in the webpage in advance, so that the labor cost can be reduced, and the method and the device have good universality for different styles of webpages.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a web page content extraction method and an electronic device. Background Art

[0002] In the process of network content operation, it is usually necessary to extract the main text content from the web page. Nowadays, the content styles of Internet media are becoming more and more diverse, and the complexity of web pages is also increasing day by day. How to achieve convenient and efficient web page text content extraction has become an urgent problem to be solved.

[0003] In the prior art, when extracting the body content of a web page, it is generally done manually. The operator manually removes the captions, links, and titles on the web page, retains the body content, and then extracts the body content of the web page through methods such as text recognition.

[0004] However, this method relies on manual operation by operators, which increases the work and maintenance costs of operators, and the extraction efficiency also needs to be improved urgently. Summary of the invention

[0005] The purpose of the present application is to provide a web page content extraction method and electronic device to address the deficiencies in the above-mentioned prior art, so as to solve the problems of high cost and low efficiency in web page text extraction in the prior art.

[0006] To achieve the above objectives, the technical solutions adopted in this application are as follows:

[0007] In a first aspect, the present application provides a method for extracting webpage content, the method comprising:

[0008] Performing paragraph extraction on the web page to be extracted to obtain at least one page paragraph of the web page to be extracted, each of the page paragraphs including at least one tag content and at least one non-tag content;

[0009] According to the tag contents of each of the page paragraphs, determining whether the non-tag contents in each of the page paragraphs are text contents, and if so, extracting the text contents and treating the page paragraph as a target text paragraph;

[0010] The at least one target text paragraph is merged according to the page layout information of the web page to be extracted to obtain a text extraction result of the web page to be extracted.

[0011] Optionally, the step of extracting paragraphs from the web page to be extracted to obtain at least one page paragraph of the web page to be extracted includes:

[0012] Using the network address of the web page to be extracted as an input parameter of the target conversion method, calling the target conversion method to convert the web page data into source code text;

[0013] A page element extraction method is called to extract at least one page paragraph from the source code text.

[0014] Optionally, determining whether the non-tag content in each of the page paragraphs is text content according to the tag content of each of the page paragraphs, and if so, extracting the text content and taking the page paragraph as a target text paragraph includes:

[0015] Filtering each of the page paragraphs according to the tag content of each of the page paragraphs to obtain at least one paragraph to be selected;

[0016] Determine whether the non-label content in each of the to-be-selected paragraphs is text content according to the non-label content in each of the to-be-selected paragraphs;

[0017] If so, the text content is extracted, and the to-be-selected paragraph is used as a target text paragraph.

[0018] Optionally, determining whether the non-label content in each of the to-be-selected paragraphs is text content according to the non-label content in each of the to-be-selected paragraphs includes:

[0019] According to the title tag, obtain the title text of the web page to be extracted;

[0020] Performing semantic analysis on the non-label content in the paragraph to be selected and the adjacent paragraphs of the paragraph to be selected to obtain a first semantic analysis result of the paragraph to be selected and a second semantic analysis result of the adjacent paragraphs;

[0021] Determine, according to the first semantic analysis result, the second semantic analysis result and the title text, a relevance result corresponding to the non-label content of the to-be-selected paragraph;

[0022] Determining the position of the to-be-selected paragraph in the to-be-extracted web page according to the tag content of the to-be-selected paragraph;

[0023] According to the relevance result and the position of the paragraph to be selected in the web page to be extracted, it is determined whether the non-label content in the paragraph to be selected is the main text content.

[0024] Optionally, determining the relevance result corresponding to the non-label content of the to-be-selected paragraph according to the first semantic analysis result, the second semantic analysis result and the title text includes:

[0025] Performing semantic relevance analysis on the first semantic analysis result and the second semantic analysis result to obtain a first relevance;

[0026] Performing semantic relevance analysis on the first semantic analysis result and the title text to obtain a second relevance;

[0027] A relevance result corresponding to the non-label content of the to-be-selected paragraph is obtained according to the first relevance and the second relevance.

[0028] Optionally, determining whether the non-label content in the to-be-selected paragraph is text content according to the relevance result and the position of the to-be-selected paragraph in the to-be-extracted webpage includes:

[0029] If the relevance result meets a preset condition, and the position of the paragraph to be selected in the web page to be extracted is within a preset position range, then the non-label content in the paragraph to be selected is determined to be the main text content.

[0030] Optionally, filtering each of the page paragraphs according to the tag content of each of the page paragraphs to obtain at least one paragraph to be selected includes:

[0031] If the tag content of the page paragraph indicates that the page paragraph is a picture, the page paragraph is removed; if the tag content of the page paragraph indicates that the page paragraph is not a picture, the page paragraph is used as a paragraph to be selected.

[0032] Optionally, the merging of the at least one target text paragraph according to the page layout information of the web page to be extracted to obtain the text extraction result of the web page to be extracted includes:

[0033] If the page layout information of the web page to be extracted indicates that the web page to be extracted is in a single-column layout, merging the target text paragraphs according to the position information of the target text paragraphs to obtain a text extraction result of the web page to be extracted;

[0034] If the page layout information of the web page to be extracted indicates that the web page to be extracted is a multi-column layout, the starting position of the web page to be extracted is determined according to the title tag of the web page to be extracted, and each target text paragraph is merged according to the position information of each text paragraph and the starting position of the web page to be extracted to obtain the text extraction result of the web page to be extracted.

[0035] Optionally, before merging the at least one target text paragraph according to the page layout information of the web page to be extracted to obtain the text extraction result of the web page to be extracted, the method further includes:

[0036] Acquire the attribute information of each layer tag in the web page to be extracted, and determine the page layout information of the web page to be extracted according to the attribute information of the layer tag, and / or acquire the cascading style sheet of the web page to be extracted, and determine the page layout information of the web page to be extracted according to the cascading style sheet.

[0037] In a second aspect, the present application provides a webpage content extraction device, the device comprising:

[0038] An extraction module, configured to extract paragraphs from a web page to be extracted, and obtain at least one page paragraph of the web page to be extracted, wherein each page paragraph includes at least one tag content and at least one non-tag content;

[0039] A determination module, for determining whether the non-tag content in each of the page paragraphs is text content according to the tag content of each of the page paragraphs, and if so, extracting the text content and treating the page paragraph as a target text paragraph;

[0040] The merging module is used to merge the at least one target text paragraph according to the page layout information of the web page to be extracted, so as to obtain the text extraction result of the web page to be extracted.

[0041] Optionally, the extraction module is specifically used for:

[0042] Using the network address of the web page to be extracted as an input parameter of the target conversion method, calling the target conversion method to convert the web page data into source code text;

[0043] A page element extraction method is called to extract at least one page paragraph from the source code text.

[0044] Optionally, the determining module is specifically used to:

[0045] Filtering each of the page paragraphs according to the tag content of each of the page paragraphs to obtain at least one paragraph to be selected;

[0046] Determine whether the non-label content in each of the to-be-selected paragraphs is text content according to the non-label content in each of the to-be-selected paragraphs;

[0047] If so, the text content is extracted, and the to-be-selected paragraph is used as a target text paragraph.

[0048] Optionally, the determining module is specifically used to:

[0049] According to the title tag, obtain the title text of the web page to be extracted;

[0050] Performing semantic analysis on the non-label content in the paragraph to be selected and the adjacent paragraphs of the paragraph to be selected to obtain a first semantic analysis result of the paragraph to be selected and a second semantic analysis result of the adjacent paragraphs;

[0051] Determine, according to the first semantic analysis result, the second semantic analysis result and the title text, a relevance result corresponding to the non-label content of the to-be-selected paragraph;

[0052] Determining the position of the to-be-selected paragraph in the to-be-extracted web page according to the tag content of the to-be-selected paragraph;

[0053] According to the relevance result and the position of the paragraph to be selected in the web page to be extracted, it is determined whether the non-label content in the paragraph to be selected is the main text content.

[0054] Optionally, the determining module is specifically used to:

[0055] Performing semantic relevance analysis on the first semantic analysis result and the second semantic analysis result to obtain a first relevance;

[0056] Performing semantic relevance analysis on the first semantic analysis result and the title text to obtain a second relevance;

[0057] A relevance result corresponding to the non-label content of the to-be-selected paragraph is obtained according to the first relevance and the second relevance.

[0058] Optionally, the determining module is specifically used to:

[0059] If the relevance result meets a preset condition, and the position of the paragraph to be selected in the web page to be extracted is within a preset position range, then the non-label content in the paragraph to be selected is determined to be the main text content.

[0060] Optionally, the determining module is specifically used to:

[0061] If the tag content of the page paragraph indicates that the page paragraph is a picture, the page paragraph is removed; if the tag content of the page paragraph indicates that the page paragraph is not a picture, the page paragraph is used as a paragraph to be selected.

[0062] Optionally, the merging module is specifically used for:

[0063] If the page layout information of the web page to be extracted indicates that the web page to be extracted is in a single-column layout, merging the target text paragraphs according to the position information of the target text paragraphs to obtain a text extraction result of the web page to be extracted;

[0064] If the page layout information of the web page to be extracted indicates that the web page to be extracted is a multi-column layout, the starting position of the web page to be extracted is determined according to the title tag of the web page to be extracted, and each target text paragraph is merged according to the position information of each text paragraph and the starting position of the web page to be extracted to obtain the text extraction result of the web page to be extracted.

[0065] Optionally, the merging module is specifically used for:

[0066] Acquire the attribute information of each layer tag in the web page to be extracted, and determine the page layout information of the web page to be extracted according to the attribute information of the layer tag, and / or acquire the cascading style sheet of the web page to be extracted, and determine the page layout information of the web page to be extracted according to the cascading style sheet.

[0067] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of a web page content extraction method as described in any one of the first aspects.

[0068] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a web page content extraction method as described in any one of the first aspects are executed.

[0069] The beneficial effects of the present application are: performing paragraph extraction on the web page to be extracted, obtaining at least one page paragraph of the web page to be extracted, each page paragraph including at least one tag content and at least one non-tag content, determining whether the non-tag content in each page paragraph is text content according to the tag content of each page paragraph, if so, extracting the text content, and taking the page paragraph as a target text paragraph, merging at least one target text paragraph according to the page layout information of the web page to be extracted, and obtaining the text extraction result of the web page to be extracted. By parsing the web page content and obtaining the text content of the web page based on the tag recognition in the web page content, automatic text content extraction can be achieved, and the present application does not need to pre-delete the captions, links, etc. in the web page, so it can reduce labor costs, and has good versatility for web pages of different styles.

[0070] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0072] Figure 1 A flow chart of a webpage content extraction method provided by an embodiment of the present application is shown;

[0073] Figure 2 A flowchart of extracting page paragraphs provided by an embodiment of the present application is shown;

[0074] Figure 3 A flowchart for determining a target text paragraph provided by an embodiment of the present application is shown;

[0075] Figure 4 A flowchart for determining text content provided by an embodiment of the present application is shown;

[0076] Figure 5 A flow chart of determining a correlation result provided by an embodiment of the present application is shown;

[0077] Figure 6 A flow chart of merging processing provided by an embodiment of the present application is shown;

[0078] Figure 7 A schematic diagram of the structure of a webpage content extraction device provided in an embodiment of the present application is shown;

[0079] Figure 8 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0080] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0081] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0082] A web page may include advertisements, titles, and text content. When performing network operation and maintenance, it is often necessary to extract the text content from the web page. However, how to accurately and efficiently extract the text content from the complex web page content has become an urgent problem to be solved.

[0083] In the prior art, maintenance personnel usually manually delete captions, links, titles, etc. in web pages, and then extract the text content from the remaining content. However, this method greatly increases the work cost and maintenance cost of operators.

[0084] Based on this, the present application proposes a webpage content extraction method, which first extracts paragraphs from a webpage, then identifies the extracted paragraphs based on the tag content to obtain the target text paragraphs, and finally merges the target text paragraphs based on the layout of the webpage to obtain the text extraction result of the webpage. Compared with the method of manually removing captions, links, and titles before extracting the text content in the prior art, the present application can realize fully automatic text content extraction, and is not limited by the style layout of the webpage, and has good versatility.

[0085] Next, combine Figure 1 , the webpage content extraction method of the present application is described, and the execution subject of the method can be an electronic device with processing capabilities, such as Figure 1 As shown, the method includes:

[0086] S101 , extracting paragraphs from a web page to be extracted to obtain at least one page paragraph of the web page to be extracted, each page paragraph including at least one tag content and at least one non-tag content.

[0087] Before extracting paragraphs from a web page to be extracted, source code data of the web page to be extracted may be obtained. As a possible implementation, an operator may access the web page to be extracted through the web page address of the web page to be extracted, and obtain the source code of the web page to be extracted as the source code data of the web page to be extracted.

[0088] After the source code data of the web page to be extracted is obtained, the source code data can be converted into a document class, and at least one page paragraph of the web page to be extracted can be extracted from the document class. The page paragraph can be a text paragraph, a picture, etc. whose distance in the web page to be extracted is less than a preset threshold.

[0089] As a possible implementation method, the URL address of the web page to be extracted can be parsed by Jsoup to obtain the document class of the web page to be extracted, and then the document class of the web page to be extracted can be parsed to obtain multiple page paragraphs of the web page to be extracted.

[0090] The tag content in a page paragraph includes: text tags, image tags, font attribute tags, etc. The page paragraph also includes at least one non-tag content, which can include: text, image address, etc. The page paragraph can be The paragraph between the tags.

[0091] S102: Determine whether the non-tag content in each page paragraph is text content according to the tag content of each page paragraph. If so, extract the text content and take the page paragraph as a target text paragraph.

[0092] It should be noted that the non-tag content text in the page paragraphs is not necessarily the main content, but may also include the name of the picture in the web page, related search text in the web page, comment text in the web page, etc. Therefore, it is necessary to further determine whether each page paragraph is the main content based on the tag content of each page paragraph.

[0093] When determining the main text content based on the tag content, the present application can also combine the semantics of the text for analysis, and determine whether the non-tag content in the page paragraph is the main text content based on the relevance between the non-tag content in the page paragraph and the title of the web page to be extracted.

[0094] If the non-tag content is text content, the non-tag content of the page paragraph can be subjected to text extraction, the extracted text content is used as a target text paragraph, and the position information of the page paragraph corresponding to the target text paragraph in the web page to be extracted is recorded.

[0095] As a possible implementation method, this step can be performed on the page paragraphs from front to back according to their order in the source code data of the web page to be extracted, and an incremental identifier or the position coordinates of the target text paragraph in the web page to be extracted can be added to the obtained target text paragraphs, so as to merge the target text paragraphs according to the identifier or the position coordinates.

[0096] S103: According to the page layout information of the web page to be extracted, at least one target text paragraph is merged to obtain a text extraction result of the web page to be extracted.

[0097] The page layout information of the web page to be extracted may indicate the layout of the web page to be extracted, such as single column, double column or multiple columns. It should be understood that when merging texts of web pages with different layouts, the merging process needs to be performed in order based on the position of the text in the web page to obtain a text extraction result with the same order as the web page to be extracted.

[0098] In another possible implementation, after the page layout information is determined, the target text paragraphs may be arranged according to the relevance between the target text paragraphs to obtain the text extraction result of the web page to be extracted.

[0099] It should be noted that after arranging the target text paragraphs in order, the format of the merged text can be optimized, such as unifying the text font, text size and paragraph format, to ensure that the merged text information is complete, fluent and logically clear.

[0100] In the embodiment of the present application, paragraph extraction is performed on the web page to be extracted to obtain at least one page paragraph of the web page to be extracted, each page paragraph includes at least one tag content and at least one non-tag content, and according to the tag content of each page paragraph, it is determined whether the non-tag content in each page paragraph is the main text content, and if so, the main text content is extracted, and the page paragraph is used as a target main text paragraph, and according to the page layout information of the web page to be extracted, at least one target main text paragraph is merged to obtain the main text extraction result of the web page to be extracted. By parsing the web page content and obtaining the main text content of the web page based on the tag recognition in the web page content, automatic main text content extraction can be achieved, and the present application does not need to pre-delete the captions, links, etc. in the web page, so it can reduce labor costs, and has good versatility for web pages of different styles.

[0101] The following is a further description of extracting paragraphs from the web page to be extracted to obtain at least one page paragraph of the web page to be extracted. Figure 2 As shown, the above step S101 includes:

[0102] S201, using the network address of the web page to be extracted as an input parameter of the target conversion method, calling the target conversion method, and converting the web page data into source code text.

[0103] The source code text may be a document class of a web page to be extracted. The network address of the web page to be extracted may be a URL address of the web page to be extracted. The target conversion method may be a Document doc=Jsoup.parse(rawContent) method. The source code data of the web page to be extracted may be obtained according to the URL address of the web page to be extracted. By calling the Document doc=Jsoup.parse(rawContent) method, the source code data of the web page to be extracted may be converted into a document.

[0104] S202: Call a page element extraction method to extract at least one page paragraph from the source code text.

[0105] The page element extraction method may be Elements links = doc.getElementsByTag (TAG_P) method. By calling the page element extraction method, at least one page paragraph may be extracted from the source code text. Specifically, by calling the page element extraction method, the Jsoup library may be used to extract elements with specific tags from the source code text. In this application, the returned tag may be , that is, the page paragraph in the source text.

[0106] The following is a further explanation of the above-mentioned method of determining whether the non-tag content in each page paragraph is the body content according to the tag content of each page paragraph, and if so, extracting the body content and taking the page paragraph as a target body paragraph, such as Figure 3 As shown, the above step S102 includes:

[0107] S301 : Filter each page paragraph according to the tag content of each page paragraph to obtain at least one paragraph to be selected.

[0108] After obtaining the page paragraph, you can perform a multi-dimensional analysis on the page paragraph to determine whether the page paragraph is the main text content.

[0109] As a possible implementation method, you can filter according to the tag content of the page paragraphs, filter out the page paragraphs with picture content, and then filter the page paragraphs based on the tag content of the page paragraphs, and select the page paragraph with the tag content indicating text content as a candidate paragraph.

[0110] S302: Determine whether the non-tag content in each of the paragraphs to be selected is text content according to the non-tag content in each of the paragraphs to be selected.

[0111] It should be noted that the above S301 step is only a preliminary filtering based on the tag content. The non-tag content in the selected paragraph may be text, empty text or picture information. Therefore, the non-tag content in the selected paragraph can be judged again, and the selected paragraphs with empty text and non-tag content as picture information can be eliminated.

[0112] Optionally, in the present application, semantic analysis can also be performed on non-tag content, including semantic similarity analysis of the semantics of the text in each candidate paragraph and the title of the web page to be extracted, and semantic similarity analysis of each candidate paragraph and its adjacent candidate paragraphs, and determining the main text content based on the similarity.

[0113] S303: If yes, extract the text content and take the to-be-selected paragraph as a target text paragraph.

[0114] If the non-tag content in the paragraph to be selected is body content, the non-tag content can be extracted, and the paragraph to be selected is used as a target body paragraph, and the position information of the paragraph to be selected in the web page to be extracted is recorded.

[0115] The following is a further explanation of the above-mentioned determination of whether the non-label content in each candidate paragraph is the main text content according to the non-label content in each candidate paragraph. Figure 4 As shown, the above step S302 includes:

[0116] S401, obtaining the title text of the web page to be extracted according to the title tag.

[0117] The title tag can be " <title> "and"< / title> ", the title text can be " <title> "and"< / title> ".

[0118] S402 , performing semantic analysis on non-label content in the paragraph to be selected and adjacent paragraphs of the paragraph to be selected, to obtain a first semantic analysis result of the paragraph to be selected and a second semantic analysis result of the adjacent paragraphs.

[0119] The non-tag content in the selected paragraph can be " "and" " The adjacent paragraph of the paragraph to be selected may be the previous paragraph to be selected or the next paragraph to be selected of the paragraph to be selected in the web page to be extracted.

[0120] By performing semantic analysis on the non-label content in the selected paragraph and the non-label content of the adjacent paragraphs of the selected paragraph, a first semantic analysis result of the non-label content of the selected paragraph and a second semantic analysis result of the non-label content of the adjacent paragraphs can be obtained.

[0121] The first semantic analysis result may be a feature vector of non-label content of the paragraph to be selected, and the second semantic analysis result may be a feature vector of non-label content of a paragraph adjacent to the paragraph to be selected.

[0122] S403: Determine the relevance result corresponding to the non-label content of the paragraph to be selected according to the first semantic analysis result, the second semantic analysis result and the title text.

[0123] The relevance results corresponding to the non-label content include the relevance results between the selected paragraph and the adjacent paragraphs and the relevance results between the selected paragraph and the title text.

[0124] By calculating the similarity between the first semantic analysis result and the second semantic analysis result, the correlation result between the selected paragraph and the adjacent paragraph can be obtained. By calculating the similarity between the first semantic analysis result and the title text, the correlation result between the selected paragraph and the title text can be obtained.

[0125] The relevance result is used to describe the degree of association between paragraphs. It should be understood that the body content generally has a greater degree of association with the title, and the degree of association between adjacent body contents is also greater. Therefore, based on the relevance result, it can be determined whether the non-label content in the selected paragraph is the body content.

[0126] S404: Determine the position of the paragraph to be selected in the web page to be extracted according to the tag content of the paragraph to be selected.

[0127] The tag content can indicate the font size, font and other information of the text in the selected paragraph. Therefore, based on the tag content of the selected paragraph, it can be determined whether the selected paragraph is a title. If so, the order of the titles can be determined according to the order of each tag, and then the selected paragraphs between the titles are marked according to their order in the web page to be extracted. The mark is used to indicate the position of the selected paragraph in the web page to be extracted.

[0128] S405: Determine whether the non-label content in the to-be-selected paragraph is text content according to the relevance result and the position of the to-be-selected paragraph in the to-be-extracted web page.

[0129] Optionally, if the relevance result indicates that the relevance of the selected paragraph to the title and the adjacent paragraphs is high, and the position of the selected paragraph in the web page to be extracted is within the layout range of the body content, then the non-label content in the selected paragraph can be determined to be the body content. On the contrary, if the relevance result indicates that the relevance of the selected paragraph to the title and the adjacent paragraphs is low, or the position of the selected paragraph in the web page to be extracted is not within the layout range of the body content, then the non-label content in the selected paragraph can be considered not to be the body content.

[0130] The following is a further explanation of the above-mentioned determination of the relevance result corresponding to the non-label content of the selected paragraph based on the first semantic analysis result, the second semantic analysis result and the title text. Figure 5 As shown, the above step S403 includes:

[0131] S501: Perform semantic relevance analysis on the first semantic analysis result and the second semantic analysis result to obtain a first relevance.

[0132] The first correlation is used to characterize the correlation between the non-label content of the selected paragraph and the non-label content of the adjacent paragraphs of the selected paragraph. The semantic correlation analysis of the first semantic analysis result and the second semantic analysis result can be performed by calculating the cosine similarity between the feature vector of the non-label content of the selected paragraph and the feature vector of the non-label content of the adjacent paragraph, and measuring the semantic similarity between the selected paragraph and the adjacent paragraph by the cosine similarity.

[0133] S502: Perform semantic relevance analysis on the first semantic analysis result and the title text to obtain a second relevance.

[0134] The second relevance is used to characterize the relevance between the non-label content of the selected paragraph and the title text. The semantic relevance analysis of the first semantic analysis result and the title text can be performed by calculating the cosine similarity between the feature vector of the non-label content of the selected paragraph and the feature vector of the title text, and measuring the semantic similarity between the selected paragraph and the title text by the cosine similarity.

[0135] S503: Obtain a relevance result corresponding to the non-label content of the paragraph to be selected according to the first relevance and the second relevance.

[0136] As a possible implementation, the first relevance and the second relevance may be weighted and the weighted result may be used as the relevance result corresponding to the non-label content of the selected paragraph. The weight of the first relevance and the weight of the second relevance may be set by the user, and this application does not limit this.

[0137] It should be noted that in some web pages, there may be a large title and multiple sub-titles. Generally speaking, the sub-titles have a higher correlation with the paragraph to be selected. At this time, the similarity result between the paragraph to be selected and the sub-title can be used as the second correlation. The font and font size of the sub-title are generally different from the text content. After the sub-title is identified, the paragraph to be selected between adjacent sub-titles can be used as the paragraph to be selected for the previous sub-title in the adjacent sub-titles, and the semantic similarity between the paragraph to be selected and the previous sub-title is calculated.

[0138] The following is a further explanation of determining whether the non-label content in the selected paragraph is the main text content according to the relevance result and the position of the selected paragraph in the web page to be extracted, including:

[0139] If the relevance result meets the preset conditions, and the position of the paragraph to be selected in the web page to be extracted is within a preset position range, then the non-tag content in the paragraph to be selected is determined to be the main text content.

[0140] The correlation result satisfies the preset condition, which may be that the weighted result of the correlation is greater than a preset threshold, or that the first correlation is greater than a first preset threshold and the second correlation is greater than a second preset threshold.

[0141] The preset position range of the web page to be extracted may be a position range in which the text content may be predicted to exist after performing layout analysis and identification on the web page to be extracted.

[0142] By analyzing the relevance between the selected paragraph and the title text and adjacent paragraphs, and combining the relevance results of semantic analysis with the position information of the selected paragraph in the web page to be extracted, the accuracy of identifying the text content in the web page can be improved.

[0143] The above step of filtering each page paragraph according to the tag content of each page paragraph to obtain at least one paragraph to be selected includes:

[0144] If the tag content of the page paragraph indicates that the page paragraph is a picture, the page paragraph is removed; if the tag content of the page paragraph indicates that the page paragraph is not a picture, the page paragraph is taken as a candidate paragraph.

[0145] If the tag content of a page paragraph includes a string of images, such as "img src", the page paragraph can be removed.

[0146] For example, assume that the content of the page paragraph is:

[0147] “ <p align＝"center"style＝"text-align:center；">

[0148] <imgsrc="https: / / i.guan.cn / news / mainland / 2024 / 12 / 11 / 20241211112904534.jpg" / >

[0149] ”

[0150] Among them, if a character string containing a referenced image is included, it means that the page paragraph contains image information, and the page paragraph can be removed at this time.

[0151] If the tag content of the page paragraph does not contain a character string of an image or a character string that references an image, the page paragraph can be retained as a candidate paragraph.

[0152] The following is a further explanation of the above-mentioned method of merging at least one target text paragraph according to the page layout information of the web page to be extracted to obtain the text extraction result of the web page to be extracted. Figure 6 As shown, the above step S103 includes:

[0153] S601. If the page layout information of the web page to be extracted indicates that the web page to be extracted is in a single-column layout, each target text paragraph is merged according to the position information of each target text paragraph to obtain a text extraction result of the web page to be extracted.

[0154] Among them, the page layout information can be The style attribute of the tag, which can also be the display attribute in CSS. The style attribute of the tag can indicate whether the element is in left-right layout, and the display attribute in CSS can also indicate whether the column layout is a normal single-column layout or a left-right double-column layout.

[0155] If the page layout information of the web page to be extracted indicates that the web page to be extracted is a single-column layout, the target text paragraphs can be arranged and merged according to the order of the target text paragraphs in the web page to be extracted and the sequence identifiers added previously to obtain the text extraction result of the web page to be extracted.

[0156] It should be noted that, when performing the merging process, the tag content of each target text paragraph may be removed, and the non-tag content may be merged and processed, thereby obtaining a text extraction result that does not contain tags.

[0157] S602. If the page layout information of the web page to be extracted indicates that the web page to be extracted is a multi-column layout, the starting position of the web page to be extracted is determined according to the title tag of the web page to be extracted, and each target text paragraph is merged according to the position information of each text paragraph and the starting position of the web page to be extracted to obtain the text extraction result of the web page to be extracted.

[0158] If the page layout information of the web page to be extracted indicates that the web page to be extracted is a multi-column layout, the starting position of the main text content can be determined as the first column on the left or the first column on the right based on the position of the title of the web page to be extracted, and the merging can be started from the determined starting position. When the starting position is the first column on the left, the merging is carried out in order from left to right. When the starting position is the first column on the right, the merging is carried out in order from right to left, so as to obtain the main text extraction result of the web page to be extracted.

[0159] As an example, when determining the starting position, you can determine the label <title> Alignment, if the label< / title> <title> To align the text to the left, you can use the first target text paragraph in the first column on the left as the starting position.< / title>

[0160] Before merging at least one target text paragraph according to the page layout information of the web page to be extracted to obtain the text extraction result of the web page to be extracted, the process of obtaining the page layout information includes:

[0161] Obtaining attribute information of each layer tag in the web page to be extracted, and determining page layout information of the web page to be extracted according to the attribute information of the layer tags, and / or obtaining a cascading style sheet of the web page to be extracted, and determining page layout information of the web page to be extracted according to the cascading style sheet.

[0162] The attribute information of the layer label can be The style attribute float field of the tag. If the value of the float field is left or right, it means that the page elements of the web page to be extracted are in a left-right layout.

[0163] If the page layout information cannot be determined based on the float field, the page layout information can be determined based on the value of the display attribute in the Cascading Style Sheet CSS of the web page to be extracted. Specifically, the display:inline-block attribute can display multiple elements side by side, and sets the horizontal arrangement rules and flexbox or grid layout related attributes. Among them, the grid-template-columns attribute is used to define the column layout. The size and number of columns and rows of the grid container defined by display:inline-block can be determined based on the grid-template-columns attribute, and then the page layout can be determined to be normal or left-right layout based on the number of columns. If the number of columns is single, it can be determined that the web page to be extracted is a single-column layout. If the number of columns is multiple, it can be determined that the web page to be extracted is a multi-column layout.

[0164] Based on the same inventive concept, the embodiment of the present application also provides a web page content extraction device corresponding to the web page content extraction method. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned web page content extraction method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0165] Figure 7 The schematic diagram of the structure of a webpage content extraction device provided in an embodiment of the present application is shown. The device includes: an extraction module 701, a determination module 702 and a merging module 703.

[0166] An extraction module 701 is used to extract paragraphs from a web page to be extracted, and obtain at least one page paragraph of the web page to be extracted, each page paragraph including at least one tag content and at least one non-tag content;

[0167] The determination module 702 is used to determine whether the non-tag content in each page paragraph is the main text content according to the tag content of each page paragraph, and if so, extract the main text content and take the page paragraph as a target main text paragraph;

[0168] The merging module 703 is used to merge at least one target text paragraph according to the page layout information of the web page to be extracted, so as to obtain the text extraction result of the web page to be extracted.

[0169] Optionally, the extraction module 701 is specifically used for:

[0170] Using the network address of the web page to be extracted as an input parameter of the target conversion method, calling the target conversion method to convert the web page data into source code text;

[0171] Call the page element extraction method to extract at least one page paragraph from the source text.

[0172] Optionally, the determination module 702 is specifically configured to:

[0173] Filter each page paragraph according to the tag content of each page paragraph to obtain at least one paragraph to be selected;

[0174] According to the non-label content in each of the paragraphs to be selected, determining whether the non-label content in each of the paragraphs to be selected is the main text content;

[0175] If so, the body content is extracted and the selected paragraph is used as a target body paragraph.

[0176] Optionally, the determination module 702 is specifically configured to:

[0177] According to the title tag, obtain the title text of the web page to be extracted;

[0178] Performing semantic analysis on non-label content in the paragraph to be selected and adjacent paragraphs of the paragraph to be selected, obtaining a first semantic analysis result of the paragraph to be selected and a second semantic analysis result of the adjacent paragraph;

[0179] Determine the relevance result corresponding to the non-label content of the to-be-selected paragraph according to the first semantic analysis result, the second semantic analysis result and the title text;

[0180] Determine the position of the to-be-selected paragraph in the web page to be extracted according to the tag content of the to-be-selected paragraph;

[0181] According to the relevance result and the position of the paragraph to be selected in the web page to be extracted, it is determined whether the non-label content in the paragraph to be selected is the main text content.

[0182] Optionally, the determination module 702 is specifically configured to:

[0183] Performing semantic relevance analysis on the first semantic analysis result and the second semantic analysis result to obtain a first relevance;

[0184] Performing semantic relevance analysis on the first semantic analysis result and the title text to obtain a second relevance;

[0185] According to the first relevance and the second relevance, a relevance result corresponding to the non-label content of the selected paragraph is obtained.

[0186] Optionally, the determination module 702 is specifically configured to:

[0187] If the relevance result meets the preset conditions, and the position of the paragraph to be selected in the web page to be extracted is within a preset position range, then the non-tag content in the paragraph to be selected is determined to be the main text content.

[0188] Optionally, the determination module 702 is specifically configured to:

[0189] If the tag content of the page paragraph indicates that the page paragraph is a picture, the page paragraph is removed; if the tag content of the page paragraph indicates that the page paragraph is not a picture, the page paragraph is taken as a candidate paragraph.

[0190] Optionally, the merging module 703 is specifically used for:

[0191] If the page layout information of the web page to be extracted indicates that the web page to be extracted is a single-column layout, each target text paragraph is merged according to the position information of each target text paragraph to obtain a text extraction result of the web page to be extracted;

[0192] If the page layout information of the web page to be extracted indicates that the web page to be extracted is a multi-column layout, the starting position of the web page to be extracted is determined according to the title tag of the web page to be extracted, and each target text paragraph is merged according to the position information of each text paragraph and the starting position of the web page to be extracted to obtain the text extraction result of the web page to be extracted.

[0193] Optionally, the merging module 703 is specifically used for:

[0194] Obtaining attribute information of each layer tag in the web page to be extracted, and determining page layout information of the web page to be extracted according to the attribute information of the layer tags, and / or obtaining a cascading style sheet of the web page to be extracted, and determining page layout information of the web page to be extracted according to the cascading style sheet.

[0195] The embodiment of the present application can realize automatic extraction of the main content of the web page by parsing the content of the web page and obtaining the main content of the web page based on the tag recognition in the web page content. The present application does not need to pre-delete the captions, links, etc. in the web page, thereby reducing labor costs and having good versatility for web pages of different styles.

[0196] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown, comprising: a processor 801, a storage medium 802 and a bus 803, wherein the storage medium 802 stores machine-readable instructions executable by the processor 801, and when the electronic device runs a web page content extraction method such as in the embodiment, the processor 801 communicates with the storage medium 802 via the bus 803, and the processor 801 executes the machine-readable instructions, and the processor 801 executes the preamble of the method item to execute the steps in the above-mentioned web page content extraction method.

[0197] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed when a processor is running, and the processor executes the steps in the above-mentioned web page content extraction method.

[0198] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment, which will not be repeated here.

[0199] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0200] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0201] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0202] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0203] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0204] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A web page content extraction method, characterized in that: include: Performing paragraph extraction on the web page to be extracted to obtain at least one page paragraph of the web page to be extracted, each of the page paragraphs including at least one tag content and at least one non-tag content; According to the tag contents of each of the page paragraphs, determining whether the non-tag contents in each of the page paragraphs are text contents, and if so, extracting the text contents and treating the page paragraph as a target text paragraph; The at least one target text paragraph is merged according to the page layout information of the web page to be extracted to obtain a text extraction result of the web page to be extracted.

2. The method according to claim 1, characterized in that The step of extracting paragraphs from a web page to be extracted to obtain at least one page paragraph of the web page to be extracted includes: Using the network address of the web page to be extracted as an input parameter of the target conversion method, calling the target conversion method to convert the web page data into source code text; A page element extraction method is called to extract at least one page paragraph from the source code text.

3. The method according to claim 1, characterized in that The step of determining whether the non-tag content in each of the page paragraphs is text content according to the tag content of each of the page paragraphs, and if so, extracting the text content and treating the page paragraph as a target text paragraph includes: Filtering each of the page paragraphs according to the tag content of each of the page paragraphs to obtain at least one paragraph to be selected; Determine whether the non-label content in each of the to-be-selected paragraphs is text content according to the non-label content in each of the to-be-selected paragraphs; If so, the text content is extracted, and the to-be-selected paragraph is used as a target text paragraph.

4. The method according to claim 3, characterized in that The step of determining whether the non-label content in each of the to-be-selected paragraphs is text content according to the non-label content in each of the to-be-selected paragraphs includes: According to the title tag, obtain the title text of the web page to be extracted; Performing semantic analysis on the non-label content in the paragraph to be selected and the adjacent paragraphs of the paragraph to be selected to obtain a first semantic analysis result of the paragraph to be selected and a second semantic analysis result of the adjacent paragraphs; Determine, according to the first semantic analysis result, the second semantic analysis result and the title text, a relevance result corresponding to the non-label content of the to-be-selected paragraph; Determining the position of the to-be-selected paragraph in the to-be-extracted web page according to the tag content of the to-be-selected paragraph; According to the relevance result and the position of the paragraph to be selected in the web page to be extracted, it is determined whether the non-label content in the paragraph to be selected is the main text content.

5. The method according to claim 4, characterized in that The determining, according to the first semantic analysis result, the second semantic analysis result and the title text, a relevance result corresponding to the non-label content of the to-be-selected paragraph includes: Performing semantic relevance analysis on the first semantic analysis result and the second semantic analysis result to obtain a first relevance; Performing semantic relevance analysis on the first semantic analysis result and the title text to obtain a second relevance; A relevance result corresponding to the non-label content of the to-be-selected paragraph is obtained according to the first relevance and the second relevance.

6. The method according to claim 4, characterized in that The step of determining whether the non-label content in the to-be-selected paragraph is text content according to the relevance result and the position of the to-be-selected paragraph in the to-be-extracted webpage includes: If the relevance result meets a preset condition, and the position of the paragraph to be selected in the web page to be extracted is within a preset position range, then the non-label content in the paragraph to be selected is determined to be the main text content.

7. The method according to claim 3, characterized in that The filtering of each of the page paragraphs according to the tag contents of each of the page paragraphs to obtain at least one to-be-selected paragraph includes: If the tag content of the page paragraph indicates that the page paragraph is a picture, the page paragraph is removed; if the tag content of the page paragraph indicates that the page paragraph is not a picture, the page paragraph is used as a paragraph to be selected.

8. The method according to claim 1, characterized in that The step of merging the at least one target text paragraph according to the page layout information of the web page to be extracted to obtain the text extraction result of the web page to be extracted includes: If the page layout information of the web page to be extracted indicates that the web page to be extracted is in a single-column layout, merging the target text paragraphs according to the position information of the target text paragraphs to obtain a text extraction result of the web page to be extracted; If the page layout information of the web page to be extracted indicates that the web page to be extracted is a multi-column layout, the starting position of the web page to be extracted is determined according to the title tag of the web page to be extracted, and each target text paragraph is merged according to the position information of each text paragraph and the starting position of the web page to be extracted to obtain the text extraction result of the web page to be extracted.

9. The method according to claim 8, characterized in that Before merging the at least one target text paragraph according to the page layout information of the web page to be extracted to obtain the text extraction result of the web page to be extracted, the method further includes: Acquire the attribute information of each layer tag in the web page to be extracted, and determine the page layout information of the web page to be extracted according to the attribute information of the layer tag, and / or acquire the cascading style sheet of the web page to be extracted, and determine the page layout information of the web page to be extracted according to the cascading style sheet.

10. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the web page content extraction method as described in any one of claims 1 to 9.

Images