Method and related device for obtaining corpus based on webpage

Abstract

The application discloses a method for obtaining corpus based on a webpage and a related device, and is applied to the technical field of data processing. The evaluation parameters of each label data block are obtained from each label data block based on the features of each label data block in the source code of a target webpage. The target label data block is obtained based on the evaluation parameters. The text in the target label data block is extracted as corpus. The features include at least one of a label type and a character density. Because the data in the source code of most webpages are in the form of label data blocks, the features are extracted from the dimension of the label data blocks, the evaluation parameters are obtained, and then the target label data block is obtained and the corpus is extracted therefrom. The method not only realizes the purpose of extracting corpus from a webpage, but also can be applied to more types of webpages, and thus has strong versatility.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and related apparatus for obtaining corpus based on web pages. Background Technology

[0002] With the rapid development of the internet, web pages, as the primary carrier of information dissemination, have experienced explosive growth in content data. The text within web pages can serve as corpus for training large-scale language models.

[0003] Therefore, how to obtain corpora from web pages is a problem worth studying. Summary of the Invention

[0004] To address the aforementioned problems, this application provides a method and related apparatus for obtaining corpus based on web pages, the specific solution of which is as follows:

[0005] The first aspect of this application provides a method for obtaining corpus based on web pages, including:

[0006] Based on the characteristics of each tag data block in the source code of the target webpage, the evaluation parameters of each tag data block are obtained. The characteristics include at least one of tag type and character density. The tag type includes text or non-text.

[0007] Based on the evaluation parameters, the target label data block is obtained from each label data block;

[0008] The text in the target label data block is extracted as corpus.

[0009] In one possible implementation, before obtaining the evaluation parameters of each tag data block based on the characteristics of each tag data block in the source code of the target webpage, the method further includes: converting the source code of the target webpage into structured data, wherein each tag data block in the structured data serves as a node of the structured data.

[0010] In one possible implementation, the feature further includes:

[0011] Evaluation parameters of the child nodes of the node.

[0012] In one possible implementation, obtaining the evaluation parameters for each tag data block based on the characteristics of each tag data block in the source code of the target webpage includes:

[0013] The tag type of the tag data block is the main text, and the evaluation parameter takes a first value; the tag type of the tag data block is the non-main text, and the evaluation parameter takes a second value.

[0014] Based on at least one of the character density and child node evaluation parameters, the first value or the second value is adjusted to obtain a third value, wherein the child node is a child node of a node in structured data, and the structured data is converted from the source code of the target webpage.

[0015] In one possible implementation, the character density includes at least one of: letter density and punctuation density.

[0016] In one possible implementation, extracting the text from the target tag block as corpus includes:

[0017] If the text in the target label data block meets the following conditions, the text is extracted as corpus. The conditions include at least one of the following: the number of characters is greater than a first threshold, the word frequency is not greater than a second threshold, and the correlation between the title text and the non-title text is greater than a third threshold.

[0018] In one possible implementation, before obtaining the evaluation parameters of each tag data block based on the characteristics of each tag data block in the source code of the target webpage, the method further includes:

[0019] The first webpage is determined to be the target webpage based on at least one of the following: the number of hyperlinks, the character density, and the number of list clusters in the first webpage.

[0020] A second aspect of this application provides an apparatus for acquiring corpus based on web pages, comprising:

[0021] The first acquisition module is used to acquire evaluation parameters of each tag data block based on the characteristics of each tag data block in the source code of the target webpage. The characteristics include at least one of tag type and character density, and the tag type includes text or non-text.

[0022] The second acquisition module is used to acquire the target label data block from each label data block based on the evaluation parameters;

[0023] The extraction module is used to extract the text from the target label data block as corpus.

[0024] A third aspect of this application provides an electronic device comprising:

[0025] Memory, used to store program code;

[0026] A processor is configured to run the program code to implement the method for obtaining corpus based on web pages provided in the first aspect of this application.

[0027] A third aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the webpage-based corpus acquisition method provided in the first aspect.

[0028] A fourth aspect of this application is a computer program product comprising computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the webpage-based corpus acquisition method provided in the first aspect.

[0029] The method and related apparatus for obtaining corpus based on web pages provided in this application obtain evaluation parameters for each tag data block based on the features of each tag data block in the source code of the target web page, obtain the target tag data block based on the evaluation parameters, and extract the text in the target tag data block as corpus. The features include at least one of tag type and character density. Since the data in the source code of most web pages is presented in the form of tag data blocks, the method of extracting features and obtaining evaluation parameters from the dimension of tag data blocks, and then obtaining the target tag data block and extracting corpus from it, not only achieves the purpose of extracting corpus from web pages, but can also be applied to more types of web pages, thus having strong versatility. Attached Figure Description

[0030] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0031] Figure 1 A flowchart illustrating a method for obtaining corpus based on web pages, provided in an embodiment of this application;

[0032] Figure 2 This is an example diagram of the tag data block in the source code of a webpage;

[0033] Figure 3 A flowchart illustrating yet another method for obtaining corpus based on web pages, provided in an embodiment of this application;

[0034] Figure 4 This is a flowchart illustrating another method for obtaining corpus based on web pages, provided in an embodiment of this application.

[0035] Figure 5 This is a schematic diagram of the structure of a device for obtaining corpus based on web pages, provided in an embodiment of this application;

[0036] Figure 6 This is a structural example diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0038] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0039] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0040] Web pages are typically written in HTML source code (hereinafter referred to as source code), and the content presented on the web page is distributed in corresponding locations within the source code. For example, the source code includes navigation bar code, body text code, and comment section code, among which the navigation bar code contains the text displayed in the navigation bar of the web page, such as "News" and "Entertainment," and the body text code contains the body text of the web page.

[0041] During their research, the inventors discovered that not all text in webpage source code is suitable as corpus for training large language models. For example, longer text on webpages is more suitable for training large language models, making the main body text of the webpage more suitable as corpus, while the text displayed in the navigation bar (usually short text) is not suitable. Therefore, it is necessary to extract suitable text from the webpage source code.

[0042] Currently, some tools exist that extract text from the source code of certain web pages based on the location of the code containing the main text. However, these tools are only applicable to these specific web pages. For other web pages where the location of the main text varies significantly, they cannot accurately extract the main text, and therefore cannot extract suitable text for use as corpus. It is evident that these tools have poor versatility.

[0043] The embodiments of this application provide a method and related apparatus for obtaining corpus based on web pages, which can extract text suitable as corpus from web page source code, and at least has the advantage of strong versatility.

[0044] Figure 1 This application provides an embodiment of a method for obtaining corpus based on web pages, comprising the following steps:

[0045] S11. Based on the characteristics of each tag data block in the source code of the target webpage, obtain the evaluation parameters of each tag data block.

[0046] The source code of the target webpage refers to the source code of the webpage from which the corpus is to be extracted.

[0047] Web page source code typically contains semantic tags, such as article (usually called an article, indicating the area containing the main text content) and footer. Semantic tags indicate the area in the web page where the text with the tag is displayed.

[0048] A tagged data block refers to at least one line of code with semantic tags. Figure 2 For example, Figure 2 This shows a tag data block with the semantic label "footer," which includes start indication information. <footer> and end instruction information< / footer> The text "2023 My Site. All Rights Reserved." between the start and end instructions is the text displayed at the bottom of the webpage (represented by the footer).

[0049] Features include at least one of tag type and character density.

[0050] Tag type refers to the type of semantic tag, which can be either text or non-text.

[0051] In some implementations, semantic tags that may appear in the webpage source code are pre-classified into either text or non-text types. For example, the semantic tags "article," "content," and "main" are classified into text types, while the semantic tags "header" and "footer" are classified into non-text types. The classification is based on the probability of containing long text. For instance, the text contained in the source code of the "article" type is more likely to be text, so "article" is classified into the text type.

[0052] Understandably, the type of a tag can be determined as text or non-text based on the type to which the semantic tags of the tag data block belong.

[0053] Since the main text is more suitable as corpus, the classification of types is helpful in extracting texts that are more suitable as corpus.

[0054] A character is a character in the text contained in a tag data block. A character in any tag data block may be text, punctuation, or both.

[0055] Character density can be the number of characters per unit area (e.g., the area is counted in pixels).

[0056] Character density reflects the density (or sparsity) of characters, and thus the likelihood of containing long text.

[0057] As can be seen, the evaluation parameters are used to reflect whether the text in the labeled data block is suitable as corpus. Specifically, the evaluation parameters are used to reflect the probability that the text in the labeled data block is long text.

[0058] The specific method for obtaining evaluation parameters from features will be described in subsequent embodiments.

[0059] S12. Based on the evaluation parameters, obtain the target label data block from each label data block.

[0060] The target label data block is the label data block from which the text is to be extracted.

[0061] As mentioned earlier, the evaluation parameters are used to reflect whether the text in the labeled data block is suitable as corpus. Therefore, based on the evaluation parameters, it is possible to obtain target labeled data blocks that are suitable for extracting corpus.

[0062] S13. Extract the text from the target label data block as corpus.

[0063] The method provided in this embodiment obtains evaluation parameters for each tag data block based on the characteristics of each tag data block in the source code of the target webpage. Based on the evaluation parameters, the target tag data block is obtained, and the text in the target tag data block is extracted as corpus. Since the data in most webpages is presented in the form of tag data blocks, the method of extracting features and obtaining evaluation parameters from the dimension of tag data blocks, and then obtaining the target tag data block and extracting corpus from it, not only achieves the purpose of extracting corpus from webpages, but can also be applied to more types of webpages, thus having strong versatility.

[0064] Furthermore, the label type includes text or non-text, so the label type can reflect whether the label data block is text or non-text. Generally, text is more likely to contain long text. Character density can reflect the density (or sparsity) of characters and also the possibility of containing long text. Therefore, the evaluation parameters obtained based on features can reflect the possibility that each label data block contains long text. Thus, the target label data block obtained based on the evaluation parameters is more likely to contain long text. In other words, it lays the foundation for obtaining long text, and the text extracted from it is more suitable as corpus.

[0065] Figure 3 This application provides another method for obtaining corpus based on web pages. In this embodiment, for Figure 1 The process shown will be explained in more detail.

[0066] Figure 3 The process includes the following steps:

[0067] S21. Convert the source code of the target webpage into structured data.

[0068] Structured data refers to data that is structured according to a predefined model or organized in a predefined manner.

[0069] Structured data can be viewed as a tree composed of nodes, with each tag data block in the source code of the target webpage being converted as a node of the structured data.

[0070] The source code of a webpage is unstructured data. The purpose of converting the source code of a webpage into structured data is to make the hierarchical relationship of each tag data block clearer, so as to facilitate subsequent processing, such as improving processing efficiency and reducing the possibility of errors.

[0071] In some implementations, an example of structured data is the Document Object Model (DOM) tree. After the source code of the target webpage is converted into a DOM tree, the tag data blocks in the source code constitute the nodes of the DOM tree. It can be understood that the nodes have a hierarchical relationship. The next level node of any node becomes a child node, and any node is a child node of the previous level node. Therefore, the tag data blocks in the source code have a hierarchical relationship.

[0072] S22. Based on the characteristics of each tag data block in the source code of the target webpage, obtain the evaluation parameters of each tag data block.

[0073] The source code of the target webpage in this step is structured data.

[0074] The characteristics of any labeled data block include: label type, character density, and evaluation parameters of the node's child nodes.

[0075] For an explanation of the label types, please refer to the above examples, which will not be repeated here.

[0076] In some implementations, character density includes at least one of letter density and punctuation density, and character density can reflect whether the text is long.

[0077] The evaluation parameters of a node's child nodes can reflect the likelihood that the child nodes are long text.

[0078] In some implementations, the methods for obtaining evaluation parameters based on the above three factors include:

[0079] If the tag type is text, the evaluation parameter takes the first value; if the tag type is not text, the evaluation parameter takes the second value. For example, if the evaluation parameter is a score, the first value is numerical A, and the second value is numerical B. A is greater than B; that is, if the tag type is text, a higher score is assigned, and if the tag type is not text, a lower score is assigned. As described in the above embodiment, for any given tag data block, the type to which the tag type belongs is the tag type.

[0080] Based on at least one of the evaluation parameters of character density and child nodes, adjust the first or second value to obtain the third value.

[0081] In some implementations, a first coefficient is obtained based on character density, a second coefficient is obtained based on the evaluation parameters of child nodes, and the first and second coefficients are used to adjust the score obtained based on the label type to obtain a third value.

[0082] Because higher character density generally indicates longer text and makes it more suitable for corpus, a higher character density results in a larger first coefficient. For example, a first sub-coefficient less than 1 can be obtained based on character density; a higher character density results in a larger first sub-coefficient. Similarly, a second sub-coefficient less than 1 can be obtained based on punctuation density; a higher punctuation density results in a larger second sub-coefficient. Multiplying the first and second sub-coefficients yields the first coefficient.

[0083] Understandably, separating text density and punctuation density in the statistics helps to more accurately evaluate whether the labeled data block contains long text.

[0084] The evaluation parameters of a child node represent the probability that the text in the labeled data block of the child node is long text. It can be understood that if multiple evaluation parameters of a child node all indicate a low probability of long text, then the probability that the text in the labeled data block of the parent node of the child node is also low. Therefore, if the evaluation parameters of the parent node of the child node indicate a high probability of long text, it means that one of the child nodes of the parent node may be long text, but other child nodes are not long text. Such labeled data blocks are not suitable as corpus because they contain short text.

[0085] Therefore, in this embodiment, the target child node is determined based on the evaluation parameters of the child node. For example, the child node with a score greater than the child node score threshold is taken as the target child node. If the number of target child nodes is greater than the number of child nodes threshold, then C is taken as the second coefficient. Otherwise, D is taken as the second coefficient. Both C and D are values ​​less than 1 and C is greater than D. That is to say, if most of the child nodes under a node are target child nodes, that is, most of the child nodes contain text that may be long text, then the node is suitable as corpus, and the second coefficient takes a larger value. Otherwise, the second coefficient takes a smaller value.

[0086] Understandably, using the evaluation parameters of child nodes as one of the criteria for obtaining the evaluation parameters of parent nodes can remove unevenly distributed label data blocks in long texts, which is beneficial for filtering out texts that are more suitable as corpus.

[0087] For example, the way to adjust the score based on the label type using the first and second coefficients is to multiply the first and second coefficients by the score based on the label type to obtain a third value, which is the final score.

[0088] As can be seen, the final score integrates multiple dimensions of factors, including tag type, character density, and evaluation parameters of child nodes, and can more comprehensively and accurately reflect whether it is a long text.

[0089] It is understood that the above calculation method is only an example and not a limitation. Any calculation method that obtains the evaluation parameters based on at least one of the evaluation parameters of the tag type, character density and child nodes is within the scope of this application.

[0090] S23. Based on the evaluation parameters, obtain the target label data block.

[0091] As explained in S21, for example, the evaluation parameter is a score. The higher the score, the greater the likelihood that the label data block contains long text. Therefore, for example, based on the relationship between the evaluation parameter and the evaluation parameter threshold, the target label data block is obtained. For example, the label data block with an evaluation parameter greater than the evaluation parameter threshold is taken as the target label data block.

[0092] S24. Extract the text from the target label data block as corpus.

[0093] The method provided in this embodiment, in addition to being highly versatile, also evaluates the label data blocks from multiple dimensions to obtain evaluation parameters, thereby obtaining target label data blocks containing the main text more comprehensively and accurately, and thus obtaining text that is more suitable as corpus.

[0094] Figure 4This is another method for obtaining corpus based on web pages disclosed in the embodiments of this application. Compared with the above embodiments, it adds preprocessing and postprocessing, with the aim of saving computing resources and obtaining more suitable corpus.

[0095] Figure 4 The process includes the following steps:

[0096] S31. The webpage is determined to be the target webpage based on at least one of the following: the number of hyperlinks in the webpage, the character density, and the number of list clusters.

[0097] Web pages are diverse and complex, and some web pages contain mostly short text, such as directory pages or list pages. The likelihood of extracting suitable text for use as corpus from these web pages is low. Therefore, in order to save computing power in subsequent steps and improve processing efficiency, this step filters out useful web pages, i.e. target web pages.

[0098] For ease of distinction, any webpage can be called the first webpage, and the useful webpages selected from the first webpage are called the target webpages. That is to say, S31 determines the first webpage as the target webpage based on at least one of the following: the number of hyperlinks, the character density, and the number of list clusters in the first webpage.

[0099] In this step, it is determined whether a webpage is the target webpage based on at least one of the following: the number of hyperlinks in the webpage, the character density, and the number of list clusters.

[0100] The more hyperlinks a webpage has, the more likely it is a directory or list page. The lower the character density, the more short text it contains. The more list clusters there are, the more lists it contains. Since lists are short texts, they contain more text.

[0101] In some implementations, thresholds can be pre-configured for the number of hyperlinks, character density, and number of list clusters. These parameters are then compared with the corresponding thresholds, and the webpage is determined to be the target webpage based on the comparison results.

[0102] For example, a webpage filtering model is pre-trained. The training process is as follows: select one thousand webpage samples. Each webpage sample is manually labeled as a detail page or a non-detail page (such as a directory page or list page). That is, the samples contain positive examples and negative examples. Positive examples are webpages labeled as detail pages, and negative examples are webpages labeled as directory pages or list pages.

[0103] After each webpage sample is input into the model, the model extracts the features of each webpage sample. The features of the webpage sample include at least one of the following: the number of hyperlinks, character density, and the number of list clusters in the webpage. It can be understood that the more features included, the more accurate the trained model will be.

[0104] The model outputs results based on the extracted features, indicating whether the webpage sample is a detail page or not. The model parameters are adjusted based on the difference between the results and the annotations.

[0105] Through the above training, the model learns the features of the detail page, thus enabling it to filter out detail pages.

[0106] Based on this, the specific implementation of S31 is as follows: input the webpage into the model, for example, input the URL of the webpage into the model so that the model can obtain the webpage, obtain the output result of the model, and use the webpage of the detail page indicated by the result as the target webpage.

[0107] It is understandable that the training method of the above model and the method of obtaining the target webpage based on the model are only examples.

[0108] S32. Based on the characteristics of each tag data block in the source code of the target webpage, obtain the evaluation parameters of each tag data block.

[0109] The features include at least one of tag type and character density, and the tag type includes body text or non-body text.

[0110] S33. Based on the evaluation parameters, obtain the target label data block.

[0111] The specific implementation methods of S32-S33 can be found in the above embodiments, and will not be repeated here.

[0112] S34. If the text in the target label data block meets the conditions, extract the text as corpus.

[0113] The conditions include at least one of the following: the number of characters is greater than a first threshold, the word frequency is not greater than a second threshold, and the relevance between the title text and the non-title text is greater than a third threshold. The title text refers to the content used as the title in the text of the target label data block, while the non-title text refers to all other content in the text of the target label data block besides the title.

[0114] It is understandable that the above conditions are based on the content of the text, that is, based on some characteristics of the text content, the probability of the text being a long text is determined.

[0115] A word count greater than the first threshold indicates a large word count, meeting the characteristics of long text. A word frequency not exceeding the second threshold indicates that the frequency of a particular word repetition is not very high, suggesting that the content may be formal and suitable as long text for corpus. In other words, if the word count is large but the frequency of repeated words is high, it is not suitable as corpus. A correlation between the title text and the non-title text greater than the third threshold indicates that the title and body text are more likely to match. In other words, text with a mismatch between title and body text is not suitable as corpus.

[0116] Understandably, the more items included in the conditions, the more suitable the extracted text will be as corpus, but the computational load can be greater. In practice, the items included in the conditions can be configured as needed.

[0117] The advantage of this step is that it further filters texts from the perspective of text content to make them more suitable as corpus, thus making the final obtained texts more suitable as corpus.

[0118] Understandably, if the text in the target label data block does not meet the conditions, it means that its content is not suitable as corpus and should be discarded.

[0119] The process provided in this embodiment first filters web pages to identify useful ones before calculating evaluation parameters, then calculates evaluation parameters for the useful web pages, and after obtaining target label data blocks based on the evaluation parameters, extracts corpus based on the text content in the target label data blocks. This not only reduces the amount of computation but also improves the quality of the corpus.

[0120] It is understandable that the method provided in the above embodiments may still fail to obtain suitable text for some web pages. In response, some implementations pre-configure a special web page database, such as writing the URLs of web pages from which suitable text cannot be extracted into the special web page database. Based on this, before the above embodiments, the following steps are also included: determining whether the web page is a special web page based on the URL of the web page; if not, executing the method provided in the above embodiments to extract the corpus; if so, calling the method for special web pages to extract the corpus.

[0121] For example, for a special webpage, the method is to call the template information of the main text in the pre-configured special webpage, such as the information of the area occupied by the main text, and then extract the content of the main text from the special webpage.

[0122] Based on this, before extracting the corpus, the web pages are classified, and the corpus is extracted using corresponding methods for specific web pages, thereby further expanding the applicable scenarios of the web page-based corpus extraction method provided in the embodiments of this application.

[0123] Figure 5 This application provides an embodiment of a device for obtaining corpus based on web pages, which includes a first acquisition module, a second acquisition module, and an extraction module.

[0124] The first acquisition module is used to obtain evaluation parameters for each tag data block based on the characteristics of each tag data block in the source code of the target webpage. The characteristics include at least one of tag type and character density. The tag type includes text or non-text. The second acquisition module is used to obtain the target tag data block from the tag data blocks based on the evaluation parameters. The extraction module is used to extract the text from the target tag data block as corpus.

[0125] In some implementations, the first acquisition module is further configured to: convert the source code of the target webpage into structured data before acquiring the evaluation parameters of each tag data block based on the characteristics of each tag data block in the source code of the target webpage, wherein each tag data block in the structured data serves as a node of the structured data.

[0126] In some implementations, the feature further includes: evaluation parameters of the child nodes of the node.

[0127] In some implementations, the first acquisition module acquires the evaluation parameters of each tag data block based on the characteristics of each tag data block in the source code of the target webpage. This includes: if the tag type of the tag data block is the main text, the evaluation parameter takes a first value; if the tag type of the tag data block is non-main text, the evaluation parameter takes a second value; based on at least one of the character density and the evaluation parameters of the child nodes, the first or second value is adjusted to obtain a third value, where the child nodes are child nodes of nodes in structured data, and the structured data is converted from the source code of the target webpage.

[0128] In some implementations, the character density includes at least one of: letter density and punctuation density.

[0129] In some implementations, the extraction module extracts text from the target label data block as corpus in the following ways: when the text in the target label data block meets certain conditions, the text is extracted as corpus, and the conditions include at least one of the following: the number of characters is greater than a first threshold, the word frequency is not greater than a second threshold, and the correlation between the title text and the non-title text is greater than a third threshold.

[0130] In some implementations, the apparatus further includes a filtering module, configured to determine that the first webpage is the target webpage based on at least one of the following: the number of hyperlinks, character density, and the number of list clusters in the first webpage, before obtaining the evaluation parameters of each tag data block based on the characteristics of each tag data block in the source code of the target webpage.

[0131] For details on the specific implementation of the functions of each module, please refer to the above method implementation examples, which will not be repeated here.

[0132] The device for obtaining corpus based on web pages provided in this embodiment can extract text as corpus from the source code of web pages, has high versatility, and the extracted text is more suitable as corpus.

[0133] Figure 6 This is a structural example diagram of an electronic device provided in an embodiment of this application, with reference to... Figure 6The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0134] The electronic device includes a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0135] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0136] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the sound signal processing methods provided in this application.

[0137] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the sound signal processing methods provided in this application.

[0138] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0140] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0141] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0142] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0143] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for obtaining corpus based on web pages, characterized in that, include: Based on the characteristics of each tag data block in the source code of the target webpage, the evaluation parameters of each tag data block are obtained. The characteristics include at least one of tag type and character density. The tag type includes text or non-text. Based on the evaluation parameters, the target label data block is obtained from each label data block; The text in the target label data block is extracted as corpus.

2. The method according to claim 1, characterized in that, Before obtaining the evaluation parameters of each tag data block based on the characteristics of each tag data block in the source code of the target webpage, the method further includes: The source code of the target webpage is converted into structured data, and each tag data block in the structured data serves as a node of the structured data.

3. The method according to claim 2, characterized in that, The features also include: Evaluation parameters of the child nodes of the node.

4. The method according to any one of claims 1-3, characterized in that, The method of obtaining evaluation parameters for each tag data block based on the characteristics of each tag data block in the source code of the target webpage includes: The tag type of the tag data block is the main text, and the evaluation parameter takes a first value; the tag type of the tag data block is the non-main text, and the evaluation parameter takes a second value. Based on at least one of the character density and child node evaluation parameters, the first value or the second value is adjusted to obtain a third value, wherein the child node is a child node of a node in structured data, and the structured data is converted from the source code of the target webpage.

5. The method according to any one of claims 1-4, characterized in that, The step of extracting the text from the target tag block as corpus includes: If the text in the target label data block meets the following conditions, the text is extracted as corpus. The conditions include at least one of the following: the number of characters is greater than a first threshold, the word frequency is not greater than a second threshold, and the correlation between the title text and the non-title text is greater than a third threshold.

6. The method according to any one of claims 1-5, characterized in that, Before obtaining the evaluation parameters of each tag data block based on the characteristics of each tag data block in the source code of the target webpage, the method further includes: The first webpage is determined to be the target webpage based on at least one of the following: the number of hyperlinks, the character density, and the number of list clusters in the first webpage.

7. A device for acquiring corpus based on web pages, characterized in that, include: The first acquisition module is used to acquire evaluation parameters of each tag data block based on the characteristics of each tag data block in the source code of the target webpage. The characteristics include at least one of tag type and character density, and the tag type includes text or non-text. The second acquisition module is used to acquire the target label data block from each label data block based on the evaluation parameters; The extraction module is used to extract the text from the target label data block as corpus.

8. An electronic device, characterized in that, include: Memory, used to store program code; A processor for running the program code to implement the method for obtaining corpus based on web pages as described in any one of claims 1-6.

9. A computer storage medium, characterized in that, The computer storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the method for obtaining corpus based on web pages as described in any one of claims 1-6.

10. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the method for obtaining corpus based on a web page as described in any one of claims 1-6.

Images