A web page information extraction method, device, equipment and storage medium
By introducing target title classifiers and entity classifiers in web page information extraction, combined with entity recognition model, the problem of lack of universality in web page content extraction in the existing technology is solved, and information extraction of web pages of different structures is realized, and the extraction is improved. Accuracy and efficiency.
Patent Information
- Application Number
- CN202210959612.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-10
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-08-10
AI Technical Summary
The existing web page content extraction scheme lacks universality, is difficult to adapt to web pages of different structures, and is unable to effectively block noise text, affecting the accuracy and efficiency of information extraction.
The dual classifier method based on the target title classifier and the target entity classifier is adopted to parse the web page, determine the title node and entity node set, and perform entity extraction through the target entity recognition model to enhance the universality and accuracy of information extraction.
It realizes flexible structured web page information extraction, improves the accuracy and universality of extraction, can effectively block noise text, and enhances the efficiency of information extraction.
Smart Images

Figure CN115238078B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of Internet technology, and in particular to a web page information extraction method, device, equipment and storage medium. Background Art
[0002] In today's Internet age, web pages have become an important information carrier for most people to understand the world. At the same time, since web pages usually contain more or less noise text, people have an increasing demand for mining effective information from web pages.
[0003] In the process of industrial data mining, the main task is to extract entities for data analysis in professional fields; but there is usually an additional task of extracting web page content. The extracted web page content must contain the extracted entities to shield irrelevant text blocks in the web page and display the context of the extracted entities locally. However, most of the existing web page content extraction schemes are only applicable to web pages with fixed structures. For data from different sites, the corresponding extraction strategy needs to be reset, which lacks universality. Therefore, it is urgently needed to be improved. Summary of the invention
[0004] The present invention provides a web page information extraction method, device, equipment and storage medium to realize information extraction for web pages with flexible structures.
[0005] According to one aspect of the present invention, a method for extracting web page information is provided, the method comprising:
[0006] Parsing the web page to be processed to obtain a target tree object and text information corresponding to a node in the target tree object;
[0007] Based on the target title classifier and the target entity classifier, the text information corresponding to the nodes in the target tree object is processed respectively, and a title node set and an entity node set are determined from the target tree object according to the processing results;
[0008] Based on the target entity recognition model, entity extraction is performed on the text information corresponding to the entity node set to obtain the entity information of the web page to be processed;
[0009] Determine the content location information of the web page to be processed according to the title node set and the entity node set
[0010] According to another aspect of the present invention, there is provided a web page information extraction device, the device comprising:
[0011] A web page information parsing module, used to parse the web page to be processed, and obtain a target tree object and text information corresponding to a node in the target tree object;
[0012] A node set determination module, for processing text information corresponding to nodes in the target tree object based on a target title classifier and a target entity classifier, and determining a title node set and an entity node set from the target tree object according to the processing results;
[0013] An entity information determination module is used to perform entity extraction on text information corresponding to an entity node set based on a target entity recognition model to obtain entity information of the web page to be processed;
[0014] The content location information determination module is used to determine the content location information of the web page to be processed according to the title node set and the entity node set.
[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0016] at least one processor; and
[0017] a memory communicatively connected to the at least one processor; wherein,
[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the web page information extraction method described in any embodiment of the present invention.
[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the web page information extraction method described in any embodiment of the present invention when executed.
[0020] The technical solution of the embodiment of the present invention parses the web page to be processed to obtain the target tree object and the text information corresponding to the nodes in the target tree object, and then processes the text information corresponding to the nodes in the target tree object based on the target title classifier and the target entity classifier, and determines the title node set and the entity node set from the target tree object according to the processing results, and then extracts the entity information corresponding to the entity node set based on the target entity recognition model to obtain the entity information of the web page to be processed, and finally determines the content location information of the web page to be processed based on the title node set and the entity node set. The above technical solution introduces the title classifier and the entity classifier, that is, the dual classifier to filter the title node and the entity node. Through these two auxiliary classifiers, information can be extracted for web pages with different structures, which enhances the versatility; at the same time, the accuracy of web page information extraction can also be improved.
[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0023] Figure 1 is a flow chart of a web page information extraction method provided according to the first embodiment of the present invention;
[0024] Figure 2A is a flow chart of a web page information extraction method provided according to Embodiment 2 of the present invention;
[0025] Figure 2B is a schematic diagram of determining a candidate content node according to Embodiment 2 of the present invention;
[0026] Figure 3 is a flow chart of a web page information extraction method provided according to Embodiment 3 of the present invention;
[0027] Figure 4 is a structural schematic diagram of a web page information extraction device provided according to a fourth embodiment of the present invention;
[0028] Figure 5 It is a structural schematic diagram of an electronic device for implementing the web page information extraction method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0030] It should be noted that the terms "sample", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0031] In addition, it should be noted that the collection, storage, use, processing, transmission, provision and disclosure of the web pages to be processed involved in the technical solution of the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0032] Embodiment 1
[0033] Figure 1 1 is a flow chart of a web page information extraction method provided according to the first embodiment of the present invention. This embodiment is applicable to the case of how to extract web page information. The method can be executed by a web page information extraction device. The web page information extraction device can be implemented in the form of hardware and / or software. The web page information extraction device can be integrated into an electronic device that carries the web page information extraction function, such as a server. Figure 1 As shown, the web page information extraction method of this embodiment may include:
[0034] S110, parsing the web page to be processed to obtain a target tree object and text information corresponding to the nodes in the target tree object.
[0035] In this embodiment, the web page to be processed refers to a web page document that needs to be extracted information, such as an HTML (Hyper Text Markup Language) document. It should be noted that an HTML document is a specific data representation of a web page, and a web page is a presentation form of an HTML document after being rendered by a browser; an HTML document contains HTML tags such as div, p, table, or CSS, JavaScript codes, etc., and in addition to valid web page content, there are also a large number of irrelevant texts; HTML is a standard tree structure, and each HTML node is organized according to a hierarchical relationship to form a document tree structure, namely a Document Object Model (DOM).
[0036] The target tree object refers to the document tree structure corresponding to the web page to be processed, that is, the DOM tree.
[0037] The text information corresponding to the nodes in the target tree object refers to the text information in the webpage corresponding to each node in the target tree object.
[0038] Specifically, the web page to be processed may be parsed based on an HTML parsing tool (such as Lxml) to obtain a target tree object corresponding to the web page to be processed and text information corresponding to a node in the target tree object.
[0039] S120: Based on the target title classifier and the target entity classifier, the text information corresponding to the nodes in the target tree object is processed respectively, and a title node set and an entity node set are determined from the target tree object according to the processing results.
[0040] In this embodiment, the target title classifier refers to a classifier used to determine whether a node in the target tree object is a title node. The target entity classifier refers to a classifier used to determine whether a node in the target tree object is an entity node.
[0041] Among them, the title node refers to a node of the title class; it should be noted that the title in the present invention can be understood as a small paragraph of the class title with a relatively obvious mark, and its text has a certain regularity, which can generally be the beginning of a section, a table header, etc. The entity node refers to a node of the entity class.
[0042] Specifically, for each node in the target tree object, the text information corresponding to the node is input into the target title classifier and the target entity classifier respectively. After being processed by the classifier, the category to which the text information corresponding to the node belongs is obtained. Then, according to the category to which the text information corresponding to the node belongs, it is determined whether the node is a title node or an entity node. Specifically, if the category to which the text information corresponding to the node belongs is the title category, then the node is a title node; if the category to which the text information corresponding to the node belongs is the entity category, then the node is an entity node. In other words, in the above manner, a set of title nodes and a set of entity nodes can be determined from the target tree object.
[0043] It should be noted that the so-called entity is the whole or partial text of the smallest component element of the HTML document, such as a word or phrase in a professional field. The so-called title is the title of the paragraph, table or chapter to which the entity belongs.
[0044] S130 . Based on the target entity recognition model, entity extraction is performed on the text information corresponding to the entity node set to obtain entity information of the web page to be processed.
[0045] In this embodiment, the target entity recognition model refers to a model used for entity recognition. The entity information refers to the entity content contained in the web page to be processed.
[0046] Specifically, the text information corresponding to each node in the entity node set can be input into the target entity recognition model to perform entity extraction to obtain the entity information of the web page to be processed.
[0047] S140: Determine content location information of the web page to be processed according to the title node set and the entity node set.
[0048] In this embodiment, the content location information refers to the node information to which the entity information belongs or corresponds. Exemplarily, the expression form can be path selection information, such as XPath path selection information. For example, the paragraph to be extracted describes an item, the target entity "hospital liquid oxygen supply service" exists in the paragraph, and there is a representative title such as "Three, main subject information" before the paragraph, that is, the node information corresponding to the paragraph is used as the content location information, which can also be understood as the approximate location information corresponding to the entity information.
[0049] Optionally, the content location information of the web page to be processed can be determined based on certain rules, according to the title node set and the entity node set. For example, the content location information of the web page to be processed can be determined based on the location determination model, according to the title node set and the entity node set.
[0050] It should be noted that the execution order of S130 and S140 in the present invention is not specifically limited, and they can be executed simultaneously, or S130 can be executed first and then S140, or S140 can be executed first and then S130.
[0051] The technical solution of the embodiment of the present invention parses the web page to be processed to obtain the target tree object and the text information corresponding to the nodes in the target tree object, and then processes the text information corresponding to the nodes in the target tree object based on the target title classifier and the target entity classifier, and determines the title node set and the entity node set from the target tree object according to the processing results, and then extracts the entity information corresponding to the entity node set based on the target entity recognition model to obtain the entity information of the web page to be processed, and finally determines the content location information of the web page to be processed based on the title node set and the entity node set. The above technical solution introduces the title classifier and the entity classifier, that is, the dual classifier to filter the title node and the entity node. Through these two auxiliary classifiers, information can be extracted for web pages with different structures, which enhances the versatility; at the same time, the accuracy of web page information extraction can also be improved.
[0052] Embodiment 2
[0053] FIG2 is a flow chart of a web page information extraction method provided according to the second embodiment of the present invention. Based on the above embodiment, this embodiment further optimizes "determining the content location information of the web page to be processed according to the title node set and the entity node set" and provides an optional implementation scheme. As shown in FIG2, the web page information extraction method of this embodiment may include:
[0054] S210: Parse the web page to be processed to obtain a target tree object and text information corresponding to the nodes in the target tree object.
[0055] S220: Based on the target title classifier and the target entity classifier, the text information corresponding to the nodes in the target tree object is processed respectively, and a title node set and an entity node set are determined from the target tree object according to the processing results.
[0056] S230: Based on the target entity recognition model, entity extraction is performed on the text information corresponding to the entity node set to obtain entity information of the web page to be processed.
[0057] S240: Perform node matching on the title node set and the entity node set to obtain a title entity node pair set.
[0058] In this embodiment, the title node pair refers to a node pair consisting of an entity node and a node of a title corresponding to the entity. The title node pair set refers to a set consisting of title node pairs in the target tree object.
[0059] Optionally, based on certain node matching rules, the title node set and the entity node set can be node matched to obtain a title entity node pair set. For example, the title node set and the entity node set can be node matched based on the distance between the nodes in the title node set and the nodes in the entity node set to obtain a title entity node pair. Specifically, for each entity node in the entity node set, the distance between the entity node and each title node in the title node set is determined, and the node in the title node set corresponding to the smallest distance is selected as the title node corresponding to the entity node, that is, the entity node and the title node corresponding to the entity node form a pair of title entity node pairs. For example, if node a is an entity node, node b is a header node, and node p is the least common ancestor node of node a and node b, then the distance between node a and node b can be determined based on the depth of the node in the target tree object, specifically, dist(a, b) = [h(a) - h(p)] + [h(b) - h(p)] = h(a) + h(b) - 2h(p), where h(n) represents the depth of node n, and dist(a, b) represents the distance between node a and node b. In particular, if node a is a descendant node of node b, then the least common ancestor node of node a and node b is node b, that is, p = b, and then dist(a, b) = h(a) - h(b).
[0060] S250: Determine a target content node set from the target tree object according to the set of title entity node pairs and the hierarchical relationship between nodes in the target tree object.
[0061] In this embodiment, the target content node refers to the node of the upper level that contains the entity node and the title node corresponding to the entity node. It can also be understood as the node corresponding to the area to which the entity content and the title corresponding to the entity content belong, such as the node corresponding to the paragraph where a certain entity content and its title are located, or the node corresponding to the table where a certain entity content and its title are located, etc.
[0062] Optionally, according to the set of title entity node pairs and the hierarchical relationship between nodes in the target tree object, a set of candidate content nodes is determined from the target tree object; and the candidate content node set is deduplicated to obtain a target content node set.
[0063] Specifically, for each title entity node pair, according to the hierarchical relationship between nodes in the target tree object, the minimum common ancestor node of the title entity node pair is determined from the target tree object; the minimum common ancestor node is used as the candidate content node of the title entity node pair. Figure 2B A specific form of the target tree object of the web page to be processed is given, such as Figure 2BThe entity node and the subtitle node shown have a minimum common ancestor node of tbody, so the tbody node is a candidate content node for the entity node and the subtitle node.
[0064] Furthermore, under the same title node, there may be multiple entity nodes. In this case, there will be the same title node in multiple title entity node pairs, so there will be multiple candidate content nodes. Therefore, the candidate content node set can be deduplicated, that is, the same candidate content nodes are removed to obtain the target content node set.
[0065] S260: Determine content location information of the web page to be processed according to the target content node set.
[0066] The technical solution of the embodiment of the present invention is to parse the web page to be processed to obtain the target tree object and the text information corresponding to the nodes in the target tree object, and then process the text information corresponding to the nodes in the target tree object based on the target title classifier and the target entity classifier, and determine the title node set and the entity node set from the target tree object according to the processing results, and then perform entity extraction on the text information corresponding to the entity node set based on the target entity recognition model to obtain the entity information of the web page to be processed, and finally perform node matching on the title node set and the entity node set to obtain the title entity node pair set, and determine the target content node set from the target tree object according to the hierarchical relationship between the title entity node pair set and the nodes in the target tree object, and determine the content location information of the web page to be processed according to the target content node set. The above technical solution determines the target content node through the hierarchical relationship between the nodes to determine the content location information, that is, to determine the area where the entity is located, so that the content location information is determined more accurately.
[0067] Embodiment 3
[0068] Figure 3 is a flow chart of a webpage information extraction method provided according to Example 3 of the present invention. Based on the above examples, this example further elaborates on the determination method of "target title classifier", "target entity classifier" and "target entity recognition model" and provides an optional implementation scheme. Figure 3 As shown, the web page information extraction method of this embodiment may include:
[0069] S310: Parse the web page to be processed to obtain a target tree object and text information corresponding to the nodes in the target tree object.
[0070] S320: Based on the target title classifier and the target entity classifier, the text information corresponding to the nodes in the target tree object is processed respectively, and a title node set and an entity node set are determined from the target tree object according to the processing results.
[0071] In this embodiment, the target title classifier and the target entity classifier can be obtained in the following manner: according to the sample classification data set, the classifier to be trained is trained to obtain the target title classifier and the target entity classifier; the sample classification data set includes text information corresponding to at least one node, and label information corresponding to at least one node.
[0072] Among them, the label information is the label of the text information corresponding to the node, that is, the label to which the node belongs, which can be a title label or an entity label. For example, for example, the paragraph to be extracted describes a project, the target entity "hospital liquid oxygen supply service" exists in the paragraph, and there is a representative title such as "Three, main label information" before the paragraph, then the words "main label information" can be annotated with the "title" label, and the words "hospital liquid oxygen supply service" can be annotated with the "entity" label; for another example, if the content to be extracted is a table, and one column of the table title is "label content", and the data corresponding to this column "IVC independent air supply system" is the target entity, then the "label content" and "IVC independent air supply system" in the table are annotated with the "title" label and the "entity" label respectively.
[0073] It should be noted that each sample data in the sample classification data set is the text information corresponding to the node, the title label corresponding to the text information, and the entity label. l ,y e ), where x(t) represents the text of node t, y l Indicates the title tag, y e Represents an entity tag.
[0074] Specifically, the classifiers to be trained can be trained separately according to the sample classification data to obtain a target title classifier and a target entity classifier.
[0075] S330: Based on the target entity recognition model, entity extraction is performed on the text information corresponding to the entity node set to obtain entity information of the web page to be processed.
[0076] In this embodiment, the sample data in the sample classification data set is also annotated based on a specific annotation tool, that is, the annotation result includes the start and end character positions of the annotated area and the path selection information relative to the DOM tree. For example, the sample classification data can be annotated through label-studio; wherein label-studio is an open source data annotation system that supports multiple data formats, can render the imported annotated HTML document, and provide users with the option of highlighting the target area by dragging and selecting text with the mouse, and the metadata of the exported annotation results provides the start and end character positions of the annotated area and the XPath path selection information relative to the DOM tree.
[0077] Specifically, from the sample classification data set, data with entity label information is extracted to construct a sample recognition data set, i.e. (x(e), r(e)), where e represents the node of the entity label, x(e) and r(e) represent the original text of node e (i.e. the text information corresponding to the node) and the text of the annotation result (i.e. the text information containing the annotation result (such as the start and end character positions of the annotation area and the XPath path selection information relative to the DOM tree)). Afterwards, the sample recognition data set is used to train the entity recognition model to be trained to obtain the target entity recognition model.
[0078] S340: Determine content location information of the web page to be processed according to the title node set and the entity node set.
[0079] The technical solution of the embodiment of the present invention parses the web page to be processed to obtain the target tree object and the text information corresponding to the nodes in the target tree object, and then processes the text information corresponding to the nodes in the target tree object based on the target title classifier and the target entity classifier, and determines the title node set and the entity node set from the target tree object according to the processing results, and then extracts the entity information corresponding to the entity node set based on the target entity recognition model to obtain the entity information of the web page to be processed, and finally determines the content location information of the web page to be processed based on the title node set and the entity node set. The above technical solution introduces the title classifier and the entity classifier, that is, the dual classifier to filter the title node and the entity node. Through these two auxiliary classifiers, information can be extracted for web pages with different structures, which enhances the versatility; at the same time, the accuracy of web page information extraction can also be improved.
[0080] Here, it should be noted that the scope of application of the present invention is to simultaneously perform entity extraction tasks and content extraction tasks of web pages, and the extracted content needs to have representative title features and contain the extracted entities. The use of the present invention under these conditions is protected.
[0081] Embodiment 4
[0082] Figure 4 1 is a schematic diagram of the structure of a web page information extraction device according to Embodiment 4 of the present invention. This embodiment is applicable to the situation of how to extract web page information. The web page information extraction device can be implemented in the form of hardware and / or software. The web page information extraction device can be integrated into an electronic device that carries the web page information extraction function, such as a server. Figure 4 As shown, the web page information extraction device of this embodiment may include:
[0083] The web page information parsing module 410 is used to parse the web page to be processed and obtain the target tree object and the text information corresponding to the node in the target tree object;
[0084] The node set determination module 420 is used to process the text information corresponding to the nodes in the target tree object based on the target title classifier and the target entity classifier, and determine the title node set and the entity node set from the target tree object according to the processing results;
[0085] An entity information determination module 430 is used to perform entity extraction on text information corresponding to an entity node set based on a target entity recognition model to obtain entity information of a web page to be processed;
[0086] The content location information determination module 440 is used to determine the content location information of the web page to be processed according to the title node set and the entity node set.
[0087] The technical solution of the embodiment of the present invention parses the web page to be processed to obtain the target tree object and the text information corresponding to the nodes in the target tree object, and then processes the text information corresponding to the nodes in the target tree object based on the target title classifier and the target entity classifier, and determines the title node set and the entity node set from the target tree object according to the processing results, and then extracts the entity information corresponding to the entity node set based on the target entity recognition model to obtain the entity information of the web page to be processed, and finally determines the content location information of the web page to be processed based on the title node set and the entity node set. The above technical solution introduces the title classifier and the entity classifier, that is, the dual classifier to filter the title node and the entity node. Through these two auxiliary classifiers, information can be extracted for web pages with different structures, which enhances the versatility; at the same time, the accuracy of web page information extraction can also be improved.
[0088] Optionally, the content location information determination module 440 includes:
[0089] A node pair determination unit, used for performing node matching on the title node set and the entity node set to obtain a title entity node pair set;
[0090] A content node set determination unit, configured to determine a target content node set from a target tree object according to a set of title entity node pairs and a hierarchical relationship between nodes in the target tree object;
[0091] The content location information determination unit is used to determine the content location information of the web page to be processed according to the target content node set.
[0092] Optionally, the node pair determination unit is specifically used for:
[0093] According to the distance between the nodes in the title node set and the nodes in the entity node set, node matching is performed on the title node set and the entity node set to obtain a title entity node pair.
[0094] Optionally, the content node set determination unit is specifically used for:
[0095] Determine a candidate content node set from the target tree object according to the set of title entity node pairs and the hierarchical relationship between nodes in the target tree object;
[0096] The candidate content node set is deduplicated to obtain a target content node set.
[0097] Optionally, the content node set determining unit is further specifically configured to:
[0098] For each title entity node pair, according to the hierarchical relationship between nodes in the target tree object, determine the minimum common ancestor node of the title entity node pair from the target tree object;
[0099] The minimum common ancestor node is used as the candidate content node of the title entity node pair.
[0100] Optionally, the device further comprises:
[0101] The classifier determination module is used to train the classifier to be trained according to the sample classification data set to obtain the target title classifier and the target entity classifier; the sample classification data set includes text information corresponding to at least one node and label information corresponding to at least one node.
[0102] Optionally, the device further includes an entity recognition model determination module, which is used to:
[0103] From the sample classification data set, extract the data with label information as entity labels to construct a sample recognition data set;
[0104] The sample recognition data set is used to train the entity recognition model to be trained to obtain the target entity recognition model.
[0105] The web page information extraction device provided in the embodiment of the present invention can execute the web page information extraction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0106] Embodiment 5
[0107] Figure 5 is a schematic diagram of the structure of an electronic device for implementing the web page information extraction method according to an embodiment of the present invention, Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0108] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0109] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0110] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a web page information extraction method.
[0111] In some embodiments, the web page information extraction method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the web page information extraction method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the web page information extraction method in any other appropriate manner (for example, by means of firmware).
[0112] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0113] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0114] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0115] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0116] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0117] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0118] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0119] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A web page information extraction method, It is characterized in that include: Parsing the web page to be processed to obtain a target tree object and text information corresponding to a node in the target tree object; Based on the target title classifier and the target entity classifier, the text information corresponding to the nodes in the target tree object is processed respectively, and a title node set and an entity node set are determined from the target tree object according to the processing results; Based on the target entity recognition model, entity extraction is performed on the text information corresponding to the entity node set to obtain the entity information of the web page to be processed; Performing node matching on the title node set and the entity node set based on the distance between the nodes to obtain a title entity node pair set; Determine a target content node set from the target tree object according to the set of title entity node pairs and the hierarchical relationship between nodes in the target tree object; Determining content location information of the web page to be processed according to the target content node set; The content location information refers to the node information to which the entity information belongs or corresponds.
2. The method according to claim 1, It is characterized in that The step of determining a target content node set from the target tree object according to the set of title entity node pairs and the hierarchical relationship between nodes in the target tree object comprises: Determine a candidate content node set from the target tree object according to the set of title entity node pairs and the hierarchical relationship between nodes in the target tree object; The candidate content node set is deduplicated to obtain a target content node set.
3. The method according to claim 2, It is characterized in that The step of determining a candidate content node set from the target tree object according to the set of title entity node pairs and the hierarchical relationship between nodes in the target tree object comprises: For each title entity node pair, determining the minimum common ancestor node of the title entity node pair from the target tree object according to the hierarchical relationship between the nodes in the target tree object; The minimum common ancestor node is used as a candidate content node for the title entity node pair.
4. The method according to claim 1, It is characterized in that Also includes: According to the sample classification data set, the classifier to be trained is trained to obtain the target title classifier and the target entity classifier; The sample classification data set includes text information corresponding to at least one node and label information corresponding to the at least one node.
5. The method according to claim 1, It is characterized in that Also includes: From the sample classification data set, extract the data with label information as entity labels to construct a sample recognition data set; The sample recognition data set is used to train the entity recognition model to be trained to obtain a target entity recognition model.
6. A web page information extraction device, It is characterized in that include: A web page information parsing module, used to parse the web page to be processed, and obtain a target tree object and text information corresponding to a node in the target tree object; A node set determination module, for processing text information corresponding to nodes in the target tree object based on a target title classifier and a target entity classifier, and determining a title node set and an entity node set from the target tree object according to the processing results; An entity information determination module is used to perform entity extraction on text information corresponding to an entity node set based on a target entity recognition model to obtain entity information of the web page to be processed; a content location information determination module, configured to perform node matching on the title node set and the entity node set based on the distance between the nodes to obtain a title entity node pair set; determine a target content node set from the target tree object according to the title entity node pair set and the hierarchical relationship between the nodes in the target tree object; and determine the content location information of the web page to be processed according to the target content node set; The content location information refers to the node information to which the entity information belongs or corresponds.
7. An electronic device, It is characterized in that The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the web page information extraction method according to any one of claims 1 to 5.
8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the web page information extraction method according to any one of claims 1 to 5 when executed.
