A web page analysis method and system for a vertical domain knowledge graph and a medium
By using node embedding algorithms and classification models, the problem of low efficiency and accuracy in webpage parsing in vertical domain knowledge graphs is solved, enabling fast and accurate extraction of webpage information, adapting to external changes, and improving parsing efficiency and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU SHIYUAN ELECTRONICS CO LTD
- Filing Date
- 2021-07-21
- Publication Date
- 2026-06-05
AI Technical Summary
When building knowledge graphs for vertical domains, existing technologies struggle to efficiently and accurately parse complex web page structures, resulting in long parsing times and low accuracy.
A node embedding algorithm is used for webpage parsing. Combined with a classification model and a vertical domain knowledge graph, the algorithm quickly identifies and extracts important information in the vertical neighborhood through preprocessing, node tree construction, vectorization learning, and simplification of classification results.
It improves the efficiency and accuracy of webpage parsing, adapts to external changes, saves parsing time, and enhances the semantic expression capabilities of nodes, meeting the needs of knowledge graph construction.
Smart Images
Figure CN115687556B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of knowledge graph technology, and more specifically, to a webpage parsing method, system, and medium for vertical domain knowledge graphs. Background Technology
[0002] Knowledge graphs currently provide an effective way to represent, organize, manage, and apply massive amounts of data on the internet, making knowledge representation more intelligent. Examples include recommendation search, intelligent reasoning, and intelligent question answering. The application of knowledge graphs requires substantial data support. For instance, general-purpose knowledge graphs need a large amount of encyclopedia entries, terms, and news information. Vertical domain knowledge graphs require a wealth of expert knowledge, financial information, and research reports within that domain. Therefore, efficiently and quickly crawling external information and processing it into the triples required for knowledge graphs is particularly important.
[0003] Currently, to better serve their internal supply chains, companies are building industry chain knowledge graphs in various vertical sectors. This allows them to better understand the upstream and downstream relationships of a product, quickly analyze corresponding manufacturers, and grasp real-time changes. However, building industry chain graphs requires crawling large amounts of external information, and different information sources have different web page structures, making it technically challenging.
[0004] Furthermore, traditional webpage parsing typically involves converting the webpage structure into a node tree structure and traversing the tree to parse the webpage nodes. However, with increasingly rich webpage elements and complex structures, the number of nodes in the constructed node tree is also growing. Parsing nodes using a complete traversal approach limits parsing time to some extent. Moreover, for webpage parsing aimed at building knowledge graphs for vertical domains, the crawled webpage information is generally related data within that specific domain, and simply parsing the node tree from the webpage structure yields unsatisfactory accuracy. Summary of the Invention
[0005] Based on the aforementioned technical deficiencies, this invention aims to perform webpage parsing and extraction based on a node embedding algorithm, quickly remove useless webpages using a classification model to accelerate webpage parsing efficiency, and combine it with a knowledge graph of a vertical domain to quickly identify and extract important information within a specific vertical neighborhood on a webpage.
[0006] To achieve the above technical objectives, this application provides a webpage parsing method for vertical domain knowledge graphs, comprising the following steps:
[0007] Preprocess the webpage;
[0008] A node tree is constructed based on the preprocessed webpage, and the relationships between the nodes and their contexts, as well as the class tags that identify different webpage elements, are obtained.
[0009] The node embedding algorithm enables vectorization learning for each node in the node tree, thereby obtaining the embedding representation of each node.
[0010] Obtain the knowledge graph of the first vertical domain;
[0011] The first vertical domain knowledge graph, the embedded representation, and the class label are input into the trained classification model to obtain the classification result;
[0012] The node tree is simplified based on the classification results.
[0013] Specifically, the preprocessing includes removing advertisements, friend links, and pop-up information from the webpage.
[0014] Preferably, the webpage parsing method for vertical domain knowledge graphs further includes extracting information from the simplified node tree to supplement the first vertical domain knowledge graph, which serves as the second vertical domain knowledge graph.
[0015] Furthermore, the node embedding-based algorithm enables vectorization learning for each node in the node tree, including:
[0016] Initialize each node in the node tree as a 256-dimensional random vector;
[0017] The random vector corresponding to each node in the node tree is input into the trained vector learning model;
[0018] The trained vector learning model outputs an embedding representation of each node, wherein the embedding representation is a vector that has a long short-term context relationship with each node.
[0019] Furthermore, the method for obtaining the first vertical domain knowledge graph includes:
[0020] Extract the text information of each node in the node tree;
[0021] Named entity recognition is performed on the text information of each node to obtain a first vertical domain knowledge graph.
[0022] Furthermore, information is extracted from the simplified node tree to supplement the first vertical domain knowledge graph, forming the second vertical domain knowledge graph, including:
[0023] Extract the text information of each node in the simplified node tree;
[0024] Named entity recognition is performed on the text information of each node to obtain a third vertical domain knowledge graph;
[0025] Based on the entity link graph database involved in the third vertical domain knowledge graph;
[0026] The attribute features associated with the entity are added as new nodes to the first vertical domain knowledge graph from the graph database, forming the second vertical domain knowledge graph.
[0027] Preferably, the vector learning model will be trained and continuously optimized during the training process using a loss function, wherein the loss function is:
[0028]
[0029] Where N represents the number of nodes in the node tree, J represents the number of long and short-term context nodes of the nth node, and P(e j |e n ) represents conditional probability.
[0030] A second aspect of the present invention provides a webpage parsing system for vertical domain knowledge graphs, the system comprising:
[0031] The preprocessing module is used to preprocess web pages;
[0032] The node tree construction module is used to build a node tree based on the preprocessed webpage, and obtain the node tree nodes and their context relationships, as well as the class tags that identify different webpage elements;
[0033] The vector learning module is used to learn the vectorization of each node in the node tree based on the node embedding algorithm, so as to obtain the embedding representation of each node.
[0034] The graph acquisition module is used to acquire the knowledge graph of the first vertical domain.
[0035] The classification module is used to input the first vertical domain knowledge graph, the embedding representation, and the class label into the trained classification model to obtain the classification result;
[0036] A simplification module is used to simplify the node tree based on the classification results.
[0037] A third aspect of the present invention provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, which, when executed by the processor, cause the processor to perform the following steps:
[0038] Preprocess the webpage;
[0039] A node tree is constructed based on the preprocessed webpage, and the relationship between the node tree nodes and their context, as well as the class tags that identify different webpage elements, are obtained.
[0040] The node embedding algorithm enables vectorization learning for each node in the node tree, thereby obtaining the embedding representation of each node.
[0041] Obtain the knowledge graph of the first vertical domain;
[0042] The first vertical domain knowledge graph, the embedded representation, and the class label are input into the trained classification model to obtain the classification result;
[0043] The node tree is simplified based on the classification results.
[0044] A fourth aspect of the present invention provides a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the following steps:
[0045] Preprocess the webpage;
[0046] A node tree is constructed based on the preprocessed webpage, and the relationship between the node tree nodes and their context, as well as the class tags that identify different webpage elements, are obtained.
[0047] The node embedding algorithm enables vectorization learning for each node in the node tree, thereby obtaining the embedding representation of each node.
[0048] Obtain the knowledge graph of the first vertical domain;
[0049] The first vertical domain knowledge graph, the embedded representation, and the class label are input into the trained classification model to obtain the classification result;
[0050] The node tree is simplified based on the classification results.
[0051] The beneficial effects of this application are as follows: The method provided by this invention can better overcome the limitations of logical rules, utilize external features to parse web pages, better adapt to external changes, and save parsing time. In particular, this application considers the contextual structure of the web page itself and combines it with vertical domain knowledge graph information, thereby increasing the semantic expression capability of nodes and improving the accuracy of parsing. In addition, since the process of constructing a vertical domain knowledge graph to parse web pages is an iterative and reinforcing process, the entire parsing process is made more in line with the needs of knowledge graph construction while accelerating web page parsing. Attached Figure Description
[0052] Figure 1 A schematic diagram of the method flow of Embodiment 1 of this application is shown;
[0053] Figure 2 A schematic diagram of the method flow of Embodiment 2 of this application is shown;
[0054] Figure 3 A schematic diagram of the node tree structure in Embodiment 2 of this application is shown;
[0055] Figure 4 A schematic diagram of the vector learning model structure in Embodiment 2 of this application is shown;
[0056] Figure 5 A schematic diagram of the classification model structure in Embodiment 2 of this application is shown;
[0057] Figure 6 A schematic diagram of the system structure of Embodiment 3 of this application is shown;
[0058] Figure 7 This illustration shows a schematic diagram of the structure of an electronic device according to an embodiment of this application;
[0059] Figure 8 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation
[0060] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application. It will be apparent to those skilled in the art that this application can be implemented without one or more of these details. In other instances, some technical features well-known in the art have not been described to avoid confusion with this application.
[0061] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or combinations thereof.
[0062] Exemplary embodiments according to this application will now be described in more detail with reference to the accompanying drawings. However, these exemplary embodiments may be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. The drawings are not drawn to scale, and some details may be enlarged and omitted for clarity. The shapes of the various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.
[0063] Example 1:
[0064] This embodiment implements a webpage parsing method oriented towards vertical domain knowledge graphs, such as... Figure 1 As shown, it includes the following steps:
[0065] S1. Preprocess the webpage;
[0066] S2. Construct a node tree based on the preprocessed webpage to obtain the node tree nodes and their context relationships, as well as the class tags that identify different webpage elements;
[0067] S3. Based on the node embedding algorithm, each node in the node tree is vectorized and learned to obtain the embedding representation of each node;
[0068] S4. Obtain the knowledge graph of the first vertical domain;
[0069] S5. Input the first vertical domain knowledge graph, the embedded representation, and the class label into the trained classification model to obtain the classification result;
[0070] S6. Simplify the node tree based on the classification results.
[0071] In this context, the node tree is also represented as the DOM tree, the node embedding algorithm can be represented as the NodeEmbedding algorithm, the embedding representation is the embedding representation, and the class tag is also represented as the class tag. Figure 1 The same applies to the Chinese and other embodiments.
[0072] Specifically, preprocessing includes removing advertisements, friend links, and pop-up messages from web pages.
[0073] Preferably, the webpage parsing method for vertical domain knowledge graphs further includes extracting information from the simplified node tree to supplement the first vertical domain knowledge graph, which serves as the second vertical domain knowledge graph.
[0074] Furthermore, based on the node embedding algorithm, each node in the node tree undergoes vectorization learning, including:
[0075] Initialize each node in the node tree as a 256-dimensional random vector;
[0076] The random vector corresponding to each node in the node tree is input into the trained vector learning model;
[0077] The trained vector learning model outputs the embedding representation of each node, where the embedding representation is a vector that has long-short-term contextual relationships with each node.
[0078] Furthermore, methods for obtaining the knowledge graph of the first vertical domain include:
[0079] Extract the text information of each node in the node tree;
[0080] Named entity recognition (NER) is performed on the text information of each node to obtain the first vertical domain knowledge graph.
[0081] Furthermore, information is extracted from the simplified node tree to supplement the first vertical domain knowledge graph, forming the second vertical domain knowledge graph, including:
[0082] Extract the text information of each node in the simplified node tree;
[0083] Named entity recognition is performed on the text information of each node to obtain a knowledge graph of the third vertical domain;
[0084] Based on the entity link graph database involved in the knowledge graph of the third vertical domain;
[0085] The attribute features associated with entities in the graph database are used as new nodes to supplement the first vertical domain knowledge graph, which then becomes the second vertical domain knowledge graph.
[0086] Preferably, the vector learning model will be trained and continuously optimized during the training process using a loss function, which is:
[0087]
[0088] Where N represents the number of nodes in the node tree, J represents the number of long and short-term context nodes of the nth node, and P(e j |e n The conditional probability is represented by (). During training, cross-entropy can be used to represent the conditional probability between two nodes. The entire network is trained unsupervised using the long and short-term correlations of nodes to create the vector representation of the current node. Therefore, the embedding representation of each node contains rich contextual relationships and webpage semantic information.
[0089] In addition to training the embedding representation of each node in the node tree, all web page elements with class tags in each webpage are also collected. Because elements with class tags have obvious category information in the webpage structure, they can be represented by embedding vectors for use in subsequent classification model training. Here, random initialization is used directly for embedding, and then it is trained together with the classification model. The classification model is mainly used to determine whether a node is useful in the construction of the knowledge graph for the current vertical domain. This embodiment uses a binary classification model, discarding useless nodes to reduce the number of parsed nodes.
[0090] This embodiment utilizes node embedding algorithms and classification models to quickly and accurately simplify the node tree structure during webpage parsing, thus accelerating parsing speed and accuracy. Introducing vertical domain graph information during webpage parsing allows the classification model to better learn from the information constructed in the vertical domain graph. Through continuous iteration, the model can gradually better understand relevant webpage information within the vertical domain, thereby improving parsing accuracy. The embedding vectors of webpage nodes can also be applied to webpage similarity and classification. Therefore, when constructing a vertical domain graph, similar webpage content within that domain can be quickly found, accelerating the crawling of knowledge graph information.
[0091] Example 2:
[0092] This embodiment implements a webpage parsing method oriented towards vertical domain knowledge graphs, such as... Figure 2 As shown, the process includes: webpage preprocessing, constructing a node tree (DOM tree), node embedding (NodeEmbedding vector learning), classification using a classification model, simplifying the node tree, extracting webpage information, and creating a vertical domain knowledge graph. Detailed explanations of the steps are as follows.
[0093] The first step is to preprocess the webpage. Preprocessing includes removing advertisements, backlinks, and pop-up messages from the webpage, including promotional messages and other useless information.
[0094] The second step involves constructing a DOM tree based on the preprocessed webpages, obtaining the DOM tree nodes, their context relationships, and the class tags that identify different webpage elements. In webpage parsing, each webpage is equivalent to a DOM tree; the HTML elements on the webpage are the tree nodes, and the text and images on the webpage are the information carried by each node. Furthermore, the paths between nodes form context relationships, such as... Figure 3 The node tree structure shown consists of different nodes, including a root node and leaf nodes. C1, C2, C3, and C4 identify different classes. For example, C3 can represent the "information" class, and C4 can represent the "surface capacitor".
[0095] The third step involves using the NodeEmbedding algorithm to perform vectorization learning on each node in the DOM tree, obtaining the embedding representation of each node. Each node in the DOM tree is initialized as a 256-dimensional random vector; the random vector corresponding to each node in the DOM tree is input into the trained vector learning model; the trained vector learning model outputs the embedding representation of each node, where the embedding representation is a vector with long-short-term contextual relationships to each node.
[0096] Specifically, the vector learning model is a Long-Short Skip-gram network. Figure 4 This is the network structure diagram of the Long-Short Skip-gram module, where θ(t)∈R256 can be viewed as a node vector in a node tree, initially obtained by one-hot encoding. t represents the node number. s represents the short-term hierarchical relationship node number of t, and l represents the long-term hierarchical relationship node number of t. As can be seen from the DOM tree structure, the parent and child nodes of a node contain a significant amount of related information about the current node. In the webpage structure, since the DOM tree structure is often quite deep, a node may have related relationships with multiple nodes before or after it. Therefore, based on the statistical information of each webpage, the mode of the long-term hierarchical relationships is obtained, thus yielding the long-term hierarchical related nodes of t. W∈R256x256, where W is a weight matrix, and the vector embedding representation of node t is obtained by multiplying e(t) and W.
[0097] Preferably, the vector learning model will be trained and continuously optimized during the training process using a loss function, which is:
[0098]
[0099] Where N represents the number of nodes in the node tree, J represents the number of long and short-term context nodes of the nth node, and P(e j |e n The conditional probability is represented by (). During training, cross-entropy can be used to represent the conditional probability between two nodes. The entire network is trained unsupervised using the long and short-term correlations of nodes to create the vector representation of the current node. Therefore, the embedding representation of each node contains rich contextual relationships and webpage semantic information.
[0100] In addition to training the embedding representation of each node in the DOM tree, we also collect all web page elements with class tags in each web page. This is because elements with class tags have obvious category information in the web page structure, so they can be embedded and used for training the subsequent classification model. Here, we directly use random initialization to perform embedding, and then train it together with the classification model.
[0101] The fourth step is to obtain the knowledge graph for the first vertical domain. This includes: extracting the text information from each node in the DOM tree; performing named entity recognition (NER) on the text information of each node to obtain the knowledge graph for the first vertical domain. NER can utilize a pre-trained NER model. For example, in the semiconductor field, if the text information in a node on a webpage is: "The price increase of surface mount capacitors has led to an increase in production costs for downstream industries," a pre-trained NER model can identify that this text information contains the entity "surface mount capacitor."
[0102] The fifth step involves inputting the first vertical domain knowledge graph, the embedded representation, and the class labels into the trained classification model to obtain the classification result. The classification model is primarily used to determine whether a node is useful in the construction of the current vertical domain knowledge graph. This embodiment employs a binary classification model, discarding useless nodes directly to reduce the number of parsed nodes. Figure 5 The diagram shows the structure of the classification model. Using the wide & deep approach, when predicting the class of node e(t), as follows... Figure 5 On the left, the context nodes are first linearly combined with the current node, enabling the model to learn the linear combination relationships between different nodes, resulting in a generalized linear model. And... Figure 5 The right side is a fully connected generalization model; MLP stands for Multilayer Perceptron, which provides the model with generalization capabilities. Finally, the left side (contextual linear combination information) and the right side (node semantic category vector representation information) are concatenated using the sigmoid activation function, resulting in a 0-1 variable as the final output. 0 represents an invalid node, and 1 represents a valid node.
[0103] Step 6: Simplify the node tree based on the classification results. The binary classification model discards useless nodes directly, that is, discards the nodes corresponding to 0 and retains the nodes corresponding to 1 as useful nodes.
[0104] The seventh step involves extracting information from the simplified node tree to supplement the first vertical domain knowledge graph, thus creating the second vertical domain knowledge graph. Specifically, this includes: extracting the text information of each node in the simplified node tree; performing named entity recognition on the text information of each node to obtain the third vertical domain knowledge graph; based on the entity link graph database involved in the third vertical domain knowledge graph; and supplementing the first vertical domain knowledge graph with the attribute features associated with entities as new nodes from the graph database, thus creating the second vertical domain knowledge graph.
[0105] For example, in the semiconductor field, the text information within a certain node on a webpage might be: "The price increase of surface mount capacitors has led to increased production costs in downstream industries." Using a trained Negative Entity Recognition (NER) model, this text information can be identified as containing the entity "[surface mount capacitor]". Then, an entity linking model is used to link it to the "[surface mount capacitor]" entity in the vertical domain knowledge graph. In this vertical domain knowledge graph, the entity "[surface mount capacitor]" may have many attributes, such as voltage, current, price, and upstream / downstream supply chains, already stored in the graph database. These attributes are retrieved from the graph and used as supplementary features for the current node's text, making the current node contain more semantic features relevant to the vertical domain. Subsequently, this new knowledge graph, i.e., the second vertical domain knowledge graph, is input into a classification model for classification.
[0106] It should be noted that when inputting the vertical domain knowledge graph, embedding representation, and class labels into the trained classification model, the vertical domain knowledge graph (including the first and second vertical domain knowledge graphs) and class labels, like the embedding representation, need to be converted into vectors. Through continuous iteration, the entire model can gradually better understand the relevant web page information in the vertical domain, thereby improving the parsing accuracy.
[0107] Example 3:
[0108] This embodiment implements a webpage parsing system oriented towards vertical domain knowledge graphs, such as... Figure 6 As shown, it includes:
[0109] Preprocessing module 601 is used to preprocess web pages;
[0110] The node tree construction module 602 is used to construct a node tree based on the preprocessed web page, and obtain the node tree, its context relationship, and class tags that identify different web page elements.
[0111] The vector learning module 603 is used to perform vectorization learning on each node in the node tree based on node embedding to obtain the embedding representation of each node;
[0112] Graph acquisition module 604 is used to acquire the knowledge graph of the first vertical domain.
[0113] The classification module 605 is used to input the first vertical domain knowledge graph, the embedding representation, and the class label into the trained classification model to obtain the classification result;
[0114] The simplification module 606 is used to simplify the node tree based on the classification results.
[0115] As a possible implementation, this embodiment may also include a supplementary module for extracting information from the simplified node tree to supplement the first vertical domain knowledge graph, serving as the second vertical domain knowledge graph.
[0116] Please refer to the following. Figure 7 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 7 As shown, the electronic device 2 includes: a processor 200, a memory 201, a bus 202, and a communication interface 203. The processor 200, the communication interface 203, and the memory 201 are connected via the bus 202. The memory 201 stores a computer program that can run on the processor 200. When the processor 200 runs the computer program, it executes the web page parsing method for vertical domain knowledge graphs provided in any of the foregoing embodiments of this application.
[0117] The memory 201 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 203 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.
[0118] Bus 202 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. Memory 201 is used to store programs. After receiving execution instructions, processor 200 executes the programs. The webpage parsing method for vertical domain knowledge graphs disclosed in any of the foregoing embodiments of this application can be applied to processor 200, or implemented by processor 200.
[0119] The processor 200 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 200 or by instructions in software form. The processor 200 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 201. The processor 200 reads the information in memory 201 and, in conjunction with its hardware, completes the steps of the above method.
[0120] The electronic device provided in this application embodiment and the webpage parsing method for vertical domain knowledge graphs provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.
[0121] This application also provides a computer-readable storage medium corresponding to the webpage parsing method for vertical domain knowledge graphs provided in the foregoing embodiments. Please refer to... Figure 8 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the web page parsing method for vertical domain knowledge graphs provided in any of the foregoing embodiments.
[0122] Examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which will not be described in detail here.
[0123] It should be noted that the algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. Various general-purpose devices can also be used in conjunction with the teachings herein. The required structure for constructing such devices is readily apparent from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages is for the purpose of disclosing the best mode of implementation of this application. Numerous specific details are set forth in the specification provided herein. However, it is to be understood that embodiments of this application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification. Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more aspects of the invention, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of this application. However, this disclosed approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim.
[0124] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A webpage parsing method for vertical domain knowledge graphs, characterized in that, Includes the following steps: Preprocess the webpage; A node tree is constructed based on the preprocessed webpage, and the relationships between the nodes and their contexts, as well as the class tags that identify different webpage elements, are obtained. The node embedding algorithm enables vectorization learning for each node in the node tree, thereby obtaining the embedding representation of each node. Obtain the knowledge graph of the first vertical domain; The first vertical domain knowledge graph, the embedded representation, and the class label are input into the trained classification model to obtain the classification result; The node tree is simplified based on the classification results; The method further includes extracting information from the simplified node tree to supplement the first vertical domain knowledge graph, serving as the second vertical domain knowledge graph, specifically: Extract the text information of each node in the simplified node tree; Named entity recognition is performed on the text information of each node to obtain a third vertical domain knowledge graph; Based on the entity link graph database involved in the third vertical domain knowledge graph; The attribute features associated with the entity are added as new nodes to the first vertical domain knowledge graph from the graph database, forming the second vertical domain knowledge graph.
2. The webpage parsing method for vertical domain knowledge graphs according to claim 1, characterized in that, The preprocessing includes removing advertisements, friend links, and pop-up information from the webpage.
3. The webpage parsing method for vertical domain knowledge graphs according to claim 1, characterized in that, The node embedding-based algorithm enables vectorized learning for each node in the node tree, including: Initialize each node in the node tree as a 256-dimensional random vector; The random vector corresponding to each node in the node tree is input into the trained vector learning model; The trained vector learning model outputs an embedding representation of each node, wherein the embedding representation is a vector that has a long short-term context relationship with each node.
4. The webpage parsing method for vertical domain knowledge graphs according to claim 1, characterized in that, The method for obtaining the first vertical domain knowledge graph includes: Extract the text information of each node in the node tree; Named entity recognition is performed on the text information of each node to obtain a first vertical domain knowledge graph.
5. The webpage parsing method for vertical domain knowledge graphs according to claim 3, characterized in that, The vector learning model is optimized during training using a loss function, which is: Where N represents the number of nodes in the node tree, J represents the number of long-short-term context nodes of the nth node, and P(e j |e n ) represents conditional probability.
6. A webpage parsing system for vertical domain knowledge graphs, characterized in that, The system includes: The preprocessing module is used to preprocess web pages; The node tree building module is used to build a node tree based on the preprocessed web page, and obtain the node tree, its context relationship, and class tags that identify different web page elements. The vector learning module is used to learn the vectorization of each node in the node tree based on the node embedding algorithm, so as to obtain the embedding representation of each node. The graph acquisition module is used to acquire the knowledge graph of the first vertical domain. The classification module is used to input the first vertical domain knowledge graph, the embedding representation, and the class label into the trained classification model to obtain the classification result; A simplification module is used to simplify the node tree based on the classification results; The system also includes a supplementation module, used to extract information from the simplified node tree to supplement the first vertical domain knowledge graph, serving as the second vertical domain knowledge graph. The supplementation module specifically performs the following steps: Extract the text information of each node in the simplified node tree; Named entity recognition is performed on the text information of each node to obtain a third vertical domain knowledge graph; Based on the entity link graph database involved in the third vertical domain knowledge graph; The attribute features associated with the entity are added as new nodes to the first vertical domain knowledge graph from the graph database, forming the second vertical domain knowledge graph.
7. A computer device, comprising a memory and a processor, characterized in that, The memory stores computer-readable instructions that, when executed by the processor, cause the processor to perform the steps of the method as claimed in any one of claims 1 to 5.
8. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the steps of the method as claimed in any one of claims 1 to 5.