Method and device for identifying field relationships of network pages

Abstract

The present application relates to a method and device for identifying field relationships of a web page. In one embodiment, by obtaining a node tree of a web page and performing a depth-first traversal on the node tree, the relationship between nodes can be retained, and then the text nodes obtained by the depth-first traversal are extracted, and according to the relationship between different text nodes in the node tree, it is determined whether there is a corresponding relationship between the target fields extracted from the text nodes. In this way, without the need to use other relationship models, the corresponding relationship between fields is identified through the structural relationship of the node tree, which reduces the time for field relationship identification, improves the efficiency of information extraction, and improves the accuracy of field relationship identification.

Description

Technical Field

[0001] The present disclosure relates to the field of information analysis, and in particular to a method and device for identifying field relationships of a web page. Background Art

[0002] With the continuous development of Internet information technology, the amount of Internet information is expanding. With the further popularization of mobile Internet, the information generated by the Internet is growing exponentially every moment. Faced with such a huge Internet information database, how to better understand this information has always been an important issue in the field of information analysis.

[0003] The information currently provided on the Internet is basically web pages in HTML format, and HTML is a display-oriented markup language, the main purpose of which is to facilitate the display of web pages. For information analysis, after extracting the text information in the web page, it is necessary to first extract and classify some field information in the text information, and then further analyze the obtained field information according to the needs. After the field is extracted in the prior art, the field relationship can be determined based on the context information training relationship model, and the field relationship of the web page can also be determined based on the distance between the fields. However, when the context information is insufficient, the method of determining only by the distance between the fields has the problem of the distance being too far and exceeding the judgment standard. Therefore, how to effectively judge and identify the field relationship in the text is a key problem in the information analysis process. Summary of the invention

[0004] Based on this, in order to solve the above technical problems, a method and device for identifying field relationships of a web page are provided. The technical solution disclosed in the present invention is as follows:

[0005] According to one aspect of an embodiment of the present disclosure, a method for identifying field relationships of a web page is provided, comprising:

[0006] Get the node tree of the target web page;

[0007] Performing a depth-first traversal on the node tree to obtain the webpage nodes of the target webpage;

[0008] Performing field extraction on the text nodes in the web page nodes to obtain target fields;

[0009] Based on the node position relationship of the nodes corresponding to the target fields in the node tree, the corresponding relationship between the target fields is determined.

[0010] In one embodiment, the step of extracting fields from text nodes in the web page node to obtain target fields includes:

[0011] Taking a text node that appears alone in the webpage node as a first area to be extracted;

[0012] Taking the text nodes that appear continuously in the webpage nodes as the second area to be extracted;

[0013] Field extraction is performed on the first area to be extracted and / or the second area to be extracted according to a preset extraction method to obtain a target field.

[0014] In one embodiment, determining the correspondence between the target fields based on the positional relationship of the nodes corresponding to the target fields in the node tree includes:

[0015] Obtain the position of the node corresponding to the target field in the node tree;

[0016] Traversing the non-text nodes in the node tree, and taking the non-text nodes containing at least two of the target fields as relationship nodes;

[0017] Based on the levels of the relationship nodes in the node tree and the contents of the target fields, it is determined whether there is a corresponding relationship between the target fields.

[0018] In one embodiment, determining whether there is a corresponding relationship between the target fields based on the level of the relationship node in the node tree and the content of the target field includes:

[0019] Obtaining the level of the relationship node in the node tree;

[0020] Selecting the relationship nodes in order from the lower layer to the upper layer, and determining whether there is a conflict relationship between the target fields under the selected relationship nodes;

[0021] When there is no conflict relationship between the target fields, it is determined that there is a corresponding relationship between the target fields.

[0022] In one embodiment, obtaining the node tree of the target web page includes:

[0023] Convert the target web page into an intermediate object that can be processed by the preset language algorithm;

[0024] Nodes are obtained from the intermediate object according to the preset language algorithm, and the obtained nodes are converted into a node tree.

[0025] In one of the embodiments, the preset language algorithm includes xpath language.

[0026] According to another aspect of an embodiment of the present disclosure, a device for identifying field relationships of a web page is provided, including:

[0027] A node tree acquisition module is used to acquire the node tree of the target web page;

[0028] A depth traversal module, used to perform a depth-first traversal on the node tree to obtain the webpage nodes of the target webpage;

[0029] A field extraction module, used to extract fields from text nodes in the web page nodes to obtain target fields;

[0030] The relationship identification module is used to determine the corresponding relationship between the target fields based on the node position relationship between the nodes corresponding to the target fields in the node tree.

[0031] According to another aspect of an embodiment of the present disclosure, a computer device is further provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0032] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is further provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0033] According to another aspect of an embodiment of the present disclosure, a computer program product is further provided, including a computer program, wherein the steps of the above method are implemented when the computer program is executed by a processor.

[0034] In the technical solution provided by the embodiment of the present disclosure, by obtaining the node tree of the web page and performing a depth-first traversal on the node tree, the relationship between the nodes can be retained, and then the text nodes obtained by the depth-first traversal are extracted, and according to the relationship between different text nodes in the node tree, it is determined whether there is a corresponding relationship between the target fields extracted from the text nodes. In this way, without the need to use other relationship models, the corresponding relationship between the fields is identified through the structural relationship of the node tree, which reduces the time for field relationship identification, improves the efficiency of information extraction, and improves the accuracy of field relationship identification.

[0035] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the implementation methods of this specification or the technical solutions in the prior art, the drawings required for use in the implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some implementation methods recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0037] Figure 1 is a flow chart of a method for identifying field relationships of a network page in an embodiment;

[0038] Figure 2 is a schematic diagram of a node tree obtained by converting a web page in one embodiment;

[0039] Figure 3 In one embodiment, Figure 2 Schematic diagram of the result of depth-first traversal of the node tree in;

[0040] Figure 4 is a schematic diagram of a process of extracting fields from text nodes in an embodiment;

[0041] Figure 5 is a schematic diagram of a node tree obtained by converting a web page in another embodiment;

[0042] Figure 6 is a schematic diagram of a process for determining the corresponding relationship between target fields according to the position relationship of nodes in a node tree in an embodiment;

[0043] Figure 7 is a schematic diagram of a process of obtaining a node tree of a target web page in an embodiment;

[0044] Figure 8 is a schematic diagram of the structure of a field relationship identification device for a network page in an embodiment;

[0045] Fig. 9 The figure is a schematic diagram of the internal structure of a computer device in an embodiment. DETAILED DESCRIPTION

[0046] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0047] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described here can be implemented in an order other than those illustrated or described here. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0048] The terms "vertical", "horizontal", "left", "right", "up", "down", "front", "rear", "circumferential", "direction of travel" and similar expressions used in this document are based on the orientation or position relationship shown in the drawings and are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0049] Unless otherwise defined, the technical and scientific terms used herein may have the same meaning as that generally understood by those skilled in the art to which the technical field of the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The terms "and / or", "and / or", "at least one of..." used herein include any and all combinations of one or more related listed items. It should be noted that the connection, connection, etc. described in the present disclosure may be a direct connection through an interface or pin between devices, may be a connection through a lead, or may be a wireless connection (communication connection).

[0050] In big data analysis, in addition to extracting fields, it is also particularly important to determine the relationship between fields in the information extraction task. In the prior art, the web page is usually converted, invalid nodes are removed, and fields are extracted from the nodes containing the text information. Then, the extracted fields are identified by combining context judgment, field distance judgment, etc. to determine whether there is a corresponding relationship between these fields. However, in cases where the context information is concise or lacks detailed description, setting the relationship judgment standard based solely on the distance between fields is often not compatible with all situations. When the distance between two fields is too far, it is impossible to determine whether there is a certain relationship between the two. In this regard, there is a method that proposes to use a field relationship model to identify the relationship between extracted fields, but the web page types are complex, the relationship model has low compatibility, the matching process is cumbersome, and the accuracy is not guaranteed.

[0051] To address the above issues, a new method is urgently needed to effectively identify the correspondence between fields.

[0052] According to one aspect of the embodiment of the present disclosure, Figure 1 As shown, a method for identifying field relationships of a network page is provided, comprising:

[0053] Step S202, obtaining a node tree of a target web page.

[0054] The target web page may be the original web page used to extract information, and the node tree may be a DOM (Document Object Model) tree. DOM is a standard API for processing HTML and XML files. The DOM tree provides an access model to the entire HTML document or XML document, and treats the document as a tree structure. Each node of the tree represents an HTML tag or a text item within a tag. The nodes in the node tree have a hierarchical relationship with each other, and this relationship can be described by parent nodes, child nodes, and sibling nodes (also called brother nodes).

[0055] Specifically, the target web page can be crawled by a crawler tool or a preset language algorithm to obtain the child node and parent node information of each node, recursively traverse the entire web page, and represent the node information as a node tree.

[0056] Step S204: perform a depth-first traversal on the node tree to obtain the webpage nodes of the target webpage.

[0057] Among them, depth-first traversal is an algorithm that can be used to access nodes. In the depth-first traversal algorithm, a node is used as the vertex, and each branch is followed along this node to the end of the child node, and no other adjacent nodes are visited before reaching the end of one of the branches. When the child node at the end is reached, the previous parent node of the child node is first returned, and another child node of the parent node is selected to repeat the process. The web page node can be a node string formed by connecting several nodes that retain the depth-first traversal order.

[0058] Specifically, a depth-first traversal of the node tree can obtain a string of web page nodes in a specific order, wherein the specific order is consistent with the access order of the depth-first traversal algorithm. The web page nodes include text nodes and non-text nodes, and the non-text nodes include element nodes and attribute nodes.

[0059] Step S206, performing field extraction on the text node in the web page node to obtain a target field.

[0060] Specifically, according to the demand for data, the fields of the text nodes in the nodes can be extracted by using field extraction algorithms and the like to obtain several target fields. The field extraction algorithm can be a named entity recognition method. Among them, the named entity recognition method (Named Entity Recognition, referred to as NER) is a method that can extract specific fields such as place names, organization names, date and time from unstructured input text, and can extract more categories of fields according to demand, such as product name, model, price, etc.

[0061] In some specific implementations, for example, when a numeric field between 3 and 8 digits needs to be extracted, a corresponding script can be written in Python to automatically extract a numeric field with a specific number of digits in a text node. In some other implementations, text with a specific number of characters can also be extracted according to an algorithm.

[0062] Step S208: determining the correspondence between the target fields based on the node position relationship between the nodes corresponding to the target fields in the node tree.

[0063] The target field corresponding node may be a text node from which the target field is extracted.

[0064] Specifically, the position of the text node to which the target field belongs in the node tree can reflect the position of the target field in the node tree. The positional relationship of different text nodes in the node tree can be used to obtain the relevance of the fields they contain. The closer the level of the common parent node of these text nodes is to their own level, the stronger the relevance. For example, when two target fields are extracted from two different text nodes, and the upper level of these two text nodes in the node tree is the same parent node, it can be considered that there is a corresponding relationship between the two target fields.

[0065] In the technical solution provided by the embodiment of the present disclosure, by obtaining the node tree of the web page and performing a depth-first traversal on the node tree, the relationship between the nodes can be retained, and then the text nodes obtained by the depth-first traversal are extracted, and according to the relationship between different text nodes in the node tree, it is determined whether there is a corresponding relationship between the target fields extracted from the text nodes. In this way, without the need to use other relationship models, the corresponding relationship between the fields is identified through the structural relationship of the node tree, which reduces the time for field relationship identification, improves the efficiency of information extraction, and improves the accuracy of field relationship identification.

[0066] Figure 2 FIG. 1 is a schematic diagram of a node tree obtained by converting a web page in one embodiment. Figure 2 As shown, Html, Div, H1, p, and Strong represent non-text nodes in the node tree, and Text represents a text node in the node tree.

[0067] Figure 3 In one embodiment, Figure 2 A schematic diagram of the result of depth-first traversal of the node tree in . Figure 3 As shown, Figure 2 The first node of the depth-first traversal of the middle node tree is the initial Html node, and then the Div node of the next level and the H1 node under the Div node are visited in sequence, and finally a string of nodes with a specific order is obtained by visiting them in sequence. Figure 3 The order of arrows in the diagram indicates the order of access in depth-first traversal.

[0068] In one embodiment, Figure 4 As shown, the field extraction of the text node in the web page node to obtain the target field includes:

[0069] Step S2062: taking the text node that appears alone in the web page node as the first area to be extracted.

[0070] Specifically, after obtaining a specific sequence of web page nodes through depth-first traversal, if the previous node and the next node of a text node are both non-text nodes, the text node is used as the first area to be extracted. Figure 3In the example, the first Text that appears in the first line is a text node, and its previous node H1 and the next node Div are both non-text nodes, so this Text can be used as the first area to be extracted.

[0071] Step S2064: taking the text nodes that appear continuously in the web page nodes as the second area to be extracted.

[0072] Specifically, after obtaining a specific sequence of web page nodes through depth-first traversal, if at least one of the previous node and the next node of a text node is a text node, these adjacent text nodes are regarded as a set and the set is used as the second area to be extracted. Figure 3 In the figure, the texts in the second row appear in pairs, and each two adjacent texts can be used as a second area to be extracted.

[0073] Step S2066: Perform field extraction on the first area to be extracted and / or the second area to be extracted according to a preset extraction method to obtain a target field.

[0074] Specifically, according to the need for field extraction, a specific extraction method can be used to extract fields in the first area to be extracted and the second area to be extracted, or one of the first area to be extracted and the second area to be extracted can be selected for extraction to obtain the required target fields. In some other embodiments, a partial area in the second area to be extracted can also be selected for field extraction, for example, the second area to be extracted containing at least three text nodes can be subjected to field extraction. Among them, the specific extraction method can be implemented through Crawlab (crawler management platform), or it can be a field extraction program or script written in Python, Java or other languages.

[0075] It should be understood that, according to the method of this embodiment, several first areas to be extracted and several second areas to be extracted can be obtained.

[0076] In the above embodiment, the nodes obtained by depth-first traversal retain the order of relations in the node tree, and the text nodes therein are divided twice, and the text nodes with closer order and stronger relationship are merged into an extraction set. In this way, when extracting fields that appear in batches, the extraction area can be quickly locked, improving the efficiency of extracting and identifying fields.

[0077] Figure 5 FIG. 1 is a schematic diagram of a node tree obtained by converting a web page in another embodiment. Figure 5 As shown, Html, Div1~Div3, H1, p1~p6, Strong represent non-text nodes in the node tree, and Text1~Text13 represent text nodes in the node tree.

[0078] In one embodiment, Figure 6 As shown, the determining the correspondence between the target fields based on the positional relationship of the nodes corresponding to the target fields in the node tree includes:

[0079] Step S2082, obtaining the position of the node corresponding to the target field in the node tree.

[0080] The target field corresponding node may be a text node from which the target field is extracted.

[0081] Specifically, when extracting the target field from the text node, the name of the text node can be recorded, and the corresponding position of the text node can be found in the node tree. In some other implementations, the extraction position of the text node can also be recorded when extracting the target field, and the corresponding position can be found in the node tree according to the access order of the depth-first traversal.

[0082] Step S2084, traverse the non-text nodes in the node tree, and take the non-text nodes containing at least two of the target fields as relationship nodes.

[0083] It should be noted that each target field originates from a text node. The text node is considered to contain the target field, and the parent node of the text node is also considered to contain the target field. Accordingly, the parent node of the parent node in the node tree, the parent node of the parent node, and the root node are all considered to contain the target field. The non-text node containing at least two target fields means that the non-text node contains at least two child nodes, and both of the two child nodes contain the target field.

[0084] Specifically, the algorithm can be used to traverse all non-text nodes in the node tree, and the number of child nodes containing the target field under each non-text node is calculated. If at least two child nodes under a non-text node contain the target field, the non-text node is used as a relationship node.

[0085] For example, in Figure 5 In , if Text2~Text7 are all text nodes that contain one target field, then p1 is a non-text node that contains one target field, and the same is true for p2~p6. Since Div2 contains p1, p2, and p3, Div2 is a non-text node that contains three target fields, and the same is true for Div3. Correspondingly, Div1 is the parent node of Div2 and Div3, Html is the parent node of Div1, and Div1 and Html are both non-text nodes that contain six target fields. Figure 5 In the example, the four nodes Html, Div1, Div2, and Div3 can all be used as relationship nodes.

[0086] Step S2086: Determine whether there is a corresponding relationship between the target fields based on the levels of the relationship nodes in the node tree and the contents of the target fields.

[0087] Among them, the levels can be obtained according to the relationship between the parent node and the child node in the node tree. For example, the parent node at the root of the node tree can be used as the first-level node, the child nodes of the first-level node are the second-level nodes, the child nodes of the second-level nodes are the third-level nodes, and so on, until the end node of the node tree is reached.

[0088] Specifically, a threshold can be set according to the node tree. When the difference between the level of the relationship node and the level of the target field does not exceed the threshold, it can be considered that the target field under the relationship node may have a greater connection, and a simple algorithm is used to determine the correspondence between the fields in combination with the content of the target field. For example, if the initial parent node is a first-layer node and the last child node is a ninth-layer node, the threshold can be set to 3. When the relationship node is at the seventh layer, the target fields under the relationship node are all text nodes at the ninth layer, and the level difference between the two is 2. It can be considered that there may be a correspondence between these target fields. At this time, the content of the target field can be simply judged by field type, etc. For example, if the target fields are all numeric fields, it can be determined that there is a correspondence between the target fields under the relationship node.

[0089] It should be noted that the correspondence relationship mentioned in the present disclosure can represent that some terms or fields are determined to have a strong correlation through algorithms and other means in the process of big data analysis. In the node tree formed by web page conversion, in the process of extending from the root node of the node tree to the end node, the function, content, meaning, etc. represented by the node are continuously refined as the number of node layers increases. Therefore, it can be considered that if some nodes near the end level are close to each other, the contents contained in these nodes will have a certain connection, and the closer the node is to the end, the stronger the connection.

[0090] In the above embodiment, the number of target fields included under the non-text node can be calculated to establish the relationship node, and whether there is a corresponding relationship between the target fields can be determined according to the level of the relationship node in the node tree. In this way, the relationship of the extracted fields can be effectively identified according to the relationship represented by the node tree and the position of the node corresponding to the target field, and the recognition efficiency and accuracy are improved.

[0091] In one embodiment, determining whether there is a corresponding relationship between the target fields based on the level of the relationship node in the node tree and the content of the target field includes:

[0092] Obtaining the level of the relationship node in the node tree;

[0093] Selecting the relationship nodes in order from the lower layer to the upper layer, and determining whether there is a conflict relationship between the target fields under the selected relationship nodes;

[0094] When there is no conflict relationship between the target fields, it is determined that there is a corresponding relationship between the target fields.

[0095] Specifically, the levels of all relationship nodes in the node tree can be obtained, and starting from the relationship nodes in the lower layer, it is determined whether there is a conflict relationship between the target fields under the relationship node. In some embodiments, if a relationship node contains two target fields, one of which is a user name and the other is a user identification number, it can be considered that there is no conflict relationship between the two target fields. In some other embodiments, if a relationship node contains three target fields, one of which is a user name and the other two are user identification numbers, it can be considered that there is a conflict relationship between the three target fields.

[0096] For example: Figure 5 In the embodiment described, if Text2 to Text7 are all text nodes containing one target field, then p1 to p6 are non-text nodes containing one target field, Div2 and Div3 are non-text nodes containing three target fields, and Div1 and Html are non-text nodes containing six target fields. Figure 5 In the figure, the four nodes Html, Div1, Div2, and Div3 can all be used as relationship nodes. When judging the relationship between fields, start with Div2. If there is no conflict relationship between the three target fields extracted from Text2, Text3, and Text4, respectively, then you can record the corresponding relationship between the three target fields. Check whether there are any unjudged fields, and you will find that there are target fields extracted from the three nodes Text5, Text6, and Text7 that need to be judged. Starting from Div3, if there is no conflict relationship between the three target fields extracted from Text5, Text6, and Text7, respectively, then you can record the existence of a corresponding relationship. At this time, the corresponding relationship between all target fields has been determined, and there is no need to judge the two relationship nodes Div1 and Html, and the field relationship recognition is completed.

[0097] In the above embodiment, according to the upper and lower hierarchical relationship of the relationship nodes, the lower-level relationship nodes can be selected first to judge the conflict relationship between the target fields, so that the corresponding relationship between the target fields can be quickly determined. In this way, after confirming the relationship between the fields through the lower-level relationship nodes, there is no need to judge the upper-level relationship nodes, which reduces the amount of calculation and speeds up the field recognition speed.

[0098] In one embodiment, Figure 7As shown, the node tree of obtaining the target web page includes:

[0099] Step S2022: converting the target webpage into an intermediate object that can be processed by a preset language algorithm.

[0100] The preset language algorithm may be an algorithm written in the xpath (XML Path Language) language, or may be a regular expression or beautifulsoup4 (a Python library that can extract data from HTML or XML files) or other language algorithms that can capture nodes. The intermediate object is an object that is consistent with the processing format of the preset language algorithm.

[0101] Step S2024: Obtain nodes from the intermediate object according to the preset language algorithm, and convert the obtained nodes into a node tree.

[0102] Specifically, algorithms can be used to identify and capture intermediate objects, obtain nodes, and form a node tree.

[0103] In the above embodiment, the nodes of different types of web pages can be quickly obtained through language algorithms such as xpath and regular expressions and presented in the form of a node tree, which has strong compatibility and faster information extraction speed.

[0104] In one embodiment, the preset language algorithm includes xpath language.

[0105] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0106] According to another aspect of the present disclosure, Figure 8 As shown, a field relationship identification device for a network page is also provided, comprising:

[0107] A node tree acquisition module 502 is used to acquire a node tree of a target web page;

[0108] A depth traversal module 504 is used to perform a depth-first traversal on the node tree to obtain the webpage nodes of the target webpage;

[0109] A field extraction module 506 is used to extract fields from text nodes in the web page node to obtain target fields;

[0110] The relationship identification module 508 is used to determine the corresponding relationship between the target fields based on the node position relationship between the nodes corresponding to the target fields in the node tree.

[0111] For the specific definition of the above-mentioned identification device, please refer to the definition of the above-mentioned identification method above, which will not be repeated here. According to the above-mentioned identification method, the identification device can add a first module, a second module, etc. to implement the steps in the corresponding method embodiment. The various modules in the above-mentioned identification device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0112] According to another aspect of the embodiment of the present disclosure, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, the above-mentioned generation method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0113] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0114] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0115] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0116] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the embodiments provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but is not limited thereto.

[0117] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that those skilled in the art will easily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0119] It will be understood that the present disclosure is not limited to the exact construction that has been described and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for identifying field relationships of a web page, characterized in that: include: Get the node tree of the target web page; Performing a depth-first traversal on the node tree to obtain the webpage nodes of the target webpage; Performing field extraction on the text nodes in the web page nodes to obtain target fields; Determine the correspondence between the target fields based on the node position relationship of the nodes corresponding to the target fields in the node tree; The method of determining the correspondence between the target fields based on the positional relationship of the nodes corresponding to the target fields in the node tree includes: obtaining the position of the nodes corresponding to the target fields in the node tree; traversing the non-text nodes in the node tree, and taking the non-text nodes containing at least two of the target fields as relationship nodes; and determining whether there is a correspondence between the target fields based on the levels of the relationship nodes in the node tree and the contents of the target fields.

2. The identification method according to claim 1, characterized in that: The step of extracting fields from the text nodes in the web page nodes to obtain target fields includes: Taking a text node that appears alone in the webpage node as a first area to be extracted; Taking the text nodes that appear continuously in the webpage nodes as the second area to be extracted; Field extraction is performed on the first area to be extracted and / or the second area to be extracted according to a preset extraction method to obtain a target field.

3. The identification method according to claim 1, characterized in that: The determining whether there is a corresponding relationship between the target fields based on the level of the relationship node in the node tree and the content of the target field includes: Obtaining the level of the relationship node in the node tree; Selecting the relationship nodes in order from the lower layer to the upper layer, and determining whether there is a conflict relationship between the target fields under the selected relationship nodes; When there is no conflict relationship between the target fields, it is determined that there is a corresponding relationship between the target fields.

4. The method according to claim 1, characterized in that: The determining whether there is a corresponding relationship between the target fields based on the level of the relationship node in the node tree and the content of the target field includes: When the difference between the level at which the relationship node is located and the level at which the target field is located is less than or equal to a preset threshold, it is determined whether there is a corresponding relationship between the target fields according to the content of the target field.

5. The identification method according to claim 1, characterized in that: The node tree of obtaining the target web page includes: Convert the target web page into an intermediate object that can be processed by the preset language algorithm; Nodes are obtained from the intermediate object according to the preset language algorithm, and the obtained nodes are converted into a node tree.

6. The identification method according to claim 5, characterized in that: The preset language algorithm includes the xpath language.

7. A device for identifying field relationships of a web page, characterized in that: include: A node tree acquisition module is used to acquire the node tree of the target web page; A depth traversal module, used to perform a depth-first traversal on the node tree to obtain the webpage nodes of the target webpage; A field extraction module, used to extract fields from text nodes in the web page nodes to obtain target fields; A relationship identification module is used to determine the corresponding relationship between the target fields based on the node position relationship of the nodes corresponding to the target fields in the node tree; wherein the relationship identification module is also used to: obtain the position of the node corresponding to the target field in the node tree; traverse the non-text nodes in the node tree, and use the non-text nodes containing at least two of the target fields as relationship nodes; Based on the levels of the relationship nodes in the node tree and the contents of the target fields, it is determined whether there is a corresponding relationship between the target fields.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images