Data structure generation method and system for page document and electronic equipment
Through document parsing and tag parsing methods in Python environment, navigation tags are used as the starting point to generate structural nodes according to the tag type, solving the problem of interfering tags in complex HTML documents and improving the readability of data structures.
Patent Information
- Application Number
- CN202510137414.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-07
- Publication Date
- 2025-05-09
AI Technical Summary
When handling complex multi-layer nested HTML documents, automated parsing is difficult to accurately identify and remove interfering tags, resulting in low readability of data structures.
Using document analysis and tag analysis methods in Python environment, the navigation label is used as the starting point, and the label to be recorded or the new current label is determined according to the label type, the structural node is generated, and the page document label is parsed one by one to generate data structures and remove interference labels.
It realizes the automatic generation of data structures while improving the readability of data structures, effectively removing interfering tags in complex HTML documents.
Smart Images
Figure CN119961540A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of document parsing, and in particular to a method, system and electronic device for generating a data structure for a page document. Background Art
[0002] In the digital age, HTML (HyperText Markup Language) and other web documents have become the core carriers of Internet information display. Through the tag structure to organize and present the page content, the page can display information according to the predetermined layout and format, providing users with an intuitive and convenient browsing experience. It is widely used in corporate websites, online manuals, e-commerce platforms and various information portals. With the rapid development of Internet technology, the page content has gradually expanded from simple text and pictures to multimedia content including video, audio, animation, interactive elements, etc., and the complexity of page documents has also increased significantly, resulting in the increasingly complex hierarchical structure contained in HTML documents and other page documents, and a sharp increase in the types and number of tags.
[0003] When faced with complex multi-layer nested structures and mixed texts, manual data structure extraction is inefficient and cannot meet the needs of large-scale, high-efficiency information processing. Parsing and extracting page documents using machine learning, natural language processing and other technologies can improve the efficiency of data structure acquisition to a certain extent. However, due to the complexity and diversity of page document structures, automated methods often find it difficult to accurately identify and remove interfering tags in data structures, resulting in the extracted data structures containing a large amount of irrelevant or redundant content, which affects users' browsing and usage experience and has low readability. Summary of the invention
[0004] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.
[0005] In view of the above-mentioned shortcomings of the prior art, the present application provides a data structure generation method, system and electronic device for page documents, so as to remove interference tags while realizing automatic generation of data structure, thereby improving the readability of the data structure.
[0006] The present application provides a method for generating a data structure for a page document, comprising: obtaining a navigation tag in a page document, and using the navigation tag as a current tag; in response to the current tag, obtaining one or more subtags corresponding to the current tag from the page document, determining each of the subtags as a tag to be recorded or a new current tag according to the tag type, and generating a structure node according to the tag to be recorded; generating a data structure corresponding to the page document according to a structure node set, wherein the structure node set stores the structure nodes.
[0007] In one embodiment of the present application, obtaining navigation tags in a page document includes: passing the page document into the document parsing method, wherein the document parsing method is used to parse the page document using a preset document parsing algorithm to obtain a parsing result, and extract the navigation tags from the parsing result; the document parsing method is built based on a Python environment.
[0008] In one embodiment of the present application, the navigation tag is used as the current tag, including: using the navigation tag as a loop start tag; in response to the loop start tag, generating a set of structural nodes corresponding to the loop start tag, and using the loop start tag as the current tag; in response to the current tag, passing the current tag and the set of structural nodes into a tag parsing method, wherein the tag parsing method is used to obtain one or more sub-tags corresponding to the current tag from the page document, determine each of the sub-tags as a tag to be recorded or a new current tag according to the tag type, and generate a structural node according to the tag to be recorded; the tag parsing method is built based on the Python environment.
[0009] In one embodiment of the present application, each sub-tag is determined as a tag to be recorded or a new current tag according to the tag type, including at least one of the following: if the number of the sub-tags is one, and the tag type corresponding to the sub-tag includes a text type or a link type, the sub-tag is determined as a tag to be recorded; if the number of the sub-tags is one, and the tag type corresponding to the sub-tag does not include a text type and a link type, the sub-tag is determined as a new current tag; if the number of the sub-tags is multiple, and the tag structure corresponding to each of the sub-tags is a nested structure, the sub-tag corresponding to the text type is determined as a tag to be recorded, and the sub-tags other than the tag to be recorded are determined as new loop start tags; if the number of the sub-tags is multiple, and the tag structure corresponding to each of the sub-tags is a parallel structure, each of the sub-tags is determined as a new current tag.
[0010] In one embodiment of the present application, the first-order sub-tag is determined as the first tag, and the tag structure is determined in at least one of the following ways: if the tag type of the first tag includes a text type, the tag structure corresponding to each of the sub-tags is determined as a nested structure; if the tag type of the first tag does not include a text type, the tag structure corresponding to each of the sub-tags is determined as a parallel structure.
[0011] In one embodiment of the present application, the structure node is composed of at least the following parts: a title field, used to carry the tag name corresponding to the tag to be recorded; a path field, used to carry the path address in the link tag, wherein the link tag is any sub-tag, and the tag type corresponding to the link tag includes the link type; a node set field, used to carry the structure node set corresponding to the loop start tag in the sub-tag.
[0012] In one embodiment of the present application, a structure node is generated according to the tag to be recorded, including: in response to the current tag, generating a structure node corresponding to the current tag according to the JSON structure; if there is a tag to be recorded in the sub-tag corresponding to the current tag, recording the tag information of the tag to be recorded through the structure node, and storing the structure node in a target set, wherein the target set is a set of structure nodes passed into the tag parsing method.
[0013] In one embodiment of the present application, the method further includes: constructing a text filtering method based on a Python environment; passing the sub-tags into the text filtering method in the form of a sub-tag set, wherein the text filtering method is used to delete interfering text from the sub-tags.
[0014] The present application provides a data structure generation system for page documents, including: a document parsing module, configured to obtain a navigation tag in a page document and use the navigation tag as a current tag; a tag parsing module, configured to obtain one or more sub-tags corresponding to the current tag from the page document in response to the current tag, determine each of the sub-tags as a to-be-recorded tag or a new current tag according to the tag type, and generate a structure node according to the to-be-recorded tag; a generation module, configured to generate a data structure corresponding to the page document according to a structure node set, wherein the structure node set stores the structure nodes.
[0015] The present application provides an electronic device, comprising: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above method.
[0016] Beneficial effects of this application:
[0017] By taking the navigation tag in the page document as the current tag, and responding to the current tag, determining the subtag corresponding to the current tag as the tag to be recorded or the new current tag according to the tag type, the data structure corresponding to the page document is generated according to the structure node generated based on the tag to be recorded. In this way, taking the navigation tag in the page document as the parsing starting point, and taking the current tag as the loop starting point, the structure node is generated according to the tag to be recorded, and the tag of the interference item is taken as the new current tag, the document tags in the page document are parsed one by one in a loop, so as to generate the data structure corresponding to the page document according to the structure node, and the interference tags are removed while realizing the automatic generation of the data structure, thereby improving the readability of the data structure. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a flowchart of a method for generating a data structure for a page document in an embodiment of the present application;
[0019] Figure 2 is a flowchart of a current tag acquisition method in an embodiment of the present application;
[0020] Figure 3 is a flow chart of another method for generating a data structure for a page document in an embodiment of the present application;
[0021] Figure 4 This is a nested structure diagram corresponding to a page document in an embodiment of the present application;
[0022] Figure 5 It is a structural diagram of a data structure corresponding to a page document in an embodiment of the present application;
[0023] Figure 6 It is a structural diagram of a data structure generation system for page documents in an embodiment of the present application;
[0024] Figure 7 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION
[0025] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and sub-samples in the embodiments can be combined with each other without conflict.
[0026] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0027] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.
[0028] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.
[0029] Unless otherwise stated, the term "plurality" means two or more.
[0030] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.
[0031] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.
[0032] Combination Figure 1 As shown, the embodiment of the present disclosure provides a method for generating a data structure for a page document, comprising:
[0033] Step S101, obtaining a navigation tag in a page document, and using the navigation tag as a current tag;
[0034] Step S102, in response to the current tag, obtaining one or more sub-tags corresponding to the current tag from the page document, determining each sub-tag as a tag to be recorded or a new current tag according to the tag type, and generating a structure node according to the tag to be recorded;
[0035] Step S103, generating a data structure corresponding to the page document according to the structure node set;
[0036] The structure node set stores structure nodes.
[0037] The data structure generation method for a page document provided by the embodiment of the present disclosure is adopted. By taking the navigation tag in the page document as the current tag, and responding to the current tag, determining the subtag corresponding to the current tag as the tag to be recorded or the new current tag according to the tag type, the data structure corresponding to the page document is generated according to the structure node generated based on the tag to be recorded. In this way, taking the navigation tag in the page document as the parsing starting point, and taking the current tag as the loop starting point, a structure node is generated according to the tag to be recorded, and taking the tag of the interference item as the new current tag, the document tags in the page document are parsed one by one in a loop manner, thereby generating the data structure corresponding to the page document according to the structure node, removing the interference tags while realizing the automatic generation of the data structure, and improving the readability of the data structure.
[0038] Combination Figure 2 As shown, in step S101, obtaining the navigation tag in the page document includes:
[0039] Step S10101, constructing a document parsing method based on the Python environment;
[0040] Step S10102, passing the page document into the document parsing method;
[0041] The document parsing method is used to parse the page document using a preset document parsing algorithm to obtain a parsing result, and to extract a navigation tag from the parsing result.
[0042] In some embodiments, a document parsing method (parse_html_to_json method) is defined based on a Python environment, wherein a method input parameter of the document parsing method is an HTML document, and a method output parameter of the document parsing method is a document parsing result; through the document parsing method, BeautifulSoup is used as a document parsing algorithm to parse the HTML document to obtain a document parsing result, wherein the document parsing result includes one or more navigation tags.
[0043] Combination Figure 2 As shown, in step S101, the navigation tag is used as the current tag, including:
[0044] Step S10103, constructing a tag parsing method based on the Python environment, and using the navigation tag as the loop start tag;
[0045] Step S10104, in response to the loop start tag, generating a set of structure nodes corresponding to the loop start tag, and using the loop start tag as the current tag;
[0046] Step S10105, in response to the current tag, passing the current tag and the structure node set into the tag parsing method;
[0047] Among them, the tag parsing method is used to obtain one or more sub-tags corresponding to the current tag from the page document, determine each sub-tag as a tag to be recorded or a new current tag according to the tag type, and generate a structure node according to the tag to be recorded.
[0048] In some embodiments, a label parsing method (parse_element method) is defined based on the Python environment, wherein the method input parameters of the label parsing method include the current label and the structure node set of the loop body where the current label is located, and the method output parameters of the label parsing method include the label parsing result, and the label parsing result includes at least a part of the structure node set, the new current label, and the new loop start label.
[0049] In some embodiments, a navigation tag or a tag within a tag group is used as a loop start tag, and tags within the same nested body are parsed one by one to form a loop body; the loop start tag is used as the start mark of the loop body to generate a set of structural nodes corresponding to the loop body; the tags within the nested body are parsed one by one, and the structural nodes obtained from the loop body are stored in the set of structural nodes corresponding to the loop body, thereby distinguishing different loop bodies for easy structural parsing.
[0050] In some embodiments, current tags except navigation tags are interference tags, which are only used as placeholders for data structures without considering specific tag data. Therefore, interference tags use the structure node set in the same loop body and do not need to establish a new structure node set.
[0051] Optionally, in step S10105, the method further includes: constructing a text filtering method based on the Python environment; passing the sub-tags into the text filtering method in the form of a sub-tag set, wherein the text filtering method is used to delete interfering text from the sub-tags.
[0052] In some embodiments, a text filtering method (parse_filter_tag method) is defined based on a Python environment, wherein the method input parameter of the text filtering method is a sub-tag set, and the method output parameter of the text filtering method is a real tag set; one or more sub-tags corresponding to the current tag are obtained from the page document, and the interference text is removed from all sub-tags in the sub-tag set by the text filtering method to obtain a real tag set, and the real tag set is updated with the sub-tag set to perform tag parsing on the real tag set.
[0053] In some embodiments, if the subtag is a string element, the interference text includes line breaks and blank characters at the beginning and end of the text.
[0054] Optionally, each subtag is determined as a tag to be recorded or a new current tag according to the tag type, including at least one of the following: if the number of subtags is one, and the tag type corresponding to the subtag includes a text type or a link type, the subtag is determined as a tag to be recorded; if the number of subtags is one, and the tag type corresponding to the subtag does not include a text type and a link type, the subtag is determined as a new current tag; if the number of subtags is multiple, and the tag structure corresponding to each subtag is a nested structure, the subtag corresponding to the text type is determined as a tag to be recorded, and the subtags other than the tag to be recorded are determined as new loop start tags; if the number of subtags is multiple, and the tag structure corresponding to each subtag is a parallel structure, each subtag is determined as a new current tag.
[0055] In this way, in the process of automatically generating data structures, nested structures and parallel structures are intelligently identified, and interfering tags are ignored in a leapfrog manner, without the need to identify unnecessary tags, thereby maximizing compatibility and data structure accuracy.
[0056] In some embodiments, the loop start tag is used as the loop starting point, and the loop terminal will eventually be the parsing of a single sub-tag; if the sub-tag is of text type or link type, the tag information of the sub-tag is recorded through the structure node, and the structure node is stored in the structure node set of the same loop body to end the loop, wherein the text type sub-tag is an independent title tag, and the link type sub-tag is a hyperlink tag; if the sub-tag is not a text tag or a hyperlink tag, then a new sub-tag is nested in the sub-tag, which is an interference tag, and the sub-tag needs to be used as the new current tag to parse the tag nested in the sub-tag.
[0057] Optionally, the first-order sub-tag is determined as the first tag, and the tag structure is determined in at least one of the following ways: if the tag type of the first tag includes a text type, the tag structure corresponding to each sub-tag is determined as a nested structure; if the tag type of the first tag does not include a text type, the tag structure corresponding to each sub-tag is determined as a parallel structure.
[0058] In some embodiments, if there are multiple sub-tags, the sub-tags are arranged based on a preset order, and the sub-tags are determined in sequence from front to back in the preset order as the first-order sub-tag, the second-order sub-tag, the third-order sub-tag, and so on; wherein, when the sub-tags in the sub-tag set belong to a nested structure, the first-order sub-tag is a text type, which is used to carry the tag information corresponding to the nested structure.
[0059] In some embodiments, if there are multiple sub-tags, it is queried whether the first-ranked sub-tag is of text type; if the first-ranked sub-tag is of text type, it means that the sub-tags in the sub-tag set are of nested structure, that is, the sub-tag includes a group title tag and one or more intra-group tags, the group title tag is recorded in the structure node, and the intra-group tag is used as the loop start tag to start a new loop body; if the first-ranked sub-tag is not of text type, it means that the sub-tags in the sub-tag set are of parallel structure, each sub-tag is nested with tags, which are interference tags, and each sub-tag is used as a new current tag for further loop parsing.
[0060] Optionally, the structure node consists of at least the following parts: a title field, used to carry the tag name corresponding to the tag to be recorded; a path field, used to carry the path address in the link tag, wherein the link tag is any sub-tag, and the tag type corresponding to the link tag includes the link type; a node set field, used to carry the structure node set corresponding to the loop start tag in the sub-tag.
[0061] In some embodiments, a structure node set (items) is established for each loop body, and the structure node set is used to store the structure nodes (items) of the same loop body, wherein the structure node includes a title field (menu), a path field (path) and a node set field (children). The title field carries the label text in the title tag or hyperlink tag, the path field carries the link path in the hyperlink tag, and the node set field carries the structure node set corresponding to each label in the group, so as to characterize the connection relationship between the loop bodies.
[0062] Optionally, a structure node is generated according to the tag to be recorded, including: in response to the current tag, generating a structure node corresponding to the current tag according to the JSON structure; if there is a sub-tag corresponding to the current tag to be recorded, the tag information of the tag to be recorded is recorded through the structure node, and the structure node is stored in a target collection, wherein the target collection is a set of structure nodes passed into the tag parsing method.
[0063] In some embodiments, if in the current loop body, the number of subtags is one and the subtag is a text tag, the text tag is recorded through the title field of the structure node; the structure node is stored in the structure node set corresponding to the current loop body, and the loop ends.
[0064] In some embodiments, if in the current loop body, the number of sub-tags is one, and the sub-tag is a hyperlink tag, the tag text of the hyperlink tag is recorded through the title field of the structure node, and the connection path of the hyperlink tag is recorded through the path field of the structure node; the structure node is stored in the structure node set corresponding to the current loop body, and the loop ends.
[0065] In some embodiments, if in the current loop body, the number of subtags is one, and the subtag is not a text tag or a hyperlink tag, the subtag is nested with a new tag; the subtag and the set of structural nodes corresponding to the current loop body are passed to the tag parsing method.
[0066] In some embodiments, if in the current loop body, there are multiple sub-tags and the sub-tags belong to a nested structure, a structure node set is established for each of the intra-group tags in the sub-tags; the group title tags in the sub-tags and the structure node sets corresponding to the intra-group tags in the sub-tags are recorded through structure nodes, and the structure node records are stored in the structure node set of the current loop body; and a tag parsing method is called for the intra-group tags in the sub-tags and the structure node sets corresponding to the intra-group tags in the sub-tags.
[0067] In some embodiments, if there are multiple sub-tags in the current loop body and the sub-tags belong to a parallel structure, it is not necessary to record the sub-tags through structural nodes to achieve the purpose of removing interfering tags. Each sub-tag is combined with the structural node set of the current loop body and passed into the label parsing method.
[0068] Combination Figure 3 As shown, the embodiment of the present application provides a data structure generation for page documents, including:
[0069] Step S301, obtaining a navigation tag by parsing the page document;
[0070] Among them, the navigation tag is used as the loop start tag;
[0071] Step S302, generating a set of structure nodes corresponding to the loop start tag, and taking the loop start tag as the current tag;
[0072] Step S303, generating a structural node corresponding to the current tag, and obtaining a sub-tag corresponding to the current tag from the page document;
[0073] Step S304, determine whether the number of sub-tags is one, if yes, jump to step S305, if no, jump to step S309;
[0074] Step S305, determining whether the sub-tag is a text tag or a hyperlink tag, if so, jump to step S306, if not, jump to step S308;
[0075] Step S306, recording the sub-tag through the structure node, and adding the structure node to the structure node set;
[0076] Step S307: Generate a data structure corresponding to the page document according to each set of structural nodes.
[0077] Step S308: Use the sub-tag as the new current tag and jump to step S303.
[0078] Step S309, determine whether the first-order sub-tag is a text tag, if so, jump to step S310, if not, jump to step S312;
[0079] Step S310, recording the structure node set corresponding to the group title label and the group internal label through the structure node;
[0080] Step S311, taking the tag in the group as a new loop start tag, and jumping to step S302.
[0081] Step S312: determine each sub-tag as a new current tag, and jump to step S303.
[0082] The data structure generation method for a page document provided by the embodiment of the present disclosure is adopted. By taking the navigation tag in the page document as the current tag, and responding to the current tag, determining the subtag corresponding to the current tag as the tag to be recorded or the new current tag according to the tag type, the data structure corresponding to the page document is generated according to the structure node generated based on the tag to be recorded. In this way, taking the navigation tag in the page document as the parsing starting point, and taking the current tag as the loop starting point, a structure node is generated according to the tag to be recorded, and taking the tag of the interference item as the new current tag, the document tags in the page document are parsed one by one in a loop manner, thereby generating the data structure corresponding to the page document according to the structure node, removing the interference tags while realizing the automatic generation of the data structure, and improving the readability of the data structure.
[0083] In some embodiments, the nested structure diagram corresponding to the page document is as follows: Figure 4 As shown, the page document includes navigation tags, interference tags, real tag lists, title tags and hyperlink tags. Each square module is a document tag, and the left tag and the right tag connected to the left tag belong to the parent tag and the child tag, wherein the child tag list and the interference tag are respectively annotated in different forms; based on the data structure for the page document, the tags other than the hyperlink tag and the title tag are reflected as placeholders without considering the specific name of the tag, the interference tags are ignored, and the nested structure is cyclically parsed. The generation method generates a data structure according to the page document, wherein the data structure is as shown in Figure 5 shown.
[0084] Combination Figure 6 As shown, an embodiment of the present application provides a data structure generation system for page documents, including a document parsing module 601, a tag parsing module 602 and a generation module 603.
[0085] The document parsing module 601 is configured to obtain the navigation tag in the page document and use the navigation tag as the current tag.
[0086] The tag parsing module 602 is configured to obtain one or more subtags corresponding to the current tag from the page document in response to the current tag, determine each subtag as a tag to be recorded or a new current tag according to the tag type, and generate a structure node according to the tag to be recorded.
[0087] The generating module 603 is configured to generate a data structure corresponding to the page document according to the structure node set, wherein the structure node set stores structure nodes.
[0088] The data structure generation system for page documents provided by the embodiment of the present disclosure is adopted. By taking the navigation tag in the page document as the current tag, and responding to the current tag, determining the subtag corresponding to the current tag as the tag to be recorded or the new current tag according to the tag type, the data structure corresponding to the page document is generated according to the structure node generated based on the tag to be recorded. In this way, taking the navigation tag in the page document as the parsing starting point, and taking the current tag as the loop starting point, the structure node is generated according to the tag to be recorded, and taking the tag of the interference item as the new current tag, the document tags in the page document are parsed one by one in a loop manner, so as to generate the data structure corresponding to the page document according to the structure node, and removing the interference tags while realizing the automatic generation of the data structure, thereby improving the readability of the data structure.
[0089] An embodiment of the present disclosure further provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above method.
[0090] Figure 7 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 7 The computer system 700 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0091] like Figure 7As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703, such as executing the method in the above embodiment. In the RAM 703, various programs and data required for system operation are also stored. The CPU 701, the ROM 702 and the RAM 703 are connected to each other through the bus 704. The input / output (I / O) interface 705 is also connected to the bus 704.
[0092] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.
[0093] The electronic device disclosed in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run the computer program so that the electronic device executes each step of the above method.
[0094] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation can be changed. Parts and sub-samples of some embodiments may be included in or replace parts and sub-samples of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of a stated sub-sample, whole, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other sub-samples, wholes, steps, operations, elements, components and / or groups of these. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.
[0095] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0096] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some sub-samples can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.
[0097] The flowchart and block diagram in the accompanying drawings show the possible architecture, functions and operations of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.
Claims
1. A method for generating a data structure for a page document, characterized in that: include: Get the navigation tag in the page document, and use the navigation tag as the current tag; In response to the current tag, one or more subtags corresponding to the current tag are obtained from the page document, each of the subtags is determined as a tag to be recorded or a new current tag according to the tag type, and a structure node is generated according to the tag to be recorded; A data structure corresponding to the page document is generated according to a structure node set, wherein the structure node set stores the structure nodes.
2. The method according to claim 1, characterized in that Get the navigation tags in the page document, including: The page document is passed into the document parsing method, wherein the document parsing method is used to parse the page document using a preset document parsing algorithm to obtain a parsing result, and extract navigation tags from the parsing result; the document parsing method is built based on a Python environment.
3. The method according to claim 1, characterized in that Taking the navigation tag as the current tag includes: Using the navigation tag as a loop start tag; In response to a loop start tag, generating a structure node set corresponding to the loop start tag, and using the loop start tag as a current tag; In response to the current tag, the current tag and the set of structural nodes are passed into a tag parsing method, wherein the tag parsing method is used to obtain one or more sub-tags corresponding to the current tag from the page document, determine each of the sub-tags as a tag to be recorded or a new current tag according to the tag type, and generate a structural node according to the tag to be recorded; the tag parsing method is built based on the Python environment.
4. The method according to claim 3, characterized in that Determining each of the subtags as a tag to be recorded or a new current tag according to the tag type includes at least one of the following: If the number of the subtag is one, and the tag type corresponding to the subtag includes a text type or a link type, the subtag is determined as a tag to be recorded; If the number of the sub-tag is one, and the tag type corresponding to the sub-tag does not include a text type and a link type, the sub-tag is determined as a new current tag; If there are multiple subtags, and the tag structure corresponding to each subtag is a nested structure, the subtag corresponding to the text type is determined as the tag to be recorded, and the subtags other than the tag to be recorded are determined as new loop start tags; If there are multiple sub-tags, and the tag structure corresponding to each sub-tag is a parallel structure, each sub-tag is determined as a new current tag.
5. The method according to claim 4, characterized in that The first-order subtag is determined as the first tag, and the tag structure is determined by at least one of the following methods: If the tag type of the first tag includes a text type, determining the tag structure corresponding to each of the subtags to be a nested structure; If the tag type of the first tag does not include a text type, the tag structure corresponding to each of the sub-tags is determined to be a parallel structure.
6. The method according to claim 4, characterized in that The structural node is composed of at least one of the following: A title field, used to carry the tag name corresponding to the tag to be recorded; A path field, used to carry a path address in a link tag, wherein the link tag is any sub-tag, and the tag type corresponding to the link tag includes a link type; The node set field is used to carry the structure node set corresponding to the loop start tag in the subtag.
7. The method according to claim 3, characterized in that Generating a structure node according to the tag to be recorded includes: In response to the current tag, generating a structure node corresponding to the current tag according to the JSON structure; If there is a tag to be recorded in the subtag corresponding to the current tag, the tag information of the tag to be recorded is recorded through the structure node, and the structure node is stored in the target set, wherein the target set is the structure node set passed into the tag parsing method.
8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Build text filtering methods based on Python environment; The sub-tags are passed to the text filtering method in the form of a sub-tag set, wherein the text filtering method is used to delete interfering text from the sub-tags.
9. A data structure generation system for page documents, characterized in that: include: The document parsing module is configured to obtain a navigation tag in a page document and use the navigation tag as a current tag; A tag parsing module is configured to, in response to a current tag, obtain one or more subtags corresponding to the current tag from the page document, determine each of the subtags as a tag to be recorded or a new current tag according to the tag type, and generate a structure node according to the tag to be recorded; The generating module is configured to generate a data structure corresponding to the page document according to a structure node set, wherein the structure node set stores the structure nodes.
10. An electronic device, characterized in that: include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method according to any one of claims 1 to 8.