Web page parsing method, device, electronic device and medium

Abstract

The present application provides a web page parsing method, device, electronic device and medium, wherein the method comprises: obtaining a document object model of a target web page to be parsed; loading a target page parsing rule configured by a user in a preset format; obtaining a tag type in the target page parsing rule; for each tag type, using a parsing method corresponding to the tag type, parsing web page data belonging to the tag type in the document object model. The embodiments of the present application can improve the versatility of web page parsing and reduce a large amount of repetitive development work for developers.

Description

Technical Field

[0001] The present application relates to the technical field of web page parsing, and in particular to a web page parsing method, device, electronic device and medium. Background Art

[0002] At present, the web page parsing method is to write a set of independent parsing modules according to the structure of the specific web page. It is difficult to reuse the code when parsing different web pages, which will cause a lot of duplication of work for developers. In addition, the parsing of web pages is completed by developers. If the requirements need to be changed (such as web page revision issues), users often need to communicate frequently with developers.

[0003] The web page data that needs to be parsed in data collection work has two main structural layout forms: list ( ),sheet( 、 ) and other row-level tags. However, when faced with data parsing tasks for many different websites (pages), due to the variety of page structure layouts, it is necessary to write specific parsing methods for each page, which results in a lot of relatively repetitive development work. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a web page parsing method, device, electronic device and medium to improve the versatility of web page parsing and reduce a lot of repetitive development work for developers.

[0005] In a first aspect, an embodiment of the present application provides a web page parsing method, comprising:

[0006] Obtain the document object model of the target web page to be parsed;

[0007] Load the target page parsing rules configured by the user in a preset format;

[0008] Obtaining the tag type in the target page parsing rule;

[0009] For each tag type, a parsing method corresponding to the tag type is used to parse the web page data belonging to the tag type in the document object model.

[0010] In a possible implementation, the tag type includes at least one of a row-level tag, a block-level tag, and an inline block tag, and the block-level tag includes a list-type block-level tag and a table-type block-level tag.

[0011] In a possible implementation, for each tag type, a parsing method corresponding to the tag type is used to parse the web page data belonging to the tag type in the document object model, including:

[0012] For the row-level tag, the parsing method corresponding to the row-level tag in the target page parsing rule is adopted to parse the web page data belonging to the row-level tag in the document object model.

[0013] In a possible implementation, for each tag type, a parsing method corresponding to the tag type is used to parse the web page data belonging to the tag type in the document object model, including:

[0014] For a block-level tag of a list type, locate all subtags of the block-level tag of the list type in the document object model;

[0015] If the list-type block-level tag is not configured with document parameters, the text data in all sub-tags of the block-level tag is obtained.

[0016] In a possible implementation, the method further includes:

[0017] If the list-type block-level tag is configured with a document parameter, traverse each item in the document parameter, split each sub-tag of the block-level tag, and obtain dictionary-type sub-tag data;

[0018] Merge the dictionary-type sub-tag data corresponding to all sub-tags of the block-level tag.

[0019] In a possible implementation, for each tag type, a parsing method corresponding to the tag type is used to parse the web page data belonging to the tag type in the document object model, including:

[0020] For a block-level tag of a table type, locate all subtags of the block-level tag of the table type in the document object model;

[0021] Each sub-tag of the block-level tag is split, and the split data is assembled into dictionary-type data.

[0022] In a possible implementation, obtaining a document object model of a target web page to be parsed includes:

[0023] Obtain the source code of the target web page to be parsed;

[0024] The source code of the target web page is assembled into a document object model.

[0025] In a second aspect, an embodiment of the present application provides a web page parsing device, including:

[0026] An object acquisition module is used to acquire the document object model of the target web page to be parsed;

[0027] A rule loading module is used to load the target page parsing rules configured by the user in a preset format;

[0028] A tag acquisition module, used to acquire the tag type in the target page parsing rule;

[0029] The web page parsing module is used to parse the web page data belonging to the tag type in the document object model by adopting a parsing method corresponding to the tag type for each tag type.

[0030] In a possible implementation, the tag type includes at least one of a row-level tag, a list-type block-level tag, and a table-type block-level tag.

[0031] In a possible implementation, the web page parsing module is specifically used to: for a row-level tag, use a parsing method corresponding to the row-level tag in the target page parsing rule to parse the web page data belonging to the row-level tag in the document object model.

[0032] In a possible implementation, the web page parsing module is specifically used to:

[0033] For a block-level tag of a list type, locate all subtags of the block-level tag of the list type in the document object model;

[0034] If the list-type block-level tag is not configured with document parameters, the text data in all sub-tags of the block-level tag is obtained.

[0035] In a possible implementation, the web page parsing module is further used to:

[0036] If the list-type block-level tag is configured with a document parameter, traverse each item in the document parameter, split each sub-tag of the block-level tag, and obtain dictionary-type sub-tag data;

[0037] Merge the dictionary-type sub-tag data corresponding to all sub-tags of the block-level tag.

[0038] In a possible implementation, the web page parsing module is specifically used to:

[0039] For a block-level tag of a table type, locate all subtags of the block-level tag of the table type in the document object model;

[0040] Each sub-tag of the block-level tag is split, and the split data is assembled into dictionary-type data.

[0041] In a possible implementation, the object acquisition module includes:

[0042] An acquisition unit, used for acquiring the source code of the target web page to be parsed;

[0043] The assembling unit is used to assemble the source code of the target web page into a document object model.

[0044] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.

[0045] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are executed.

[0046] The web page parsing method provided by the embodiment of the present application first obtains the document object model of the target web page to be parsed; secondly, loads the target page parsing rules configured by the user in a preset format; again obtains the tag type in the target page parsing rules; and finally, for each tag type, adopts the parsing method corresponding to the tag type to parse the web page data belonging to the tag type in the document object model. Since the user can configure the target page parsing rules in a preset format, that is, the page parsing rules can be flexibly changed according to the user's needs, the versatility of web page parsing can be improved, developers can avoid repeated development work, and save frequent communication with developers.

[0047] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flow chart of a web page parsing method provided by an embodiment of the present application is shown;

[0050] Figure 2 A screenshot of a simulated web page provided in an embodiment of the present application is shown;

[0051] Figure 3 A schematic diagram of the structure of a web page parsing device provided in an embodiment of the present application is shown;

[0052] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0053] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0054] Considering that the existing web page parsing methods all write a set of independent parsing modules according to the structure of the specific web page. It is difficult to reuse the code when parsing different web pages, which will cause developers to have a lot of duplication of work. In addition, the parsing of web pages is completed by developers. If the requirements need to be changed (such as web page revision issues), users often need to communicate frequently with developers.

[0055] The web page data that needs to be parsed in data collection work has two main structural layout forms: list ( ) and other block-level tags and hyperlinks or titles ( ),sheet( 、 ) and other row-level tags. However, when faced with data parsing tasks for many different websites (pages), due to the variety of page structure layouts, it is necessary to write a specific parsing method for each page, which results in a large amount of relatively repetitive development work. Based on this, the embodiments of the present application provide a web page parsing method, device, electronic device, and medium, which are described below through embodiments.

[0056] To facilitate understanding of this embodiment, a web page parsing method disclosed in an embodiment of the present application is first introduced in detail.

[0057] Please refer to Figure 1 , Figure 1 This is a flow chart of a web page parsing method provided by an embodiment of the present application. Figure 1 As shown, the following steps may be included:

[0058] Step S101, obtaining the document object model of the target web page to be parsed;

[0059] Step S102, loading the target page parsing rules configured by the user in a preset format;

[0060] Step S103, obtaining the tag type in the target page parsing rule;

[0061] Step S104: for each tag type, using a parsing method corresponding to the tag type, parse the web page data belonging to the tag type in the document object model.

[0062] In step S101, the HTML source code of the target web page to be parsed is first obtained, and then the HTML source code is assembled into an HTML DOM object, that is, a document object model. This embodiment is not limited to this, and other types of source code and DOM objects can also be used.

[0063] Hypertext Markup Language (HTML) source code is an uncompiled text file composed of HTML commands, which can describe text, graphics, animations, sounds, tables, links, etc. The structure of HTML consists of two parts: the head and the body. The head describes the information required by the browser, while the body contains the specific content to be described.

[0064] Document Object Model (DOM) is a standard programming interface for processing extensible markup languages ​​recommended by the W3C organization. It is an application programming interface (API) that is independent of platform and language, and can dynamically access programs and scripts to update their content, structure, and style of www documents (currently, HTML and XML documents are defined by the description part).

[0065] In step S102, the user needs to configure the target page parsing rules according to the preset format, and then load the target page parsing rules. That is, the formulation and writing of the page parsing rules are handed over to the user to complete, and the page parsing rules can be flexibly changed. Optionally, the loading process of the page parsing rules can be to submit the page parsing rules through an interface. The loading process of the page parsing rules can also be to read the already written parsing rules from a database. Since the user can configure the target page parsing rules according to the preset format, that is, the page parsing rules can be flexibly changed according to the user's needs, the versatility of web page parsing can be improved, the developer can avoid repeated development work, and save frequent communication with the developer.

[0066] Below Figure 2 Taking the page shown in the figure as an example, the target page parsing rules are introduced in detail.

[0067] Figure 2 The HTML source code of the web page corresponding to the page shown includes:

[0068]

[0069]

[0070]

[0071] The layout of data in the HTML source code of the above webpage: There are three types of data structures, namely list type, table type and single row data. It should be noted that the data of these three structures are used as examples only to teach those skilled in the art how to implement. The present embodiment of the present invention is not limited to this and may also include data of other structures, such as data of inline block structure. Assuming that the user wants to obtain data of these three structures in the page at the same time, the target page parsing rules that need to be written are as follows:

[0072]

[0073]

[0074] Among them, the key values ​​in the above target page parsing rules are shown in Table 1:

[0075] Table 1

[0076]

[0077] From the above table 1, we can see that the purpose of the above target page parsing rules is to obtain the data in the list and 、 ) and other block-level tags and hyperlinks or titles ( Get the contents of different tags separately; get the data in the table; get The URL of the tag.

[0078] In step S103, tags are classified in HTML according to the location characteristics of the tags in the document. The tag types include at least one of row-level tags, block-level tags and inline block tags. The block-level tags include list-type block-level tags and table-type block-level tags.

[0079] Row-level tags are tags that can stay on the same line with other elements and cannot set width and height, such as a, span, strong, u, em-->.

[0080] Block-level tags refer to tags that occupy a single line, do not stay on the same line with other elements, and can set width and height, for example: div, p, h1-h6, ul, li, dl, dt, dd.

[0081] In step S104, different parsing methods are used to parse the web page data in the document object model according to different tag types.

[0082] The parsing method in step S104 is introduced in detail below with complete code (python).

[0083] Complete code (python):

[0084]

[0085]

[0086] For the above complete code (python), the parsing process includes the following three situations:

[0087] The first case: for a row-level tag, the parsing method corresponding to the row-level tag in the target page parsing rule is adopted to parse the web page data belonging to the row-level tag in the document object model.

[0088] Specifically, for simple line-level tag parsing: extract data from the HTML DOM object using the xpath expression defined in the page parsing rule. Optionally, you can also use other methods that can operate HTML DOM nodes, such as cssslect, to obtain page elements.

[0089] result=dom.xpath("string({current_xpath})".format(current_xpath=xpath_e xp))

[0090] The second case: for a list-type block-level tag, locate all sub-tags of the list-type block-level tag in the document object model; if the list-type block-level tag is not configured with document parameters, obtain the text data in all sub-tags of the block-level tag. If the list-type block-level tag is configured with document parameters, traverse each item in the document parameters, split each sub-tag of the block-level tag, and obtain dictionary-type sub-tag data; merge the dictionary-type sub-tag data corresponding to all sub-tags of the block-level tag.

[0091] Specifically, firstly, the block-level tags to be parsed in the page are divided into layers, then the data of the divided sub-tags are extracted, and finally the data extracted from the sub-tags are assembled and merged into a complete data. It should be noted that in the specific implementation, the xpath formula can be used to extract the page content, or other methods that can operate HTML DOM nodes such as cssslect can be used to obtain page elements. The following example is used for explanation.

[0092] First, locate all the subtags under the block-level tag to be parsed in the HTML DOM:

[0093] children_list=dom.xpath(. / / div[@class='list-info'] / ul+' / *')

[0094] for child in children_list:

[0095] Next, you need to determine whether the elements in the subtags need to be further split. If not, directly get all the text in the subtags:

[0096] li_value=child.xpath('string(.)')

[0097] If the document parameter is configured, each sub-tag needs to be split; for each sub-tag, traverse each item configured in the document parameter. The result after parsing is that each sub-tag is a dictionary type of data, where the key is the value corresponding to the document_field defined by the user in the document, and the value is the data parsed corresponding to the document_xpath defined by the user in the document:

[0098] for field,document_exp in document.items():

[0099] li_value[field]=child.xpath("string(.{})".format(document_exp))

[0100] Finally, all sub-tag data are merged together as a complete 'list' tag data:

[0101] result.append(li_value)

[0102] The third case: for a block-level tag of a table type, all subtags of the block-level tag of the table type are located in the document object model; each subtag of the block-level tag is split, and the split data is assembled into dictionary-type data.

[0103] Specifically, firstly, the block-level tags to be parsed in the page are divided into layers, then the data of the divided sub-tags are extracted, and finally the data extracted from the sub-tags are assembled and merged into a complete data. It should be noted that in the specific implementation, the xpath formula can be used to extract the page content, or other methods that can operate HTML DOM nodes such as cssslect can be used to obtain page elements. The following example is used for explanation.

[0104] For "table" type block-level tag parsing: the result will be output in the form of key-value pairs such as {'table header':'content'}.

[0105] First, locate all the subtags under the block-level tag to be parsed in the HTML dom (each line in the list):

[0106] children_list=dom.xpath(. / / div[@class='table-info'] / ul+' / *')

[0107] for child in children_list:

[0108] Next, split the data in each row of the list and assemble the split data into dictionary-type data. If the number of elements in each row is equal to 2, the first one is used as the key and the second one is used as the value; if the number of elements in each row is greater than two, the first one is selected as the key and all the remaining values ​​are used as the value:

[0109]

[0110] Through the above analysis, the analysis results obtained include:

[0111]

[0112]

[0113] It should be noted that the parsing of row-level tags, list-type block-level tags and table-type block-level tags is taken as an example only to teach those skilled in the art how to implement the present invention. The specific implementation is not limited to this and may also include the parsing of other types of tags.

[0114] In a possible implementation, after step S104, the method further includes: synchronously returning the parsing result to the user, or outputting the parsing result to a designated storage location, such as a NoSQL database such as MongoDB or a message queue such as Kafka.

[0115] The web page parsing method provided by the embodiment of the present application first obtains the document object model of the target web page to be parsed; secondly, loads the target page parsing rules configured by the user in a preset format; again obtains the tag type in the target page parsing rules; and finally, for each tag type, adopts the parsing method corresponding to the tag type to parse the web page data belonging to the tag type in the document object model. Since the user can configure the target page parsing rules in a preset format, that is, the page parsing rules can be flexibly changed according to the user's needs, the versatility of web page parsing can be improved, developers can avoid repeated development work, and save frequent communication with developers.

[0116] Based on the same technical concept, the embodiments of the present application also provide a page parsing device, an electronic device, and a computer storage medium, etc. For details, please refer to the following embodiments.

[0117] Please refer to Figure 3 , is a schematic diagram of the structure of a web page parsing device provided in an embodiment of the present application. Figure 3 As shown, this may include:

[0118] The object acquisition module 301 is used to acquire the document object model of the target web page to be parsed;

[0119] A rule loading module 302 is used to load the target page parsing rules configured by the user in a preset format;

[0120] The tag acquisition module 303 is used to acquire the tag type in the target page parsing rule;

[0121] The web page parsing module 304 is used to parse the web page data belonging to the tag type in the document object model using a parsing method corresponding to the tag type for each tag type.

[0122] In a possible implementation, the tag type includes at least one of a row-level tag, a block-level tag, and an inline block tag, and the block-level tag includes a list-type block-level tag and a table-type block-level tag.

[0123] In a possible implementation, the web page parsing module is specifically used to: for a row-level tag, use a parsing method corresponding to the row-level tag in the target page parsing rule to parse the web page data belonging to the row-level tag in the document object model.

[0124] In a possible implementation, the web page parsing module 304 is specifically used to:

[0125] For a block-level tag of a list type, locate all subtags of the block-level tag of the list type in the document object model;

[0126] If the list-type block-level tag is not configured with document parameters, the text data in all sub-tags of the block-level tag is obtained.

[0127] In a possible implementation, the web page parsing module 304 is further configured to:

[0128] If the list-type block-level tag is configured with a document parameter, traverse each item in the document parameter, split each sub-tag of the block-level tag, and obtain dictionary-type sub-tag data;

[0129] Merge the dictionary-type sub-tag data corresponding to all sub-tags of the block-level tag.

[0130] In a possible implementation, the web page parsing module 304 is specifically used to:

[0131] For a block-level tag of a table type, locate all subtags of the block-level tag of the table type in the document object model;

[0132] Each sub-tag of the block-level tag is split, and the split data is assembled into dictionary-type data.

[0133] In a possible implementation, the object acquisition module 301 includes:

[0134] An acquisition unit, used for acquiring the source code of the target web page to be parsed;

[0135] The assembling unit is used to assemble the source code of the target web page into a document object model.

[0136] The present application embodiment discloses an electronic device, such as Figure 4 As shown, it includes: a processor 401, a memory 402 and a bus 403, wherein the memory 402 stores machine-readable instructions executable by the processor 401, and when the electronic device is running, the processor 401 communicates with the memory 402 via the bus 403. When the machine-readable instructions are executed by the processor 401, the following steps are performed:

[0137] Obtain the document object model of the target web page to be parsed;

[0138] Load the target page parsing rules configured by the user in a preset format;

[0139] Obtaining the tag type in the target page parsing rule;

[0140] For each tag type, a parsing method corresponding to the tag type is used to parse the web page data belonging to the tag type in the document object model.

[0141] In a possible implementation, the tag type includes at least one of a row-level tag, a block-level tag, and an inline block tag, and the block-level tag includes a list-type block-level tag and a table-type block-level tag.

[0142] In a possible implementation, the processor 401 parses the web page data belonging to the tag type in the document object model using a parsing method corresponding to the tag type for each tag type, including:

[0143] For the row-level tag, the parsing method corresponding to the row-level tag in the target page parsing rule is adopted to parse the web page data belonging to the row-level tag in the document object model.

[0144] In a possible implementation, the processor 401 parses the web page data belonging to the tag type in the document object model using a parsing method corresponding to the tag type for each tag type, including:

[0145] For a block-level tag of a list type, locate all subtags of the block-level tag of the list type in the document object model;

[0146] If the list-type block-level tag is not configured with document parameters, the text data in all sub-tags of the block-level tag is obtained.

[0147] In a possible implementation manner, the processor 401 is further configured to:

[0148] If the list-type block-level tag is configured with a document parameter, traverse each item in the document parameter, split each sub-tag of the block-level tag, and obtain dictionary-type sub-tag data;

[0149] Merge the dictionary-type sub-tag data corresponding to all sub-tags of the block-level tag.

[0150] In a possible implementation, the processor 401 parses the web page data belonging to the tag type in the document object model using a parsing method corresponding to the tag type for each tag type, including:

[0151] For a block-level tag of a table type, locate all subtags of the block-level tag of the table type in the document object model;

[0152] Each sub-tag of the block-level tag is split, and the split data is assembled into dictionary-type data.

[0153] In a possible implementation, the processor 401 obtains the document object model of the target webpage to be parsed, including:

[0154] Obtain the source code of the target web page to be parsed;

[0155] The source code of the target web page is assembled into a document object model.

[0156] The computer program product of the web page parsing method provided in the embodiments of the present application includes a computer-readable storage medium storing a non-volatile program code executable by a processor. The instructions included in the program code can be used to execute the method described in the previous method embodiments. The specific implementation can be found in the method embodiments and will not be repeated here.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0158] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0159] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0161] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0162] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A web page parsing method, characterized in that: include: Obtain the document object model of the target web page to be parsed; Load the target page parsing rules configured by the user in the preset format; Obtaining a tag type classified according to the location characteristics of the tag in the document in the target page parsing rule; the tag type includes at least one of a row-level tag, a block-level tag, and an inline block tag, and the block-level tag includes a list-type block-level tag and a table-type block-level tag; For each tag type, a parsing method corresponding to the tag type is used to parse the web page data belonging to the tag type in the document object model; For each tag type, adopting a parsing method corresponding to the tag type to parse the web page data belonging to the tag type in the document object model includes: For the row-level tag, the parsing method corresponding to the row-level tag in the target page parsing rule is adopted to parse the web page data belonging to the row-level tag in the document object model; For a block-level tag of a list type, locate all subtags of the block-level tag of the list type in the document object model; If the block-level tag of the list type is not configured with document parameters, then obtaining text data in all sub-tags of the block-level tag; If the list-type block-level tag is configured with a document parameter, traverse each item in the document parameter, split each sub-tag of the block-level tag, and obtain dictionary-type sub-tag data; Merge the dictionary-type sub-tag data corresponding to all sub-tags of the block-level tag; For a block-level tag of a table type, locate all subtags of the block-level tag of the table type in the document object model; Each sub-tag of the block-level tag is split, and the split data is assembled into dictionary-type data.

2. The method according to claim 1, characterized in that Get the document object model of the target web page to be parsed, including: Obtain the source code of the target web page to be parsed; The source code of the target web page is assembled into a document object model.

3. A web page parsing device, characterized in that: include: An object acquisition module is used to acquire the document object model of the target web page to be parsed; A rule loading module is used to load the target page parsing rules configured by the user in a preset format; A tag acquisition module, used to acquire the tag type classified according to the position characteristics of the tag in the document in the target page parsing rule; the tag type includes at least one of a row-level tag, a block-level tag and an inline block tag, and the block-level tag includes a list-type block-level tag and a table-type block-level tag; A web page parsing module, for parsing web page data belonging to each tag type in the document object model using a parsing method corresponding to the tag type; The web page parsing module is to parse the web page data belonging to the tag type in the document object model using a parsing method corresponding to the tag type for each tag type. The web page parsing module is specifically used to: For the row-level tag, the parsing method corresponding to the row-level tag in the target page parsing rule is adopted to parse the web page data belonging to the row-level tag in the document object model; For a block-level tag of a list type, locate all subtags of the block-level tag of the list type in the document object model; If the block-level tag of the list type is not configured with document parameters, then obtaining text data in all sub-tags of the block-level tag; If the list-type block-level tag is configured with a document parameter, traverse each item in the document parameter, split each sub-tag of the block-level tag, and obtain dictionary-type sub-tag data; Merge the dictionary-type sub-tag data corresponding to all sub-tags of the block-level tag; For a block-level tag of a table type, locate all subtags of the block-level tag of the table type in the document object model; Each sub-tag of the block-level tag is split, and the split data is assembled into dictionary-type data.

4. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of any method as claimed in claims 1 to 2.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are executed.

Images