Webpage data extraction method and device, electronic equipment and storage medium
By using a multi-level selector dynamic switching strategy in web page data extraction, the selector is automatically switched to cope with changes in page structure and element selectors, thus solving the problem of web page data extraction failure and achieving efficient and stable data collection.
Patent Information
- Application Number
- CN202510815383.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-18
- Publication Date
- 2025-10-31
AI Technical Summary
Existing technologies cannot address the problem of web page data extraction failure caused by frequent changes in page structure and element selectors on web platforms.
Data is extracted from target nodes on a webpage using a first selector. In response to the selection switching condition being met, a second selector is selected from a pre-configured set of selectors. Data is then extracted again using the second selector until valid data is extracted. This multi-level dynamic selection switching strategy requires no manual intervention.
It enables adaptive data extraction based on dynamic changes in webpage structure and element selectors, ensuring the continuity and stability of data collection, improving data extraction efficiency, and reducing manual maintenance costs.
Smart Images
Figure CN120873267A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of Internet data collection, and in particular to a method, apparatus, electronic device and storage medium for extracting web page data. Background Technology
[0002] With the development of the Internet, data analysis, price monitoring, competitor analysis and other businesses all require the extraction of key information from web pages to provide data support for market trend analysis, user behavior research, and advertising effectiveness evaluation. Existing data extraction methods can extract some key information from web pages in advance.
[0003] However, existing technologies cannot solve the problem of web page data extraction failure caused by frequent changes in page structure and element selectors on web platforms. Summary of the Invention
[0004] This application provides a web page data extraction method, apparatus, electronic device, and storage medium to solve the problem that existing technologies cannot address the failure of web page data extraction caused by frequent changes in page structure and element selectors on web platforms.
[0005] Firstly, this application provides a method for extracting web page data, the method comprising:
[0006] Use the first selector to extract data from the target nodes in the webpage;
[0007] In response to the selection switching condition being met, a second selector is selected from the pre-configured selector set;
[0008] The target node is then subjected to data extraction again using the second selector.
[0009] In one possible implementation, after extracting data from the target node again using the second selector, the method further includes:
[0010] In response to the selection switching condition being met again, a second selector is selected from the selector set, and the step of extracting data from the target node again using the second selector is executed until valid data of the target node is extracted, or all selectors in the selector set are traversed.
[0011] In one possible implementation, the response to satisfying the selector switching condition includes:
[0012] If it is determined that the current extraction result for the target node is empty or the extraction result fails the validity verification, the selector switching condition is satisfied.
[0013] In one possible implementation, the response to satisfying the selector switching condition includes:
[0014] If a change in the page structure of the webpage is detected, it is determined that the selector switching conditions are met.
[0015] In one possible implementation, prior to extracting data from the target node in the webpage using the first selector, the method further includes:
[0016] Select the set master selector as the first selector from the pre-configured selector set;
[0017] The step of selecting a second selector from a pre-configured set of selectors includes:
[0018] From the pre-configured set of selectors, select the highest priority selector that has not been selected before as the second selector, according to a predefined priority order.
[0019] In one possible implementation, the method further includes:
[0020] If data extraction from the target node fails using the second selector, the webpage source code is saved, and an exception log containing information about the data extraction process of the target node using the second selector is generated.
[0021] In one possible implementation, the change in the page structure of the webpage is detected by the following method:
[0022] The webpage is detected using a preset page structure change detection algorithm. A change in the page structure is determined when any one or more of the following conditions are detected:
[0023] The preset key page elements are missing, the positional relationship between the key page elements is abnormal, and the content of the key page elements is incomplete.
[0024] Secondly, this application provides a web page data extraction device, the device comprising:
[0025] The first extraction module is used to extract data from target nodes in a webpage using a first selector;
[0026] The selector switching module is used to select a second selector from a pre-configured set of selectors in response to the fulfillment of selector switching conditions;
[0027] The second extraction module is used to extract data from the target node again using the second selector.
[0028] In one possible implementation, the device further includes:
[0029] The iterative extraction module is used to select a second selector from the selector set again in response to the selection switching condition being met, and to perform the step of extracting data from the target node again using the second selector, until valid data of the target node is extracted, or to traverse all selectors in the selector set.
[0030] In one possible implementation, the selector switching module is specifically used for:
[0031] If it is determined that the current extraction result for the target node is empty or the extraction result fails the validity verification, the selector switching condition is satisfied.
[0032] In one possible implementation, the selector switching module is specifically used for:
[0033] If a change in the page structure of the webpage is detected, it is determined that the selector switching conditions are met.
[0034] In one possible implementation, the first extraction module is specifically used for:
[0035] Select the set master selector as the first selector from the pre-configured selector set;
[0036] The selector switching module is specifically used for:
[0037] From the pre-configured set of selectors, select the highest priority selector that has not been selected before as the second selector, according to a predefined priority order.
[0038] In one possible implementation, the device further includes:
[0039] The log saving module is used to save the webpage source code and generate an exception log containing information about the data extraction process of the second selector on the target node if the data extraction of the target node fails to be performed using the second selector.
[0040] In one possible implementation, the selector switching module detects whether the page structure of the webpage has changed by:
[0041] The webpage is detected using a preset page structure change detection algorithm. A change in the page structure is determined when any one or more of the following conditions are detected:
[0042] The preset key page elements are missing, the positional relationship between the key page elements is abnormal, and the content of the key page elements is incomplete.
[0043] Thirdly, this application provides an electronic device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to:
[0044] Use the first selector to extract data from the target nodes in the webpage;
[0045] In response to the selection switching condition being met, a second selector is selected from the pre-configured selector set;
[0046] The target node is then subjected to data extraction again using the second selector.
[0047] Fourthly, this application also provides a computer storage medium storing computer-executable instructions for executing the web page data extraction method described in any of the preceding claims of this application.
[0048] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application extracts data from target nodes in a webpage using a first selector. In response to meeting the selector switching condition, a second selector is selected from a pre-configured selector set, and the second selector is used to extract data from the target nodes again, ultimately extracting valid key data from the current webpage. This method of data extraction using adaptive selectors can address the problem of dynamic changes in webpage structure and element selectors. Through a multi-level dynamic selector switching strategy, no manual intervention is required, ensuring the continuity and stability of data collection, improving the efficiency of data extraction, and reducing manual maintenance costs. Attached Figure Description
[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0051] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0052] Figure 1 A flowchart illustrating an embodiment of a webpage data extraction method provided in this application;
[0053] Figure 2 A flowchart illustrating an embodiment of another webpage data extraction method provided in this application;
[0054] Figure 3 A flowchart illustrating another embodiment of the webpage data extraction method provided in this application;
[0055] Figure 4 A block diagram of a web page data extraction device provided in an embodiment of this application;
[0056] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0057] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0058] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.
[0059] To address the technical problem of data extraction failure caused by frequent changes in page structure and element selectors on web platforms in existing technologies, this application provides a method that extracts data from target nodes on a webpage using a first selector. In response to a selector switching condition, a second selector is selected from a pre-configured selector set, and the second selector is used to extract data from the target nodes again, ultimately extracting the valid key data from the current webpage. This adaptive selector-based data extraction method can handle the problem of dynamic changes in webpage structure and element selectors. Through a multi-level dynamic selector switching strategy, no manual intervention is required, ensuring the continuity and stability of data collection, improving data extraction efficiency, and reducing manual maintenance costs.
[0060] Figure 1 A flowchart illustrating an embodiment of a webpage data extraction method provided in this application mainly includes the following steps:
[0061] Step 101: Use the first selector to extract data from the target nodes in the webpage.
[0062] A target node refers to the node corresponding to the data item from which data extraction is currently needed. Examples include product titles, prices, list items, etc. Other examples include title tag nodes, list tag nodes, etc. The first selector mentioned above refers to the first predefined selector used to extract data from the target node on the webpage.
[0063] In web pages, the page element characteristics, page structure change frequency, and validation rules for different data items vary significantly. For example, price data is sensitive information on e-commerce websites, and the platform may frequently change the attributes related to the price tag node. Furthermore, the attribute information of the price tag node and the title tag node differs greatly. Therefore, different selectors need to be set for data extraction for different data items. In this embodiment, a selector set can be predefined for each data item, containing multiple selectors. The first selector mentioned above can be the most important selector in the selector set. Additionally, the selectors include path selectors and element selectors; this embodiment uses a path selector as an example for explanation.
[0064] For each of the target nodes mentioned above, a corresponding selector class name can be set. Then, when data extraction is required from a target node, the target node can be located directly by its selector class name, or it can be located by a specific path selector, and the data from that target node can be extracted.
[0065] In one embodiment, a first selector is used to extract data from target nodes in a webpage. Specifically, a path selector is selected from a predefined set of selectors for the current target node to extract data from the target node in the webpage.
[0066] For example, assuming the target node is a title tag node, select a path selector from a predefined set of selectors (e.g., '. / / div[contains(@class,"title--")] / / span / text()'), use this path selector to locate the title tag node, and extract the data from the title tag node.
[0067] Step 102: In response to the selection switching condition being met, select a second selector from the pre-configured selector set.
[0068] Step 103: Use the second selector to extract data from the target node again.
[0069] The following is a unified explanation of steps 102-103 above:
[0070] In one embodiment, in response to meeting the selector switching condition, a second selector is selected from a pre-configured selector set, and the second selector is used to extract data from the target node again. Specifically, when the set selector switching condition is met, a second selector is selected from a predefined selector set, and the second selector is used to extract data from the target node again. Here, the second selector is a different selector from the first selector.
[0071] This application provides an automatic selector switching mechanism that automatically triggers selector switching operations when set conditions are met. For example, if the first selector cannot extract data from the target node on the current webpage, the automatic selector switching mechanism is triggered to select a second selector from a predefined set of selectors. This multi-level automatic selector switching mechanism automatically switches selectors for data extraction without manual intervention, significantly improving the data extraction method's adaptability to dynamic changes in the current web platform's page structure and element selectors.
[0072] Furthermore, this embodiment also sets a selection strategy for the second selector. When a selector switching operation is triggered, the second selector is selected for data extraction according to the selection strategy. The above selection strategy prioritizes the use of high-precision selectors to ensure data extraction quality and significantly shortens the selector switching time. The selection strategy for the second selector will be described in detail in the relevant embodiments below.
[0073] The method provided in this application extracts data from target nodes on a webpage using a first selector. In response to a selector switching condition being met, a second selector is selected from a pre-configured selector set. This second selector is then used to extract data from the target nodes again, ultimately extracting valid key data from the current webpage. This adaptive selector-based data extraction method can address the problem of dynamic changes in webpage structure and element selectors. Through a multi-level dynamic selector switching strategy, no manual intervention is required, ensuring the continuity and stability of data collection, improving data extraction efficiency, and reducing manual maintenance costs.
[0074] Figure 2 A flowchart illustrating another embodiment of a webpage data extraction method provided in this application mainly describes the selection strategy of the selector, including the following steps:
[0075] Step 201: Select the set main selector from the pre-configured selector set as the first selector, and use the first selector to extract data from the target nodes in the webpage.
[0076] Step 202: In response to the selection switching condition being met, select the highest priority and unselected candidate from the candidate selections included in the pre-configured selection set as the second selection according to the predefined priority order.
[0077] Step 203: Use the second selector to extract data from the target node again.
[0078] The following is a unified explanation of steps 201-203 above:
[0079] Based on the detailed description of the above related embodiments, it can be understood that each data item corresponds to a selector set, and the first selector is the first selector selected from the selector set for data extraction.
[0080] In one embodiment, a pre-configured primary selector is selected as the first selector from a pre-configured set of selectors. The primary selector is the highest-priority selector preset for the target data item (such as product title, price, etc.) or target node (such as product title tag node, price tag node). The primary selector is designed based on the most frequently used and stable element features among the page elements of the current data item, aiming to achieve accurate data extraction when the page structure remains unchanged. In actual data collection, the target node of the relevant data item is generally located based on the characteristics or attributes of elements. Therefore, the primary selector in this embodiment utilizes the most stable element features or attributes among the page elements of the data item to locate the target node. Using the primary selector makes it easiest to accurately extract data, hence its highest priority.
[0081] Correspondingly, the other selectors in the pre-configured selector set are backup selectors. Specifically, backup selectors are redundant backups of the primary selector and are stored in the selector set. Compared to the primary selector, backup selectors can cover more scenarios involving changes in page structure or element selectors of target nodes. For example, to prevent bulk data collection, e-commerce platforms might modify element characteristics or attributes (such as class name, ID, and tag hierarchy characteristics) of page elements for relevant data items, making it impossible to locate the target node of the relevant data item using these characteristics or attributes. Setting multiple backup selectors can cover multiple element attributes or characteristics of the target node. Even if a certain element attribute or characteristic of the target node changes, or even if the page structure changes, other backup selectors can still be used to locate the unchanged element attributes or characteristics for data extraction.
[0082] Furthermore, alternative selectors can be divided into precise alternative selectors and general alternative selectors. Precise alternative selectors utilize element features or attributes that are less frequently used or less stable than the primary selector to locate the target node of the relevant data item. Precise alternative selectors are the selectors that most easily extract data items, second only to the primary selector, with a priority lower than the primary selector and higher than the general alternative selector. Specifically, the priority order of each selector in the selector set can be defined based on the order of the primary selector, precise alternative selector, and general alternative selector. The higher the priority of the selector, the easier it is to extract data.
[0083] In one embodiment, a primary selector is selected from a pre-configured set of selectors as the first selector, and the first selector is used to extract data from the target node in the webpage. In response to the selection switching condition being met, a backup selector with the highest priority that has not been selected before is selected from the pre-configured set of selectors according to a predefined priority order as the second selector, and the second selector is used to extract data from the target node again.
[0084] Specifically, when the page structure or element selector of a webpage changes dynamically, it indicates that the selector switching condition is met, triggering a dynamic selector switching mechanism to re-extract data. When the element selector or page structure of the target node changes, it means that the element attributes or characteristics of the target node for the relevant data item have changed. This can cause the relevant selector to fail to locate the target node using its element attributes or characteristics, leading to data extraction failure or the extraction of invalid data. In this case, it is necessary to switch selectors to extract data. The specific method for determining whether the selector switching condition is met is only briefly explained here and will be detailed in the relevant embodiments below.
[0085] Specifically, the highest-priority primary selector is chosen from a predefined set of selectors as the first selector. This first selector is used to locate the target node of the relevant data item on the webpage, and then the data from the target node is extracted. While extracting data using the first selector, if the selector switching conditions are met, the highest-priority, unselected backup selector is selected from the predefined set in priority order as the second selector. This second selector is used to locate the target node of the relevant data item, and data extraction is performed on the target node of the relevant data item again. By setting a priority order, the selector most capable of extracting data can be selected according to the priority selection strategy, thus eliminating the need for multiple selector switching and shortening the time to extract valid data.
[0086] Step 204: In response to the selection switching condition being met again, select a second selector from the selector set and perform the step of extracting data from the target node again using the second selector until valid data of the target node is extracted, or traverse all selectors in the selector set.
[0087] In one embodiment, if the selector switching condition is met, the automatic selector switching operation is triggered again, a second selector is selected from the selector set, and the second selector is used to extract data from the target node again until valid data of the target node is extracted, or the traversal of all selectors in the selector set is completed and the traversal is stopped.
[0088] Specifically, the above traversal logic involves selecting each selector sequentially from the selector set according to the second selector selection strategy. The traversal terminates when any selector extracts valid data, or when no valid data is extracted after traversing all selectors.
[0089] For example, for a list of data items, a set of selectors is defined. A first selector is selected from this set and used to extract data from the target node on the webpage. If the selector switching condition is met, a second selector is selected from the set according to the selector selection strategy, and used to extract data from the target node on the webpage. If the selector switching condition is met again, another second selector is selected from the set according to the selector selection strategy, and used to extract data from the target node on the webpage. This process continues until valid data is extracted, or until no valid data is extracted after traversing all selectors, then the traversal stops.
[0090] For example, suppose the selector set contains one primary selector and three alternative selectors, and the priority order of the selectors is: primary selector, alternative selector 1, alternative selector 2, and alternative selector 3. First, the primary selector is used as the first selector to extract data from the target node. If the selector switching condition is met, the selector switching mechanism is triggered, and alternative selector 1 is used as the second selector to extract data from the target node. If the selector switching condition is met, the selector switching mechanism is triggered, and alternative selector 2 is used as the second selector to extract data from the target node. If data is successfully extracted, an alternative selector is not reselected as the second selector for data extraction. If data is not successfully extracted, alternative selector 3 is reselected as the second selector for data extraction. If data is successfully extracted, an alternative selector is not reselected as the second selector for data extraction. If data is still not successfully extracted, at this point, there are no unselected alternative selectors left in the selector set, indicating that data extraction has failed, and the selectors in the selector set are no longer traversed.
[0091] pass Figure 2 The description of the illustrated embodiment establishes a selector priority selection strategy. In response to dynamic changes in page structure and element selectors, a dynamic selector switching mechanism is triggered, enabling dynamic adaptation of selectors for data extraction without manual intervention. This reduces manual maintenance costs and selects the most suitable selector according to priority order, reducing resource consumption. This fundamentally solves the problem of data extraction failure caused by dynamic changes in page structure and element selectors during data collection. It truly achieves adaptive data extraction of target nodes, providing a stable, continuous, and efficient adaptive data extraction solution for data analysis services.
[0092] Figure 3 The flowchart of another embodiment of the web page data extraction method provided in this application mainly describes how to determine whether the selector switching conditions are met, and the handling mechanism for data extraction failure, mainly including the following steps:
[0093] Step 301: Use the first selector to extract data from the target nodes in the webpage.
[0094] For step 301, see above. Figure 1 The relevant descriptions of the embodiments shown will not be repeated here.
[0095] Step 302: In response to the selection switching condition being met, select a second selector from the pre-configured selector set.
[0096] This application provides two methods to satisfy the selector switching conditions: one is to determine that the current extraction result of the target node is empty or the extraction result fails the validity verification, and the other is to detect that the page structure of the webpage has changed. The following is a detailed explanation of these two methods:
[0097] In one embodiment, if it is determined that the current extraction result for the target node is empty or the extraction result fails the validity verification, the selector switching condition is met, and a second selector is selected from the pre-configured selector set. Specifically, if it is determined that the current extraction result for the target node is empty (i.e., extraction failed), or the extraction result fails the validity verification (invalid data was extracted), a second selector is selected from the pre-configured selector set.
[0098] The above embodiments address situations where the element selector of the relevant data items changes. When the element selector of the target node changes, that is, when the relevant element attributes or characteristics (element class name, ID attribute, tag level characteristics) of the target node change dynamically, it means that the corresponding target node cannot be located using the element attributes or characteristics before the change, and therefore data cannot be extracted, or the extracted data is invalid. In this case, the current selector can no longer locate the element attributes or characteristics of the target node after the change, and the extracted data is empty or invalid, indicating that the selector switching condition is met, and the selector needs to be switched for data extraction.
[0099] Furthermore, the validity of the extracted data can be determined by detecting its text length, data type, and numerical range. For example, if the text length of the extracted product title does not meet the preset range, the extracted product title data has failed validation; if the data type of the extracted product price does not match the expected numeric type, the extracted product price data has failed validation; if the number of products in the extracted product list is low or below a preset threshold, the extracted product list has failed validation.
[0100] For example, if the price format of a product in the extracted data does not meet the numeric type format or the store name is empty, it indicates that the extracted data is unqualified and has failed verification. This means that the selector switching condition is met, and a second selector is selected from the pre-configured selector set according to priority for data extraction. This is merely an example; other methods can also be used to verify the validity of the extracted data, and this application embodiment does not impose any limitations on this.
[0101] In another embodiment, if a change in the page structure of a webpage is detected, it is determined that the selector switching condition is met, and a second selector is selected from a pre-configured set of selectors.
[0102] The above embodiments address situations where the page structure changes dynamically. When the page structure changes, the target node of the current data item may not exist or the content of the target node may not conform to the format of the current data item. In this case, the current selector will not be able to locate the target node, indicating that the selector switching condition is met. Therefore, a second selector is selected from the pre-configured selector set for data extraction.
[0103] In this embodiment, the page structure of a webpage can be detected to have changed in the following way: the webpage is detected by a preset page structure change detection algorithm, and the page structure of the webpage is determined to have changed when any one or more of the following situations are detected: a preset key page element is missing, the positional relationship between key page elements is abnormal, or the content in the key page element is incomplete.
[0104] For example, if the current target node is the product price tag node, then we need to detect the key element "price" of the product price tag node. If the key element is detected, it means that the current page structure has not changed. If the key element is not detected, it means that the key page element is missing, that is, the current page structure has changed.
[0105] For example, if the current target node is the product list tag node, and only 3 data items are detected in the product list, which does not meet the number of products in a normal e-commerce website, it means that the data extracted is not a valid product list, that is, the content of the extracted product list is incomplete, which means that the current page structure has changed.
[0106] For example, normally, the title tag node in an e-commerce website should be above the price tag node. If the price tag and title tag are reversed in the current page structure, it means that the current page structure has changed.
[0107] Step 303: Use the second selector to extract data from the target node again.
[0108] Step 304: If data extraction from the target node fails using the second selector, save the webpage source code and generate an exception log containing information about the data extraction process of the target node using the second selector.
[0109] The following is a unified description of steps 303-304 above:
[0110] Based on the above steps, if there is no unselected alternative selector in the selector set that can be used as the second selector for data extraction, it indicates that data extraction from the target node using the second selector has failed. In this case, data extraction from the target node is no longer performed using the second selector. Instead, an error log is recorded during the data extraction process using the second selector, and the current webpage's source code is saved so that users can analyze the reason for the data extraction failure based on the error log.
[0111] In one embodiment, a second selector is used to extract data from the target node again. If data extraction from the target node using the second selector fails, the webpage source code is saved, and an exception log containing information about the data extraction process of the second selector is generated. Specifically, during the data extraction process using the second selector, it may be necessary to select a selector from the selector set multiple times for data extraction. This process may result in multiple failures in data extraction using the second selector. Each failure is recorded in an exception log to ensure that the data collection process is not interrupted, allowing for switching to the next selector for data extraction. If no valid data is extracted after traversing all selectors in the selector set, the multiple exception logs recorded during the data extraction process are aggregated into a single target exception log and saved. Simultaneously, the source code of the current page is saved for user analysis and debugging.
[0112] Figure 3 The illustrated embodiment triggers an automatic selector switching mechanism by verifying the extracted results and detecting the current page structure. This mechanism can quickly respond and switch to a valid selector when the page structure changes or the element selector changes, ensuring the continuity and reliability of data collection, further improving the efficiency of data extraction, reducing the manual cost of data collection, and retaining anomaly logs in case of data extraction failure, providing a data foundation for system optimization and analysis.
[0113] Figure 4 A block diagram of a web page data extraction device provided in this application embodiment, the device comprising:
[0114] The first extraction module 41 is used to extract data from target nodes in a webpage using a first selector;
[0115] Selector switching module 42 is used to select a second selector from a pre-configured selector set in response to the fulfillment of selector switching conditions;
[0116] The second extraction module 43 is used to extract data from the target node again using the second selector.
[0117] In one possible implementation, the device further includes:
[0118] The iterative extraction module is used to select a second selector from the selector set again in response to the selection switching condition being met, and to perform the step of extracting data from the target node again using the second selector, until valid data of the target node is extracted, or to traverse all selectors in the selector set.
[0119] In one possible implementation, the selector switching module 42 is specifically used for:
[0120] If it is determined that the current extraction result for the target node is empty or the extraction result fails the validity verification, the selector switching condition is satisfied.
[0121] In one possible implementation, the selector switching module 42 is specifically used for:
[0122] If a change in the page structure of the webpage is detected, it is determined that the selector switching conditions are met.
[0123] In one possible implementation, the first extraction module 41 is specifically used for:
[0124] Select the set master selector as the first selector from the pre-configured selector set;
[0125] The selector switching module is specifically used for:
[0126] From the pre-configured set of selectors, select the highest priority selector that has not been selected before as the second selector, according to a predefined priority order.
[0127] In one possible implementation, the device further includes:
[0128] The log saving module is used to save the webpage source code and generate an exception log containing information about the data extraction process of the second selector on the target node if the data extraction of the target node fails to be performed using the second selector.
[0129] In one possible implementation, the selector switching module 42 detects whether the page structure of the webpage has changed by:
[0130] The webpage is detected using a preset page structure change detection algorithm. A change in the page structure is determined when any one or more of the following conditions are detected:
[0131] The preset key page elements are missing, the positional relationship between the key page elements is abnormal, and the content of the key page elements is incomplete.
[0132] like Figure 5 As shown in the figure, this application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.
[0133] Memory 113 is used to store computer programs;
[0134] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the web page data extraction method provided in any of the foregoing method embodiments, including:
[0135] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the web page data extraction method provided in any of the foregoing method embodiments.
[0136] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0138] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.
[0139] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A method for extracting webpage data, characterized in that, The method includes: Use the first selector to extract data from the target nodes in the webpage; In response to the selection switching condition being met, a second selector is selected from the pre-configured selector set; The target node is then subjected to data extraction again using the second selector.
2. The method according to claim 1, characterized in that, After extracting data from the target node again using the second selector, the method further includes: In response to the selection switching condition being met again, a second selector is selected from the selector set, and the step of extracting data from the target node again using the second selector is executed until valid data of the target node is extracted, or all selectors in the selector set are traversed.
3. The method according to claim 1 or 2, characterized in that, The response to satisfying the selector switching condition includes: If it is determined that the current extraction result for the target node is empty or the extraction result fails the validity verification, the selector switching condition is satisfied.
4. The method according to claim 1 or 2, characterized in that, The response to satisfying the selector switching condition includes: If a change in the page structure of the webpage is detected, it is determined that the selector switching conditions are met.
5. The method according to claim 1, characterized in that, Before using the first selector to extract data from the target nodes in the webpage, the method further includes: Select the set master selector as the first selector from the pre-configured selector set; The step of selecting a second selector from a pre-configured set of selectors includes: From the pre-configured set of selectors, select the highest priority selector that has not been selected before as the second selector, according to a predefined priority order.
6. The method according to claim 1, characterized in that, The method further includes: If data extraction from the target node fails using the second selector, the webpage source code is saved, and an exception log containing information about the data extraction process of the target node using the second selector is generated.
7. The method according to claim 4, characterized in that, The following methods are used to detect whether the page structure of the webpage has changed: The webpage is detected using a preset page structure change detection algorithm. A change in the page structure is determined when any one or more of the following conditions are detected: The preset key page elements are missing, the positional relationship between the key page elements is abnormal, and the content of the key page elements is incomplete.
8. A webpage data extraction device, characterized in that, The device includes: The first extraction module is used to extract data from target nodes in a webpage using a first selector; The selector switching module is used to select a second selector from a pre-configured set of selectors in response to the fulfillment of selector switching conditions; The second extraction module is used to extract data from the target node again using the second selector.
9. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a web page data extraction program stored in the memory to implement the web page data extraction method according to any one of claims 1-7.
10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the web page data extraction method according to any one of claims 1-7.